Mailscanner milter to reject high score spam at MTA level
Shawn Iverson
iversons at rushville.k12.in.us
Sun Aug 19 21:14:47 UTC 2018
Latest commit
Spamassassin/MailScanner are now comparing envelope from and from properly
:)
https://github.com/shawniverson/v5/commit/625040caedc93b2c4e78edf305b69ac8a82bc25b
Todo:
Cleanup code
Add more options to mailscanner config, such as # of milter threads, etc.
Remove hard coded items where feasible
Honor placement of Header entries in header (top or bottom)
Test Test Test (anyone interested is welcome!)
Patches for MailWatch to handle new queues and process counts
Fix bug observed where mailscanner thinks processes are still running when
stopping (but aren't, harmless, just weird)
Feedback is welcome.
On Sun, Aug 19, 2018 at 4:26 PM Shawn Iverson <iversons at rushville.k12.in.us>
wrote:
> Found the callback code for MAIL FROM:
>
> 'M' SMFIC_MAIL MAIL FROM: information
> Expected response: Accept/reject action
>
> Time for some more coding :D
>
> On Sun, Aug 19, 2018 at 4:04 PM Shawn Iverson <
> iversons at rushville.k12.in.us> wrote:
>
>> Another update
>>
>> The latest commit to my branch includes more fixes and a new thing that
>> needs handled now that a Milter is in play. When mail is submitted back to
>> postfix, it needs to be processed by other things that could reject the
>> email (such as an blocked sender). This is a problem because MailScanner
>> does not know how to handle rejects since it has always been part of the
>> queue process interacting directly with the queues, not before queue
>> process.
>>
>> During message delivery, if a reject is detected, I inject a special
>> header to the message and requeue it to MailScanner. MailScanner has a
>> chance, based on this special header to flag the message, remove the
>> special header, add diagnostic info to the header about the relay, and
>> quarantine the message. The mail admin is happy knowing that the message
>> didn't just vanish and has an opportunity to resolve the issue and release
>> it or know the disposition of the email.
>>
>> Another cool thing about this Milter Processor I discovered is you can
>> simply drop a message into /var/spool/MailScanner/milterin from
>> /var/spool/MailScanner/quarantine, and it will try to redeliver it :D
>>
>> This new code is in Message.pm and only becomes active if the milter is
>> activated.
>>
>> On Sun, Aug 19, 2018 at 9:30 AM Shawn Iverson <
>> iversons at rushville.k12.in.us> wrote:
>>
>>> Oh yeah, here's the config for Postfix:
>>>
>>> smtpd_milters = inet:127.0.0.1:33333
>>> smtpd_milter_maps = cidr:/etc/postfix/smtpd_milter_map
>>>
>>> /etc/postfix/smtpd_milter_map:
>>> 127.0.0.0/8 DISABLE
>>> ::/64 DISABLE
>>>
>>> This allows scanned emails to pass the milter, as well as notifications
>>> sent from the localhost. You do need at least Postfix version 3.2 I
>>> believe to have milter map support.
>>>
>>>
>>>
>>>
>>> On Sat, Aug 18, 2018 at 11:28 PM Shawn Iverson <
>>> iversons at rushville.k12.in.us> wrote:
>>>
>>>> MailScanner users:
>>>>
>>>> The MailScanner Milter project is coming along nicely.
>>>>
>>>> https://github.com/shawniverson/v5/commits/081118msmilter
>>>>
>>>> I am currently running this on a split relay to test the milter without
>>>> impacting production email.
>>>>
>>>> The design is fairly simple, although development has taken about 40
>>>> hours of my time. I know more about MailScanner (and perl) than I ever
>>>> have :D
>>>>
>>>> The Milter is integrated into MailScanner and forks as a branch of the
>>>> MailScanner process tree, keeping systemd happy.
>>>>
>>>> The Milter process intercepts incoming email and tells postfix to
>>>> DISCARD, which basically accepts the mail and silently drops it before
>>>> entering the queue. At the same time, the Milter writes a raw email file
>>>> to the /var/spool/MailScanner/milterin queue.
>>>>
>>>> MailScanner picks up the message batches in the milterin directory,
>>>> processes them, and spits them out to /var/spool/MailScanner/milterout
>>>> directory as raw email files.
>>>>
>>>> The MSMail Processor (new) relays the messages to postfix for further
>>>> processing over port 25. A optional localhost rule in header_checks
>>>> removes the local entry from the header before delivery.
>>>>
>>>> The benefits are that the postfix queue is not touched at all
>>>> throughout this process, making the solution (hopefully) an acceptable one
>>>> within the postfix community. It is also very fast, and the codebase for
>>>> this method is smaller than even the Postfix Processor, and MailScanner
>>>> gets its own queues, separate from postfix.
>>>>
>>>> One drawback to this method is there is no apparent way to extract the
>>>> Envelope From address (at least not yet, perhaps I am missing a milter
>>>> code), although it doesn't appear that MailScanner is all that concerned
>>>> about it and doesn't go out of its way to capture it. I think it is
>>>> important though, for spoof detection, so I will continue to research this.
>>>>
>>>> Anyone that is willing to get their feet wet and test can apply the
>>>> following files from my branch:
>>>>
>>>> (In common)
>>>> /usr/sbin/MailScanner
>>>> /usr/share/MailScanner/perl/MailScanner/Milter.pm
>>>> /usr/share/MailScanner/perl/MailScanner/MSMail.pm
>>>> /usr/share/MailScanner/perl/MailScanner/MSDiskStore.pm
>>>> /usr/share/MailScanner/perl/MailScanner/ConfigDefs.pm
>>>>
>>>> Then create the following dirs:
>>>> mkdir -p /var/spool/MailScanner/milterin
>>>> chown postfix:mtagroup /var/spool/MailScanner/milterin
>>>> mkdir -p /var/spool/MailScanner/milterout
>>>> chown postfix:mtagroup /var/spool/MailScanner/milterout
>>>>
>>>> Apply the following to /etc/MailScanner/MailScanner.conf:
>>>> Incoming Queue Dir = /var/spool/MailScanner/milterin
>>>> Outgoing Queue Dir = /var/spool/MailScanner/milterout
>>>> MTA = MSMail
>>>> MSMail Queue Type = short | long (pick one that matches your postfix
>>>> setting)
>>>>
>>>> I recommend doing this in a test or split relay environment that
>>>> blackholes email. Do not use in production yet ;)
>>>>
>>>> Known issues at the moment:
>>>> MailWatch doesn't recogize MSMail as an 'MTA' so the queue stats do not
>>>> appear
>>>> More validation and error handling is needed throughout. Weird emails
>>>> abound!
>>>> Need to know the envelope from sender. Currently hidden from the
>>>> milter, but hopefully exposable via a callback code.
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 14, 2018 at 10:56 AM Shawn Iverson <
>>>> iversons at rushville.k12.in.us> wrote:
>>>>
>>>>> Dear MailScanner users:
>>>>>
>>>>> I am officially working on creating a lightweight milter for
>>>>> MailScanner.
>>>>>
>>>>> This milter will not provide MTA protocol rejection for postfix, due
>>>>> to the severe performance penalty it would cause. All mail will be
>>>>> intercepted, accepted, and silently dropped from the postfix queue and
>>>>> placed in a MailScanner queue.
>>>>>
>>>>> I have a working prototype, and it is processing mail! It is in need
>>>>> of heavy refactoring and some bug squashing.
>>>>>
>>>>> Currently it attempts to create a postfix formatted queue file (very
>>>>> ugly, who thought up this file format???!!!). I may instead create a new
>>>>> Milter Processor for MailScanner that reduces the overhead of doing this
>>>>> and can read the incoming email in a simple line-by-line format. This may
>>>>> also increase performance overall and reduce all the conversions happening.
>>>>>
>>>>> The other side of the coin is what to do when MailScanner is done
>>>>> processing mail. Currently, it generates a postfix queue file and drops it
>>>>> into postfix incoming directory. It should not do this but instead drop
>>>>> the message into postfix using native postfix tools. That will be the next
>>>>> part I tackle as part of the Milter Processor.
>>>>>
>>>>> Why am I doing this? I want to place MailScanner back in a good
>>>>> standing with Postfix folks (at least when the milter + postfix method is
>>>>> in use).
>>>>>
>>>>> I have no plans of removing the old method but rather provide a more
>>>>> supported path for postfix users.
>>>>>
>>>>> Wish me luck. I could be heard across the neighborhood when
>>>>> MailScanner processed an email from the Milter for the first time! :D
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Aug 11, 2018 at 9:58 AM David Jones <djones at ena.com> wrote:
>>>>>
>>>>>> On 08/11/2018 08:52 AM, Shawn Iverson wrote:
>>>>>> > David,
>>>>>> >
>>>>>> > I agree that this is true, and part of my lack of motivation to do
>>>>>> it.
>>>>>> > One reason I wanted it as an option was to reconcile the ongoing
>>>>>> > conflict with the postfix community and return MailScanner to good
>>>>>> > standing to this community. Weitze has been very stern about
>>>>>> > MailScanner directly tapping the postfix queues.
>>>>>> >
>>>>>> > Perhaps an alternative option would be to create a fast MailScanner
>>>>>> > milter that behaves more like the HOLD queue. Basically just a
>>>>>> milter
>>>>>> > that immediately fires back accept to postfix and places all the
>>>>>> > messages in a MailScanner HOLD queue as opposed to a postfix HOLD
>>>>>> > queue. Doing so would maintain speed, simplicity, and be more
>>>>>> compliant
>>>>>> > with postfix. The code would also be very simple.
>>>>>> >
>>>>>> > Then, as you say, if you need MTA level functionality for SA, use
>>>>>> other
>>>>>> > software and methods.
>>>>>> >
>>>>>> >
>>>>>>
>>>>>> This light MS milter would make a lot of sense based on your goal to
>>>>>> get
>>>>>> compliant with Postfix and back "in" with the Postfix community. +1
>>>>>>
>>>>>> >
>>>>>> > On Sat, Aug 11, 2018 at 9:39 AM David Jones <djones at ena.com
>>>>>> > <mailto:djones at ena.com>> wrote:
>>>>>> >
>>>>>> > On 08/11/2018 08:15 AM, Shawn Iverson wrote:
>>>>>> > > I have been planning for a MailScanner milter for quite some
>>>>>> > time. I
>>>>>> > > have been specifically studying rpamd's milter source for
>>>>>> this
>>>>>> > purpose.
>>>>>> > > Alas, lack of time and lack of money are always an issue,
>>>>>> and I
>>>>>> > put a
>>>>>> > > lot of hours in my day job. As Jerry would say, I like to
>>>>>> eat
>>>>>> > and have
>>>>>> > > a roof over my head :D
>>>>>> > >
>>>>>> > > If I do find the time to build a milter, performance will
>>>>>> > definitely be
>>>>>> > > impacted. The reason is that postfix will have to keep each
>>>>>> session
>>>>>> > > open for the duration of scanning, and each MailScanner child
>>>>>> > would have
>>>>>> > > to issue a callback to postfix after scanning the spam so
>>>>>> that
>>>>>> > postfix
>>>>>> > > can responds to the connection appropriately (i.e. reject or
>>>>>> > accept).
>>>>>> > > This will slow down mail processing considerably. If I do
>>>>>> this,
>>>>>> > I am
>>>>>> > > going to keep the HOLD queue around, so you would have to
>>>>>> choose
>>>>>> > between
>>>>>> > > speed or MTA level rejection functionality.
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> > My gut tells me that this is going to be so slow, that it's not
>>>>>> > going to
>>>>>> > be worth the time to put into it. If you want to reject at MTA
>>>>>> time,
>>>>>> > throw in amavis-new or spamd (not rspamd) using the same
>>>>>> SpamAsssassin
>>>>>> > rules and Bayes DB to get most of the same features as
>>>>>> MailScanner
>>>>>> > during the SMTP conversation. Then the mail that gets through
>>>>>> can be
>>>>>> > filtered by MailScanner for it's extra features that make it
>>>>>> unique.
>>>>>> >
>>>>>> > I understand there are different local legal requirements
>>>>>> around the
>>>>>> > world that if email is accepted at MTA time then it has to be
>>>>>> passed on
>>>>>> > to the end user's mailbox. If you are located in one of these
>>>>>> > countries, then this would be more of an issue. But since I am
>>>>>> in a
>>>>>> > country that doesn't have this legal requirement, I do block
>>>>>> email
>>>>>> > post-MTA by MailScanner.
>>>>>> >
>>>>>> > The majority of my spam is blocked at the MTA level already by
>>>>>> highly
>>>>>> > tuned RBLs and postscreen's RBL weighting which is very, very
>>>>>> good.
>>>>>> > Only a small percentage of spam that is zero-hour or from
>>>>>> compromised
>>>>>> > accounts makes it to MailScanner.
>>>>>> >
>>>>>> > I highly recommend the Invaluement RBL. It's very accurate --
>>>>>> only
>>>>>> > 1 or
>>>>>> > 2 false positives over 5+ the years. This RBL is very cost
>>>>>> effective
>>>>>> > and has allowed me to disable all Spamhaus RBL checks in
>>>>>> SpamAssassin
>>>>>> > saving thousands of dollars a year. (We have too high a volume
>>>>>> to stay
>>>>>> > under the free usage limits of Spamhaus so we were having to
>>>>>> pay for
>>>>>> > the
>>>>>> > RBL feed.)
>>>>>> >
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > > On Tue, Aug 7, 2018 at 10:52 AM David Jones via MailScanner
>>>>>> > > <mailscanner at lists.mailscanner.info
>>>>>> > <mailto:mailscanner at lists.mailscanner.info>
>>>>>> > > <mailto:mailscanner at lists.mailscanner.info
>>>>>> > <mailto:mailscanner at lists.mailscanner.info>>> wrote:
>>>>>> > >
>>>>>> > > On 08/07/2018 05:03 AM, info at schroeffu.ch
>>>>>> > <mailto:info at schroeffu.ch> <mailto:info at schroeffu.ch
>>>>>> > <mailto:info at schroeffu.ch>>
>>>>>> > > wrote:
>>>>>> > > >
>>>>>> > > > Hi Mailscanner friends,
>>>>>> > > >
>>>>>> > > > is there any progress to make MailScanner usable as a
>>>>>> > postfix milter?
>>>>>> > > > The most biggest problem I have is, SPAM is not
>>>>>> possible to
>>>>>> > > reject when
>>>>>> > > > reaching a high score at MTA level. For my
>>>>>> understanding,
>>>>>> > connect
>>>>>> > > via
>>>>>> > > > milter instead of queue ^HOLD would be the solution.
>>>>>> > > >
>>>>>> > > > For the next decade we are still using MailScanner
>>>>>> instead
>>>>>> > of others
>>>>>> > > > like Rspamd, because MailScanner is like a mail suite
>>>>>> for mail
>>>>>> > > security,
>>>>>> > > > but if there will never be the possibility to reject
>>>>>> at
>>>>>> > MTA level
>>>>>> > > the
>>>>>> > > > high score spam, we will also change in 1-3 years
>>>>>> while
>>>>>> > replacing
>>>>>> > > the OS
>>>>>> > > > beyond.
>>>>>> > > >
>>>>>> > >
>>>>>> > > One of MailScanner's strongest features is it's batch
>>>>>> mode
>>>>>> > processing
>>>>>> > > that will allow it to handle a very high volume of mail
>>>>>> > flow. I doubt
>>>>>> > > that MailScanner will ever be changed to run as a milter
>>>>>> for this
>>>>>> > > reason.
>>>>>> > >
>>>>>> > > I tried rspamd and found it wasn't as good as the author
>>>>>> > claims so no
>>>>>> > > reason to try to use that as a milter. It also wasn't as
>>>>>> > fast as it
>>>>>> > > claims. I could not send high volumes of mail through it
>>>>>> > like I could
>>>>>> > > with MailScanner.
>>>>>> > >
>>>>>> > > If you want to block high scoring spam at the MTA level,
>>>>>> I
>>>>>> > suggest
>>>>>> > > using
>>>>>> > > amavis or spamd with the same SA rulesets as
>>>>>> MailScanner.
>>>>>> > This will
>>>>>> > > get
>>>>>> > > you most of the power of MailScanner's blocking at the
>>>>>> MTA.
>>>>>> > >
>>>>>> > > https://wiki.apache.org/spamassassin/IntegratedInMta
>>>>>> > >
>>>>>> > > If you you use postscreen and postwhite at the Postfix
>>>>>> MTA
>>>>>> > level, you
>>>>>> > > can block most of the obvious spam with a tuned list of
>>>>>> > RBLs. See the
>>>>>> > > SA users mailing list over the past year for details on
>>>>>> this
>>>>>> > from me
>>>>>> > > and
>>>>>> > > a few others.
>>>>>> > >
>>>>>> > > I suggest setting up a quick test VM with iRedmail to
>>>>>> get a good
>>>>>> > > example
>>>>>> > > of how to do TLS and amavis integration well with
>>>>>> Postfix.
>>>>>> > >
>>>>>> > > --
>>>>>> > > David Jones
>>>>>> > >
>>>>>> > >
>>>>>> > > --
>>>>>> > > MailScanner mailing list
>>>>>> > > mailscanner at lists.mailscanner.info
>>>>>> > <mailto:mailscanner at lists.mailscanner.info>
>>>>>> > > <mailto:mailscanner at lists.mailscanner.info
>>>>>> > <mailto:mailscanner at lists.mailscanner.info>>
>>>>>> > > http://lists.mailscanner.info/mailman/listinfo/mailscanner
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > > --
>>>>>> > > Shawn Iverson, CETL
>>>>>> > > Director of Technology
>>>>>> > > Rush County Schools
>>>>>> > > 765-932-3901 x1171
>>>>>> > > iversons at rushville.k12.in.us
>>>>>> > <mailto:iversons at rushville.k12.in.us>
>>>>>> > <mailto:iversons at rushville.k12.in.us
>>>>>> > <mailto:iversons at rushville.k12.in.us>>
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> > --
>>>>>> > David Jones
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Shawn Iverson, CETL
>>>>>> > Director of Technology
>>>>>> > Rush County Schools
>>>>>> > 765-932-3901 x1171
>>>>>> > iversons at rushville.k12.in.us <mailto:iversons at rushville.k12.in.us>
>>>>>> >
>>>>>> >
>>>>>>
>>>>>>
>>>>>> --
>>>>>> David Jones
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Shawn Iverson, CETL
>>>>> Director of Technology
>>>>> Rush County Schools
>>>>> 765-932-3901 x1171
>>>>> iversons at rushville.k12.in.us
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Shawn Iverson, CETL
>>>> Director of Technology
>>>> Rush County Schools
>>>> 765-932-3901 x1171
>>>> iversons at rushville.k12.in.us
>>>>
>>>>
>>>>
>>>
>>> --
>>> Shawn Iverson, CETL
>>> Director of Technology
>>> Rush County Schools
>>> 765-932-3901 x1171
>>> iversons at rushville.k12.in.us
>>>
>>>
>>>
>>
>> --
>> Shawn Iverson, CETL
>> Director of Technology
>> Rush County Schools
>> 765-932-3901 x1171
>> iversons at rushville.k12.in.us
>>
>>
>>
>
> --
> Shawn Iverson, CETL
> Director of Technology
> Rush County Schools
> 765-932-3901 x1171
> iversons at rushville.k12.in.us
>
>
>
--
Shawn Iverson, CETL
Director of Technology
Rush County Schools
765-932-3901 x1171
iversons at rushville.k12.in.us
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20180819/009b0684/attachment.html>
More information about the MailScanner
mailing list