Mailscanner milter to reject high score spam at MTA level
Shawn Iverson
iversons at rushville.k12.in.us
Mon Aug 20 15:49:46 UTC 2018
Corrected link
https://github.com/shawniverson/v5/commit/6eef93cb3cbc49f72cdca656e8a2bf655de35164
On Mon, Aug 20, 2018 at 11:49 AM Shawn Iverson <iversons at rushville.k12.in.us>
wrote:
> RFC 5321 support added.
>
> Also reviewing RFC 822, particularly handling of delivery failure notices,
> with respect to the milter.
>
>
> https://github.com/shawniverson/v5/commit/6eef93cb3cbc49f72cdca656e8a2bf655de351645321
>
>
> Currently addressing a bug with the daemon forking new children. Seems
> to be caused by the Milter being a child process of MailScanner. If I
> cannot resolve, I plan to separate the Milter into its own daemon. This
> may be better anyway and allow both to be managed independently.
>
> On Sun, Aug 19, 2018 at 5:14 PM Shawn Iverson <
> iversons at rushville.k12.in.us> wrote:
>
>> Latest commit
>>
>> Spamassassin/MailScanner are now comparing envelope from and from
>> properly :)
>>
>>
>> https://github.com/shawniverson/v5/commit/625040caedc93b2c4e78edf305b69ac8a82bc25b
>>
>> Todo:
>>
>> Cleanup code
>> Add more options to mailscanner config, such as # of milter threads, etc.
>> Remove hard coded items where feasible
>> Honor placement of Header entries in header (top or bottom)
>> Test Test Test (anyone interested is welcome!)
>> Patches for MailWatch to handle new queues and process counts
>> Fix bug observed where mailscanner thinks processes are still running
>> when stopping (but aren't, harmless, just weird)
>>
>> Feedback is welcome.
>>
>>
>>
>>
>>
>>
>>
>> On Sun, Aug 19, 2018 at 4:26 PM Shawn Iverson <
>> iversons at rushville.k12.in.us> wrote:
>>
>>> Found the callback code for MAIL FROM:
>>>
>>> 'M' SMFIC_MAIL MAIL FROM: information
>>> Expected response: Accept/reject action
>>>
>>> Time for some more coding :D
>>>
>>> On Sun, Aug 19, 2018 at 4:04 PM Shawn Iverson <
>>> iversons at rushville.k12.in.us> wrote:
>>>
>>>> Another update
>>>>
>>>> The latest commit to my branch includes more fixes and a new thing that
>>>> needs handled now that a Milter is in play. When mail is submitted back to
>>>> postfix, it needs to be processed by other things that could reject the
>>>> email (such as an blocked sender). This is a problem because MailScanner
>>>> does not know how to handle rejects since it has always been part of the
>>>> queue process interacting directly with the queues, not before queue
>>>> process.
>>>>
>>>> During message delivery, if a reject is detected, I inject a special
>>>> header to the message and requeue it to MailScanner. MailScanner has a
>>>> chance, based on this special header to flag the message, remove the
>>>> special header, add diagnostic info to the header about the relay, and
>>>> quarantine the message. The mail admin is happy knowing that the message
>>>> didn't just vanish and has an opportunity to resolve the issue and release
>>>> it or know the disposition of the email.
>>>>
>>>> Another cool thing about this Milter Processor I discovered is you can
>>>> simply drop a message into /var/spool/MailScanner/milterin from
>>>> /var/spool/MailScanner/quarantine, and it will try to redeliver it :D
>>>>
>>>> This new code is in Message.pm and only becomes active if the milter is
>>>> activated.
>>>>
>>>> On Sun, Aug 19, 2018 at 9:30 AM Shawn Iverson <
>>>> iversons at rushville.k12.in.us> wrote:
>>>>
>>>>> Oh yeah, here's the config for Postfix:
>>>>>
>>>>> smtpd_milters = inet:127.0.0.1:33333
>>>>> smtpd_milter_maps = cidr:/etc/postfix/smtpd_milter_map
>>>>>
>>>>> /etc/postfix/smtpd_milter_map:
>>>>> 127.0.0.0/8 DISABLE
>>>>> ::/64 DISABLE
>>>>>
>>>>> This allows scanned emails to pass the milter, as well as
>>>>> notifications sent from the localhost. You do need at least Postfix
>>>>> version 3.2 I believe to have milter map support.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Aug 18, 2018 at 11:28 PM Shawn Iverson <
>>>>> iversons at rushville.k12.in.us> wrote:
>>>>>
>>>>>> MailScanner users:
>>>>>>
>>>>>> The MailScanner Milter project is coming along nicely.
>>>>>>
>>>>>> https://github.com/shawniverson/v5/commits/081118msmilter
>>>>>>
>>>>>> I am currently running this on a split relay to test the milter
>>>>>> without impacting production email.
>>>>>>
>>>>>> The design is fairly simple, although development has taken about 40
>>>>>> hours of my time. I know more about MailScanner (and perl) than I ever
>>>>>> have :D
>>>>>>
>>>>>> The Milter is integrated into MailScanner and forks as a branch of
>>>>>> the MailScanner process tree, keeping systemd happy.
>>>>>>
>>>>>> The Milter process intercepts incoming email and tells postfix to
>>>>>> DISCARD, which basically accepts the mail and silently drops it before
>>>>>> entering the queue. At the same time, the Milter writes a raw email file
>>>>>> to the /var/spool/MailScanner/milterin queue.
>>>>>>
>>>>>> MailScanner picks up the message batches in the milterin directory,
>>>>>> processes them, and spits them out to /var/spool/MailScanner/milterout
>>>>>> directory as raw email files.
>>>>>>
>>>>>> The MSMail Processor (new) relays the messages to postfix for further
>>>>>> processing over port 25. A optional localhost rule in header_checks
>>>>>> removes the local entry from the header before delivery.
>>>>>>
>>>>>> The benefits are that the postfix queue is not touched at all
>>>>>> throughout this process, making the solution (hopefully) an acceptable one
>>>>>> within the postfix community. It is also very fast, and the codebase for
>>>>>> this method is smaller than even the Postfix Processor, and MailScanner
>>>>>> gets its own queues, separate from postfix.
>>>>>>
>>>>>> One drawback to this method is there is no apparent way to extract
>>>>>> the Envelope From address (at least not yet, perhaps I am missing a milter
>>>>>> code), although it doesn't appear that MailScanner is all that concerned
>>>>>> about it and doesn't go out of its way to capture it. I think it is
>>>>>> important though, for spoof detection, so I will continue to research this.
>>>>>>
>>>>>> Anyone that is willing to get their feet wet and test can apply the
>>>>>> following files from my branch:
>>>>>>
>>>>>> (In common)
>>>>>> /usr/sbin/MailScanner
>>>>>> /usr/share/MailScanner/perl/MailScanner/Milter.pm
>>>>>> /usr/share/MailScanner/perl/MailScanner/MSMail.pm
>>>>>> /usr/share/MailScanner/perl/MailScanner/MSDiskStore.pm
>>>>>> /usr/share/MailScanner/perl/MailScanner/ConfigDefs.pm
>>>>>>
>>>>>> Then create the following dirs:
>>>>>> mkdir -p /var/spool/MailScanner/milterin
>>>>>> chown postfix:mtagroup /var/spool/MailScanner/milterin
>>>>>> mkdir -p /var/spool/MailScanner/milterout
>>>>>> chown postfix:mtagroup /var/spool/MailScanner/milterout
>>>>>>
>>>>>> Apply the following to /etc/MailScanner/MailScanner.conf:
>>>>>> Incoming Queue Dir = /var/spool/MailScanner/milterin
>>>>>> Outgoing Queue Dir = /var/spool/MailScanner/milterout
>>>>>> MTA = MSMail
>>>>>> MSMail Queue Type = short | long (pick one that matches your postfix
>>>>>> setting)
>>>>>>
>>>>>> I recommend doing this in a test or split relay environment that
>>>>>> blackholes email. Do not use in production yet ;)
>>>>>>
>>>>>> Known issues at the moment:
>>>>>> MailWatch doesn't recogize MSMail as an 'MTA' so the queue stats do
>>>>>> not appear
>>>>>> More validation and error handling is needed throughout. Weird
>>>>>> emails abound!
>>>>>> Need to know the envelope from sender. Currently hidden from the
>>>>>> milter, but hopefully exposable via a callback code.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 14, 2018 at 10:56 AM Shawn Iverson <
>>>>>> iversons at rushville.k12.in.us> wrote:
>>>>>>
>>>>>>> Dear MailScanner users:
>>>>>>>
>>>>>>> I am officially working on creating a lightweight milter for
>>>>>>> MailScanner.
>>>>>>>
>>>>>>> This milter will not provide MTA protocol rejection for postfix, due
>>>>>>> to the severe performance penalty it would cause. All mail will be
>>>>>>> intercepted, accepted, and silently dropped from the postfix queue and
>>>>>>> placed in a MailScanner queue.
>>>>>>>
>>>>>>> I have a working prototype, and it is processing mail! It is in
>>>>>>> need of heavy refactoring and some bug squashing.
>>>>>>>
>>>>>>> Currently it attempts to create a postfix formatted queue file (very
>>>>>>> ugly, who thought up this file format???!!!). I may instead create a new
>>>>>>> Milter Processor for MailScanner that reduces the overhead of doing this
>>>>>>> and can read the incoming email in a simple line-by-line format. This may
>>>>>>> also increase performance overall and reduce all the conversions happening.
>>>>>>>
>>>>>>> The other side of the coin is what to do when MailScanner is done
>>>>>>> processing mail. Currently, it generates a postfix queue file and drops it
>>>>>>> into postfix incoming directory. It should not do this but instead drop
>>>>>>> the message into postfix using native postfix tools. That will be the next
>>>>>>> part I tackle as part of the Milter Processor.
>>>>>>>
>>>>>>> Why am I doing this? I want to place MailScanner back in a good
>>>>>>> standing with Postfix folks (at least when the milter + postfix method is
>>>>>>> in use).
>>>>>>>
>>>>>>> I have no plans of removing the old method but rather provide a more
>>>>>>> supported path for postfix users.
>>>>>>>
>>>>>>> Wish me luck. I could be heard across the neighborhood when
>>>>>>> MailScanner processed an email from the Milter for the first time! :D
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Aug 11, 2018 at 9:58 AM David Jones <djones at ena.com> wrote:
>>>>>>>
>>>>>>>> On 08/11/2018 08:52 AM, Shawn Iverson wrote:
>>>>>>>> > David,
>>>>>>>> >
>>>>>>>> > I agree that this is true, and part of my lack of motivation to
>>>>>>>> do it.
>>>>>>>> > One reason I wanted it as an option was to reconcile the ongoing
>>>>>>>> > conflict with the postfix community and return MailScanner to
>>>>>>>> good
>>>>>>>> > standing to this community. Weitze has been very stern about
>>>>>>>> > MailScanner directly tapping the postfix queues.
>>>>>>>> >
>>>>>>>> > Perhaps an alternative option would be to create a fast
>>>>>>>> MailScanner
>>>>>>>> > milter that behaves more like the HOLD queue. Basically just a
>>>>>>>> milter
>>>>>>>> > that immediately fires back accept to postfix and places all the
>>>>>>>> > messages in a MailScanner HOLD queue as opposed to a postfix HOLD
>>>>>>>> > queue. Doing so would maintain speed, simplicity, and be more
>>>>>>>> compliant
>>>>>>>> > with postfix. The code would also be very simple.
>>>>>>>> >
>>>>>>>> > Then, as you say, if you need MTA level functionality for SA, use
>>>>>>>> other
>>>>>>>> > software and methods.
>>>>>>>> >
>>>>>>>> >
>>>>>>>>
>>>>>>>> This light MS milter would make a lot of sense based on your goal
>>>>>>>> to get
>>>>>>>> compliant with Postfix and back "in" with the Postfix community. +1
>>>>>>>>
>>>>>>>> >
>>>>>>>> > On Sat, Aug 11, 2018 at 9:39 AM David Jones <djones at ena.com
>>>>>>>> > <mailto:djones at ena.com>> wrote:
>>>>>>>> >
>>>>>>>> > On 08/11/2018 08:15 AM, Shawn Iverson wrote:
>>>>>>>> > > I have been planning for a MailScanner milter for quite
>>>>>>>> some
>>>>>>>> > time. I
>>>>>>>> > > have been specifically studying rpamd's milter source for
>>>>>>>> this
>>>>>>>> > purpose.
>>>>>>>> > > Alas, lack of time and lack of money are always an issue,
>>>>>>>> and I
>>>>>>>> > put a
>>>>>>>> > > lot of hours in my day job. As Jerry would say, I like to
>>>>>>>> eat
>>>>>>>> > and have
>>>>>>>> > > a roof over my head :D
>>>>>>>> > >
>>>>>>>> > > If I do find the time to build a milter, performance will
>>>>>>>> > definitely be
>>>>>>>> > > impacted. The reason is that postfix will have to keep
>>>>>>>> each session
>>>>>>>> > > open for the duration of scanning, and each MailScanner
>>>>>>>> child
>>>>>>>> > would have
>>>>>>>> > > to issue a callback to postfix after scanning the spam so
>>>>>>>> that
>>>>>>>> > postfix
>>>>>>>> > > can responds to the connection appropriately (i.e. reject
>>>>>>>> or
>>>>>>>> > accept).
>>>>>>>> > > This will slow down mail processing considerably. If I do
>>>>>>>> this,
>>>>>>>> > I am
>>>>>>>> > > going to keep the HOLD queue around, so you would have to
>>>>>>>> choose
>>>>>>>> > between
>>>>>>>> > > speed or MTA level rejection functionality.
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> >
>>>>>>>> > My gut tells me that this is going to be so slow, that it's
>>>>>>>> not
>>>>>>>> > going to
>>>>>>>> > be worth the time to put into it. If you want to reject at
>>>>>>>> MTA time,
>>>>>>>> > throw in amavis-new or spamd (not rspamd) using the same
>>>>>>>> SpamAsssassin
>>>>>>>> > rules and Bayes DB to get most of the same features as
>>>>>>>> MailScanner
>>>>>>>> > during the SMTP conversation. Then the mail that gets
>>>>>>>> through can be
>>>>>>>> > filtered by MailScanner for it's extra features that make it
>>>>>>>> unique.
>>>>>>>> >
>>>>>>>> > I understand there are different local legal requirements
>>>>>>>> around the
>>>>>>>> > world that if email is accepted at MTA time then it has to be
>>>>>>>> passed on
>>>>>>>> > to the end user's mailbox. If you are located in one of these
>>>>>>>> > countries, then this would be more of an issue. But since I
>>>>>>>> am in a
>>>>>>>> > country that doesn't have this legal requirement, I do block
>>>>>>>> email
>>>>>>>> > post-MTA by MailScanner.
>>>>>>>> >
>>>>>>>> > The majority of my spam is blocked at the MTA level already
>>>>>>>> by highly
>>>>>>>> > tuned RBLs and postscreen's RBL weighting which is very, very
>>>>>>>> good.
>>>>>>>> > Only a small percentage of spam that is zero-hour or from
>>>>>>>> compromised
>>>>>>>> > accounts makes it to MailScanner.
>>>>>>>> >
>>>>>>>> > I highly recommend the Invaluement RBL. It's very accurate
>>>>>>>> -- only
>>>>>>>> > 1 or
>>>>>>>> > 2 false positives over 5+ the years. This RBL is very cost
>>>>>>>> effective
>>>>>>>> > and has allowed me to disable all Spamhaus RBL checks in
>>>>>>>> SpamAssassin
>>>>>>>> > saving thousands of dollars a year. (We have too high a
>>>>>>>> volume to stay
>>>>>>>> > under the free usage limits of Spamhaus so we were having to
>>>>>>>> pay for
>>>>>>>> > the
>>>>>>>> > RBL feed.)
>>>>>>>> >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > > On Tue, Aug 7, 2018 at 10:52 AM David Jones via MailScanner
>>>>>>>> > > <mailscanner at lists.mailscanner.info
>>>>>>>> > <mailto:mailscanner at lists.mailscanner.info>
>>>>>>>> > > <mailto:mailscanner at lists.mailscanner.info
>>>>>>>> > <mailto:mailscanner at lists.mailscanner.info>>> wrote:
>>>>>>>> > >
>>>>>>>> > > On 08/07/2018 05:03 AM, info at schroeffu.ch
>>>>>>>> > <mailto:info at schroeffu.ch> <mailto:info at schroeffu.ch
>>>>>>>> > <mailto:info at schroeffu.ch>>
>>>>>>>> > > wrote:
>>>>>>>> > > >
>>>>>>>> > > > Hi Mailscanner friends,
>>>>>>>> > > >
>>>>>>>> > > > is there any progress to make MailScanner usable as
>>>>>>>> a
>>>>>>>> > postfix milter?
>>>>>>>> > > > The most biggest problem I have is, SPAM is not
>>>>>>>> possible to
>>>>>>>> > > reject when
>>>>>>>> > > > reaching a high score at MTA level. For my
>>>>>>>> understanding,
>>>>>>>> > connect
>>>>>>>> > > via
>>>>>>>> > > > milter instead of queue ^HOLD would be the solution.
>>>>>>>> > > >
>>>>>>>> > > > For the next decade we are still using MailScanner
>>>>>>>> instead
>>>>>>>> > of others
>>>>>>>> > > > like Rspamd, because MailScanner is like a mail
>>>>>>>> suite for mail
>>>>>>>> > > security,
>>>>>>>> > > > but if there will never be the possibility to
>>>>>>>> reject at
>>>>>>>> > MTA level
>>>>>>>> > > the
>>>>>>>> > > > high score spam, we will also change in 1-3 years
>>>>>>>> while
>>>>>>>> > replacing
>>>>>>>> > > the OS
>>>>>>>> > > > beyond.
>>>>>>>> > > >
>>>>>>>> > >
>>>>>>>> > > One of MailScanner's strongest features is it's batch
>>>>>>>> mode
>>>>>>>> > processing
>>>>>>>> > > that will allow it to handle a very high volume of mail
>>>>>>>> > flow. I doubt
>>>>>>>> > > that MailScanner will ever be changed to run as a
>>>>>>>> milter for this
>>>>>>>> > > reason.
>>>>>>>> > >
>>>>>>>> > > I tried rspamd and found it wasn't as good as the
>>>>>>>> author
>>>>>>>> > claims so no
>>>>>>>> > > reason to try to use that as a milter. It also wasn't
>>>>>>>> as
>>>>>>>> > fast as it
>>>>>>>> > > claims. I could not send high volumes of mail through
>>>>>>>> it
>>>>>>>> > like I could
>>>>>>>> > > with MailScanner.
>>>>>>>> > >
>>>>>>>> > > If you want to block high scoring spam at the MTA
>>>>>>>> level, I
>>>>>>>> > suggest
>>>>>>>> > > using
>>>>>>>> > > amavis or spamd with the same SA rulesets as
>>>>>>>> MailScanner.
>>>>>>>> > This will
>>>>>>>> > > get
>>>>>>>> > > you most of the power of MailScanner's blocking at the
>>>>>>>> MTA.
>>>>>>>> > >
>>>>>>>> > > https://wiki.apache.org/spamassassin/IntegratedInMta
>>>>>>>> > >
>>>>>>>> > > If you you use postscreen and postwhite at the Postfix
>>>>>>>> MTA
>>>>>>>> > level, you
>>>>>>>> > > can block most of the obvious spam with a tuned list of
>>>>>>>> > RBLs. See the
>>>>>>>> > > SA users mailing list over the past year for details
>>>>>>>> on this
>>>>>>>> > from me
>>>>>>>> > > and
>>>>>>>> > > a few others.
>>>>>>>> > >
>>>>>>>> > > I suggest setting up a quick test VM with iRedmail to
>>>>>>>> get a good
>>>>>>>> > > example
>>>>>>>> > > of how to do TLS and amavis integration well with
>>>>>>>> Postfix.
>>>>>>>> > >
>>>>>>>> > > --
>>>>>>>> > > David Jones
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > > --
>>>>>>>> > > MailScanner mailing list
>>>>>>>> > > mailscanner at lists.mailscanner.info
>>>>>>>> > <mailto:mailscanner at lists.mailscanner.info>
>>>>>>>> > > <mailto:mailscanner at lists.mailscanner.info
>>>>>>>> > <mailto:mailscanner at lists.mailscanner.info>>
>>>>>>>> > > http://lists.mailscanner.info/mailman/listinfo/mailscanner
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > > --
>>>>>>>> > > Shawn Iverson, CETL
>>>>>>>> > > Director of Technology
>>>>>>>> > > Rush County Schools
>>>>>>>> > > 765-932-3901 x1171
>>>>>>>> > > iversons at rushville.k12.in.us
>>>>>>>> > <mailto:iversons at rushville.k12.in.us>
>>>>>>>> > <mailto:iversons at rushville.k12.in.us
>>>>>>>> > <mailto:iversons at rushville.k12.in.us>>
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > David Jones
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > Shawn Iverson, CETL
>>>>>>>> > Director of Technology
>>>>>>>> > Rush County Schools
>>>>>>>> > 765-932-3901 x1171
>>>>>>>> > iversons at rushville.k12.in.us <mailto:iversons at rushville.k12.in.us
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> David Jones
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Shawn Iverson, CETL
>>>>>>> Director of Technology
>>>>>>> Rush County Schools
>>>>>>> 765-932-3901 x1171
>>>>>>> iversons at rushville.k12.in.us
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Shawn Iverson, CETL
>>>>>> Director of Technology
>>>>>> Rush County Schools
>>>>>> 765-932-3901 x1171
>>>>>> iversons at rushville.k12.in.us
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Shawn Iverson, CETL
>>>>> Director of Technology
>>>>> Rush County Schools
>>>>> 765-932-3901 x1171
>>>>> iversons at rushville.k12.in.us
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Shawn Iverson, CETL
>>>> Director of Technology
>>>> Rush County Schools
>>>> 765-932-3901 x1171
>>>> iversons at rushville.k12.in.us
>>>>
>>>>
>>>>
>>>
>>> --
>>> Shawn Iverson, CETL
>>> Director of Technology
>>> Rush County Schools
>>> 765-932-3901 x1171
>>> iversons at rushville.k12.in.us
>>>
>>>
>>>
>>
>> --
>> Shawn Iverson, CETL
>> Director of Technology
>> Rush County Schools
>> 765-932-3901 x1171
>> iversons at rushville.k12.in.us
>>
>>
>>
>
> --
> Shawn Iverson, CETL
> Director of Technology
> Rush County Schools
> 765-932-3901 x1171
> iversons at rushville.k12.in.us
>
>
>
--
Shawn Iverson, CETL
Director of Technology
Rush County Schools
765-932-3901 x1171
iversons at rushville.k12.in.us
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20180820/9fcb465f/attachment-0001.html>
More information about the MailScanner
mailing list