Mailscanner milter to reject high score spam at MTA level

[Shawn Iverson]

RFC 5321 support added.

Also reviewing RFC 822, particularly handling of delivery failure notices,
with respect to the milter.

https://github.com/shawniverson/v5/commit/6eef93cb3cbc49f72cdca656e8a2bf655de351645321


Currently addressing a bug with the daemon forking new children.   Seems to
be caused by the Milter being a child process of MailScanner.  If I cannot
resolve, I plan to separate the Milter into its own daemon.  This may be
better anyway and allow both to be managed independently.

On Sun, Aug 19, 2018 at 5:14 PM Shawn Iverson <iversons at rushville.k12.in.us>
wrote:

> Latest commit
>
> Spamassassin/MailScanner are now comparing envelope from and from properly
> :)
>
>
> https://github.com/shawniverson/v5/commit/625040caedc93b2c4e78edf305b69ac8a82bc25b
>
> Todo:
>
> Cleanup code
> Add more options to mailscanner config, such as # of milter threads, etc.
> Remove hard coded items where feasible
> Honor placement of Header entries in header (top or bottom)
> Test Test Test (anyone interested is welcome!)
> Patches for MailWatch to handle new queues and process counts
> Fix bug observed where mailscanner thinks processes are still running when
> stopping (but aren't, harmless, just weird)
>
> Feedback is welcome.
>
>
>
>
>
>
>
> On Sun, Aug 19, 2018 at 4:26 PM Shawn Iverson <
> iversons at rushville.k12.in.us> wrote:
>
>> Found the callback code for MAIL FROM:
>>
>> 'M'     SMFIC_MAIL      MAIL FROM: information
>>                         Expected response:  Accept/reject action
>>
>> Time for some more coding :D
>>
>> On Sun, Aug 19, 2018 at 4:04 PM Shawn Iverson <
>> iversons at rushville.k12.in.us> wrote:
>>
>>> Another update
>>>
>>> The latest commit to my branch includes more fixes and a new thing that
>>> needs handled now that a Milter is in play.  When mail is submitted back to
>>> postfix, it needs to be processed by other things that could reject the
>>> email (such as an blocked sender).  This is a problem because MailScanner
>>> does not know how to handle rejects since it has always been part of the
>>> queue process interacting directly with the queues, not before queue
>>> process.
>>>
>>> During message delivery, if a reject is detected, I inject a special
>>> header to the message and requeue it to MailScanner.  MailScanner has a
>>> chance, based on this special header to flag the message, remove the
>>> special header, add diagnostic info to the header about the relay, and
>>> quarantine the message.  The mail admin is happy knowing that the message
>>> didn't just vanish and has an opportunity to resolve the issue and release
>>> it or know the disposition of the email.
>>>
>>> Another cool thing about this Milter Processor I discovered is you can
>>> simply drop a message into /var/spool/MailScanner/milterin from
>>> /var/spool/MailScanner/quarantine, and it will try to redeliver it :D
>>>
>>> This new code is in Message.pm and only becomes active if the milter is
>>> activated.
>>>
>>> On Sun, Aug 19, 2018 at 9:30 AM Shawn Iverson <
>>> iversons at rushville.k12.in.us> wrote:
>>>
>>>> Oh yeah, here's the config for Postfix:
>>>>
>>>> smtpd_milters = inet:127.0.0.1:33333
>>>> smtpd_milter_maps = cidr:/etc/postfix/smtpd_milter_map
>>>>
>>>> /etc/postfix/smtpd_milter_map:
>>>> 127.0.0.0/8 DISABLE
>>>> ::/64 DISABLE
>>>>
>>>> This allows scanned emails to pass the milter, as well as notifications
>>>> sent from the localhost.  You do need at least Postfix version 3.2 I
>>>> believe to have milter map support.
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Aug 18, 2018 at 11:28 PM Shawn Iverson <
>>>> iversons at rushville.k12.in.us> wrote:
>>>>
>>>>> MailScanner users:
>>>>>
>>>>> The MailScanner Milter project is coming along nicely.
>>>>>
>>>>> https://github.com/shawniverson/v5/commits/081118msmilter
>>>>>
>>>>> I am currently running this on a split relay to test the milter
>>>>> without impacting production email.
>>>>>
>>>>> The design is fairly simple, although development has taken about 40
>>>>> hours of my time.  I know more about MailScanner (and perl) than I ever
>>>>> have :D
>>>>>
>>>>> The Milter is integrated into MailScanner and forks as a branch of the
>>>>> MailScanner process tree, keeping systemd happy.
>>>>>
>>>>> The Milter process intercepts incoming email and tells postfix to
>>>>> DISCARD, which basically accepts the mail and silently drops it before
>>>>> entering the queue.  At the same time, the Milter writes a raw email file
>>>>> to the /var/spool/MailScanner/milterin queue.
>>>>>
>>>>> MailScanner picks up the message batches in the milterin directory,
>>>>> processes them, and spits them out to /var/spool/MailScanner/milterout
>>>>> directory as raw email files.
>>>>>
>>>>> The MSMail Processor (new) relays the messages to postfix for further
>>>>> processing over port 25.  A optional localhost rule in header_checks
>>>>> removes the local entry from the header before delivery.
>>>>>
>>>>> The benefits are that the postfix queue is not touched at all
>>>>> throughout this process, making the solution (hopefully) an acceptable one
>>>>> within the postfix community.  It is also very fast, and the codebase for
>>>>> this method is smaller than even the Postfix Processor, and MailScanner
>>>>> gets its own queues, separate from postfix.
>>>>>
>>>>> One drawback to this method is there is no apparent way to extract the
>>>>> Envelope From address (at least not yet, perhaps I am missing a milter
>>>>> code), although it doesn't appear that MailScanner is all that concerned
>>>>> about it and doesn't go out of its way to capture it.  I think it is
>>>>> important though, for spoof detection, so I will continue to research this.
>>>>>
>>>>> Anyone that is willing to get their feet wet and test can apply the
>>>>> following files from my branch:
>>>>>
>>>>> (In common)
>>>>> /usr/sbin/MailScanner
>>>>> /usr/share/MailScanner/perl/MailScanner/Milter.pm
>>>>> /usr/share/MailScanner/perl/MailScanner/MSMail.pm
>>>>> /usr/share/MailScanner/perl/MailScanner/MSDiskStore.pm
>>>>> /usr/share/MailScanner/perl/MailScanner/ConfigDefs.pm
>>>>>
>>>>> Then create the following dirs:
>>>>> mkdir -p /var/spool/MailScanner/milterin
>>>>> chown postfix:mtagroup /var/spool/MailScanner/milterin
>>>>> mkdir -p /var/spool/MailScanner/milterout
>>>>> chown postfix:mtagroup /var/spool/MailScanner/milterout
>>>>>
>>>>> Apply the following to /etc/MailScanner/MailScanner.conf:
>>>>> Incoming Queue Dir = /var/spool/MailScanner/milterin
>>>>> Outgoing Queue Dir = /var/spool/MailScanner/milterout
>>>>> MTA = MSMail
>>>>> MSMail Queue Type = short | long (pick one that matches your postfix
>>>>> setting)
>>>>>
>>>>> I recommend doing this in a test or split relay environment that
>>>>> blackholes email.  Do not use in production yet ;)
>>>>>
>>>>> Known issues at the moment:
>>>>> MailWatch doesn't recogize MSMail as an 'MTA' so the queue stats do
>>>>> not appear
>>>>> More validation and error handling is needed throughout.  Weird emails
>>>>> abound!
>>>>> Need to know the envelope from sender.  Currently hidden from the
>>>>> milter, but hopefully exposable via a callback code.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 14, 2018 at 10:56 AM Shawn Iverson <
>>>>> iversons at rushville.k12.in.us> wrote:
>>>>>
>>>>>> Dear MailScanner users:
>>>>>>
>>>>>> I am officially working on creating a lightweight milter for
>>>>>> MailScanner.
>>>>>>
>>>>>> This milter will not provide MTA protocol rejection for postfix, due
>>>>>> to the severe performance penalty it would cause.  All mail will be
>>>>>> intercepted, accepted, and silently dropped from the postfix queue and
>>>>>> placed in a MailScanner queue.
>>>>>>
>>>>>> I have a working prototype, and it is processing mail!  It is in need
>>>>>> of heavy refactoring and some bug squashing.
>>>>>>
>>>>>> Currently it attempts to create a postfix formatted queue file (very
>>>>>> ugly, who thought up this file format???!!!).  I may instead create a new
>>>>>> Milter Processor for MailScanner that reduces the overhead of doing this
>>>>>> and can read the incoming email in a simple line-by-line format.  This may
>>>>>> also increase performance overall and reduce all the conversions happening.
>>>>>>
>>>>>> The other side of the coin is what to do when MailScanner is done
>>>>>> processing mail.  Currently, it generates a postfix queue file and drops it
>>>>>> into postfix incoming directory.  It should not do this but instead drop
>>>>>> the message into postfix using native postfix tools.  That will be the next
>>>>>> part I tackle as part of the Milter Processor.
>>>>>>
>>>>>> Why am I doing this?  I want to place MailScanner back in a good
>>>>>> standing with Postfix folks (at least when the milter + postfix method is
>>>>>> in use).
>>>>>>
>>>>>> I have no plans of removing the old method but rather provide a more
>>>>>> supported path for postfix users.
>>>>>>
>>>>>> Wish me luck.  I could be heard across the neighborhood when
>>>>>> MailScanner processed an email from the Milter for the first time! :D
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Aug 11, 2018 at 9:58 AM David Jones <djones at ena.com> wrote:
>>>>>>
>>>>>>> On 08/11/2018 08:52 AM, Shawn Iverson wrote:
>>>>>>> > David,
>>>>>>> >
>>>>>>> > I agree that this is true, and part of my lack of motivation to do
>>>>>>> it.
>>>>>>> > One reason I wanted it as an option was to reconcile the ongoing
>>>>>>> > conflict with the postfix community and return MailScanner to good
>>>>>>> > standing to this community.  Weitze has been very stern about
>>>>>>> > MailScanner directly tapping the postfix queues.
>>>>>>> >
>>>>>>> > Perhaps an alternative option would be to create a fast
>>>>>>> MailScanner
>>>>>>> > milter that behaves more like the HOLD queue.  Basically just a
>>>>>>> milter
>>>>>>> > that immediately fires back accept to postfix and places all the
>>>>>>> > messages in a MailScanner HOLD queue as opposed to a postfix HOLD
>>>>>>> > queue.  Doing so would maintain speed, simplicity, and be more
>>>>>>> compliant
>>>>>>> > with postfix. The code would also be very simple.
>>>>>>> >
>>>>>>> > Then, as you say, if you need MTA level functionality for SA, use
>>>>>>> other
>>>>>>> > software and methods.
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>> This light MS milter would make a lot of sense based on your goal to
>>>>>>> get
>>>>>>> compliant with Postfix and back "in" with the Postfix community.  +1
>>>>>>>
>>>>>>> >
>>>>>>> > On Sat, Aug 11, 2018 at 9:39 AM David Jones <djones at ena.com
>>>>>>> > <mailto:djones at ena.com>> wrote:
>>>>>>> >
>>>>>>> >     On 08/11/2018 08:15 AM, Shawn Iverson wrote:
>>>>>>> >      > I have been planning for a MailScanner milter for quite some
>>>>>>> >     time.  I
>>>>>>> >      > have been specifically studying rpamd's milter source for
>>>>>>> this
>>>>>>> >     purpose.
>>>>>>> >      > Alas, lack of time and lack of money are always an issue,
>>>>>>> and I
>>>>>>> >     put a
>>>>>>> >      > lot of hours in my day job.  As Jerry would say, I like to
>>>>>>> eat
>>>>>>> >     and have
>>>>>>> >      > a roof over my head :D
>>>>>>> >      >
>>>>>>> >      > If I do find the time to build a milter, performance will
>>>>>>> >     definitely be
>>>>>>> >      > impacted.  The reason is that postfix will have to keep
>>>>>>> each session
>>>>>>> >      > open for the duration of scanning, and each MailScanner
>>>>>>> child
>>>>>>> >     would have
>>>>>>> >      > to issue a callback to postfix after scanning the spam so
>>>>>>> that
>>>>>>> >     postfix
>>>>>>> >      > can responds to the connection appropriately  (i.e. reject
>>>>>>> or
>>>>>>> >     accept).
>>>>>>> >      > This will slow down mail processing considerably.  If I do
>>>>>>> this,
>>>>>>> >     I am
>>>>>>> >      > going to keep the HOLD queue around, so you would have to
>>>>>>> choose
>>>>>>> >     between
>>>>>>> >      > speed or MTA level rejection functionality.
>>>>>>> >      >
>>>>>>> >      >
>>>>>>> >      >
>>>>>>> >
>>>>>>> >     My gut tells me that this is going to be so slow, that it's not
>>>>>>> >     going to
>>>>>>> >     be worth the time to put into it.  If you want to reject at
>>>>>>> MTA time,
>>>>>>> >     throw in amavis-new or spamd (not rspamd) using the same
>>>>>>> SpamAsssassin
>>>>>>> >     rules and Bayes DB to get most of the same features as
>>>>>>> MailScanner
>>>>>>> >     during the SMTP conversation.  Then the mail that gets through
>>>>>>> can be
>>>>>>> >     filtered by MailScanner for it's extra features that make it
>>>>>>> unique.
>>>>>>> >
>>>>>>> >     I understand there are different local legal requirements
>>>>>>> around the
>>>>>>> >     world that if email is accepted at MTA time then it has to be
>>>>>>> passed on
>>>>>>> >     to the end user's mailbox.  If you are located in one of these
>>>>>>> >     countries, then this would be more of an issue.  But since I
>>>>>>> am in a
>>>>>>> >     country that doesn't have this legal requirement, I do block
>>>>>>> email
>>>>>>> >     post-MTA by MailScanner.
>>>>>>> >
>>>>>>> >     The majority of my spam is blocked at the MTA level already by
>>>>>>> highly
>>>>>>> >     tuned RBLs and postscreen's RBL weighting which is very, very
>>>>>>> good.
>>>>>>> >     Only a small percentage of spam that is zero-hour or from
>>>>>>> compromised
>>>>>>> >     accounts makes it to MailScanner.
>>>>>>> >
>>>>>>> >     I highly recommend the Invaluement RBL.  It's very accurate --
>>>>>>> only
>>>>>>> >     1 or
>>>>>>> >     2 false positives over 5+ the years.  This RBL is very cost
>>>>>>> effective
>>>>>>> >     and has allowed me to disable all Spamhaus RBL checks in
>>>>>>> SpamAssassin
>>>>>>> >     saving thousands of dollars a year.  (We have too high a
>>>>>>> volume to stay
>>>>>>> >     under the free usage limits of Spamhaus so we were having to
>>>>>>> pay for
>>>>>>> >     the
>>>>>>> >     RBL feed.)
>>>>>>> >
>>>>>>> >      >
>>>>>>> >      >
>>>>>>> >      >
>>>>>>> >      > On Tue, Aug 7, 2018 at 10:52 AM David Jones via MailScanner
>>>>>>> >      > <mailscanner at lists.mailscanner.info
>>>>>>> >     <mailto:mailscanner at lists.mailscanner.info>
>>>>>>> >      > <mailto:mailscanner at lists.mailscanner.info
>>>>>>> >     <mailto:mailscanner at lists.mailscanner.info>>> wrote:
>>>>>>> >      >
>>>>>>> >      >     On 08/07/2018 05:03 AM, info at schroeffu.ch
>>>>>>> >     <mailto:info at schroeffu.ch> <mailto:info at schroeffu.ch
>>>>>>> >     <mailto:info at schroeffu.ch>>
>>>>>>> >      >     wrote:
>>>>>>> >      >      >
>>>>>>> >      >      > Hi Mailscanner friends,
>>>>>>> >      >      >
>>>>>>> >      >      > is there any progress to make MailScanner usable as a
>>>>>>> >     postfix milter?
>>>>>>> >      >      > The most biggest problem I have is, SPAM is not
>>>>>>> possible to
>>>>>>> >      >     reject when
>>>>>>> >      >      > reaching a high score at MTA level. For my
>>>>>>> understanding,
>>>>>>> >     connect
>>>>>>> >      >     via
>>>>>>> >      >      > milter instead of queue ^HOLD would be the solution.
>>>>>>> >      >      >
>>>>>>> >      >      > For the next decade we are still using MailScanner
>>>>>>> instead
>>>>>>> >     of others
>>>>>>> >      >      > like Rspamd, because MailScanner is like a mail
>>>>>>> suite for mail
>>>>>>> >      >     security,
>>>>>>> >      >      > but if there will never be the possibility to reject
>>>>>>> at
>>>>>>> >     MTA level
>>>>>>> >      >     the
>>>>>>> >      >      > high score spam, we will also change in 1-3 years
>>>>>>> while
>>>>>>> >     replacing
>>>>>>> >      >     the OS
>>>>>>> >      >      > beyond.
>>>>>>> >      >      >
>>>>>>> >      >
>>>>>>> >      >     One of MailScanner's strongest features is it's batch
>>>>>>> mode
>>>>>>> >     processing
>>>>>>> >      >     that will allow it to handle a very high volume of mail
>>>>>>> >     flow.  I doubt
>>>>>>> >      >     that MailScanner will ever be changed to run as a
>>>>>>> milter for this
>>>>>>> >      >     reason.
>>>>>>> >      >
>>>>>>> >      >     I tried rspamd and found it wasn't as good as the author
>>>>>>> >     claims so no
>>>>>>> >      >     reason to try to use that as a milter.  It also wasn't
>>>>>>> as
>>>>>>> >     fast as it
>>>>>>> >      >     claims.  I could not send high volumes of mail through
>>>>>>> it
>>>>>>> >     like I could
>>>>>>> >      >     with MailScanner.
>>>>>>> >      >
>>>>>>> >      >     If you want to block high scoring spam at the MTA
>>>>>>> level, I
>>>>>>> >     suggest
>>>>>>> >      >     using
>>>>>>> >      >     amavis or spamd with the same SA rulesets as
>>>>>>> MailScanner.
>>>>>>> >     This will
>>>>>>> >      >     get
>>>>>>> >      >     you most of the power of MailScanner's blocking at the
>>>>>>> MTA.
>>>>>>> >      >
>>>>>>> >      > https://wiki.apache.org/spamassassin/IntegratedInMta
>>>>>>> >      >
>>>>>>> >      >     If you you use postscreen and postwhite at the Postfix
>>>>>>> MTA
>>>>>>> >     level, you
>>>>>>> >      >     can block most of the obvious spam with a tuned list of
>>>>>>> >     RBLs.  See the
>>>>>>> >      >     SA users mailing list over the past year for details on
>>>>>>> this
>>>>>>> >     from me
>>>>>>> >      >     and
>>>>>>> >      >     a few others.
>>>>>>> >      >
>>>>>>> >      >     I suggest setting up a quick test VM with iRedmail to
>>>>>>> get a good
>>>>>>> >      >     example
>>>>>>> >      >     of how to do TLS and amavis integration well with
>>>>>>> Postfix.
>>>>>>> >      >
>>>>>>> >      >     --
>>>>>>> >      >     David Jones
>>>>>>> >      >
>>>>>>> >      >
>>>>>>> >      >     --
>>>>>>> >      >     MailScanner mailing list
>>>>>>> >      > mailscanner at lists.mailscanner.info
>>>>>>> >     <mailto:mailscanner at lists.mailscanner.info>
>>>>>>> >      >     <mailto:mailscanner at lists.mailscanner.info
>>>>>>> >     <mailto:mailscanner at lists.mailscanner.info>>
>>>>>>> >      > http://lists.mailscanner.info/mailman/listinfo/mailscanner
>>>>>>> >      >
>>>>>>> >      >
>>>>>>> >      >
>>>>>>> >      > --
>>>>>>> >      > Shawn Iverson, CETL
>>>>>>> >      > Director of Technology
>>>>>>> >      > Rush County Schools
>>>>>>> >      > 765-932-3901 x1171
>>>>>>> >      > iversons at rushville.k12.in.us
>>>>>>> >     <mailto:iversons at rushville.k12.in.us>
>>>>>>> >     <mailto:iversons at rushville.k12.in.us
>>>>>>> >     <mailto:iversons at rushville.k12.in.us>>
>>>>>>> >      >
>>>>>>> >      >
>>>>>>> >
>>>>>>> >     --
>>>>>>> >     David Jones
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > Shawn Iverson, CETL
>>>>>>> > Director of Technology
>>>>>>> > Rush County Schools
>>>>>>> > 765-932-3901 x1171
>>>>>>> > iversons at rushville.k12.in.us <mailto:iversons at rushville.k12.in.us>
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> David Jones
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Shawn Iverson, CETL
>>>>>> Director of Technology
>>>>>> Rush County Schools
>>>>>> 765-932-3901 x1171
>>>>>> iversons at rushville.k12.in.us
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Shawn Iverson, CETL
>>>>> Director of Technology
>>>>> Rush County Schools
>>>>> 765-932-3901 x1171
>>>>> iversons at rushville.k12.in.us
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Shawn Iverson, CETL
>>>> Director of Technology
>>>> Rush County Schools
>>>> 765-932-3901 x1171
>>>> iversons at rushville.k12.in.us
>>>>
>>>>
>>>>
>>>
>>> --
>>> Shawn Iverson, CETL
>>> Director of Technology
>>> Rush County Schools
>>> 765-932-3901 x1171
>>> iversons at rushville.k12.in.us
>>>
>>>
>>>
>>
>> --
>> Shawn Iverson, CETL
>> Director of Technology
>> Rush County Schools
>> 765-932-3901 x1171
>> iversons at rushville.k12.in.us
>>
>>
>>
>
> --
> Shawn Iverson, CETL
> Director of Technology
> Rush County Schools
> 765-932-3901 x1171
> iversons at rushville.k12.in.us
>
>
>

-- 
Shawn Iverson, CETL
Director of Technology
Rush County Schools
765-932-3901 x1171
iversons at rushville.k12.in.us
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20180820/fc6a8545/attachment.html>