Mailscanner milter to reject high score spam at MTA level
Shawn Iverson
iversons at rushville.k12.in.us
Sun Aug 19 13:30:24 UTC 2018
Oh yeah, here's the config for Postfix:
smtpd_milters = inet:127.0.0.1:33333
smtpd_milter_maps = cidr:/etc/postfix/smtpd_milter_map
/etc/postfix/smtpd_milter_map:
127.0.0.0/8 DISABLE
::/64 DISABLE
This allows scanned emails to pass the milter, as well as notifications
sent from the localhost. You do need at least Postfix version 3.2 I
believe to have milter map support.
On Sat, Aug 18, 2018 at 11:28 PM Shawn Iverson <iversons at rushville.k12.in.us>
wrote:
> MailScanner users:
>
> The MailScanner Milter project is coming along nicely.
>
> https://github.com/shawniverson/v5/commits/081118msmilter
>
> I am currently running this on a split relay to test the milter without
> impacting production email.
>
> The design is fairly simple, although development has taken about 40 hours
> of my time. I know more about MailScanner (and perl) than I ever have :D
>
> The Milter is integrated into MailScanner and forks as a branch of the
> MailScanner process tree, keeping systemd happy.
>
> The Milter process intercepts incoming email and tells postfix to DISCARD,
> which basically accepts the mail and silently drops it before entering the
> queue. At the same time, the Milter writes a raw email file to the
> /var/spool/MailScanner/milterin queue.
>
> MailScanner picks up the message batches in the milterin directory,
> processes them, and spits them out to /var/spool/MailScanner/milterout
> directory as raw email files.
>
> The MSMail Processor (new) relays the messages to postfix for further
> processing over port 25. A optional localhost rule in header_checks
> removes the local entry from the header before delivery.
>
> The benefits are that the postfix queue is not touched at all throughout
> this process, making the solution (hopefully) an acceptable one within the
> postfix community. It is also very fast, and the codebase for this method
> is smaller than even the Postfix Processor, and MailScanner gets its own
> queues, separate from postfix.
>
> One drawback to this method is there is no apparent way to extract the
> Envelope From address (at least not yet, perhaps I am missing a milter
> code), although it doesn't appear that MailScanner is all that concerned
> about it and doesn't go out of its way to capture it. I think it is
> important though, for spoof detection, so I will continue to research this.
>
> Anyone that is willing to get their feet wet and test can apply the
> following files from my branch:
>
> (In common)
> /usr/sbin/MailScanner
> /usr/share/MailScanner/perl/MailScanner/Milter.pm
> /usr/share/MailScanner/perl/MailScanner/MSMail.pm
> /usr/share/MailScanner/perl/MailScanner/MSDiskStore.pm
> /usr/share/MailScanner/perl/MailScanner/ConfigDefs.pm
>
> Then create the following dirs:
> mkdir -p /var/spool/MailScanner/milterin
> chown postfix:mtagroup /var/spool/MailScanner/milterin
> mkdir -p /var/spool/MailScanner/milterout
> chown postfix:mtagroup /var/spool/MailScanner/milterout
>
> Apply the following to /etc/MailScanner/MailScanner.conf:
> Incoming Queue Dir = /var/spool/MailScanner/milterin
> Outgoing Queue Dir = /var/spool/MailScanner/milterout
> MTA = MSMail
> MSMail Queue Type = short | long (pick one that matches your postfix
> setting)
>
> I recommend doing this in a test or split relay environment that
> blackholes email. Do not use in production yet ;)
>
> Known issues at the moment:
> MailWatch doesn't recogize MSMail as an 'MTA' so the queue stats do not
> appear
> More validation and error handling is needed throughout. Weird emails
> abound!
> Need to know the envelope from sender. Currently hidden from the milter,
> but hopefully exposable via a callback code.
>
>
>
>
> On Tue, Aug 14, 2018 at 10:56 AM Shawn Iverson <
> iversons at rushville.k12.in.us> wrote:
>
>> Dear MailScanner users:
>>
>> I am officially working on creating a lightweight milter for
>> MailScanner.
>>
>> This milter will not provide MTA protocol rejection for postfix, due to
>> the severe performance penalty it would cause. All mail will be
>> intercepted, accepted, and silently dropped from the postfix queue and
>> placed in a MailScanner queue.
>>
>> I have a working prototype, and it is processing mail! It is in need of
>> heavy refactoring and some bug squashing.
>>
>> Currently it attempts to create a postfix formatted queue file (very
>> ugly, who thought up this file format???!!!). I may instead create a new
>> Milter Processor for MailScanner that reduces the overhead of doing this
>> and can read the incoming email in a simple line-by-line format. This may
>> also increase performance overall and reduce all the conversions happening.
>>
>> The other side of the coin is what to do when MailScanner is done
>> processing mail. Currently, it generates a postfix queue file and drops it
>> into postfix incoming directory. It should not do this but instead drop
>> the message into postfix using native postfix tools. That will be the next
>> part I tackle as part of the Milter Processor.
>>
>> Why am I doing this? I want to place MailScanner back in a good standing
>> with Postfix folks (at least when the milter + postfix method is in use).
>>
>> I have no plans of removing the old method but rather provide a more
>> supported path for postfix users.
>>
>> Wish me luck. I could be heard across the neighborhood when MailScanner
>> processed an email from the Milter for the first time! :D
>>
>>
>>
>>
>> On Sat, Aug 11, 2018 at 9:58 AM David Jones <djones at ena.com> wrote:
>>
>>> On 08/11/2018 08:52 AM, Shawn Iverson wrote:
>>> > David,
>>> >
>>> > I agree that this is true, and part of my lack of motivation to do
>>> it.
>>> > One reason I wanted it as an option was to reconcile the ongoing
>>> > conflict with the postfix community and return MailScanner to good
>>> > standing to this community. Weitze has been very stern about
>>> > MailScanner directly tapping the postfix queues.
>>> >
>>> > Perhaps an alternative option would be to create a fast MailScanner
>>> > milter that behaves more like the HOLD queue. Basically just a milter
>>> > that immediately fires back accept to postfix and places all the
>>> > messages in a MailScanner HOLD queue as opposed to a postfix HOLD
>>> > queue. Doing so would maintain speed, simplicity, and be more
>>> compliant
>>> > with postfix. The code would also be very simple.
>>> >
>>> > Then, as you say, if you need MTA level functionality for SA, use
>>> other
>>> > software and methods.
>>> >
>>> >
>>>
>>> This light MS milter would make a lot of sense based on your goal to get
>>> compliant with Postfix and back "in" with the Postfix community. +1
>>>
>>> >
>>> > On Sat, Aug 11, 2018 at 9:39 AM David Jones <djones at ena.com
>>> > <mailto:djones at ena.com>> wrote:
>>> >
>>> > On 08/11/2018 08:15 AM, Shawn Iverson wrote:
>>> > > I have been planning for a MailScanner milter for quite some
>>> > time. I
>>> > > have been specifically studying rpamd's milter source for this
>>> > purpose.
>>> > > Alas, lack of time and lack of money are always an issue, and I
>>> > put a
>>> > > lot of hours in my day job. As Jerry would say, I like to eat
>>> > and have
>>> > > a roof over my head :D
>>> > >
>>> > > If I do find the time to build a milter, performance will
>>> > definitely be
>>> > > impacted. The reason is that postfix will have to keep each
>>> session
>>> > > open for the duration of scanning, and each MailScanner child
>>> > would have
>>> > > to issue a callback to postfix after scanning the spam so that
>>> > postfix
>>> > > can responds to the connection appropriately (i.e. reject or
>>> > accept).
>>> > > This will slow down mail processing considerably. If I do this,
>>> > I am
>>> > > going to keep the HOLD queue around, so you would have to choose
>>> > between
>>> > > speed or MTA level rejection functionality.
>>> > >
>>> > >
>>> > >
>>> >
>>> > My gut tells me that this is going to be so slow, that it's not
>>> > going to
>>> > be worth the time to put into it. If you want to reject at MTA
>>> time,
>>> > throw in amavis-new or spamd (not rspamd) using the same
>>> SpamAsssassin
>>> > rules and Bayes DB to get most of the same features as MailScanner
>>> > during the SMTP conversation. Then the mail that gets through can
>>> be
>>> > filtered by MailScanner for it's extra features that make it
>>> unique.
>>> >
>>> > I understand there are different local legal requirements around
>>> the
>>> > world that if email is accepted at MTA time then it has to be
>>> passed on
>>> > to the end user's mailbox. If you are located in one of these
>>> > countries, then this would be more of an issue. But since I am in
>>> a
>>> > country that doesn't have this legal requirement, I do block email
>>> > post-MTA by MailScanner.
>>> >
>>> > The majority of my spam is blocked at the MTA level already by
>>> highly
>>> > tuned RBLs and postscreen's RBL weighting which is very, very good.
>>> > Only a small percentage of spam that is zero-hour or from
>>> compromised
>>> > accounts makes it to MailScanner.
>>> >
>>> > I highly recommend the Invaluement RBL. It's very accurate -- only
>>> > 1 or
>>> > 2 false positives over 5+ the years. This RBL is very cost
>>> effective
>>> > and has allowed me to disable all Spamhaus RBL checks in
>>> SpamAssassin
>>> > saving thousands of dollars a year. (We have too high a volume to
>>> stay
>>> > under the free usage limits of Spamhaus so we were having to pay
>>> for
>>> > the
>>> > RBL feed.)
>>> >
>>> > >
>>> > >
>>> > >
>>> > > On Tue, Aug 7, 2018 at 10:52 AM David Jones via MailScanner
>>> > > <mailscanner at lists.mailscanner.info
>>> > <mailto:mailscanner at lists.mailscanner.info>
>>> > > <mailto:mailscanner at lists.mailscanner.info
>>> > <mailto:mailscanner at lists.mailscanner.info>>> wrote:
>>> > >
>>> > > On 08/07/2018 05:03 AM, info at schroeffu.ch
>>> > <mailto:info at schroeffu.ch> <mailto:info at schroeffu.ch
>>> > <mailto:info at schroeffu.ch>>
>>> > > wrote:
>>> > > >
>>> > > > Hi Mailscanner friends,
>>> > > >
>>> > > > is there any progress to make MailScanner usable as a
>>> > postfix milter?
>>> > > > The most biggest problem I have is, SPAM is not possible
>>> to
>>> > > reject when
>>> > > > reaching a high score at MTA level. For my understanding,
>>> > connect
>>> > > via
>>> > > > milter instead of queue ^HOLD would be the solution.
>>> > > >
>>> > > > For the next decade we are still using MailScanner
>>> instead
>>> > of others
>>> > > > like Rspamd, because MailScanner is like a mail suite
>>> for mail
>>> > > security,
>>> > > > but if there will never be the possibility to reject at
>>> > MTA level
>>> > > the
>>> > > > high score spam, we will also change in 1-3 years while
>>> > replacing
>>> > > the OS
>>> > > > beyond.
>>> > > >
>>> > >
>>> > > One of MailScanner's strongest features is it's batch mode
>>> > processing
>>> > > that will allow it to handle a very high volume of mail
>>> > flow. I doubt
>>> > > that MailScanner will ever be changed to run as a milter
>>> for this
>>> > > reason.
>>> > >
>>> > > I tried rspamd and found it wasn't as good as the author
>>> > claims so no
>>> > > reason to try to use that as a milter. It also wasn't as
>>> > fast as it
>>> > > claims. I could not send high volumes of mail through it
>>> > like I could
>>> > > with MailScanner.
>>> > >
>>> > > If you want to block high scoring spam at the MTA level, I
>>> > suggest
>>> > > using
>>> > > amavis or spamd with the same SA rulesets as MailScanner.
>>> > This will
>>> > > get
>>> > > you most of the power of MailScanner's blocking at the MTA.
>>> > >
>>> > > https://wiki.apache.org/spamassassin/IntegratedInMta
>>> > >
>>> > > If you you use postscreen and postwhite at the Postfix MTA
>>> > level, you
>>> > > can block most of the obvious spam with a tuned list of
>>> > RBLs. See the
>>> > > SA users mailing list over the past year for details on this
>>> > from me
>>> > > and
>>> > > a few others.
>>> > >
>>> > > I suggest setting up a quick test VM with iRedmail to get a
>>> good
>>> > > example
>>> > > of how to do TLS and amavis integration well with Postfix.
>>> > >
>>> > > --
>>> > > David Jones
>>> > >
>>> > >
>>> > > --
>>> > > MailScanner mailing list
>>> > > mailscanner at lists.mailscanner.info
>>> > <mailto:mailscanner at lists.mailscanner.info>
>>> > > <mailto:mailscanner at lists.mailscanner.info
>>> > <mailto:mailscanner at lists.mailscanner.info>>
>>> > > http://lists.mailscanner.info/mailman/listinfo/mailscanner
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Shawn Iverson, CETL
>>> > > Director of Technology
>>> > > Rush County Schools
>>> > > 765-932-3901 x1171
>>> > > iversons at rushville.k12.in.us
>>> > <mailto:iversons at rushville.k12.in.us>
>>> > <mailto:iversons at rushville.k12.in.us
>>> > <mailto:iversons at rushville.k12.in.us>>
>>> > >
>>> > >
>>> >
>>> > --
>>> > David Jones
>>> >
>>> >
>>> >
>>> > --
>>> > Shawn Iverson, CETL
>>> > Director of Technology
>>> > Rush County Schools
>>> > 765-932-3901 x1171
>>> > iversons at rushville.k12.in.us <mailto:iversons at rushville.k12.in.us>
>>> >
>>> >
>>>
>>>
>>> --
>>> David Jones
>>>
>>
>>
>> --
>> Shawn Iverson, CETL
>> Director of Technology
>> Rush County Schools
>> 765-932-3901 x1171
>> iversons at rushville.k12.in.us
>>
>>
>>
>
> --
> Shawn Iverson, CETL
> Director of Technology
> Rush County Schools
> 765-932-3901 x1171
> iversons at rushville.k12.in.us
>
>
>
--
Shawn Iverson, CETL
Director of Technology
Rush County Schools
765-932-3901 x1171
iversons at rushville.k12.in.us
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20180819/d4db6f48/attachment.html>
More information about the MailScanner
mailing list