Mailscanner milter to reject high score spam at MTA level
Shawn Iverson
iversons at rushville.k12.in.us
Sun Aug 19 20:26:11 UTC 2018
Found the callback code for MAIL FROM:
'M' SMFIC_MAIL MAIL FROM: information
Expected response: Accept/reject action
Time for some more coding :D
On Sun, Aug 19, 2018 at 4:04 PM Shawn Iverson <iversons at rushville.k12.in.us>
wrote:
> Another update
>
> The latest commit to my branch includes more fixes and a new thing that
> needs handled now that a Milter is in play. When mail is submitted back to
> postfix, it needs to be processed by other things that could reject the
> email (such as an blocked sender). This is a problem because MailScanner
> does not know how to handle rejects since it has always been part of the
> queue process interacting directly with the queues, not before queue
> process.
>
> During message delivery, if a reject is detected, I inject a special
> header to the message and requeue it to MailScanner. MailScanner has a
> chance, based on this special header to flag the message, remove the
> special header, add diagnostic info to the header about the relay, and
> quarantine the message. The mail admin is happy knowing that the message
> didn't just vanish and has an opportunity to resolve the issue and release
> it or know the disposition of the email.
>
> Another cool thing about this Milter Processor I discovered is you can
> simply drop a message into /var/spool/MailScanner/milterin from
> /var/spool/MailScanner/quarantine, and it will try to redeliver it :D
>
> This new code is in Message.pm and only becomes active if the milter is
> activated.
>
> On Sun, Aug 19, 2018 at 9:30 AM Shawn Iverson <
> iversons at rushville.k12.in.us> wrote:
>
>> Oh yeah, here's the config for Postfix:
>>
>> smtpd_milters = inet:127.0.0.1:33333
>> smtpd_milter_maps = cidr:/etc/postfix/smtpd_milter_map
>>
>> /etc/postfix/smtpd_milter_map:
>> 127.0.0.0/8 DISABLE
>> ::/64 DISABLE
>>
>> This allows scanned emails to pass the milter, as well as notifications
>> sent from the localhost. You do need at least Postfix version 3.2 I
>> believe to have milter map support.
>>
>>
>>
>>
>> On Sat, Aug 18, 2018 at 11:28 PM Shawn Iverson <
>> iversons at rushville.k12.in.us> wrote:
>>
>>> MailScanner users:
>>>
>>> The MailScanner Milter project is coming along nicely.
>>>
>>> https://github.com/shawniverson/v5/commits/081118msmilter
>>>
>>> I am currently running this on a split relay to test the milter without
>>> impacting production email.
>>>
>>> The design is fairly simple, although development has taken about 40
>>> hours of my time. I know more about MailScanner (and perl) than I ever
>>> have :D
>>>
>>> The Milter is integrated into MailScanner and forks as a branch of the
>>> MailScanner process tree, keeping systemd happy.
>>>
>>> The Milter process intercepts incoming email and tells postfix to
>>> DISCARD, which basically accepts the mail and silently drops it before
>>> entering the queue. At the same time, the Milter writes a raw email file
>>> to the /var/spool/MailScanner/milterin queue.
>>>
>>> MailScanner picks up the message batches in the milterin directory,
>>> processes them, and spits them out to /var/spool/MailScanner/milterout
>>> directory as raw email files.
>>>
>>> The MSMail Processor (new) relays the messages to postfix for further
>>> processing over port 25. A optional localhost rule in header_checks
>>> removes the local entry from the header before delivery.
>>>
>>> The benefits are that the postfix queue is not touched at all throughout
>>> this process, making the solution (hopefully) an acceptable one within the
>>> postfix community. It is also very fast, and the codebase for this method
>>> is smaller than even the Postfix Processor, and MailScanner gets its own
>>> queues, separate from postfix.
>>>
>>> One drawback to this method is there is no apparent way to extract the
>>> Envelope From address (at least not yet, perhaps I am missing a milter
>>> code), although it doesn't appear that MailScanner is all that concerned
>>> about it and doesn't go out of its way to capture it. I think it is
>>> important though, for spoof detection, so I will continue to research this.
>>>
>>> Anyone that is willing to get their feet wet and test can apply the
>>> following files from my branch:
>>>
>>> (In common)
>>> /usr/sbin/MailScanner
>>> /usr/share/MailScanner/perl/MailScanner/Milter.pm
>>> /usr/share/MailScanner/perl/MailScanner/MSMail.pm
>>> /usr/share/MailScanner/perl/MailScanner/MSDiskStore.pm
>>> /usr/share/MailScanner/perl/MailScanner/ConfigDefs.pm
>>>
>>> Then create the following dirs:
>>> mkdir -p /var/spool/MailScanner/milterin
>>> chown postfix:mtagroup /var/spool/MailScanner/milterin
>>> mkdir -p /var/spool/MailScanner/milterout
>>> chown postfix:mtagroup /var/spool/MailScanner/milterout
>>>
>>> Apply the following to /etc/MailScanner/MailScanner.conf:
>>> Incoming Queue Dir = /var/spool/MailScanner/milterin
>>> Outgoing Queue Dir = /var/spool/MailScanner/milterout
>>> MTA = MSMail
>>> MSMail Queue Type = short | long (pick one that matches your postfix
>>> setting)
>>>
>>> I recommend doing this in a test or split relay environment that
>>> blackholes email. Do not use in production yet ;)
>>>
>>> Known issues at the moment:
>>> MailWatch doesn't recogize MSMail as an 'MTA' so the queue stats do not
>>> appear
>>> More validation and error handling is needed throughout. Weird emails
>>> abound!
>>> Need to know the envelope from sender. Currently hidden from the
>>> milter, but hopefully exposable via a callback code.
>>>
>>>
>>>
>>>
>>> On Tue, Aug 14, 2018 at 10:56 AM Shawn Iverson <
>>> iversons at rushville.k12.in.us> wrote:
>>>
>>>> Dear MailScanner users:
>>>>
>>>> I am officially working on creating a lightweight milter for
>>>> MailScanner.
>>>>
>>>> This milter will not provide MTA protocol rejection for postfix, due to
>>>> the severe performance penalty it would cause. All mail will be
>>>> intercepted, accepted, and silently dropped from the postfix queue and
>>>> placed in a MailScanner queue.
>>>>
>>>> I have a working prototype, and it is processing mail! It is in need
>>>> of heavy refactoring and some bug squashing.
>>>>
>>>> Currently it attempts to create a postfix formatted queue file (very
>>>> ugly, who thought up this file format???!!!). I may instead create a new
>>>> Milter Processor for MailScanner that reduces the overhead of doing this
>>>> and can read the incoming email in a simple line-by-line format. This may
>>>> also increase performance overall and reduce all the conversions happening.
>>>>
>>>> The other side of the coin is what to do when MailScanner is done
>>>> processing mail. Currently, it generates a postfix queue file and drops it
>>>> into postfix incoming directory. It should not do this but instead drop
>>>> the message into postfix using native postfix tools. That will be the next
>>>> part I tackle as part of the Milter Processor.
>>>>
>>>> Why am I doing this? I want to place MailScanner back in a good
>>>> standing with Postfix folks (at least when the milter + postfix method is
>>>> in use).
>>>>
>>>> I have no plans of removing the old method but rather provide a more
>>>> supported path for postfix users.
>>>>
>>>> Wish me luck. I could be heard across the neighborhood when
>>>> MailScanner processed an email from the Milter for the first time! :D
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Aug 11, 2018 at 9:58 AM David Jones <djones at ena.com> wrote:
>>>>
>>>>> On 08/11/2018 08:52 AM, Shawn Iverson wrote:
>>>>> > David,
>>>>> >
>>>>> > I agree that this is true, and part of my lack of motivation to do
>>>>> it.
>>>>> > One reason I wanted it as an option was to reconcile the ongoing
>>>>> > conflict with the postfix community and return MailScanner to good
>>>>> > standing to this community. Weitze has been very stern about
>>>>> > MailScanner directly tapping the postfix queues.
>>>>> >
>>>>> > Perhaps an alternative option would be to create a fast MailScanner
>>>>> > milter that behaves more like the HOLD queue. Basically just a
>>>>> milter
>>>>> > that immediately fires back accept to postfix and places all the
>>>>> > messages in a MailScanner HOLD queue as opposed to a postfix HOLD
>>>>> > queue. Doing so would maintain speed, simplicity, and be more
>>>>> compliant
>>>>> > with postfix. The code would also be very simple.
>>>>> >
>>>>> > Then, as you say, if you need MTA level functionality for SA, use
>>>>> other
>>>>> > software and methods.
>>>>> >
>>>>> >
>>>>>
>>>>> This light MS milter would make a lot of sense based on your goal to
>>>>> get
>>>>> compliant with Postfix and back "in" with the Postfix community. +1
>>>>>
>>>>> >
>>>>> > On Sat, Aug 11, 2018 at 9:39 AM David Jones <djones at ena.com
>>>>> > <mailto:djones at ena.com>> wrote:
>>>>> >
>>>>> > On 08/11/2018 08:15 AM, Shawn Iverson wrote:
>>>>> > > I have been planning for a MailScanner milter for quite some
>>>>> > time. I
>>>>> > > have been specifically studying rpamd's milter source for this
>>>>> > purpose.
>>>>> > > Alas, lack of time and lack of money are always an issue, and
>>>>> I
>>>>> > put a
>>>>> > > lot of hours in my day job. As Jerry would say, I like to eat
>>>>> > and have
>>>>> > > a roof over my head :D
>>>>> > >
>>>>> > > If I do find the time to build a milter, performance will
>>>>> > definitely be
>>>>> > > impacted. The reason is that postfix will have to keep each
>>>>> session
>>>>> > > open for the duration of scanning, and each MailScanner child
>>>>> > would have
>>>>> > > to issue a callback to postfix after scanning the spam so that
>>>>> > postfix
>>>>> > > can responds to the connection appropriately (i.e. reject or
>>>>> > accept).
>>>>> > > This will slow down mail processing considerably. If I do
>>>>> this,
>>>>> > I am
>>>>> > > going to keep the HOLD queue around, so you would have to
>>>>> choose
>>>>> > between
>>>>> > > speed or MTA level rejection functionality.
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>> > My gut tells me that this is going to be so slow, that it's not
>>>>> > going to
>>>>> > be worth the time to put into it. If you want to reject at MTA
>>>>> time,
>>>>> > throw in amavis-new or spamd (not rspamd) using the same
>>>>> SpamAsssassin
>>>>> > rules and Bayes DB to get most of the same features as
>>>>> MailScanner
>>>>> > during the SMTP conversation. Then the mail that gets through
>>>>> can be
>>>>> > filtered by MailScanner for it's extra features that make it
>>>>> unique.
>>>>> >
>>>>> > I understand there are different local legal requirements around
>>>>> the
>>>>> > world that if email is accepted at MTA time then it has to be
>>>>> passed on
>>>>> > to the end user's mailbox. If you are located in one of these
>>>>> > countries, then this would be more of an issue. But since I am
>>>>> in a
>>>>> > country that doesn't have this legal requirement, I do block
>>>>> email
>>>>> > post-MTA by MailScanner.
>>>>> >
>>>>> > The majority of my spam is blocked at the MTA level already by
>>>>> highly
>>>>> > tuned RBLs and postscreen's RBL weighting which is very, very
>>>>> good.
>>>>> > Only a small percentage of spam that is zero-hour or from
>>>>> compromised
>>>>> > accounts makes it to MailScanner.
>>>>> >
>>>>> > I highly recommend the Invaluement RBL. It's very accurate --
>>>>> only
>>>>> > 1 or
>>>>> > 2 false positives over 5+ the years. This RBL is very cost
>>>>> effective
>>>>> > and has allowed me to disable all Spamhaus RBL checks in
>>>>> SpamAssassin
>>>>> > saving thousands of dollars a year. (We have too high a volume
>>>>> to stay
>>>>> > under the free usage limits of Spamhaus so we were having to pay
>>>>> for
>>>>> > the
>>>>> > RBL feed.)
>>>>> >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > On Tue, Aug 7, 2018 at 10:52 AM David Jones via MailScanner
>>>>> > > <mailscanner at lists.mailscanner.info
>>>>> > <mailto:mailscanner at lists.mailscanner.info>
>>>>> > > <mailto:mailscanner at lists.mailscanner.info
>>>>> > <mailto:mailscanner at lists.mailscanner.info>>> wrote:
>>>>> > >
>>>>> > > On 08/07/2018 05:03 AM, info at schroeffu.ch
>>>>> > <mailto:info at schroeffu.ch> <mailto:info at schroeffu.ch
>>>>> > <mailto:info at schroeffu.ch>>
>>>>> > > wrote:
>>>>> > > >
>>>>> > > > Hi Mailscanner friends,
>>>>> > > >
>>>>> > > > is there any progress to make MailScanner usable as a
>>>>> > postfix milter?
>>>>> > > > The most biggest problem I have is, SPAM is not
>>>>> possible to
>>>>> > > reject when
>>>>> > > > reaching a high score at MTA level. For my
>>>>> understanding,
>>>>> > connect
>>>>> > > via
>>>>> > > > milter instead of queue ^HOLD would be the solution.
>>>>> > > >
>>>>> > > > For the next decade we are still using MailScanner
>>>>> instead
>>>>> > of others
>>>>> > > > like Rspamd, because MailScanner is like a mail suite
>>>>> for mail
>>>>> > > security,
>>>>> > > > but if there will never be the possibility to reject at
>>>>> > MTA level
>>>>> > > the
>>>>> > > > high score spam, we will also change in 1-3 years while
>>>>> > replacing
>>>>> > > the OS
>>>>> > > > beyond.
>>>>> > > >
>>>>> > >
>>>>> > > One of MailScanner's strongest features is it's batch mode
>>>>> > processing
>>>>> > > that will allow it to handle a very high volume of mail
>>>>> > flow. I doubt
>>>>> > > that MailScanner will ever be changed to run as a milter
>>>>> for this
>>>>> > > reason.
>>>>> > >
>>>>> > > I tried rspamd and found it wasn't as good as the author
>>>>> > claims so no
>>>>> > > reason to try to use that as a milter. It also wasn't as
>>>>> > fast as it
>>>>> > > claims. I could not send high volumes of mail through it
>>>>> > like I could
>>>>> > > with MailScanner.
>>>>> > >
>>>>> > > If you want to block high scoring spam at the MTA level, I
>>>>> > suggest
>>>>> > > using
>>>>> > > amavis or spamd with the same SA rulesets as MailScanner.
>>>>> > This will
>>>>> > > get
>>>>> > > you most of the power of MailScanner's blocking at the
>>>>> MTA.
>>>>> > >
>>>>> > > https://wiki.apache.org/spamassassin/IntegratedInMta
>>>>> > >
>>>>> > > If you you use postscreen and postwhite at the Postfix MTA
>>>>> > level, you
>>>>> > > can block most of the obvious spam with a tuned list of
>>>>> > RBLs. See the
>>>>> > > SA users mailing list over the past year for details on
>>>>> this
>>>>> > from me
>>>>> > > and
>>>>> > > a few others.
>>>>> > >
>>>>> > > I suggest setting up a quick test VM with iRedmail to get
>>>>> a good
>>>>> > > example
>>>>> > > of how to do TLS and amavis integration well with Postfix.
>>>>> > >
>>>>> > > --
>>>>> > > David Jones
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > > MailScanner mailing list
>>>>> > > mailscanner at lists.mailscanner.info
>>>>> > <mailto:mailscanner at lists.mailscanner.info>
>>>>> > > <mailto:mailscanner at lists.mailscanner.info
>>>>> > <mailto:mailscanner at lists.mailscanner.info>>
>>>>> > > http://lists.mailscanner.info/mailman/listinfo/mailscanner
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > > Shawn Iverson, CETL
>>>>> > > Director of Technology
>>>>> > > Rush County Schools
>>>>> > > 765-932-3901 x1171
>>>>> > > iversons at rushville.k12.in.us
>>>>> > <mailto:iversons at rushville.k12.in.us>
>>>>> > <mailto:iversons at rushville.k12.in.us
>>>>> > <mailto:iversons at rushville.k12.in.us>>
>>>>> > >
>>>>> > >
>>>>> >
>>>>> > --
>>>>> > David Jones
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Shawn Iverson, CETL
>>>>> > Director of Technology
>>>>> > Rush County Schools
>>>>> > 765-932-3901 x1171
>>>>> > iversons at rushville.k12.in.us <mailto:iversons at rushville.k12.in.us>
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>> --
>>>>> David Jones
>>>>>
>>>>
>>>>
>>>> --
>>>> Shawn Iverson, CETL
>>>> Director of Technology
>>>> Rush County Schools
>>>> 765-932-3901 x1171
>>>> iversons at rushville.k12.in.us
>>>>
>>>>
>>>>
>>>
>>> --
>>> Shawn Iverson, CETL
>>> Director of Technology
>>> Rush County Schools
>>> 765-932-3901 x1171
>>> iversons at rushville.k12.in.us
>>>
>>>
>>>
>>
>> --
>> Shawn Iverson, CETL
>> Director of Technology
>> Rush County Schools
>> 765-932-3901 x1171
>> iversons at rushville.k12.in.us
>>
>>
>>
>
> --
> Shawn Iverson, CETL
> Director of Technology
> Rush County Schools
> 765-932-3901 x1171
> iversons at rushville.k12.in.us
>
>
>
--
Shawn Iverson, CETL
Director of Technology
Rush County Schools
765-932-3901 x1171
iversons at rushville.k12.in.us
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20180819/74af0df2/attachment-0001.html>
More information about the MailScanner
mailing list