[Fwd: Some highlights about BarricadeMX]

[Steve Freegard]

Cross posting Anthony's reply to Rob as Anthony is not a member of this 
list.

-------- Original Message --------
Subject: 	Some highlights about BarricadeMX

Date: 	Thu, 14 Jun 2007 21:16:02 +0200
From: 	Anthony Howe <achowe at snert.com>
To: 	Steve Freegard <steve.freegard at fsl.com>
CC: 	barricademx at snertsoft.info


Steve, please forward my reply to the MailScanner list. I'm posting to 
the BarricadeMX list as it might be of some interest there too.

Rob Poe (from the MailScanner list) wrote:
> So, I understand that Fort and Snert have partnered up to stop spam
> at the MTA ...... and obviously it will be a proprietary method, but
> inquiring minds still want to know .. Anyone have any kind of idea on
> the techniques?

The usual suspects. BarricadeMX blends many of the SnertSoft milters 
into one comprehensive package.

a) Enhanced grey-listing which applies two optimisations (one to address 
mail server pools sharing a common queue ie. gmail.com; the other, once 
a message from a host passes greylisting messages from that host/domain 
for all subsequent messages will pass, ie. pay the grey-list delay ONCE 
only); see

http://www.snertsoft.com/smtp/smtpf/smtpf-cf.html#smtpf_grey_list

b) Enhanced Message-ID as Email Watermark, which can be used to filter 
backscatter and skip content filtering if a message is identified as 
reply to previously sent mail.

http://www.snertsoft.com/smtp/smtpf/smtpf-cf.html#smtpf_emew

MailScanner had a patch applied to take advantage of EMEW. If combined 
with dns-gl it would be possible to by-pass both pre and post-DATA 
filtering. Of course we have dns-bl and dns-wl support too.

c) Some other interesting tests are "client is MX" to softens PTR 
requirements and IP-in-PTR ie. looks like dynamic IP tests.

http://www.snertsoft.com/smtp/smtpf/smtpf-cf.html#smtpf_client_is_mx

d) Another interesting combination is multiline welcome banner combined 
with 500ms or 1s command-pause will catch out

e) The multicast/unicast cache deals with the problem of sharing 
call-ahead, grey-listing, and other global data between MXes. It is fast 
and effective.

f) Client Rate Throttling, Global Rate Throttle, & Client Concurrency. 
Nothing particularly spectacular there, sendmail already has this and 
ours is similar, though we apply ours in the accept() loop.

g) URLBL application to HELO and MAIL FROM:. Not a new idea, but not 
widely used either.

h) Tar-pitting of negatives SMTP responses. Handy for dictionary like 
probing during a SMTP session.

i) EHLO no HELO and schizophrenic HELO checks. By disabling ESMTP you 
catch some spamware that assume EHLO and fail to fall back on HELO when 
rejected. Also if they EHLO/HELO with one name, but fail to EHLO/HELO 
with the same name on the next or subsequent attempts you can reject 
too. Note that RFC 2821 HELO/EHLO is equivalent to RSET once the first 
HELO is accepted.

http://www.snertsoft.com/smtp/smtpf/smtpf-cf.html#smtpf_smtp (last 
paragraph)

There is more. So much I forget what is novel and what has been done 
before. You have to read the documentation to get see for yourself.

> I have 2 clients.  One is a trucking firm.  They get a lot of trade
> industry magazines (which SpamAssassin hates) and a lot of people in
> the trucking industry who think it fine that they have a 400x400
> inline .gif with their signature and other misc. stuff, in an email
> with a "Hey give me a call if you want me to service your loads" ...
> yeah SA hates that too..

BarricadeMX can do some real-time content filtering such as clamav, 
spamassassin, and uribl tests in which it will reject or discard. 
However, it is limited by the proxy design in that it cannot tag nor 
quarantine messages based on content (we don't use temp. or queue 
files). milter-spamc, MailScanner, or similar is still required for that 
level of functionality. So things like inline images and their like will 
require tweaking of SpamAssassin as with MailScanner.

> The other is a law firm.  They want Artificial Intelligence for their
> spam filter.  Everything that ISN'T SPAM gets delivered and
> everything that IS SPAM doesn't  -- no errors.

No anti-spam technique is perfect. Expectations of such are unrealistic. 
 We just don't have enough good AI for such yet. (I'll note it as an 
enhancement request though; insert Spock's brain.)

Depending on your paranoia and the set of BarricadeMX options you apply, 
you can tone down the filtering or make it really strict. The FSL web 
user interface for BarricadeMX provides three default levels of 
strength, after which you can customise.

Regardless of the options you apply, a MailScanner or SpamAssassin 
installation will still benefit as the machine will see less system 
load. In your example, you turn on only those tests you are happy with 
and then let a second level filter like MailScanner have its turn. 
BarricadeMX was originally intended as a pre-filter for MailScanner to 
reduce system load by catching the low hanging fruit. What we found with 
BarricadeMX in beta testing, was it would catch the low hanging fruit on 
really tall trees :-)

> Of course, if you go too strict on filtering you can lose emails, and
> if you go too lose you end up with horse sex in your inbox ... how
> does this product compare to current methods?
> 
> Greylisting has helped tremendously, but there are still some
> seriously BRAIN DEAD admins running some seriously BRAIN DEAD email
> apps that don't ever retry.. So for the two above clients, it didn't
> work so well.

There is no suitable solution for brain dead brokeness that fails to 
follow RFC 2821 recommended mail delievery strategies. If a customer 
chooses to use broken software and the developer won't fix it, then all 
you can do is white list them if you like them enough.

> (oh, and why do people make 10 outbound mail gateways, so a message
> that retries goes between the 10 different hosts, getting a greylist
> deny every time, but te delivery retry is > 1 minute) ???

We can fix ignorance, we can't fix stupidity. Really short retry times 
less than 5 minutes are IMHO are just stupid and the result of pointy 
clickity type mentalities performing system administration tasks in 
which they are not competent in. However, rate throttling and the 
multicast/unicast cache does help some aspects of this problem.

Another feature of BarricadeMX are the runtime and hourly stats (version 
1.1 will have rolling 60 minutes stats too). Below is an example from a 
test machine.

Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
220-XXX.XXX.XXX ESMTP Welcome to smtpf
220 Copyright 2006, 2007 by SnertSoft. All rights reserved.
stats
214-2.0.0 smtpf/1.0.146 (runtime)
214-2.0.0 start-time=Thu, 14 Jun 2007 11:57:12 -0400
214-2.0.0 age=10493
214-2.0.0 active-connections=11
214-2.0.0 high-connections=23 (100.00%)
214-2.0.0 high-connections-per-second=5 (100.00%)
214-2.0.0 high-session-time=350 (100.00%)
214-2.0.0 total-KB=973 (100.00%)
214-2.0.0 CLIENTS=5724 (100.00%)
214-2.0.0 dropped=4422 (77.25%)
214-2.0.0 data-354=76 (1.33%)
214-2.0.0 client-io-error=1076 (18.80%)
214-2.0.0 client-timeout=11 (0.19%)
214-2.0.0 server-io-error=113 (1.97%)
214-2.0.0 admin-commands=75 (1.31%)
214-2.0.0 auth-pass=0 (0.00%)
214-2.0.0 auth-fail=0 (0.00%)
214-2.0.0 bogus-helo=2 (0.03%)
214-2.0.0 concurrent=0 (0.00%)
214-2.0.0 connect-bl=12 (0.21%)
214-2.0.0 connect-lan=0 (0.00%)
214-2.0.0 connect-localhost=76 (1.33%)
214-2.0.0 connect-relay=76 (1.33%)
214-2.0.0 connect-wl=0 (0.00%)
214-2.0.0 dns-bl=2750 (48.04%)
214-2.0.0 dns-gl=38 (0.66%)
214-2.0.0 dns-wl=0 (0.00%)
214-2.0.0 ehlo-no-helo=1508 (26.35%)
214-2.0.0 helo-claims-us=0 (0.00%)
214-2.0.0 helo-ip-mismatch=110 (1.92%)
214-2.0.0 helo-schizophrenic=7 (0.12%)
214-2.0.0 idle-retest-timer=0 (0.00%)
214-2.0.0 rate-client=51 (0.89%)
214-2.0.0 rate-throttle=0 (0.00%)
214-2.0.0 client-ip-in-ptr=0 (0.00%)
214-2.0.0 client-ptr-required=873 (15.25%)
214-2.0.0 client-ptr-required-error=60 (1.05%)
214-2.0.0 rfc2821-strict-helo=30 (0.52%)
214-2.0.0 smtp-command-non-ascii=0 (0.00%)
214-2.0.0 smtp-command-pause=51 (0.89%)
214-2.0.0 smtp-drop-after=0 (0.00%)
214-2.0.0 smtp-drop-unknown=6 (0.10%)
214-2.0.0 smtp-enable-esmtp=2813 (49.14%)
214-2.0.0 smtp-greet-pause=65 (1.14%)
214-2.0.0 smtp-reject-delay=0 (0.00%)
214-2.0.0 uri-bl-helo=34 (0.59%)
214-2.0.0 uri-bl-ptr=206 (3.60%)
214-2.0.0 SENDERS=3411 (100.00%)
214-2.0.0 null-sender=18 (0.53%)
214-2.0.0 call-back-cache=0 (0.00%)
214-2.0.0 call-back-made=0 (0.00%)
214-2.0.0 cli-envelope=0 (0.00%)
214-2.0.0 client-is-mx=130 (3.81%)
214-2.0.0 grey-continue=40 (1.17%)
214-2.0.0 grey-tempfail=171 (5.01%)
214-2.0.0 mail-bl=1 (0.03%)
214-2.0.0 mail-wl=0 (0.00%)
214-2.0.0 mail-parse=4 (0.12%)
214-2.0.0 require-sender-mx=0 (0.00%)
214-2.0.0 require-sender-mx-error=0 (0.00%)
214-2.0.0 siq-query-cache=0 (0.00%)
214-2.0.0 siq-query-made=0 (0.00%)
214-2.0.0 siq-score-reject=0 (0.00%)
214-2.0.0 siq-score-tag=0 (0.00%)
214-2.0.0 spf-pass=97 (2.84%)
214-2.0.0 spf-fail=9 (0.26%)
214-2.0.0 spf-none=220 (6.45%)
214-2.0.0 spf-neutral=1 (0.03%)
214-2.0.0 spf-softfail=7 (0.21%)
214-2.0.0 spf-perm-error=0 (0.00%)
214-2.0.0 spf-temp-error=0 (0.00%)
214-2.0.0 uri-bl-mail=64 (1.88%)
214-2.0.0 RECIPIENTS=257 (100.00%)
214-2.0.0 rcpt-reject=9 (3.50%)
214-2.0.0 one-rcpt-per-null=0 (0.00%)
214-2.0.0 rcpt-bl=0 (0.00%)
214-2.0.0 rcpt-wl=0 (0.00%)
214-2.0.0 rcpt-parse=0 (0.00%)
214-2.0.0 MESSAGES=80 (100.00%)
214-2.0.0 msg-accept=60 (75.00%)
214-2.0.0 msg-discard=0 (0.00%)
214-2.0.0 msg-drop=0 (0.00%)
214-2.0.0 msg-reject=20 (25.00%)
214-2.0.0 dsn-sent=0 (0.00%)
214-2.0.0 7bit-headers=0 (0.00%)
214-2.0.0 cli-content=0 (0.00%)
214-2.0.0 infected=0 (0.00%)
214-2.0.0 junk-mail=0 (0.00%)
214-2.0.0 line-length=0 (0.00%)
214-2.0.0 message-limit=0 (0.00%)
214-2.0.0 message-size=0 (0.00%)
214-2.0.0 ret-pass=0 (0.00%)
214-2.0.0 ret-fail=11 (13.75%)
214-2.0.0 ret-ttl=0 (0.00%)
214-2.0.0 strict-dot=0 (0.00%)
214-2.0.0 uri-bl=9 (11.25%)
214-2.0.0 uri-max-limit=0 (0.00%)
214-2.0.0 uri-max-test=4 (5.00%)
214 2.0.0 End.

-- 
Anthony C Howe          Skype: SirWumpus                    SnertSoft
+33 6 11 89 73 78         ICQ: 7116561      Sendmail Milter Solutions
http://www.snert.com/                       http://www.snertsoft.com/