ICSA labs anti-spam tests

[Matt Kettler]

Peter Peters wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Are there any ideas on the anti-spam tests conducted by ICSA Labs?
> How would MS score with the testrules at
> https://www.icsalabs.com/icsa/docs/html/communities/Antispamcriteriav095.pdf?
> 

99% of what's in that document is a function of the operating system or other 
system utilities (ie: syslogd, ntpd) not MailScanner..

Most of it is log formats, dates, authentication, time sync, etc..

The parts that do seem applicable are:

1) log messages for actions taken (ie: deletion), which mailscanner does.
2) detection rate: well, without their corpus it's hard to tell. This also 
depends a LOT on what tools you use with MailScanner. Do you use SpamAssassin? 
Any RBL's at the MailScanner level?

They require detect rate of 95% or higher and FP rate of 0.001% or lower.

I can tell you that in SA's own testing, SpamAssassin's FP rate is too high for 
that. However, the accuracy of SA's own test corpus is probably not accurate 
enough to ensure that less than 0.001% of the mail in the nonspam pool isn't 
actually mis-placed spam.

The SA corpus is hand classified, but humans make mistakes. To achieve 0.001%, 
you'd have to make fewer than 1 mistakes in 1 million emails. That's *way* 
beyond the bounds of human error.

The only way I can see to get numbers like that is to run the test, look every 
one of the misclassified messages, kick out the ones that are actually spam upon 
re-review, then re-run the test. However, that borders on fitting your data to 
your test. The SA team does this to a very limited degree, but it's not a 
process taken far enough to get down to 1 in a million accuracy. They review the 
ones that seem to score really high, or that hit rules that don't seem like they 
should ever hit nonspam mail, but not every misclassified message.

You'd also need a corpus of over 1 million fresh nonspam emails to detect errors 
so small, which the SA team does not have. The 3.2 mass-checks were based on 
roughly 500k nonspam's and 950k spams.