ICSA labs anti-spam tests
mkettler at evi-inc.com
Thu Apr 10 17:07:56 IST 2008
Peter Peters wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> Are there any ideas on the anti-spam tests conducted by ICSA Labs?
> How would MS score with the testrules at
99% of what's in that document is a function of the operating system or other
system utilities (ie: syslogd, ntpd) not MailScanner..
Most of it is log formats, dates, authentication, time sync, etc..
The parts that do seem applicable are:
1) log messages for actions taken (ie: deletion), which mailscanner does.
2) detection rate: well, without their corpus it's hard to tell. This also
depends a LOT on what tools you use with MailScanner. Do you use SpamAssassin?
Any RBL's at the MailScanner level?
They require detect rate of 95% or higher and FP rate of 0.001% or lower.
I can tell you that in SA's own testing, SpamAssassin's FP rate is too high for
that. However, the accuracy of SA's own test corpus is probably not accurate
enough to ensure that less than 0.001% of the mail in the nonspam pool isn't
actually mis-placed spam.
The SA corpus is hand classified, but humans make mistakes. To achieve 0.001%,
you'd have to make fewer than 1 mistakes in 1 million emails. That's *way*
beyond the bounds of human error.
The only way I can see to get numbers like that is to run the test, look every
one of the misclassified messages, kick out the ones that are actually spam upon
re-review, then re-run the test. However, that borders on fitting your data to
your test. The SA team does this to a very limited degree, but it's not a
process taken far enough to get down to 1 in a million accuracy. They review the
ones that seem to score really high, or that hit rules that don't seem like they
should ever hit nonspam mail, but not every misclassified message.
You'd also need a corpus of over 1 million fresh nonspam emails to detect errors
so small, which the SA team does not have. The 3.2 mass-checks were based on
roughly 500k nonspam's and 950k spams.
More information about the MailScanner