It seems that viruses CAN slip through MailScanner under high load!

[Brian M Hoy]

Hello,

An unfortunate combination of events resulted in one of our mail servers
trying to handle an email load that it simply should not have been
subjected to.  The gateway was running MailScanner 4.13-3, spamassassin
and sendmail at the time - I have since upgraded MailScanner and
spamassassin.

To summarise the order of events:

06:02am Message arrived and entered the Mailscanner input queue
08:57am Mailscanner process that scanned this particular message started
09:31am McAfee virus scanner process started by Mailscanner
09:36am Mailscanner gave up waiting for McAfee to complete
09:43am McAfee is still running and found the virus in the message
10:13am Mailscanner finished scanning/spam tagging batch of messages
10:15am Sendmail delivered the original message (with virus) from
Mailscanner's output queue.

The detailed logs are at the bottom of this message.  The net result is
that for a period of time we had viruses coming in (which were luckily
caught on the desktops).

It appears that any form of denial of service (attack or self inflicted)
on the mail server can result in viruses entering the network.  These
would include:

- inadequate hardware in the mail server (eg. low memory and IDE disks)
- mail loops involving large messages (each of which need scanning)
- higher than normal, and sustained, email flows (e.g. Sobig.f)
- "next hop" mail server temporarily down, creating huge output queue
- external DOS attack on the mail server

Is it possible to force MailScanner to wait for the virus scanner to
complete?  I realise that this would halt the flow of email if the virus
scanner started spinning, but it would guarantee that all messages are
virus scanned.

I have another observation that I would like to discuss.  When
MailScanner checks the input queue, it stats all files in the queue (in
order to sort them by time for fairness in processing).  Under extreme
load with many incoming messages (think mail loops), there comes a point
when the extra load of stat() ing potentially thousands of files to only
peel off 100 for processing, degrades throughput beyond the point of no
return - then the virus scanning scenario above comes into play.

Perhaps if Mailscanner sensed the system load, and if it was too high,
simply peeled off the first 100 messages for processing without any
concern for fairness.  This would hopefully remove enough "overhead"
load to keep mail moving.  When the load goes down again, then revert to
the current behaviour.

MailScanner is an excellent program and we would not want to be without
it, however some consideration to its behaviour under extreme conditions
would easily make it worldclass!

Hope this helps.

Regards,
Brian



Aug 25 06:02:58 gate2 sm-mta[946]: h7OI2lip000946: from=<George.Baltsa at tfn.com>, size=100367, class=0, nrcpts=1, msgid=<200308241802.h7OI2lip000946 at gate2.opus.co.nz>, proto=ESMTP, daemon=MTA, relay=[211.92.144.53]
Aug 25 06:02:58 gate2 sm-mta[946]: h7OI2lip000946: to=<neil.tane at opus.co.nz>, delay=00:00:05, mailer=smtp, pri=30352, stat=queued
Aug 25 08:56:04 gate2 MailScanner[3546]: MailScanner E-Mail Virus Scanner version 4.13-3 starting...
Aug 25 08:57:04 gate2 MailScanner[3546]: Using locktype = flock
Aug 25 08:57:17 gate2 MailScanner[3546]: New Batch: Found 1127 messages waiting
Aug 25 08:57:17 gate2 MailScanner[3546]: New Batch: Scanning 100 messages, 34997472 bytes
Aug 25 08:57:46 gate2 MailScanner[3546]: Spam Checks: Found 9 spam messages
Aug 25 09:31:38 gate2 MailScanner[3546]: Virus and Content Scanning: Starting
Aug 25 09:36:40 gate2 MailScanner[3546]: Commercial scanner mcafee timed out!
Aug 25 09:36:40 gate2 MailScanner[3546]: Virus Scanning: Denial Of Service attack detected!
Aug 25 09:39:29 gate2 MailScanner[3546]: /var/spool/MailScanner/incoming/3546/h7OIJGip001155/application.pif        Found the W32/Sobig.f at MM virus !!!
Aug 25 09:39:29 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1 infections
Aug 25 09:42:47 gate2 MailScanner[3546]: /var/spool/MailScanner/incoming/3546/h7OI3Gip000956/your_document.pif        Found the W32/Sobig.f at MM virus !!!
Aug 25 09:42:47 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1 infections
Aug 25 09:43:10 gate2 MailScanner[3546]: /var/spool/MailScanner/incoming/3546/h7OIGTip001125/your_document.pif        Found the W32/Sobig.f at MM virus !!!
Aug 25 09:43:10 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1 infections
Aug 25 09:43:11 gate2 MailScanner[3546]: /var/spool/MailScanner/incoming/3546/h7OI2lip000946/your_document.pif        Found the W32/Sobig.f at MM virus !!!
Aug 25 09:43:11 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1 infections
Aug 25 09:43:13 gate2 MailScanner[3546]: /var/spool/MailScanner/incoming/3546/h7OHj4ip000671/your_document.pif        Found the W32/Sobig.f at MM virus !!!
Aug 25 09:43:13 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1 infections
Aug 25 09:47:20 gate2 MailScanner[3546]: /var/spool/MailScanner/incoming/3546/h7OI43ip000965/application.pif        Found the W32/Sobig.f at MM virus !!!
Aug 25 09:47:20 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1 infections
Aug 25 09:47:43 gate2 MailScanner[3546]: /var/spool/MailScanner/incoming/3546/h7OIDUip001096/document_9446.pif        Found the W32/Sobig.f at MM virus !!!
Aug 25 09:47:43 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1 infections
Aug 25 09:58:31 gate2 MailScanner[3546]: /var/spool/MailScanner/incoming/3546/h7OHoOip000740/your_document.pif        Found the W32/Sobig.f at MM virus !!!
Aug 25 09:58:31 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1 infections
Aug 25 09:59:44 gate2 MailScanner[3546]: /var/spool/MailScanner/incoming/3546/h7OHuLip000825/details.pif        Found the W32/Sobig.f at MM virus !!!
Aug 25 09:59:45 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1 infections
Aug 25 10:00:10 gate2 MailScanner[3546]: /var/spool/MailScanner/incoming/3546/h7OHnRip000710/thank_you.pif        Found the W32/Sobig.f at MM virus !!!
Aug 25 10:00:11 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1 infections
Aug 25 10:13:58 gate2 MailScanner[3546]: Uninfected: Delivered 100 messages
Aug 25 10:15:32 gate2 MailScanner[3546]: New Batch: Found 1158 messages waiting
Aug 25 10:15:32 gate2 MailScanner[3546]: New Batch: Scanning 100 messages, 31632836 bytes
Aug 25 10:15:55 gate2 sendmail[5478]: h7OI2lip000946: to=<neil.tane at opus.co.nz>, delay=04:13:02, xdelay=00:00:01, mailer=smtp, pri=120352, relay=ausv01.opus.co.nz. [151.135.24.1], dsn=2.0.0, stat=Sent (h7OME7qP030748 Message accepted for delivery)
Aug 25 10:18:56 gate2 MailScanner[3546]: SpamAssassin timed out and was killed, consecutive failure 1 of 20
Aug 25 10:20:57 gate2 MailScanner[3546]: SpamAssassin timed out and was killed, consecutive failure 2 of 20
Aug 25 10:24:28 gate2 MailScanner[3546]: Spam Checks: Found 4 spam messages
Aug 25 10:32:08 gate2 MailScanner[3546]: MailScanner child caught a SIGHUP