It seems that viruses CAN slip through MailScanner under high load!

[Mike Kercher]

I *think* you can tune sendmail so that if the system load reaches x.xx, it
will stop accepting mail.  Might be worth looking into.  You might also
consider lowering the number of messages MS works with per batch.  I believe
the default is 100.  Try lowering that to say 30 or so.

Mike


-----Original Message-----
From: MailScanner mailing list [mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf
Of Brian M Hoy
Sent: Wednesday, August 27, 2003 7:29 PM
To: MAILSCANNER at JISCMAIL.AC.UK
Subject: It seems that viruses CAN slip through MailScanner under high load!


Hello,

An unfortunate combination of events resulted in one of our mail servers
trying to handle an email load that it simply should not have been subjected
to.  The gateway was running MailScanner 4.13-3, spamassassin and sendmail
at the time - I have since upgraded MailScanner and spamassassin.

To summarise the order of events:

06:02am Message arrived and entered the Mailscanner input queue 08:57am
Mailscanner process that scanned this particular message started 09:31am
McAfee virus scanner process started by Mailscanner 09:36am Mailscanner gave
up waiting for McAfee to complete 09:43am McAfee is still running and found
the virus in the message 10:13am Mailscanner finished scanning/spam tagging
batch of messages 10:15am Sendmail delivered the original message (with
virus) from Mailscanner's output queue.

The detailed logs are at the bottom of this message.  The net result is that
for a period of time we had viruses coming in (which were luckily caught on
the desktops).

It appears that any form of denial of service (attack or self inflicted) on
the mail server can result in viruses entering the network.  These would
include:

- inadequate hardware in the mail server (eg. low memory and IDE disks)
- mail loops involving large messages (each of which need scanning)
- higher than normal, and sustained, email flows (e.g. Sobig.f)
- "next hop" mail server temporarily down, creating huge output queue
- external DOS attack on the mail server

Is it possible to force MailScanner to wait for the virus scanner to
complete?  I realise that this would halt the flow of email if the virus
scanner started spinning, but it would guarantee that all messages are virus
scanned.

I have another observation that I would like to discuss.  When MailScanner
checks the input queue, it stats all files in the queue (in order to sort
them by time for fairness in processing).  Under extreme load with many
incoming messages (think mail loops), there comes a point when the extra
load of stat() ing potentially thousands of files to only peel off 100 for
processing, degrades throughput beyond the point of no return - then the
virus scanning scenario above comes into play.

Perhaps if Mailscanner sensed the system load, and if it was too high,
simply peeled off the first 100 messages for processing without any concern
for fairness.  This would hopefully remove enough "overhead" load to keep
mail moving.  When the load goes down again, then revert to the current
behaviour.

MailScanner is an excellent program and we would not want to be without it,
however some consideration to its behaviour under extreme conditions would
easily make it worldclass!

Hope this helps.

Regards,
Brian



Aug 25 06:02:58 gate2 sm-mta[946]: h7OI2lip000946:
from=<George.Baltsa at tfn.com>, size=100367, class=0, nrcpts=1,
msgid=<200308241802.h7OI2lip000946 at gate2.opus.co.nz>, proto=ESMTP,
daemon=MTA, relay=[211.92.144.53] Aug 25 06:02:58 gate2 sm-mta[946]:
h7OI2lip000946: to=<neil.tane at opus.co.nz>, delay=00:00:05, mailer=smtp,
pri=30352, stat=queued Aug 25 08:56:04 gate2 MailScanner[3546]: MailScanner
E-Mail Virus Scanner version 4.13-3 starting... Aug 25 08:57:04 gate2
MailScanner[3546]: Using locktype = flock Aug 25 08:57:17 gate2
MailScanner[3546]: New Batch: Found 1127 messages waiting Aug 25 08:57:17
gate2 MailScanner[3546]: New Batch: Scanning 100 messages, 34997472 bytes
Aug 25 08:57:46 gate2 MailScanner[3546]: Spam Checks: Found 9 spam messages
Aug 25 09:31:38 gate2 MailScanner[3546]: Virus and Content Scanning:
Starting Aug 25 09:36:40 gate2 MailScanner[3546]: Commercial scanner mcafee
timed out! Aug 25 09:36:40 gate2 MailScanner[3546]: Virus Scanning: Denial
Of Service attack detected!
Aug 25 09:39:29 gate2 MailScanner[3546]:
/var/spool/MailScanner/incoming/3546/h7OIJGip001155/application.pif
Found the W32/Sobig.f at MM virus !!!
Aug 25 09:39:29 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1
infections
Aug 25 09:42:47 gate2 MailScanner[3546]:
/var/spool/MailScanner/incoming/3546/h7OI3Gip000956/your_document.pif
Found the W32/Sobig.f at MM virus !!!
Aug 25 09:42:47 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1
infections
Aug 25 09:43:10 gate2 MailScanner[3546]:
/var/spool/MailScanner/incoming/3546/h7OIGTip001125/your_document.pif
Found the W32/Sobig.f at MM virus !!!
Aug 25 09:43:10 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1
infections
Aug 25 09:43:11 gate2 MailScanner[3546]:
/var/spool/MailScanner/incoming/3546/h7OI2lip000946/your_document.pif
Found the W32/Sobig.f at MM virus !!!
Aug 25 09:43:11 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1
infections
Aug 25 09:43:13 gate2 MailScanner[3546]:
/var/spool/MailScanner/incoming/3546/h7OHj4ip000671/your_document.pif
Found the W32/Sobig.f at MM virus !!!
Aug 25 09:43:13 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1
infections
Aug 25 09:47:20 gate2 MailScanner[3546]:
/var/spool/MailScanner/incoming/3546/h7OI43ip000965/application.pif
Found the W32/Sobig.f at MM virus !!!
Aug 25 09:47:20 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1
infections
Aug 25 09:47:43 gate2 MailScanner[3546]:
/var/spool/MailScanner/incoming/3546/h7OIDUip001096/document_9446.pif
Found the W32/Sobig.f at MM virus !!!
Aug 25 09:47:43 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1
infections
Aug 25 09:58:31 gate2 MailScanner[3546]:
/var/spool/MailScanner/incoming/3546/h7OHoOip000740/your_document.pif
Found the W32/Sobig.f at MM virus !!!
Aug 25 09:58:31 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1
infections
Aug 25 09:59:44 gate2 MailScanner[3546]:
/var/spool/MailScanner/incoming/3546/h7OHuLip000825/details.pif        Found
the W32/Sobig.f at MM virus !!!
Aug 25 09:59:45 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1
infections
Aug 25 10:00:10 gate2 MailScanner[3546]:
/var/spool/MailScanner/incoming/3546/h7OHnRip000710/thank_you.pif
Found the W32/Sobig.f at MM virus !!!
Aug 25 10:00:11 gate2 MailScanner[3546]: Virus Scanning: mcafee found 1
infections Aug 25 10:13:58 gate2 MailScanner[3546]: Uninfected: Delivered
100 messages Aug 25 10:15:32 gate2 MailScanner[3546]: New Batch: Found 1158
messages waiting Aug 25 10:15:32 gate2 MailScanner[3546]: New Batch:
Scanning 100 messages, 31632836 bytes Aug 25 10:15:55 gate2 sendmail[5478]:
h7OI2lip000946: to=<neil.tane at opus.co.nz>, delay=04:13:02, xdelay=00:00:01,
mailer=smtp, pri=120352, relay=ausv01.opus.co.nz. [151.135.24.1], dsn=2.0.0,
stat=Sent (h7OME7qP030748 Message accepted for delivery) Aug 25 10:18:56
gate2 MailScanner[3546]: SpamAssassin timed out and was killed, consecutive
failure 1 of 20 Aug 25 10:20:57 gate2 MailScanner[3546]: SpamAssassin timed
out and was killed, consecutive failure 2 of 20 Aug 25 10:24:28 gate2
MailScanner[3546]: Spam Checks: Found 4 spam messages Aug 25 10:32:08 gate2
MailScanner[3546]: MailScanner child caught a SIGHUP