Mailscanner child freezes

[Glenn Steen]

2008/11/14 Jeffrey Haas <jeffrey at life.illinois.edu>:
> I'm having trouble with a MailScanner child process freezing up.  If
> anyone has any suggestions on how to investigate this, I'd be grateful.
>
> On Tuesday, I upgraded a MailScanner/ClamAV installation to the latest
> versions hoping to get past the recently announced buffer overflow in
> ClamAV.  I installed using the .tgz files for MailScanner &
> SpamAssassin/ClamAV on the mailscanner.info.  I configured MailScanner
> to use clamd.
>
> Everything went well at the time (as it usually does - thanks Julian!).
>  However this morning, I found that MailScanner had stopped processing
> mail.  I was in a bit of a panic at the time, so I simply restarted
> MailScanner, and some mail started flowing, but then things froze up
> again.
>
> I was thinking that there was a message that was gumming up the works
> somewhere, so I set in MailScanner.conf:
>
> #Max Unscanned Messages Per Scan = 30
> #Max Unsafe Messages Per Scan = 30
> Max Unscanned Messages Per Scan = 1
> Max Unsafe Messages Per Scan = 1
>
> That kept the good mail from getting tangled up with the bad.
>
> Running 'ps auwx|grep MailScanner', I find:
>
> postfix  31809  0.0  0.4  25616 20088 ?        Ss   17:59   0:00
> MailScanner: starting child
> postfix  31810 95.1  1.1  50884 47328 ?        R    17:59 333:13
> MailScanner: cleaning messages
> postfix  21305  0.0  1.2  56152 51988 ?        S    22:02   0:03
> MailScanner: waiting for messages
> ...
>
>
> Process 31810 picks up a message to clean, but can't complete for some
> reason. Inspecting /var/spool/MailScanner/incoming, I can see the
> contents of the message.
>
> It is a bounce message from a Mailman list which contains an attachment,
> 'text.zip'.  I can run clamscan manually on the files extracted from the
> message and it reports 'Worm.Mydoom.M FOUND'.
>
> I thought perhaps this was an issue with clamd, since that is a bit new
> to me.  (I've used the Mail::ClamAV module for many years.)  So, I
> reconfigured to 'Virus Scanners = clamav' to have MailScanner invoke
> clamscan.  I still get the same behavior with that.  One child process
> grabs the troublesome message, and then stays in the 'cleaning messages'
> state indefinitely.  The CPU utilization is 100% for that process.  I've
> left it running for about 6 hours now, but there's no change.  I think
> my freeze up of the entire server this morning was perhaps, all of the
> children (5) getting tied up in this way.
>
> This is an Ubuntu 7.10 system with postfix 2.4.5 & perl 5.8.8 installed
> from .deb packages.  I think everything else of importance came from the
> .tgz files.
>
> The last messages from process 31810 are:
>
> Nov 13 17:59:17 les MailScanner[31810]: MailScanner E-Mail Virus Scanner
> version 4.72.5 starting...
> Nov 13 17:59:17 les MailScanner[31810]: SpamAssassin temporary working
> directory is /var/spool/MailScanner/incoming/SpamAssassin-Temp
> Nov 13 17:59:17 les MailScanner[31810]: Using SpamAssassin results cache
> Nov 13 17:59:17 les MailScanner[31810]: Connected to SpamAssassin cache
> database
> Nov 13 17:59:17 les MailScanner[31810]: Expired 6 records from the
> SpamAssassin cache
> Nov 13 17:59:18 les MailScanner[31810]: Using locktype = flock
> Nov 13 17:59:18 les MailScanner[31810]: New Batch: Found 40 messages
> waiting
> Nov 13 17:59:18 les MailScanner[31810]: New Batch: Scanning 1 messages,
> 45362 bytes
> Nov 13 17:59:18 les MailScanner[31810]: Spam Checks: Starting
> Nov 13 17:59:18 les MailScanner[31810]: Message 2D5154E85A3.D9D39 from
> 127.0.0.1 (mailman-bounces at life.illinois.edu) is whitelisted
> Nov 13 17:59:18 les MailScanner[31810]: SpamAssassin cache hit for
> message 2D5154E85A3.D9D39
> Nov 13 17:59:18 les MailScanner[31810]: Message 2D5154E85A3.D9D39 from
> 127.0.0.1 (mailman-bounces at life.illinois.edu) to uiuc.edu is not spam
> (whitelisted), SpamAssassin (cached, score=-1.44, required 6,
> autolearn=not spam, ALL_TRUSTED -1.44)
> Nov 13 17:59:18 les MailScanner[31810]: Filename Checks: Possible MS-Dos
> program shortcut attack (2D5154E85A3.D9D39 text.htm
>                                                  .pif)
> Nov 13 17:59:18 les MailScanner[31810]: Filetype Checks: No executables
> (2D5154E85A3.D9D39 text.htm
>                          .pif)
> Nov 13 17:59:18 les MailScanner[31810]: Other Checks: Found 2 problems
> Nov 13 17:59:18 les MailScanner[31810]: Virus and Content Scanning:
> Starting
> Nov 13 17:59:19 les MailScanner[31810]: ./2D5154E85A3.D9D39.message:
> Worm.Mydoom.M FOUND
> Nov 13 17:59:19 les MailScanner[31810]: ./2D5154E85A3.D9D39/text1.zip:
> Worm.Mydoom.M FOUND
> Nov 13 17:59:19 les MailScanner[31810]:
> ./2D5154E85A3.D9D39/text.htm.pif: Worm.Mydoom.M FOUND
> Nov 13 17:59:19 les MailScanner[31810]: ./2D5154E85A3.D9D39/text.zip:
> Worm.Mydoom.M FOUND
> Nov 13 17:59:20 les MailScanner[31810]: Virus Scanning: ClamAV found 4
> infections
>
> Any ideas how to prevent this from happening?  Thanks for any suggestions.
>
Hi Jeffrey,

If you attach to one of the children "getting stuck" with strace, what
does it seem to be doing? Further, looking with something simple, like
top or sar... Is it continually "eating memory" (ie leaking....)?
Since you know which messages cause this, could you lift a couple from
the hold queue and either send them to Jules or to me, so that we
could look at what our systems think of them...?
My gut feeling is that there is some problem with some perl module,
but ... that's just a gut reaction:-). What does "MailScanner -v" say?
Also, simple things like "MailScanner --lint" and definitely
"MailScanner --debug" could perhaps reveal something interesting:).

As I'm sure you know, we've had one ... instance of children "freezing
up"/"looping forever" with the milter support (specific to PF, I
missed a place to handle things:-), but that was fixed a while back
and shouldn't be affecting things with something this new. Obligatory
question is "do you run any milters?", more for completeness than
anything:-).

Cheers
-- 
-- Glenn
email: glenn < dot > steen < at > gmail < dot > com
work: glenn < dot > steen < at > ap1 < dot > se