Mailscanner suddenly started taking up 100% of availablememory - Final Resolution

[Paul Houselander]

This sounds very much like the problem I had last month. See message thread "Server Crash" in the March Archives.

It caused my server to fall over everytime I started mailscanner until I manually removed the message from the queue.

Would be interested in any suggestions as to stop it happening again.

Think I still have a copy of the message somewhere it would help in fixing.

Paul

>>> andrewh at CQG.COM 04/19/02 10:19pm >>>
Ok, so here's what caused the problem...

One of our employees set up a procmail recipe that forwarded to his local
mail box and to his AOL account. Sometime yesterday he sent a multipart mime
e-mail to his account that AOL rejected and a bounce message was sent to his
account. His account forwarded the bounce back to AOL, which got sent back
to him with the first two messages attached. This continued until the
message had several thousand sub MIME attachments. Every time one came in,
about 3 times an hour, mailscanner would open the entire mail up to all it's
thousands of sub parts, scan them all for viruses, and then continue on.

The problem was that some bug in either perl or mailscanner wasn't releasing
some part of the memory required to do this and so every mail ate about 50MB
of memory.

I'm not sure what part of Mailscanner, if it is even mailscanner, has this
bug, but it probably should be researched.

Thank you,
Andrew Hoying

> -----Original Message-----
> From: MailScanner mailing list [mailto:MAILSCANNER at JISCMAIL.AC.UK]On 
> Behalf Of Andrew Hoying
> Sent: Friday, April 19, 2002 1:24 PM
> To: MAILSCANNER at JISCMAIL.AC.UK 
> Subject: Re: Mailscanner suddenly started taking up 100% of available
> memory
>
>
> I guess I spoke to soon, while rebooting it fixed the problem for about a
> half hour, it started consuming memory again and after 20 minutes since I
> last restarted the mailscanner process it is using 170MB of memory, with
> 94MB of locked memory. I really need some help troubleshooting
> this one, so
> any suggestions would be greatly appreciated.
>
> Thank you,
> Andrew
>
> > -----Original Message-----
> > From: MailScanner mailing list [mailto:MAILSCANNER at JISCMAIL.AC.UK]On 
> > Behalf Of Andrew Hoying
> > Sent: Friday, April 19, 2002 12:01 PM
> > To: MAILSCANNER at JISCMAIL.AC.UK 
> > Subject: Re: Mailscanner suddenly started taking up 100% of available
> > memory
> >
> >
> > Well, when reverting to an older dat file didn't fix the
> problem, I broke
> > down and rebooted the system. When it came back up everything
> worked fine.
> > I'm not sure what caused the problem in the first place, but
> will keep my
> > eye on the system for any further troubles. It's running redhat 7.1 with
> > kernel version 2.4.14 patched with the grsec patch version 1.8.9.
> >
> > Andrew
> >
> > > -----Original Message-----
> > > From: MailScanner mailing list [mailto:MAILSCANNER at JISCMAIL.AC.UK]On 
> > > Behalf Of Andrew Hoying
> > > Sent: Friday, April 19, 2002 11:20 AM
> > > To: MAILSCANNER at JISCMAIL.AC.UK 
> > > Subject: Re: Mailscanner suddenly started taking up 100% of available
> > > memory
> > >
> > >
> > > Some more information.
> > >
> > > Here is the process after only 9 minutes:
> > >
> > > USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
> > > root      7644 21.0 68.1 107408 85776 ?      S    11:07   1:51
> > > /usr/bin/perl
> > > /usr/local/MailScanner/bin/mailscanner
> > > Fri Apr 19 11:16:14 MDT 2002
> > >
> > > And here is the changed files in the last 3 days:
> > >
> > > [/usr]# find ./ -mtime -3
> > > ./local/uvscan
> > > ./local/uvscan/scan.dat
> > > ./local/uvscan/names.dat
> > > ./local/uvscan/clean.dat
> > > ./local/uvscan/readme.txt
> > > ./local/uvscan/file_id.diz
> > > ./local/uvscan/mcscript.ini
> > > ./local/uvscan/packing.lst
> > > ./local/uvscan/pkgdesc.ini
> > > ./local/uvscan/validate.exe
> > > ./local/uvscan/delta.ini
> > > ./local/uvscan/dat-4197.tar
> > > ./local/uvscan/reseller.txt
> > > ./local/MailScanner/etc
> > > ./local/MailScanner/var/virus.pid
> > >
> > > It's to the point where I've been restarting mailscanner every 15
> > > minutes to
> > > keep the system running smoothly.
> > >
> > > Any help at all is greatly appreciated.
> > >
> > > Thank you,
> > > Andrew
> > >
> > > > -----Original Message-----
> > > > From: MailScanner mailing list [mailto:MAILSCANNER at JISCMAIL.AC.UK]On 
> > > > Behalf Of Andrew Hoying
> > > > Sent: Friday, April 19, 2002 9:30 AM
> > > > To: MAILSCANNER at JISCMAIL.AC.UK 
> > > > Subject: Mailscanner suddenly started taking up 100% of
> > available memory
> > > >
> > > >
> > > > Yesterday morning, for the first time, mailscanner began
> > > consuming several
> > > > hundred megabytes of memory every hour until the system ran out
> > > of memory
> > > > and killed mailscanner. The process would then repeat every 4
> > > hours or so.
> > > > Regardless of the fact that I should have resource limits
> in place to
> > > > prevent mailscanner from using all the memory, I've never seen
> > > > this behavior
> > > > before. I've been running mailscanner on this server since
> > > October of last
> > > > year without any similar problems. I am currently running
> > > version  3.13-2.
> > > > The only thing that changed, and I'm pretty sure that this is
> > > the culprit,
> > > > is that mcafee auto-updated to dat version 4197 on Wednesday.
> > > I'm not sure
> > > > where to begin troubleshooting this particular problem, and
> > certainly am
> > > > open to suggestions.
> > > >
> > > > Thank you,
> > > > Andrew
> > > >
> > >
> >
>