Upgraded to 4.67.6, MailScanner scans a batch then hangs at 100 percent CPU

[Julian Field]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Glenn Steen wrote:
> On 12/03/2008, Steve Crumley <scrumley at secure-enterprise.com> wrote:
>   
>>  > -----Original Message-----
>>  > From: mailscanner-bounces at lists.mailscanner.info
>>  > [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf
>>
>>     
>>> Of Julian Field
>>>       
>>  > Sent: Tuesday, March 11, 2008 6:50 PM
>>  > To: MailScanner discussion
>>  > Subject: Re: Upgraded to 4.67.6, MailScanner scans a batch
>>  > then hangs at 100 percent CPU
>>  >
>>
>>     
>>> * PGP Signed by an unverified key: 03/11/08 at 18:50:26
>>>       
>>  >
>>  >
>>  >
>>  > Steve Crumley wrote:
>>  > >
>>  > >
>>  > >
>>  > >> -----Original Message-----
>>  > >> From: mailscanner-bounces at lists.mailscanner.info
>>  > >> [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf
>>  > >> Of Glenn Steen
>>  > >> Sent: Tuesday, March 11, 2008 4:32 PM
>>  > >> To: MailScanner discussion
>>  > >> Subject: Re: Upgraded to 4.67.6,MailScanner scans a batch
>>  > >> then hangs at 100 percent CPU
>>  > >>
>>  > >> On 11/03/2008, Steve Crumley
>>  > <scrumley at secure-enterprise.com> wrote:
>>  > >>
>>  > >>>  > -----Original Message-----
>>  > >>>  > From: mailscanner-bounces at lists.mailscanner.info
>>  > >>>  > [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf
>>  > >>>
>>  > >>>
>>  > >>>> Of Glenn Steen
>>  > >>>>
>>  > >>>  > Sent: Tuesday, March 11, 2008 1:21 PM
>>  > >>>  > To: MailScanner discussion
>>  > >>>  > Subject: Re: Upgraded to 4.67.6,MailScanner scans a batch
>>  > >>>  > then hangs at 100 percent CPU
>>  > >>>  >
>>  > >>>  > On 11/03/2008, Steve Crumley
>>  > >>>
>>  > >> <scrumley at secure-enterprise.com> wrote:
>>  > >>
>>  > >>>  > >
>>  > >>>  > >
>>  > >>>  > >  > -----Original Message-----
>>  > >>>  > >  > From: mailscanner-bounces at lists.mailscanner.info
>>  > >>>  > >  > [mailto:mailscanner-bounces at lists.mailscanner.info]
>>  > >>>
>>  > >> On Behalf
>>  > >>
>>  > >>>  > >  > Of --[ UxBoD ]--
>>  > >>>  > >
>>  > >>>  > > > Sent: Tuesday, March 11, 2008 11:29 AM
>>  > >>>  > >  > To: MailScanner discussion
>>  > >>>  > >  > Subject: Re: Upgraded to 4.67.6, MailScanner scans a batch
>>  > >>>  > >  > then hangs at 100 percent CPU
>>  > >>>  > >  >
>>  > >>>  > >
>>  > >>>  > > > do you have strace installed on the server ? if so when the
>>  > >>>  > >  > process is running at 100% CPU connect to it and
>>  > see what it
>>  > >>>  > >  > is doing.  I had this before, but for the life of
>>  > >>>
>>  > >> me I cannot
>>  > >>
>>  > >>>  > >  > remember what I changed to fix it :(
>>  > >>>  > >  >
>>  > >>>  > >  > Things to check :-
>>  > >>>  > >  >
>>  > >>>  > >  > 1) Permissions, are they all correct
>>  > >>>  > >  > 2) Check MailScanner.conf again just to make sure no typos
>>  > >>>  > >  >
>>  > >>>  > >  > Regards,
>>  > >>>  > >  >
>>  > >>>  > >  > --
>>  > >>>  > >
>>  > >>>  > >
>>  > >>>  > > Here is the output from strace:
>>  > >>>  > >
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >  waitpid(-1, 0xbff09448, WNOHANG)        = 0
>>  > >>>  > >
>>  > >>>  > >
>>  > >>>  > >
>>  > >>>  > >
>>  > >>>  > >  The system had been running fine for over a year, I
>>  > >>>
>>  > >> can't find any
>>  > >>
>>  > >>>  > >  permission or setting change thats doing this, but
>>  > I could be
>>  > >>>  > >  overlooking something.
>>  > >>>  > >  Thanks,
>>  > >>>  > >  -Steve
>>  > >>>  > >
>>  > >>>  > Could perhaps be a busted SQLite SA cache? What does
>>  > >>>
>>  > >> analyse_s<TAB> (I
>>  > >>
>>  > >>>  > don't remember if it is sacache or spamassassin_cache
>>  > >>>
>>  > >> ... the command
>>  > >>
>>  > >>>  > completion should take care of it:-) say? If it looks
>>  > >>>
>>  > >> fishy, simply
>>  > >>
>>  > >>>  > delete the SA cache file and restart MS.
>>  > >>>  >
>>  > >>>  > You've run MailScanner --lint, right? Nothing obvious
>>  > from that?
>>  > >>>  >
>>  > >>>  > Oh, and what av scanners do you use? Obviously not
>>  > >>>
>>  > >> clamavmodule, but
>>  > >>
>>  > >>>  > perhaps clamav or clamd? are those OK?
>>  > >>>  >
>>  > >>>  > Cheers
>>  > >>>  > --
>>  > >>>  > -- Glenn
>>  > >>>  > email: glenn < dot > steen < at > gmail < dot > com
>>  > >>>  > work: glenn < dot > steen < at > ap1 < dot > se
>>  > >>>
>>  > >>>
>>  > >>>> --
>>  > >>>>
>>  > >>>  > MailScanner mailing list
>>  > >>>  > mailscanner at lists.mailscanner.info
>>  > >>>  > http://lists.mailscanner.info/mailman/listinfo/mailscanner
>>  > >>>  >
>>  > >>>  > Before posting, read http://wiki.mailscanner.info/posting
>>  > >>>  >
>>  > >>>  > Support MailScanner development - buy the book off the website!
>>  > >>>  >
>>  > >>>
>>  > >>>
>>  > >>>
>>  > >>> analyse_SpamAssassin_cache looks clean, MailScanner --lint
>>  > >>>
>>  > >> is clean too.
>>  > >>
>>  > >>>  I'm running clamd for AV but I've set virus scanning to no
>>  > >>>
>>  > >> while working
>>  > >>
>>  > >>>  on this.
>>  > >>>
>>  > >>> Thanks,
>>  > >>>  -Steve
>>  > >>>
>>  > >> Couldn't be something easily mended, huh:-)....
>>  > >>
>>  > >> What you seem to have attached to above (with strace) would be the
>>  > >> main MailScanner process, since it basically just wait for it's
>>  > >> children to end... Or is it? What does a ps listing show (one that
>>  > >> show the command argument list, since Jules rewrite it to
>>  > show what it
>>  > >> thinks it is basically doing)?
>>  > >> Do the children restart endlessly when hung? How many children are
>>  > >> there, and in what state?
>>  > >> Cheers
>>  > >> -- Glenn
>>  > >>
>>  > >
>>  > >
>>  > >
>>  > > When I first started it with 8 children, they all end up
>>  > quickly hanging
>>  > > and consuming CPU.  For now, I've set it to 1 child and I've been
>>  > > running in debug mode.  The ps gives us a good clue!  Its the only
>>  > > mailscanner process and it reports "MailScanner: extracting
>>  > attachments"
>>  > >
>>  > > Thanks,
>>  > > -Steve
>>  > >
>>  > In which case go into "sub Explode" in
>>  > /usr/lib/MailScanner/MailScanner/Message.pm, and add some
>>  > "print STDERR"
>>  > lines to generate tracing output so you can see how far it gets. When
>>  > you do a "MailScanner --debug" it will show you the STDERR
>>  > debug output
>>  > in the terminal session.
>>  >
>>  > Jules
>>  >
>>
>>
>> There's something very screwed up with my perl.  I've put "print"s in
>>  MailScanner around the call to Explode and I put a print first thing in
>>  Explode.  I get the output right before the call but nothing from
>>  explode itself and we never return to MailScanner.
>>
>>  I really appreciate everyone's help with this.
>>  Thanks,
>>  -Steve
>>
>>     
> I wonder if STDERR is unbuffered (too lazy/tired to go look it up...:)
> ... Jules? Else you might need do that to get reliable error
> printing...
>   
STDERR is unbuffered.

Jules

- -- 
Julian Field MEng CITP CEng
www.MailScanner.info
Buy the MailScanner book at www.MailScanner.info/store

MailScanner customisation, or any advanced system administration help?
Contact me at Jules at Jules.FM

PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
PGP public key: http://www.jules.fm/julesfm.asc


-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.8.1 (Build 2523)
Comment: Use Thunderbird Enigmail to verify this message
Charset: ISO-8859-1

wj8DBQFH2DJGEfZZRxQVtlQRAujLAJ4ytKdJ3TqZrSPJDuyHKSomDJG13QCbBmN6
E1kEwpWmNuq7SzIB3mBdBTQ=
=22gB
-----END PGP SIGNATURE-----

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.