OT: mailscanner & oom killer on fc2

Martin Hepworth martinh at SOLID-STATE-LOGIC.COM
Wed Mar 30 14:58:31 IST 2005 

Vlad

check in the message for old messages. There maybe some message in there
that's causing the thing to fall over.

IE, stop MS, move everything out of the inbound queue, start MS. If
everythings back to normal drip in the queue files back and see if one
of them triggers the bad.

--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300


Vlad Mazek wrote:
> Hello folks,
>
> Troubleshooting MailScanner on friends machine and I seem to be running
> into a problem that may be related to MailScanner  but I can't quite
> figure out what is the issue (hardware or software) and my Googling so
> far has been futile, thus the OT message. System is stock Fedora FC2
> with 2.6.10. As you read this message please keep in mind that I'm not
> drunk, this is what I'm seeing:
>
> At start MailScanner process simply hits the roof:
> ----
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
> COMMAND
> 29947 root      25   0  137m 131m 3188 R 92.4  8.6   0:15.15
> MailScanner
> 27854 root      25   0  766m 760m 2828 R 92.1 50.1   6:46.87
> MailScanner
> ----
> Running it in debug mode does not reveal anything unusual.
>
> As the system starts to run out of memory, MailScanner keeps on spawning
> - Its set to 5 children, I've seen it go as high up as 40. It keeps on
> growing until it allocates all available ram:
> ----
> Tasks:  90 total,   4 running,  86 sleeping,   0 stopped,   0 zombie
> Cpu(s): 94.5% us,  5.5% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,
> 0.0% si
> Mem:   1554180k total,  1485752k used,    68428k free,     4084k buffers
> Swap:        0k total,        0k used,        0k free,    31016k cached
> ----
>
> This is where oom killer steps in. MailScanner eventually runs out of
> memory and gets killed. Another MailScanner thread starts in its place,
> until they eat up all available memory. Thats when oom killer steps in
> and things get ugly very fast:
>
> ----
> Mar 30 07:44:21 mail1 root: Process did not exit cleanly, returned 255
> with signal 0
> Mar 30 07:46:14 mail1 kernel: oom-killer: gfp_mask=0x1d2
> Mar 30 07:46:17 mail1 kernel: Mem-info:
> Mar 30 07:46:23 mail1 kernel: DMA per-cpu:
> Mar 30 07:48:06 mail1 kernel: cpu 0 hot: low 2, high 6, batch 1
> Mar 30 07:48:42 mail1 shutdown: shutting down for system reboot
> Mar 30 07:48:42 mail1 kernel: cpu 0 cold: low 0, high 2, batch 1
> Mar 30 07:48:45 mail1 sshd(pam_unix)[30028]: session opened for user
> root by (uid=0)
> Mar 30 07:48:45 mail1 kernel: cpu 1 hot: low 2, high 6, batch 1
> Mar 30 07:48:45 mail1 sshd(pam_unix)[30038]: session opened for user
> root by (uid=0)
> Mar 30 07:48:45 mail1 kernel: cpu 1 cold: low 0, high 2, batch 1
> Mar 30 07:48:46 mail1 kernel: Normal per-cpu:
> Mar 30 07:48:46 mail1 kernel: cpu 0 hot: low 32, high 96, batch 16
> Mar 30 07:48:46 mail1 kernel: cpu 0 cold: low 0, high 32, batch 16
> Mar 30 07:48:46 mail1 kernel: cpu 1 hot: low 32, high 96, batch 16
> Mar 30 07:48:46 mail1 kernel: cpu 1 cold: low 0, high 32, batch 16
> Mar 30 07:48:46 mail1 kernel: HighMem per-cpu:
> Mar 30 07:48:46 mail1 kernel: cpu 0 hot: low 32, high 96, batch 16
> Mar 30 07:48:47 mail1 kernel: cpu 0 cold: low 0, high 32, batch 16
> Mar 30 07:48:47 mail1 kernel: cpu 1 hot: low 32, high 96, batch 16
> Mar 30 07:48:47 mail1 kernel: cpu 1 cold: low 0, high 32, batch 16
> Mar 30 07:48:47 mail1 kernel:
> Mar 30 07:48:47 mail1 kernel: Free pages:        4276kB (512kB HighMem)
> Mar 30 07:48:47 mail1 kernel: Active:379452 inactive:120 dirty:0
> writeback:0 unstable:0 free:1069 slab:3159 mapped:379503 pagetable$
> Mar 30 07:48:47 mail1 kernel: DMA free:68kB min:68kB low:84kB high:100kB
> active:12552kB inactive:0kB present:16384kB pages_scanned:$
> Mar 30 07:48:47 mail1 kernel: protections[]: 0 0 0
> Mar 30 07:48:47 mail1 kernel: Normal free:3696kB min:3756kB low:4692kB
> high:5632kB active:859620kB inactive:148kB present:901120kB $
> Mar 30 07:48:47 mail1 kernel: protections[]: 0 0 0
> Mar 30 07:48:47 mail1 kernel: HighMem free:512kB min:512kB low:640kB
> high:768kB active:645708kB inactive:332kB present:654528kB pag$
> Mar 30 07:48:47 mail1 kernel: protections[]: 0 0 0
> Mar 30 07:48:47 mail1 kernel: DMA: 1*4kB 0*8kB 0*16kB 0*32kB 1*64kB
> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68kB
> Mar 30 07:48:47 mail1 kernel: Normal: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB
> 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
> Mar 30 07:48:47 mail1 kernel: HighMem: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
> 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 512kB
> Mar 30 07:48:47 mail1 kernel: Swap cache: add 0, delete 0, find 0/0,
> race 0+0
> Mar 30 07:48:47 mail1 kernel: Free swap:            0kB
> Mar 30 07:48:47 mail1 kernel: 393008 pages of RAM
> Mar 30 07:48:47 mail1 kernel: 163632 pages of HIGHMEM
> Mar 30 07:48:47 mail1 kernel: 4479 reserved pages
> Mar 30 07:48:47 mail1 kernel: 163514 pages shared
> Mar 30 07:48:48 mail1 kernel: 0 pages swap cached
> Mar 30 07:48:48 mail1 kernel: Out of Memory: Killed process 26299
> (MailScanner).
> ----
>
> If I disable oom killer (echo "2" >/proc/sys/vm/overcommit_memory) I do
> not get the nasty oom message above but processes still die due to the
> lack of memory. MailScanner again spawns out of control and the system
> is basically trashed.
>
> Considering that no new software was installed recently and that the
> system was running without a problem for over 4 months, could this be a
> hardware issue? MailScanner is stock install as well, nothing fancy
> (MailScanner, SA, dcc)
>
> -Vlad
> ExchangeDefender.com
>
> ------------------------ MailScanner list ------------------------
> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
> 'leave mailscanner' in the body of the email.
> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>
> Support MailScanner development - buy the book off the website!

**********************************************************************

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.

**********************************************************************

------------------------ MailScanner list ------------------------
To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
'leave mailscanner' in the body of the email.
Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).

Support MailScanner development - buy the book off the website!

More information about the MailScanner mailing list