OT: mailscanner & oom killer on fc2

Vlad Mazek vlad at MAZEK.COM
Wed Mar 30 14:43:12 IST 2005 

    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "US-ASCII" character set.  ]
    [ Some characters may be displayed incorrectly. ]

Hello folks,

Troubleshooting MailScanner on friends machine and I seem to be running
into a problem that may be related to MailScanner  but I can't quite
figure out what is the issue (hardware or software) and my Googling so
far has been futile, thus the OT message. System is stock Fedora FC2
with 2.6.10. As you read this message please keep in mind that I'm not
drunk, this is what I'm seeing:

At start MailScanner process simply hits the roof:
----
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
COMMAND
29947 root      25   0  137m 131m 3188 R 92.4  8.6   0:15.15
MailScanner
27854 root      25   0  766m 760m 2828 R 92.1 50.1   6:46.87
MailScanner
----
Running it in debug mode does not reveal anything unusual.

As the system starts to run out of memory, MailScanner keeps on spawning
- Its set to 5 children, I've seen it go as high up as 40. It keeps on
growing until it allocates all available ram:
----
Tasks:  90 total,   4 running,  86 sleeping,   0 stopped,   0 zombie
Cpu(s): 94.5% us,  5.5% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   1554180k total,  1485752k used,    68428k free,     4084k buffers
Swap:        0k total,        0k used,        0k free,    31016k cached
----

This is where oom killer steps in. MailScanner eventually runs out of
memory and gets killed. Another MailScanner thread starts in its place,
until they eat up all available memory. Thats when oom killer steps in
and things get ugly very fast:

----
Mar 30 07:44:21 mail1 root: Process did not exit cleanly, returned 255
with signal 0
Mar 30 07:46:14 mail1 kernel: oom-killer: gfp_mask=0x1d2
Mar 30 07:46:17 mail1 kernel: Mem-info:
Mar 30 07:46:23 mail1 kernel: DMA per-cpu:
Mar 30 07:48:06 mail1 kernel: cpu 0 hot: low 2, high 6, batch 1
Mar 30 07:48:42 mail1 shutdown: shutting down for system reboot
Mar 30 07:48:42 mail1 kernel: cpu 0 cold: low 0, high 2, batch 1
Mar 30 07:48:45 mail1 sshd(pam_unix)[30028]: session opened for user
root by (uid=0)
Mar 30 07:48:45 mail1 kernel: cpu 1 hot: low 2, high 6, batch 1
Mar 30 07:48:45 mail1 sshd(pam_unix)[30038]: session opened for user
root by (uid=0)
Mar 30 07:48:45 mail1 kernel: cpu 1 cold: low 0, high 2, batch 1
Mar 30 07:48:46 mail1 kernel: Normal per-cpu:
Mar 30 07:48:46 mail1 kernel: cpu 0 hot: low 32, high 96, batch 16
Mar 30 07:48:46 mail1 kernel: cpu 0 cold: low 0, high 32, batch 16
Mar 30 07:48:46 mail1 kernel: cpu 1 hot: low 32, high 96, batch 16
Mar 30 07:48:46 mail1 kernel: cpu 1 cold: low 0, high 32, batch 16
Mar 30 07:48:46 mail1 kernel: HighMem per-cpu:
Mar 30 07:48:46 mail1 kernel: cpu 0 hot: low 32, high 96, batch 16
Mar 30 07:48:47 mail1 kernel: cpu 0 cold: low 0, high 32, batch 16
Mar 30 07:48:47 mail1 kernel: cpu 1 hot: low 32, high 96, batch 16
Mar 30 07:48:47 mail1 kernel: cpu 1 cold: low 0, high 32, batch 16
Mar 30 07:48:47 mail1 kernel:
Mar 30 07:48:47 mail1 kernel: Free pages:        4276kB (512kB HighMem)
Mar 30 07:48:47 mail1 kernel: Active:379452 inactive:120 dirty:0
writeback:0 unstable:0 free:1069 slab:3159 mapped:379503 pagetable$
Mar 30 07:48:47 mail1 kernel: DMA free:68kB min:68kB low:84kB high:100kB
active:12552kB inactive:0kB present:16384kB pages_scanned:$
Mar 30 07:48:47 mail1 kernel: protections[]: 0 0 0
Mar 30 07:48:47 mail1 kernel: Normal free:3696kB min:3756kB low:4692kB
high:5632kB active:859620kB inactive:148kB present:901120kB $
Mar 30 07:48:47 mail1 kernel: protections[]: 0 0 0
Mar 30 07:48:47 mail1 kernel: HighMem free:512kB min:512kB low:640kB
high:768kB active:645708kB inactive:332kB present:654528kB pag$
Mar 30 07:48:47 mail1 kernel: protections[]: 0 0 0
Mar 30 07:48:47 mail1 kernel: DMA: 1*4kB 0*8kB 0*16kB 0*32kB 1*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68kB
Mar 30 07:48:47 mail1 kernel: Normal: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB
0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Mar 30 07:48:47 mail1 kernel: HighMem: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 512kB
Mar 30 07:48:47 mail1 kernel: Swap cache: add 0, delete 0, find 0/0,
race 0+0
Mar 30 07:48:47 mail1 kernel: Free swap:            0kB
Mar 30 07:48:47 mail1 kernel: 393008 pages of RAM
Mar 30 07:48:47 mail1 kernel: 163632 pages of HIGHMEM
Mar 30 07:48:47 mail1 kernel: 4479 reserved pages
Mar 30 07:48:47 mail1 kernel: 163514 pages shared
Mar 30 07:48:48 mail1 kernel: 0 pages swap cached
Mar 30 07:48:48 mail1 kernel: Out of Memory: Killed process 26299
(MailScanner).
----

If I disable oom killer (echo "2" >/proc/sys/vm/overcommit_memory) I do
not get the nasty oom message above but processes still die due to the
lack of memory. MailScanner again spawns out of control and the system
is basically trashed.

Considering that no new software was installed recently and that the
system was running without a problem for over 4 months, could this be a
hardware issue? MailScanner is stock install as well, nothing fancy
(MailScanner, SA, dcc)

-Vlad
ExchangeDefender.com

------------------------ MailScanner list ------------------------
To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
'leave mailscanner' in the body of the email.
Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).

Support MailScanner development - buy the book off the website!

More information about the MailScanner mailing list