my mail server is drowning

[Jeff A. Earickson]

Gang,

Setup: Sun E220R (2 450 MHZ CPUS, 2 GB memory), Solaris 8,
MailScanner-4.20-3, sophos 3.69, sophosavi, sendmail 8.12.9,
spamassassin 2.53 (bayes learning on).  Other stuff running
on the box: qpopper 4.0.5, mailman 2.1.2.

Problem: It can't keep up with the mail traffic.  We get roughly
25K messages a day.  I didn't really have this problem until after
sophos 3.67 came out, it has been getting worse even with sophossavi.
A backlog forms in mqueue.in in the morning and the box can't keep up
during the day.  Right now (1:30 PM), I've got slightly more than 1000
messages waiting on MailScanner.  My system load (uptime load)
is a steady a steady 10 to 12, sar shows roughly 70% usr, 30% sys
with no idle.  vmstat shows that the system is doing memory paging
like crazy (pi=5K).  Iostat is not outrageous on anything, so the issue
seems to be lack of CPU.

"top" shows that MailScanner is a real heavyweight on memory usage.
Each process is roughly 40M is size, with most of that resident in
memory.  The only other heavy memory user is named (170 to 190 MB).

I've tried tuning MS by changing the number of emails per scan (25 to 100),
the number of children (4 to 10, currently running 6).  Nothing helps.
It takes MS several minutes to chomp on 100 messages -- time enough for
100 more messages to roll in.

Any suggestions for tuning?  Any ideas why MS takes so much memory?
Is this related to the bayes learning in SA?  I've noticed that the
files in /var/spool/spamassassin are big:

-rw-r--r--   1 root     daemon   1335296 May  6 13:49 auto-whitelist
-rw-------   1 root     daemon       112 May  6 13:49 bayes.lock
-rw-------   1 root     daemon        98 May  6 13:49 bayes.lock.emerald.14132
-rw-------   1 root     daemon     45468 May  6 13:49 bayes_journal
-rw-------   1 root     daemon        54 May  6 13:49 bayes_msgcount
-rw-r--r--   1 root     daemon   10559488 May  6 13:49 bayes_seen
-rw-------   1 root     daemon   7700480 May  6 13:49 bayes_toks

and that lsof shows me that bayes_seen and bayes_toks files are used
by MS processes.  Maybe the size of these files related to the memory
usage of MS?

--- Jeff Earickson
    Colby College