Using process priority to throttle inbound SMTP

[Christopher Schanzle]

Greetings Julian & MailScanner Users.

With a production-quality mail server, you want to avoid having a
sustained situation where you have more mail coming in than you have
mail going out.  Otherwise if you get hit hard (by a spammer or script
error), you'll have very long message delivery times or worse, be taken
down due to full disks.  With MailScanner doing spam and maybe virus
checking, this is easy situation to have happen on a fairly high-volume
production box with big spikes.

I discovered it is useful to run the inbound SMTP server (all cases) and
MailScanner itself (some cases) at lower than normal process priorities
(that is, higher NICE values).

The logic is that if MailScanner is chewing up all the CPU available,
and the priority of the inbound SMTP process is lower, inbound mail will
self-throttle and reduce (nearly eliminate) getting large queues of
email waiting to be processed.  Before changing priorities, we
frequently saw 800 messages waiting to be processed...we'd eventually
catch up a while after the spike subsided, but it could take many
minutes before all email is processed.  After lowering inbound SMTP
priority, if the flood of inbound mail continues, the load quickly
climbs to higher values until sendmail starts refusing inbound
connections.  Meanwhile, messages queued will be processed almost as
quickly as normal as the process scheduling priority to do spam/virus
checking is higher, and the system remains responsive (excepting SMTP
connections, of course).

Without adjusting process priorities, having sendmail refuse connections
at numbers near or slightly above MailScanner.conf's "Max Children"
setting works fairly well (we find 3 works well on uniprocessors), but
we spike over so frequently that we get dozens of warnings about
refusing connections.  Also, we try to avoid refusing connections --
it's only viable only if your MUA's don't connect to it directly (MTA's
are fine and will connect to lower-priority MX's).  In those few
occasions where users (with interactive MUA's) connect to our external
servers, that DNS name is tied to our lowest priority MX, so it is
unlikely they will get "connection refused" messages.

Right now with lowered priority on inbound SMTP, we refuse connections
about six times each day at loads around 18 (uniprocessor Athlon XP
2800+).  But our queues never get too deep and no mail is substantially
delayed.  When this primary MX refuses connections, MTA's connect to our
MX subordinates and mail keeps flowing nicely.  I am considering
lowering the load at which we refuse connections to 5 or 7 and see if
that distributes the load better.  Of course, if we had another equally
fast system, we'd make the MX values equal to balance the inbound load,
but we're not there yet.

Also, on a sluggish VIA C3 system that does spam/anti-virus system and
interactive desktop GUI things, it takes about 7-9 seconds to process a
single message.  Lowering the process priority for MailScanner itself
keeps the mailscanner load in the background and system response
(relatively) perky.

Unfortunately, I don't see an easy way to change the priorities via
/etc/sysconfig/MailScanner.  If I try
SENDMAIL="/bin/nice -19 /usr/sbin/sendmail"

then the following check in /etc/init.d/MailScanner:

[ -f $SENDMAIL ] || exit 0

results in "bash: [: too many arguments" and we head for the closest exit.

Perhaps you will consider adding variables such as PRIORITY_INBOUND,
PRIORITY_OUTBOUND, PRIORITY_MAILSCANNER to the /etc/init.d/MailScanner
and /etc/sysconfig/MailScanner files.  Attached are diffs to 4.30.3-2
which accomplishes what I think works as expected (note the NI column):

$ ps -efl | egrep "sendmail|perl|NI"
F S UID        PID  PPID  C PRI  NI ADDR    SZ WCHAN  STIME TTY
TIME CMD
5 S root     24995     1  0  79  19    -  1543 do_sel 00:43 ?
00:00:00 sendmail: accepting
connections

1 S smmsp    25000     1  0  78  19    -  1501 pause  00:43 ?
00:00:00 sendmail: Queue runner at 00:15:00 for /var/spool/clientmqueue
1 S root     25006     1  0  65 -10    -  1501 pause  00:43 ?
00:00:00 sendmail: Queue runner at 00:15:00 for /var/spool/mqueue
1 S root     25024     1  0  75  10    -  5305 wait4  00:43 ?
00:00:00 /usr/bin/perl -I/usr/lib/MailScanner /usr/sbin/MailScanner
/etc/MailScanner/MailScanner.conf
1 S root     25025 25024  6  75  10    - 12393 nanosl 00:43 ?
00:00:09 /usr/bin/perl -I/usr/lib/MailScanner /usr/sbin/MailScanner
/etc/MailScanner/MailScanner.conf
1 S root     25030 25024  6  75  10    - 12407 nanosl 00:43 ?
00:00:09 /usr/bin/perl -I/usr/lib/MailScanner /usr/sbin/MailScanner
/etc/MailScanner/MailScanner.conf


Note that I am not necessarily endorsing setting PRIORITY_OUTBOUND to a
higher than normal priority, it was just a test to ensure the code worked.

Regards,
Chris Schanzle

-------------------------- MailScanner list ----------------------
To leave, send    leave mailscanner    to jiscmail at jiscmail.ac.uk
Before posting, please see the Most Asked Questions at
http://www.mailscanner.biz/maq/     and the archives at
http://www.jiscmail.ac.uk/lists/mailscanner.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: add-priorities.patch
Type: text/x-patch
Size: 4205 bytes
Desc: not available
Url : http://lists.mailscanner.info/pipermail/mailscanner/attachments/20040525/8594aa57/add-priorities-0001.bin