Using process priority to throttle inbound SMTP

[Christopher Schanzle]

Greetings Julian & MailScanner Users.

With a production-quality mail server, you never want to have a 
sustained situation where you have more mail coming in than you have 
mail going out.  Otherwise if you get hit hard (by a spammer or script 
error), you'll have very long message delivery times or worse, be taken 
down due to full disks.  With MailScanner doing spam and maybe virus 
checking, this is easy situation to have happen on a fairly high-volume 
production box with big spikes.

I find it useful to run the inbound SMTP server (all cases) and 
MailScanner itself (some cases) at lower than normal process priorities 
(that is, higher NICE values).

The logic is that if MailScanner is chewing up all the CPU available, 
and the priority of my inbound SMTP process is lower, inbound mail will 
self-throttle and reduce (nearly eliminate) getting large queues of 
email waiting to be processed.  Before changing priorities, we 
frequently saw 800 messages waiting to be processed...we'd eventually 
catch up a while after the spike subsided, but it could take many 
minutes before all email is processed.  After lowering inbound SMTP 
priority, if the flood of inbound mail continues, the load quickly 
climbs to higher numbers until sendmail starts refusing inbound 
connections.  Meanwhile, messages queued will be processed almost as 
quickly as normal as the process scheduling priority to do spam/virus 
checking is higher, and the system remains responsive (of course, 
excepting SMTP connections).

Without adjusting process priorities, having sendmail refuse connections 
at numbers near or slightly above MailScanner.conf's "Max Children" 
setting works fairly well (we find 3 works well on uniprocessors), but 
we spike over so frequently that we get dozens of warnings about 
refusing connections.  Also, we try to avoid refusing connections -- 
it's only viable only if your MUA's don't connect to it directly (MTA's 
are fine and will connect to lower-priority MX's).  In those few 
occasions where users (with interactive MUA's) connect to our external 
servers, that DNS name is tied to our lowest priority MX, so it is 
unlikely they will get "connection refused" messages.

Right now with lowered priority on inbound SMTP, we refuse connections 
about six times each day with a load around 18 (single processor Athlon 
XP 2800+).  But our queues never get too deep and no mail is 
substantially delayed.  When our primary MX refuses connections, MTA's 
connect to our MX subordinates and mail keeps flowing nicely.  I am 
considering lowering the load at which we refuse connections to 5 or 7 
and see if that distributes the load better.  Of course, if we had 
another equally fast system, we'd make the MX values equal to balance 
the inbound load, but we're not there yet.

Also, on a sluggish VIA C3 system that does spam/anti-virus system and 
interactive desktop GUI things, it takes about 7-9 seconds to process a 
single message.  Lowering the process priority for MailScanner itself 
keeps the mailscanner load in the background and system response 
(relatively) perky.

Unfortunately, I don't see an easy way to change the priorities via 
/etc/sysconfig/MailScanner.  If I try
SENDMAIL="/bin/nice -19 /usr/sbin/sendmail"

then the following check in /etc/init.d/MailScanner:

[ -f $SENDMAIL ] || exit 0

results in "bash: [: too many arguments" and we head for the closest exit.
 
Perhaps you will consider adding variables such as PRIORITY_INBOUND, 
PRIORITY_OUTBOUND, PRIORITY_MAILSCANNER to the /etc/init.d/MailScanner 
and /etc/sysconfig/MailScanner files.  Attached are diffs to 4.30.3-2 
which accomplishes what I think works as expected (note the NI column):

$ ps -efl | egrep "sendmail|perl|NI"
F S UID        PID  PPID  C PRI  NI ADDR    SZ WCHAN  STIME TTY          
TIME CMD
5 S root     24995     1  0  79  19    -  1543 do_sel 00:43 ?        
00:00:00 sendmail: accepting 
connections                                                                                                                             

1 S smmsp    25000     1  0  78  19    -  1501 pause  00:43 ?        
00:00:00 sendmail: Queue runner at 00:15:00 for /var/spool/clientmqueue
1 S root     25006     1  0  65 -10    -  1501 pause  00:43 ?        
00:00:00 sendmail: Queue runner at 00:15:00 for /var/spool/mqueue
1 S root     25024     1  0  75  10    -  5305 wait4  00:43 ?        
00:00:00 /usr/bin/perl -I/usr/lib/MailScanner /usr/sbin/MailScanner 
/etc/MailScanner/MailScanner.conf
1 S root     25025 25024  6  75  10    - 12393 nanosl 00:43 ?        
00:00:09 /usr/bin/perl -I/usr/lib/MailScanner /usr/sbin/MailScanner 
/etc/MailScanner/MailScanner.conf
1 S root     25030 25024  6  75  10    - 12407 nanosl 00:43 ?        
00:00:09 /usr/bin/perl -I/usr/lib/MailScanner /usr/sbin/MailScanner 
/etc/MailScanner/MailScanner.conf


Note that I am not necessarily endorsing setting PRIORITY_OUTBOUND to a 
higher than normal priority, it was just a test to ensure the code worked.

Regards,
Chris Schanzle

-------------- next part --------------
A non-text attachment was scrubbed...
Name: add-priorities.patch
Type: text/x-patch
Size: 4187 bytes
Desc: not available
Url : http://lists.mailscanner.info/pipermail/mailscanner/attachments/20040525/579ffa90/add-priorities.bin