Performance

[Steve Freegard]

Hi John,


John Schmerold wrote:
> We're seeing significant backlogs, mail is taking 2-6 hours to get thru 
> the Postfix/Mailscanner gauntlet we've setup. What's everyone else 
> seeing in terms of mail processing time?

As long as you have a sub 10 second time per message (Time of Batch / 
Batch Size), then generally things are OK.

> I've looked at the home page & WIKI, so, I'm guessing I am missing 
> something or there are new techniques not yet published on the 
> mailscanner.info
> 
> Some of my statistics are as follows:
> Server config: 2.8GHz P4, 2GB DDR2, Maxtor SATA HDD
> Mail volume: approx 7,500 messages per day
> Misc: We have set the noatime flag on spool and log partitions & use a 
> local DNS caching nameserver.

Okay - do you also have /var/spool/MailScanner/incoming on tmpfs?

You don't mention your which OS you are running - if it is a RedHat 
Clone then you can set:

TMPDIR=/dev/shm
export TMPDIR

In the MailScanner initscript, which will give a small speed-up to 
SpamAssassin which will create all of it's temporary files in tmpfs as well.

> Max Children = 2

This is too low for your hardware, this should be set to 5.

> Max Unscanned Messages Per Scan = 10
> Max Unsafe Messages Per Scan = 10

These are too low also, I would set these to at least 30.  You need a 
batch size that is equal to the number of messages that could come into 
your system on average at any one time within the queue scan interval 
for maximum efficiency.

> PostFix Configuration:

> smtpd_recipient_restrictions = reject_invalid_hostname 
> reject_non_fqdn_hostname reject_non_fqdn_sender  
> reject_non_fqdn_recipient  reject_unknown_sender_domain 
> permit_mynetworks reject_unauth_destination check_sender_access 
> hash:/etc/postfix/whitelist reject_rbl_client cbl.abuseat.org 
> reject_rbl_client zen.spamhaus.org permit

Why are you checking cbl.abuseat.org and zen.spamhaus.org? - drop the 
CBL and use Zen only as it includes all the CBL data anyway.  You could 
also add list.dsbl.org too as this catches a few compromised systems 
that Zen misses here.

As you have a large backlog anyway - I suggest that you put some more 
radical configuration into Postfix so that it doesn't get any worse.

This requires Postfix 2.3 or newer:

# This is required apparently.
smtpd_delay_reject=no

# Introduce a Sendmail equivalent GreetPause
# Use /etc/postfix/nodelay_clients as a whitelist
smtpd_client_restrictions =
  permit_mynetworks
  permit_sasl_authenticated
  check_client_access hash:/etc/postfix/nodelay_clients
  sleep 4
  reject_unauth_pipelining

# Catch the systems that honour GreetPause only
# to start PIPELINING without waiting for EHLO.
# To be properly effective a multi-line greeting
# banner is *required* (patch required for Postfix).
smtpd_helo_restrictions =
  sleep 1
  reject_unauth_pipelining

> MS Log:
> Jan 30 23:40:03 mx1 MailScanner[24752]: Batch (10 messages) processed in 
> 64.20 seconds

6.42 average per message - this looks fine.

If you are running Pyzor, disable it and you'll find this will drop this 
figure even further.

> Jan 30 23:40:03 mx1 MailScanner[24752]: New Batch: Found 7981 messages 
> waiting

Based on 5 children - this will take 6215.73 seconds to process all of 
these messages (1.7 hours).

Hope this helps.

Kind regards,
Steve.