High CPU system usage

[Joshua Hirsh]

    [ The following text is in the "iso-8859-1" character set. ]
    [ Your display is set for the "US-ASCII" character set.  ]
    [ Some characters may be displayed incorrectly. ]

> Joshua, can you share your stats about CPU % system usage 
> with a 2.6 kernel?

 This system might not be the best example, but it's the only one I have access to at the moment. It averages between 15k and 20k messages per day (it used to be around 30k to 50k, but I reject at the MTA level with RBLs now).

 My peak times are always between 9 and 10 am (twice the regular hourly average for the rest of the business day). This particular system is a quad 700 Mhz Xeon (Dell PE 6450), with 2gb of RAM and 1gb for SWAP.


 My average daily CPU usages is only 25%, but normally between 30 and 50% during peak times. The system CPU usage is between 5 and 10%, the rest is user. IOWait has never reached higher than 3% and averages around 1.5% throughout the day. This system isn't using raid though, as the support for older Dell PERC raid cards was dropped by Dell/Red Hat in the 2.6 series, so I have separate disks in use for /, /var/spool and /var/spool/MailScanner to split up the I/O usages for the system, spooler and quarantine.

 I also have ATIME disabled on /var/spool, /var/log and /var/spool/MailScanner.

 The system also runs with 10 children per CPU, and averages only 1mb in swap.


 When I had originally upgraded to a 2.6 series kernel, I was running on different hardware (a dual 933 P3), but I did see a noticeable performance boost at the time. Unfortunately, it was long enough ago that I can't recall what it was, or how much of a % change it was. I would tentatively say between a 10 and 20% difference.



 From your sar output, it does appear that you have a very small backlog for I/O, but for the amount of mail you have, that should be pretty normal (writing logs, writing mail queues, writing quarantines). If your I/O wait was dipping into the double digits, than I would definitely pinpoint the problem to your disks, but the peak you posted (just under 7%) looks fine to me.

 It seems to indicate that you're running out of processing power, but I would expect that you would be able to process more than 50k messages on a 2.8Ghz machine. The last machine I was able to stress test was a dual 3Ghz Xeon, and I peaked it out at around 1.7 million messages per day.


 The only thing I could recommend at this point is to take a closer look at what your processes are doing. It's possible that a few of them are taking the system down with them by doing extra checks they shouldn't be. Try turning on the speed timings for MS and see if you can pinpoint a specific point that's slowing down your system.


 Cheers,
-Joshua

------------------------ MailScanner list ------------------------
To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
'leave mailscanner' in the body of the email.
Before posting, read the Wiki (http://wiki.mailscanner.info/) and
the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).

Support MailScanner development - buy the book off the website!