MailScanner/Spamassassin slow after version upgrade

[Julian Field]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Also, currently to disable watermarking, set *all* the watermarking 
options to no. Not just "Add Watermark". That problem will hopefully go 
away by the time I release it as stable.

donald.dawson at bakerbotts.com wrote:
> Here are the differences between the old and new conf file:
>
> # diff MailScanner.conf old/MailScanner.conf.07262007 
> < Zip Attachments = no
> < Attachments Zip Filename = MessageAttachments.zip
> < Attachments Min Total Size To Zip = 100k
> < Attachment Extensions Not To Zip = .zip .rar .gz .tgz .jpg .jpeg .mpg
> .mpe .mpeg .mp3 .rpm .htm .html .eml
> < Monitors for ClamAV Updates = /usr/local/share/clamav/*.inc/*
> /usr/local/share/clamav/*.cvd
>   
>> Monitors for ClamAV Updates = /usr/local/share/clamav/*.cvd
>>     
> < Clamd Port = 3310
> < Clamd Socket = /tmp/clamd
> < Clamd Lock File = # /var/lock/subsys/clamd
> < Clamd Use Threads = no
> < Known Web Bug Servers = msgtag.com
> < Signature Image Filename = %report-dir%/sig.jpg
> < Signature Image <img> Filename = signature.jpg
> < Attach Image To Signature = no
> < Attach Image To HTML Message Only = yes
> < Spam List = # spamhaus-ZEN # You can un-comment this to enable them
>   
>> Spam List = SBL+XBL
>>     
> < Ignore Spam Whitelist If Recipients Exceed = 40
>   
>> Ignore Spam Whitelist If Recipients Exceed = 75
>>     
> < Add Watermark = yes
> < Check Watermarks = yes
> < Treat Invalid Watermarks as Spam = spam
> < Watermark Secret = %org-name%-Secretfrog
> < Watermark Lifetime = 604800
> < Watermark Header = X-%org-name%-MailScanner-Watermark:
> < Max SpamAssassin Size = 100k
>   
>> Max SpamAssassin Size = 90k
>>     
> < Max SpamAssassin Timeouts = 10
>   
>> Max SpamAssassin Timeouts = 20
>>     
> < SpamAssassin Rule Actions =
> < SpamAssassin Temporary Dir =
> /var/spool/MailScanner/incoming/SpamAssassin-Temp
> < MailScanner Version Number = 4.62.6
>   
>> MailScanner Version Number = 4.59.4
>>     
>
> I could downgrade, but I am reticent to do so.
>
> What I may do is work on another mail server and upgrade in pieces.  
>
> 1)  Upgrade MS, but not SA.
> 2)  turn off watermarking at first
> 3)  leave rule sets the same, then implement changes one by one
> 4)  sa-compile after the other tests
> 5)  upgrade SA
>
> -----Original Message-----
> From: mailscanner-bounces at lists.mailscanner.info
> [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf Of Julian
> Field
> Sent: Monday, July 30, 2007 1:10 PM
> To: MailScanner discussion
> Subject: Re: MailScanner/Spamassassin slow after version upgrade
>
>
>
> * PGP Bad Signature, Signed by an unverified key: 07/30/07 at 19:09:46
>
> Also, what new settings did it add when you did the 
> upgrade_MailScanner_conf?
> Are any of those options on when you would prefer them to be off?
> The new code should only ever look up new config options, and not 
> execute new code if you don't have them switched on.
>
> Also, how easy is it for you to downgrade again? 
> upgrade_MailScanner_conf will happily downgrade as well as upgrade.
>
> Gottschalk, David wrote:
>   
>> How many children are you running?
>>  
>> Are you running a caching DNS server for SA lookups?
>>  
>> what does 'MailScanner --debug -debug-sa' look like?
>>  
>> One thing that made a difference for me was uping my children from 5 
>> to 10. Also lowering my "Max Unsafe Messages Per Scan" to 10, and my 
>> "Max Unscanned Messages Per Scan" to 10 helped speed things up as
>>     
> well.
>   
>>  
>> Hope that helps.
>>  
>> David Gottschalk
>>  
>>
>>     
> ------------------------------------------------------------------------
>   
>> *From:* mailscanner-bounces at lists.mailscanner.info 
>> [mailto:mailscanner-bounces at lists.mailscanner.info] *On Behalf Of 
>> *donald.dawson at bakerbotts.com
>> *Sent:* Monday, July 30, 2007 1:44 PM
>> *To:* mailscanner at lists.mailscanner.info
>> *Subject:* MailScanner/Spamassassin slow after version upgrade
>>
>> The time it takes to process an email has doubled since Friday 
>> (7/26/07) when I upgraded to MS 4.62.6 and SA 3.2.2.  It was around 
>> 3.5 seconds, and now it is averaging 7 seconds.
>>
>> We use DCC and razor2 - no pyzor.
>>
>> Jul 30 12:36:35 houmx05 MailScanner[21682]: Batch (30 messages) 
>> processed in 187.50 seconds
>> Jul 30 12:37:06 houmx05 MailScanner[21662]: Batch (16 messages) 
>> processed in 105.59 seconds
>> Jul 30 12:37:24 houmx05 MailScanner[21727]: Batch (30 messages) 
>> processed in 196.20 seconds
>> Jul 30 12:37:28 houmx05 MailScanner[21776]: Batch (30 messages) 
>> processed in 211.89 seconds
>> Jul 30 12:37:38 houmx05 MailScanner[21793]: Batch (30 messages) 
>> processed in 223.64 seconds
>>
>> sar -u shows idle CPU time:
>> 12:00:01 AM   CPU    %user    %nice  %system  %iowait    %idle
>> Average:          all     10.49      0.00      4.99      1.00
>>     
> 83.51
>   
>> # uptime is normal
>>  12:38:30 up 5 days, 25 min,  1 user,  load average: 0.81, 0.79, 0.83
>>
>> I have implemented sa-compile using re2c expecting to get a 
>> performance boost.
>>
>> 'MailScanner --lint' doesn't show any errors, except noting that we 
>> have clamav processing turned off (output attached).
>>
>> <<ms.txt>>
>> I have included output from spamassassin's lint command.
>>
>> /etc/mail/spamassassin contents:
>> bakerbotts.cf   - custom local rule file
>> Botnet.cf
>> Botnet.pm
>> init.pre
>> init.pre.pre-v310
>> KAM.cf
>> local.cf
>> mailscanner.cf
>> pdfinfo.cf
>> sare-sa-update-channels.txt
>> sa-update-keys
>> v310.pre
>> v312.pre
>> v320.pre
>>
>> contents of sare-sa-update-channels.txt used by update_spamassassin:
>>
>> updates.spamassassin.org
>> 70_sare_adult.cf.sare.sa-update.dostech.net
>> 70_sare_bayes_poison_nxm.cf.sare.sa-update.dostech.net
>> 70_sare_evilnum0.cf.sare.sa-update.dostech.net
>> 70_sare_genlsubj0.cf.sare.sa-update.dostech.net
>> 70_sare_genlsubj1.cf.sare.sa-update.dostech.net
>> 70_sare_genlsubj2.cf.sare.sa-update.dostech.net
>> 70_sare_header.cf.sare.sa-update.dostech.net
>> 70_sare_highrisk.cf.sare.sa-update.dostech.net
>> 70_sare_html.cf.sare.sa-update.dostech.net
>> 70_sare_obfu.cf.sare.sa-update.dostech.net
>> 70_sare_oem.cf.sare.sa-update.dostech.net
>> 70_sare_random.cf.sare.sa-update.dostech.net
>> 70_sare_specific.cf.sare.sa-update.dostech.net
>> 70_sare_spoof.cf.sare.sa-update.dostech.net
>> 70_sare_stocks.cf.sare.sa-update.dostech.net
>> 70_sare_unsub.cf.sare.sa-update.dostech.net
>> 70_sare_uri0.cf.sare.sa-update.dostech.net
>> 70_sare_uri1.cf.sare.sa-update.dostech.net
>> 70_sare_uri2.cf.sare.sa-update.dostech.net
>> 70_sare_whitelist.cf.sare.sa-update.dostech.net
>> 70_sare_whitelist_rcvd.cf.sare.sa-update.dostech.net
>> 70_sare_whitelist_spf.cf.sare.sa-update.dostech.net
>> 72_sare_bml_post25x.cf.sare.sa-update.dostech.net
>> 72_sare_redirect_post3.0.0.cf.sare.sa-update.dostech.net
>> 88_FVGT_headers.cf.sare.sa-update.dostech.net
>> 99_FVGT_Tripwire.cf.sare.sa-update.dostech.net
>> 99_sare_fraud_post25x.cf.sare.sa-update.dostech.net
>>
>> Any ideas would be greatly appreciated.
>>
>> Thanks,
>> Donald
>>
>> Donald Dawson
>> Security Administrator
>> Baker Botts L.L.P.
>> 713-229-2183
>>
>>     
>
> Jules
>
>   

Jules

- -- 
Julian Field MEng CITP
www.MailScanner.info
Buy the MailScanner book at www.MailScanner.info/store

MailScanner customisation, or any advanced system administration help?
Contact me at Jules at Jules.FM

PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
For all your IT requirements visit www.transtec.co.uk


-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.6.2 (Build 2014)
Charset: ISO-8859-1

wj8DBQFGrjMPEfZZRxQVtlQRAn8TAKC/LeeqyN22LViGl8q3qn6uwyWKMACg7MiB
vgyIaHs+LOFMsjwfFyV8C6k=
=T7lo
-----END PGP SIGNATURE-----

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
For all your IT requirements visit www.transtec.co.uk