Performance between SpamAssassin 3.2.1 and 3.2.2 and 3.2.3

Wed Aug 15 18:29:48 IST 2007

Ok it's 1EST here and the huge delays that I found on Fri, Mon and Tues
when I was running 3.2.3 are not occurring.  My mqueue.in has remained
swift and steady and the number of Batch 30s is down to only 13 after
24k messsages processed by MS so far today.  Today, I'm running with SA
3.2.3 but with the DNS.pm from 3.2.1.  Previous days with SA 3.2.3, my
batch 30's were in the 200-300 range because of the delays introduced in
3.2.2-.3.  Previously at this time with 3.2.2 and 3.2.3, my mqueue.in
would be backed up to 700-900 messages which although is normal load for
this time of day, it wasn't normal for the queue to be backing up.  

So I'm betting it is the changes made to the DNS.pm module.  I'll try
patching 3.2.3 with
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5589 tonight and
see what happens tomorrow.

-----Original Message-----
From: mailscanner-bounces at lists.mailscanner.info
[mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf Of Daniel
Eiland
Sent: Wednesday, August 15, 2007 10:50 AM
To: 'MailScanner discussion'
Subject: RE: Performance between SpamAssassin 3.2.1 and 3.2.2 and 3.2.3

I am also seeing a dramatic performance decrease.
My mailserver does around 30k emails a day, no problem with my previous
spamassassin install, but after upgrading to 3.2.3 my queue is always
around 1000 e-mails and emails are scanned with a delay of around 2
hours.

The strange thing is my system is 70-90% idle, so a DNS problem sounds
very likely to me.

In my maillog I log the speed, I get around 300 bytes per second! That's
terribly slow. With a similar server, but with 3.2.1i get 45000 bytes
per second.

There must be something seriously wrong I think.
I tried disabling plugins, but no significant difference.

I hope someone has a solution for this.

Daniel

-----Oorspronkelijk bericht-----
Van: mailscanner-bounces at lists.mailscanner.info
[mailto:mailscanner-bounces at lists.mailscanner.info] Namens Rose, Bobby
Verzonden: woensdag 15 augustus 2007 13:53
Aan: MailScanner discussion
Onderwerp: RE: Performance between SpamAssassin 3.2.1 and 3.2.2 and
3.2.3

My change was done last night before I knew about this dns async issue
that was introduced in 3.2.2.  I know I saw mention of the async issue
with ASN but not DNS in general.  I'll try patching 3.2.3 later today
after I've confirmed that that is where the delay in SA is coming from.
It appears to be resolved after I made my change which I only made based
on the changelogs and after diff'ing all the SA pm to confirm that there
weren't any changes elsewhere that replacing Dns.pm would effect.

I'm just surprised that no one else isn't noticing the performance
differences based on MailScanner processing times between versions. 

Bobby Rose
Senior Systems Administrator
MSIS Network Operations
Wayne State University School of Medicine

-----Original Message-----
From: mailscanner-bounces at lists.mailscanner.info
[mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf Of Randal,
Phil
Sent: Wednesday, August 15, 2007 7:29 AM
To: MailScanner discussion
Subject: RE: Performance between SpamAssassin 3.2.1 and 3.2.2 and 3.2.3

Oh, I would have tried 3.2.3 with the aforementioned patch to see if
that made a difference too, because that patch will end up in 3.2.4.

Is there any chance that you can do that?  The patch affects more than
DNS.pm, though.

If there's still an issue on 3.2.3 with the patch from bug 5589 then you
should add a note to that bug and raise the issue on the Spamassassin
Users mailing list too.

Cheers,

Phil

--
Phil Randal
Network Engineer
Herefordshire Council
Hereford, UK  

> -----Original Message-----
> From: mailscanner-bounces at lists.mailscanner.info
> [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf Of Rose,

> Bobby
> Sent: 15 August 2007 12:14
> To: MailScanner discussion
> Subject: RE: Performance between SpamAssassin 3.2.1 and 3.2.2 and
> 3.2.3
> 
> Yeh that might be it.  I checked the changelog and saw that the DNS 
> changes were the only real big change between 3.2.1. and
> 3.2.2 so what I
> did was installed 3.2.3 and replaced the Dns.pm with the version from 
> 3.2.1.  I'll know if that is the problem lies there later on today as 
> traffic load increases.
> 
> I grepped my logs from between the versions focusing on batches of 30 
> and I clearly see differences of 100 to 150 secs between 3.2.1 and the

> later versions.  If anyone wants to check their times use this
> 	egrep -e"Batch \(30 messages\) processed in" maillog
> 
> -=B
> 
> -----Original Message-----
> From: mailscanner-bounces at lists.mailscanner.info
> [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf Of 
> Randal, Phil
> Sent: Wednesday, August 15, 2007 5:38 AM
> To: MailScanner discussion
> Subject: RE: Performance between SpamAssassin 3.2.1 and 3.2.2 and
> 3.2.3
> 
> Running spamassassin -D -t <quarantined file name> reveals a lot.
> 
> I'm running SA with the patch from
> 
>   http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5589
> 
> and the debug output made that one pretty obvious.
> 
> Cheers,
> 
> Phil
> 
> --
> Phil Randal
> Network Engineer
> Herefordshire Council
> Hereford, UK
> 
> > -----Original Message-----
> > From: mailscanner-bounces at lists.mailscanner.info
> > [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf Of 
> > Leland J. Steinke
> > Sent: 14 August 2007 21:18
> > To: MailScanner discussion
> > Subject: Re: Performance between SpamAssassin 3.2.1 and 3.2.2 and
> > 3.2.3
> > 
> > Randal, Phil wrote:
> > > I'm getting very slow DNS lookups from
> > combined-HIB.dnsiplists.completewhois.com.
> > > 
> > > That might be a factor.
> > > 
> > > Does using
> > > 
> > > score __RCVD_IN_WHOIS 0
> > > score RCVD_IN_WHOIS_INVALID 0
> > > score URIBL_COMPLETEWHOIS 0
> > > 
> > > solve the performance problem?
> > 
> > Much to my recent regret, I have avoided subscribing to any 
> > spamassassin support lists, opting instead to let all of you more 
> > adventurous types work out the bugs while I just implement what is 
> > recommended on the MS list.  Would information such as the
> above be on
> 
> > the SA users list?  Is there a FAQ with known "gotchas" 
> with running
> > SA (3.2.2 in our case), where this is listed?  How in
> Perdition do you
> 
> > track that "combined-HIB.dnsiplists.completewhois.com." is slow?
> > 
> > And it's only Tuesday!
> > 
> > 
> > thanks,
> > Leland