Massive queue buildup

Tue Nov 14 23:18:57 GMT 2006

Not to hijack a thread here, but what are the optimal batch size 
numbers? We're averaging 60-70 seconds per 10 message batch, not sure 
what is eating up the resources but where are everyone elses numbers at?

-Vlad

Chandler, Jay wrote:
> Actually, I can do one better than that-- I've got Mailscanner MRTG
> installed.
>
> Right now it's right around 1700 bytes a second, and it's been pretty
> consistently there. 
>
>
> --
> Jay Chandler
> Network Administrator, Chapman University
> 714.628.7249 / chandler at chapman.edu
> Today's Excuse: We are Microsoft.  What you are experiencing is not a
> problem; it is an undocumented feature. 
>
> -----Original Message-----
> From: mailscanner-bounces at lists.mailscanner.info
> [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf Of Brent
> Addis
> Sent: Monday, November 13, 2006 5:35 PM
> To: MailScanner discussion
> Subject: RE: Massive queue buildup
>
> Thats sounds like the one you want.
>  
> Check your syslog, it should show you your batch processing speed
>  
> IE:
>  
> Nov 14 14:30:10 mx3 MailScanner[5500]: Spam Checks completed at 826
> bytes per second Nov 14 14:30:10 mx3 MailScanner[5500]: Virus and
> Content Scanning: Starting Nov 14 14:30:13 mx3 MailScanner[5500]: Virus
> Scanning completed at 4633 bytes per second Nov 14 14:30:13 mx3
> MailScanner[5500]: Uninfected: Delivered 1 messages Nov 14 14:30:13 mx3
> MailScanner[5500]: Virus Processing completed at 692908 bytes per second
> Nov 14 14:30:13 mx3 MailScanner[5500]: Batch completed at 700 bytes per
> second Nov 14 14:30:13 mx3 MailScanner[5500]: Batch (1 message)
> processed in 9.86 seconds Nov 14 14:30:13 mx3 MailScanner[5500]: Logging
> message 1Gjn7U-0000rl-44 to SQL Nov 14 14:30:13 mx3 MailScanner[5500]:
> "Always Looked Up Last" took 0.00 seconds
>  
> Do you have the spamassassin cache enabled?
>
>  
>
> ________________________________
>
> From: mailscanner-bounces at lists.mailscanner.info on behalf of Chandler,
> Jay
> Sent: Tue 11/14/2006 2:15 PM
> To: MailScanner discussion
> Subject: RE: Massive queue buildup
>
>
>
> I don't show any match for anything approaching that, other than "log
> speed," which is set to yes for MRTG.
>
>
>
>
> --
> Jay Chandler
> Network Administrator, Chapman University
> 714.628.7249 / chandler at chapman.edu
> Today's Excuse: not approved by the FCC
>
> -----Original Message-----
> From: mailscanner-bounces at lists.mailscanner.info
> [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf Of Brent
> Addis
> Sent: Monday, November 13, 2006 4:55 PM
> To: MailScanner discussion
> Subject: RE: Massive queue buildup
>
> ok, what about batch processing speed? Try enabling that within
> MailScanner.conf
>
> ________________________________
>
> From: mailscanner-bounces at lists.mailscanner.info on behalf of Chandler,
> Jay
> Sent: Tue 11/14/2006 1:52 PM
> To: MailScanner discussion
> Subject: RE: Massive queue buildup
>
>
>
> brewer# time spamassassin -C
> /usr/local/etc/mail/spamassassin/mailscanner.conf --lint
> 3.092u 0.386s 0:09.71 35.7%     10+37348k 0+0io 0pf+0w
> brewer# time spamassassin -C /usr/local/etc/mail/spamassassin/ --lint
> 4.135u 0.318s 0:04.97 89.3%     10+48865k 0+0io 0pf+0w
> brewer#
>
> Looks good from here.
>
> --
> Jay Chandler
> Network Administrator, Chapman University
> 714.628.7249 / chandler at chapman.edu
> Today's Excuse: terrorist activities
>
> -----Original Message-----
> From: mailscanner-bounces at lists.mailscanner.info
> [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf Of Brent
> Addis
> Sent: Monday, November 13, 2006 4:13 PM
> To: MailScanner discussion
> Subject: RE: Massive queue buildup
>
> 3 seconds?
>
> Not bad!
>
> However, are you sure that includes the mailscanner config? (It might be
> in /etc/spamassassin or /etc/mail/spamassassin)
>
> if not, include the config file in your lint (I think its either -C or
> -P /path/to/spam.assassin.conf)
>
> ________________________________
>
> From: mailscanner-bounces at lists.mailscanner.info on behalf of Chandler,
> Jay
> Sent: Tue 11/14/2006 1:09 PM
> To: MailScanner discussion
> Subject: RE: Massive queue buildup
>
>
>
> Time spamassassin -D --lint takes three point seven seconds-- I don't
> think that's where the holdup is occuring.
>
> Our DNS server is local and on the same netblock as the mailserver in
> question.
>
> We have a few RBLs at connecttime, but they seem to be holding up well.
>
> It's a bit of a stumper...
>
>
> --
> Jay Chandler
> Network Administrator, Chapman University
> 714.628.7249 / chandler at chapman.edu
> Today's Excuse: your keyboard's space bar is generating spurious
> keycodes.
>
> -----Original Message-----
> From: mailscanner-bounces at lists.mailscanner.info
> [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf Of Brent
> Addis
> Sent: Monday, November 13, 2006 3:48 PM
> To: MailScanner discussion
> Subject: RE: Massive queue buildup
>
> if you want to only check spamassassin (generally the slowest part with
> lots of rules) "time spamassassin -D --lint" is a good start. as long as
> your mailscanner.cf is in a reasonable place it should pick it up. If
> you keep an eye on where it seems to take a while, it should help you
> track down the problem.
>
> Or, there is an option within mailscanner to turn on batch process time.
> I forget the name however its in the last quarter of the config
> somewhere.
>
> are you running bayes? If you are, and its standard db files on the
> server, I would recommend migrating these to an sql server somewhere.
> This helped scan times a fair bit too. Another upside is this also means
> you can have multiple servers using the same db.
>
> With RBL's, don't forget spamassassin also does RBL checking so make
> sure your not doing twice the lookups you need to.
>
> 100k a day should be fine on the hardware you have.
>
>
>
> ________________________________
>
> From: mailscanner-bounces at lists.mailscanner.info on behalf of Jay
> Chandler
> Sent: Tue 11/14/2006 12:39 PM
> To: MailScanner discussion
> Subject: Re: Massive queue buildup
>
>
>
> On Nov 13, 2006, at 2:42 PM, Brent Addis wrote:
>
>
>         Check your not running one of those massive blacklists from
> SARE.
>
>         I was running one for a while while testing and a similar thing
> happened. removing it dropped my average scan time from ~2 1/2 minutes
> to 11 seconds per message.
>
>
> Where does one determine how long the average scan time is?
>
>
>         Other ideas:
>
>         - Check your dns servers are capable of standing up to the
> amount of dns requests you are making. Running something like nscd
> locally is a good idea.
>
>
> I suspect they are, but I'll verify this.
>
>
>         - Are you running very many RBL's within mailscanner? Try
> disabling these and see if it helps.
>
>
> Three or four-- nothing insane.
>
>
>         - Are you running any type of recipient verification? (as in,
> checking that the person being sent the mail actually exists). If not,
> try turning it on. I am unsure what it is called within postfix as I
> don't use it.
>
>
> We are.  Messages to undefined users fault to a 5xx error.
>
>
>         - Check out
> http://wiki.mailscanner.info/doku.php?id=maq:index#optimization_tips
> which has a few others as well.
>
>
> Thanks!
>
>
>         How much mail are you handling a day? I have a couple of single
> cpu 3.0 ghz machines comfortably handling many thousands of messages a
> day.
>
>
>
> Right now, about 100K a day.
>
>
> Thanks for the help!
>
>
>
> --
> Jay Chandler
> Network Administrator, Chapman University
> 714-628-7249 / chandler at chapman.edu
> "Bother," said Pooh as he struggled with /etc/sendmail.cf, "it never
> does quite what I want.  I wish Christopher Robin was here." -- Peter Da
> Silva in a.s.r.
>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/mailman/listinfo/mailscanner
>
> Before posting, read http://wiki.mailscanner.info/posting
>
> Support MailScanner development - buy the book off the website!
>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/mailman/listinfo/mailscanner
>
> Before posting, read http://wiki.mailscanner.info/posting
>
> Support MailScanner development - buy the book off the website!
>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/mailman/listinfo/mailscanner
>
> Before posting, read http://wiki.mailscanner.info/posting
>
> Support MailScanner development - buy the book off the website!
>
>
>   