Action when queue messages exceed Criticial Queue Size

[Sigurbjörn Birkir Lárusson]

> -----Original Message-----
> From: MailScanner mailing list 
> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf Of Julian Field
> Sent: 7. júlí 2004 16:38
> To: MAILSCANNER at JISCMAIL.AC.UK
> Subject: Re: Action when queue messages exceed Criticial Queue Size
> 
> 
> At 17:02 07/07/2004, you wrote:
> >Correct me if I'm wrong here, but this means that for 40 batches 
> >(seeing as it's decremented by one in the same code if it's 
> >0) it will 
> >keep on scanning in accelerated mode, even though the amount of 
> >messages in the queue has fallen below the CriticalQueueSize 
> limit.  In 
> >my current running mode this means that it won't hit normal 
> strict date 
> >mode until the queue size is CriticalQueueSize-40*30 or 
> >CriticalQueueSize-1200, in my case 800 messages.
> 
> You are absolutely right. The aim was to give it some 
> hysterisis so that it would continue trying to quickly clear 
> the queue until the queue had got nice and small again. With 
> a max size of 1000 and a batch size of 30, that means that 
> (by default) 5 threads will do 40*30=1200 messages each 
> before switching back to normal date-order mode. This should 
> end you up with a nice empty queue with only 1 or 2 new 
> messages appearing in it for each batch, so the delay in 
> switching back to date-order has no effect at that point.
> 
> If I switch back to date-order as soon as the queue size gets 
> below the threshold figure, it is highly likely that it will 
> just start to grow again, as processing a 2000-file directory 
> on many filesystems is a long job. I wanted to *clear* the 
> queue when it got too big, not just do part of the job as 
> that may not actually help.

Well I'm running reiserfs here under SLE (on fairly powerfull machines) so it doesn't seem to have much of an effect.  I've set the CriticalQueueSize to 10000 instead of the 2000, it is of course an emergency tactic that needs to be tuned for each site to keep the queue from killing the machine, in this particular case, I think 10000 is a number I can easily live with.  

> 
> >I would think that a better calculation (even if it's not always 
> >accurate, it's certainly more accurate than the hard coded 
> 40) would be 
> >to calculate $UnsortedBatchesLeft = 
> >($MsgsInQueue-$CriticalQueueSize)/30?
> 
> I agree 40 was a number I pulled out of thin air, but it 
> seemed a reasonable figure to me and has proved just fine in 
> several real situations where I have seen the queue-growing 
> problem occur on production sites.
> 
> Is my 40 and more arbitrary than your 30? (Rhetorical 
> question :-) Surely the number of child processes should be 
> included in your calculation as well, as they will all hit 
> queue-clearing mode pretty much at the same time.

True, I'm running 10 processes so in theory this is only about 4 complete runs for each child.

Not that it's directly relevant but I've found that setting your QueueSortOrder in sendmail to modification (oldest messages first) both matches the behaviour of MailScanner and is a lot faster than the usual priority or host based ordering (since it doesn't actually read all the files in the directory to check which to take first).  This option appeared in sendmail 8.12.

Keep up the good work on MailScanner.  So far my two machines have processed (and fully scanned) about 300 gigabytes of mail in roughly 6 days, definately not bad.

> --
> Julian Field
> www.MailScanner.info
> Professional Support Services at www.MailScanner.biz 
> MailScanner thanks transtec Computers for their support PGP 
> footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
> 
> -------------------------- MailScanner list ----------------------
> To leave, send    leave mailscanner    to jiscmail at jiscmail.ac.uk
> Before posting, please see the Most Asked Questions at
> http://www.mailscanner.biz/maq/     and the archives at
> http://www.jiscmail.ac.uk/lists/mailscanner.html
> 

-------------------------- MailScanner list ----------------------
To leave, send    leave mailscanner    to jiscmail at jiscmail.ac.uk
Before posting, please see the Most Asked Questions at
http://www.mailscanner.biz/maq/     and the archives at
http://www.jiscmail.ac.uk/lists/mailscanner.html