Action when queue messages exceed Criticial Queue Size

[Julian Field]

At 17:02 07/07/2004, you wrote:
>Correct me if I'm wrong here, but this means that for 40 batches (seeing
>as it's decremented by one in the same code if it's >0) it will keep on
>scanning in accelerated mode, even though the amount of messages in the
>queue has fallen below the CriticalQueueSize limit.  In my current running
>mode this means that it won't hit normal strict date mode until the queue
>size is CriticalQueueSize-40*30 or CriticalQueueSize-1200, in my case 800
>messages.

You are absolutely right. The aim was to give it some hysterisis so that it
would continue trying to quickly clear the queue until the queue had got
nice and small again. With a max size of 1000 and a batch size of 30, that
means that (by default) 5 threads will do 40*30=1200 messages each before
switching back to normal date-order mode. This should end you up with a
nice empty queue with only 1 or 2 new messages appearing in it for each
batch, so the delay in switching back to date-order has no effect at that
point.

If I switch back to date-order as soon as the queue size gets below the
threshold figure, it is highly likely that it will just start to grow
again, as processing a 2000-file directory on many filesystems is a long
job. I wanted to *clear* the queue when it got too big, not just do part of
the job as that may not actually help.

>I would think that a better calculation (even if it's not always accurate,
>it's certainly more accurate than the hard coded 40) would be to calculate
>$UnsortedBatchesLeft = ($MsgsInQueue-$CriticalQueueSize)/30?

I agree 40 was a number I pulled out of thin air, but it seemed a
reasonable figure to me and has proved just fine in several real situations
where I have seen the queue-growing problem occur on production sites.

Is my 40 and more arbitrary than your 30? (Rhetorical question :-)
Surely the number of child processes should be included in your calculation
as well, as they will all hit queue-clearing mode pretty much at the same
time.
--
Julian Field
www.MailScanner.info
Professional Support Services at www.MailScanner.biz
MailScanner thanks transtec Computers for their support
PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654

-------------------------- MailScanner list ----------------------
To leave, send    leave mailscanner    to jiscmail at jiscmail.ac.uk
Before posting, please see the Most Asked Questions at
http://www.mailscanner.biz/maq/     and the archives at
http://www.jiscmail.ac.uk/lists/mailscanner.html