Action when queue messages exceed Criticial Queue Size

[Sigurbjörn Birkir Lárusson]

I've been torture testing a mailscanner setup running on two IBM eServer 325's (nice machines btw) by pushing a lot of randomly sized mail to the machines.  The mail is about 30% infected (virus or spam messages) and about 70% clean.

Last night there was a high peek in the amount of messages received and there was quite a queue buildup on the machines, which is when I noticed something I found somewhat peculiar.

I had my CriticialQueueSize variable set at 2000 (seemed high enough) and this morning I noticed that the oldest mail was getting the greatest delay (which is normal, considering that when the CriticalQueueSize threshold is reached, it starts processing by directory order, and not by date order).  What did surprise me however, is when it fell under 2000 (and I expected it to start dealing with the queue oldest first) that didn't happen.

I kept watching the process (by doing a ls -lt on the mqueue) and the oldest messages were not being dealt with, it was still processing everything in directory order even when only 1000 messages were left in the Queue.

To see what was going on I looked at the Sendmail.pm code and I inserted a single log line (using MailScanner::Log::InfoLog) into the code and restarted MailScanner.  When I did that, MailScanner immediately began acting on the oldest mail first, leading me to believe that something in the code was preventing it from turning back into the normal delivery mode.

Another quick glance at the code revealed the following code stub:

$UnsortedBatchesLeft = 40
        if $CriticalQueueSize>0 && $MsgsInQueue>=$CriticalQueueSize;
      # SortedFiles is array of full pathnames now, not just filenames
      if ($UnsortedBatchesLeft>0) {
        $UnsortedBatchesLeft--;
      } else {
        @SortedFiles = sort { $ModDate{$a} <=> $ModDate{$b} } keys %ModDate;
      }

The code is easy enough to read and understand, what I don't understand is the meaning of the figure 40 (UnsortedBatchesLeft = 40) in the case where the CriticialQueueSize is set and the messages in the queue exceed the criticial queue size. 

Correct me if I'm wrong here, but this means that for 40 batches (seeing as it's decremented by one in the same code if it's >0) it will keep on scanning in accelerated mode, even though the amount of messages in the queue has fallen below the CriticalQueueSize limit.  In my current running mode this means that it won't hit normal strict date mode until the queue size is CriticalQueueSize-40*30 or CriticalQueueSize-1200, in my case 800 messages.

I would think that a better calculation (even if it's not always accurate, it's certainly more accurate than the hard coded 40) would be to calculate $UnsortedBatchesLeft = ($MsgsInQueue-$CriticalQueueSize)/30?

Sorry if this is an obvious question and has been answered before, I didn't find any reference to this particular subject on the list.

Best regards,
---
Sigurbjörn B. Lárusson <sibbi at margmidlun.is>/<sibbi at bofh.is>
Kerfisstjóri/System Administrator
Margmiðlun hf 
 

-------------------------- MailScanner list ----------------------
To leave, send    leave mailscanner    to jiscmail at jiscmail.ac.uk
Before posting, please see the Most Asked Questions at
http://www.mailscanner.biz/maq/     and the archives at
http://www.jiscmail.ac.uk/lists/mailscanner.html