Old messages no being processed

[Julian Field]

    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "US-ASCII" character set.  ]
    [ Some characters may be displayed incorrectly. ]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Terran Wright wrote:

>My problem now is that it seems that only new mails are being processed and
>the older mails in the queue are being neglected. The queue went down to
>less than 2000 mails and has now gone back up to > 7000 mails. I've
>implemented spamhaus, spamcop and various other rbls at the MTA level
>temporarily to help clear up the queue however this is not helping.
>  
>
What you are seeing is a side-effect of the failure of the "Emergency 
Queue-Clearing Mode" that MailScanner will switch into when the queue 
gets very large.

Normally, MailScanner processes messages in strict date order, oldest 
first. In order to do this, for every batch it has to read in the entire 
directory and sort it by date. On some filesystem designs this can take 
a very long time. So when the queue gets bigger than the "Max Normal 
Queue Size", MailScanner switches into a mode where it just processes 
the first messages it finds in the queue, regardless of how old they 
are. On many systems, this is a lot faster as it doesn't have to read in 
the whole dir or do any sorting.

If you think about it, you will see that if you process the first 30 and 
delete them from the incoming queue, that leaves 30 blank slots that 
will be filled by the next 30 messages to arrive in the queue. So if you 
are only just keeping up, you will end up continuously processing the 
newest messages and not clearing the backlog very much. This is actually 
exactly what most people want as it enables MailScanner to continue to 
process new mail very quickly, giving the appearance of it staying 
responsive to your users. When your incoming mail load drops (when 
everyone goes home or to sleep or whatever your users do), it will slog 
through the old backlog and sort itself out.

If you look at your system when it has a high incoming load in this 
state, it won't appear to be processing old mail very much, if at all.

So you can either leave it to get on with it, as it will deliver the old 
messages when it can. Or else you can raise the "Max Normal Queue Size" 
number a lot to force it to stay in the normal date-order processing 
mode. But that will add a large delay to all new mail as well as the old 
stuff currently in the queue. So it will actually appear to be worse to 
your users than just letting it get on with it.

It's actually quite a bit more clever about it than my explanation 
above, especially when and if it decides to switch back to normal 
oldest-first processing. It's not a trivial solution, as you need some 
hysterisis in it to avoid a positive feedback oscillating system. And 
who would have thought that an email system involved concepts explained 
in detail in 2nd-year degree level mathematics. Feedback theory, hmmm.....

Sorry that's quite so long an explanation, but the background and 
rationale for it is quite a complex behaviour model.

I suspect that most other systems out there don't quite go to such 
lengths :-)

Hope that helps explain what's happening,
Jules.

- -- 
Julian Field
www.MailScanner.info
Buy the MailScanner book at www.MailScanner.info/store
Professional Support Services at www.MailScanner.biz
MailScanner thanks transtec Computers for their support

PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654


-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.0.2 (Build 2424)

iQA/AwUBQzrw1hH2WUcUFbZUEQL4GACeOyHMB1amU5/JFGHEFvMqtzsyQgcAoNJU
pUPMNIjPKyaysP0Uib0v40aZ
=jjhm
-----END PGP SIGNATURE-----

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

------------------------ MailScanner list ------------------------
To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
'leave mailscanner' in the body of the email.
Before posting, read the Wiki (http://wiki.mailscanner.info/) and
the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).

Support MailScanner development - buy the book off the website!