Large queues

[David Lee]

On Wed, 20 Feb 2002, Ray Gardener wrote:

> Some more feedback on the subject of large queues building up.
>
> We have been running mailscanner with exim for 11 days. During most of
> this time delay for local domains were very low > 2 minutes on average
> (servicing around 20,000 messages daily).  Yesterday afternoon we
> configured two Exchange machines to use the mailscanner box as their
> smarthost. These machine typically send combined between 6000-8000 MIME
> mail items daily. After adding these machines the size of the incoming
> queue climbed dramatically > 3000 so that by 6pm items were taking 1.5
> hours to be delivered The outgoing queue never had more than 200
> messages waiting. The machine is a dual processor Sunblade 1000 and the
> load average never climbed above 0.5. The performance degradation didn't
> seem directly proportional to message throughout.
>
> This morning these delays were still happening so reading through the
> mailing list I decided to change the delivery method from batch to queue
> and deliver in background to yes.
>
> The queues very rapidly decreased and I am back to less that 2 minutes
> delay for mail to local domains; which I am happy with. system load
> averages are still less that 0.5 ;-)
>
> Mailscanner settings are:
>
> Version 3.04
> running virus scanning but no spam checking at all. (I am very wary of
> all the DNS checks that are involved and RBL checks are done by exim
> anyway).

Using combined "Deliver In Background = yes" and "Delivery Method = batch"
(or "Delivery Method = queue") is almost essential at sites with large
throughput.

But there is still a weak spot.  There can only be one MailScanner process
(in contrast to the multiple parallel sendmail/whatever processes).  So
any delays here (e.g. DNS name resolution) can cause massive build-ups in
"Incoming Queue Dir", significantly hindering the entire throughput,

If Julian has a "wish list" for MailScanner, my plea for its top item is
to allow multiple MailScanner processes to run.

We have touched on this subject before; Julian has concerns and worries,
understandably, about locking issues.  But I think it is one of those
issues that needs to be addressed, now that MailScanner is becoming a
mainstream product.  (I'm tempted to use the sales-droid "industrial
strength" jargon here...)

--

:  David Lee                                I.T. Service          :
:  Systems Programmer                       Computer Centre       :
:                                           University of Durham  :
:  http://www.dur.ac.uk/t.d.lee/            South Road            :
:                                           Durham                :
:  Phone: +44 191 374 2882                  U.K.                  :