MailScanner/Postfix message duplication - possible fix

[Lindsay Snider]

On Thu, 2003-12-18 at 13:21, Kash, Howard (Civ,ARL/CISD) wrote:
> I beginning to think you are correct.  My patch doesn't seem to help.
> Here's is one method suggested by Peter Bates for putting messages in
> the hold queue:
>
>
> > I'm using MS with Postfix in a slightly 'non-standard' way, but which
> > is working fine for 13-15K messages we deal with (actually it might be
> > more, I never bothered counting our outgoing email!)...
> >
> > I'm using a 'header_check' like so:
> >
> > In main.cf -
> > header_checks = pcre:/etc/postfix/header_checks
> >
> > In header_checks -
> >
> > /^Received:.*by .*\.lshtm.ac.uk \(Postfix\)/ HOLD
> >
> > This puts the incoming mail in the 'hold' queue, and then
> > I have in MailScanner.conf -
> >
> > Incoming Queue Dir = /var/spool/postfix/hold
> > Outgoing Queue Dir = /var/spool/postfix/incoming
>
>
> I think I'll give this a try.

We have been using the hold method here since Julian added postfix
support.  Our site is rather large and we use it across a couple of
versions of postfix 2.x.  So far, it has worked great.

>
>
> Howard
>
>
> -----Original Message-----
> From: Drew Marshall [mailto:drew at THEMARSHALLS.CO.UK]
> Sent: Tuesday, December 16, 2003 5:54 PM
> To: MAILSCANNER at JISCMAIL.AC.UK
> Subject: Re: MailScanner/Postfix message duplication - possible fix
>
>
> I wonder if this is only part of the story. Not being a programmer (Or
> even someone who 'tinkers with code'!) please forgive me if I am being
> stupid or just plain don't understand :-)
>
> The queue manager runs the queues when it's either called by receipt of
> a 1byte message from another part of Postfix or when it's inactivity
> timer times out (As set in the master.cf file). I had a play with this
> to start with and when I set the idle timer to 28 days I still got
> duplicates and the 'skipped' log entry from when MailScanner happened to
> be picking up a queued file and the queue runner had been called by
> smtpd because it had just received a message. In some instances (One
> larger message of 9Mb) this meant on my slow system that I didn't just
> get duplicates but I got the damn thing 5 times, in various states of
> delivery as it spooled into the deferred queue.
>
> Now my gamble is that moving your times to 40 seconds or even more will
> probably not cure the problem as if your system is fairly busy the queue
> runner will be almost continuously running through the deferred queue as
> it collects mail and checks for messages that are due for attempted
> redelivery (I guess this happens on every visit to the queue to ensure
> that ageing messages are not left in deferred for too long). It's that
> check that could be the problem. If MS is just about to collect the
> message when the queue runner inspects the message for age (Not worth
> locking for? Don't know?) then the two paps collide and cause the
> situation as seen. It won't matter how long you tell MS to leave the
> message there for, the queue runner could still bump into the
> collection. On my much quieter system it will probably work more
> reliably for longer as the queue runner will be called less by smtpd an
> more by the inactivity timer.
>
> One way round this could be to send the messages to the hold queue as
> the queue runner never runs in there. Now just to get the messages
> there...
>
> As I say I could be talking rubbish and I'll go away and keep going with
> what ever experiment people want to fix this issue but I though it was
> worth knocking some thoughts about.
>
> Regards
>
> Drew
--
Lindsay Snider <lindsay at pa.net>