MailScanner/Postfix message duplication - possible fix

[Kash, Howard (Civ,ARL/CISD)]

I beginning to think you are correct.  My patch doesn't seem to help.
Here's is one method suggested by Peter Bates for putting messages in
the hold queue:


> I'm using MS with Postfix in a slightly 'non-standard' way, but which
> is working fine for 13-15K messages we deal with (actually it might be
> more, I never bothered counting our outgoing email!)...
>
> I'm using a 'header_check' like so:
>
> In main.cf -
> header_checks = pcre:/etc/postfix/header_checks
>
> In header_checks -
>
> /^Received:.*by .*\.lshtm.ac.uk \(Postfix\)/ HOLD
>
> This puts the incoming mail in the 'hold' queue, and then
> I have in MailScanner.conf -
>
> Incoming Queue Dir = /var/spool/postfix/hold
> Outgoing Queue Dir = /var/spool/postfix/incoming


I think I'll give this a try.  


Howard


-----Original Message-----
From: Drew Marshall [mailto:drew at THEMARSHALLS.CO.UK] 
Sent: Tuesday, December 16, 2003 5:54 PM
To: MAILSCANNER at JISCMAIL.AC.UK
Subject: Re: MailScanner/Postfix message duplication - possible fix


I wonder if this is only part of the story. Not being a programmer (Or
even someone who 'tinkers with code'!) please forgive me if I am being
stupid or just plain don't understand :-)

The queue manager runs the queues when it's either called by receipt of
a 1byte message from another part of Postfix or when it's inactivity
timer times out (As set in the master.cf file). I had a play with this
to start with and when I set the idle timer to 28 days I still got
duplicates and the 'skipped' log entry from when MailScanner happened to
be picking up a queued file and the queue runner had been called by
smtpd because it had just received a message. In some instances (One
larger message of 9Mb) this meant on my slow system that I didn't just
get duplicates but I got the damn thing 5 times, in various states of
delivery as it spooled into the deferred queue.

Now my gamble is that moving your times to 40 seconds or even more will
probably not cure the problem as if your system is fairly busy the queue
runner will be almost continuously running through the deferred queue as
it collects mail and checks for messages that are due for attempted
redelivery (I guess this happens on every visit to the queue to ensure
that ageing messages are not left in deferred for too long). It's that
check that could be the problem. If MS is just about to collect the
message when the queue runner inspects the message for age (Not worth
locking for? Don't know?) then the two paps collide and cause the
situation as seen. It won't matter how long you tell MS to leave the
message there for, the queue runner could still bump into the
collection. On my much quieter system it will probably work more
reliably for longer as the queue runner will be called less by smtpd an
more by the inactivity timer.

One way round this could be to send the messages to the hold queue as
the queue runner never runs in there. Now just to get the messages
there...

As I say I could be talking rubbish and I'll go away and keep going with
what ever experiment people want to fix this issue but I though it was
worth knocking some thoughts about.

Regards

Drew