MailScanner/Postfix message duplication - possible fix

[Drew Marshall]

Well it's now in place and working so we'll have to wait and see. I'll
let you know what happens. Hopefully this could just be the Postfix
solution ;-)

Drew

Lindsay Snider wrote:

>On Thu, 2003-12-18 at 13:21, Kash, Howard (Civ,ARL/CISD) wrote:
>
>
>>I beginning to think you are correct.  My patch doesn't seem to help.
>>Here's is one method suggested by Peter Bates for putting messages in
>>the hold queue:
>>
>>
>>
>>
>>>I'm using MS with Postfix in a slightly 'non-standard' way, but which
>>>is working fine for 13-15K messages we deal with (actually it might be
>>>more, I never bothered counting our outgoing email!)...
>>>
>>>I'm using a 'header_check' like so:
>>>
>>>In main.cf -
>>>header_checks = pcre:/etc/postfix/header_checks
>>>
>>>In header_checks -
>>>
>>>/^Received:.*by .*\.lshtm.ac.uk \(Postfix\)/ HOLD
>>>
>>>This puts the incoming mail in the 'hold' queue, and then
>>>I have in MailScanner.conf -
>>>
>>>Incoming Queue Dir = /var/spool/postfix/hold
>>>Outgoing Queue Dir = /var/spool/postfix/incoming
>>>
>>>
>>I think I'll give this a try.
>>
>>
>
>We have been using the hold method here since Julian added postfix
>support.  Our site is rather large and we use it across a couple of
>versions of postfix 2.x.  So far, it has worked great.
>
>
>
>>Howard
>>
>>
>>-----Original Message-----
>>From: Drew Marshall [mailto:drew at THEMARSHALLS.CO.UK]
>>Sent: Tuesday, December 16, 2003 5:54 PM
>>To: MAILSCANNER at JISCMAIL.AC.UK
>>Subject: Re: MailScanner/Postfix message duplication - possible fix
>>
>>
>>I wonder if this is only part of the story. Not being a programmer (Or
>>even someone who 'tinkers with code'!) please forgive me if I am being
>>stupid or just plain don't understand :-)
>>
>>The queue manager runs the queues when it's either called by receipt of
>>a 1byte message from another part of Postfix or when it's inactivity
>>timer times out (As set in the master.cf file). I had a play with this
>>to start with and when I set the idle timer to 28 days I still got
>>duplicates and the 'skipped' log entry from when MailScanner happened to
>>be picking up a queued file and the queue runner had been called by
>>smtpd because it had just received a message. In some instances (One
>>larger message of 9Mb) this meant on my slow system that I didn't just
>>get duplicates but I got the damn thing 5 times, in various states of
>>delivery as it spooled into the deferred queue.
>>
>>Now my gamble is that moving your times to 40 seconds or even more will
>>probably not cure the problem as if your system is fairly busy the queue
>>runner will be almost continuously running through the deferred queue as
>>it collects mail and checks for messages that are due for attempted
>>redelivery (I guess this happens on every visit to the queue to ensure
>>that ageing messages are not left in deferred for too long). It's that
>>check that could be the problem. If MS is just about to collect the
>>message when the queue runner inspects the message for age (Not worth
>>locking for? Don't know?) then the two paps collide and cause the
>>situation as seen. It won't matter how long you tell MS to leave the
>>message there for, the queue runner could still bump into the
>>collection. On my much quieter system it will probably work more
>>reliably for longer as the queue runner will be called less by smtpd an
>>more by the inactivity timer.
>>
>>One way round this could be to send the messages to the hold queue as
>>the queue runner never runs in there. Now just to get the messages
>>there...
>>
>>As I say I could be talking rubbish and I'll go away and keep going with
>>what ever experiment people want to fix this issue but I though it was
>>worth knocking some thoughts about.
>>
>>Regards
>>
>>Drew
>>
>>
>--
>Lindsay Snider <lindsay at pa.net>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mailscanner.info/pipermail/mailscanner/attachments/20031219/0505d965/attachment.html