MailScanner/Postfix message duplication - possible fix 2

[Julian Field]

I came up with another idea. As you know, I can't reproduce the problem on
my systems, so you will just have to try this and see what happens.

The attached patch is for Postfix.pm
         cd /usr/lib/MailScanner/MailScanner
         patch -p0 < /tmp/Postfix.pm.patch

Please let me know how you get on with it.

At 19:32 19/12/2003, you wrote:
>Well it's now in place and working so we'll have to wait and see. I'll let
>you know what happens. Hopefully this could just be the Postfix solution ;-)
>
>Drew
>
>Lindsay Snider wrote:
>>
>>On Thu, 2003-12-18 at 13:21, Kash, Howard (Civ,ARL/CISD) wrote:
>>
>>>
>>>I beginning to think you are correct.  My patch doesn't seem to help.
>>>Here's is one method suggested by Peter Bates for putting messages in
>>>the hold queue:
>>>
>>>
>>>
>>>>
>>>>I'm using MS with Postfix in a slightly 'non-standard' way, but which
>>>>is working fine for 13-15K messages we deal with (actually it might be
>>>>more, I never bothered counting our outgoing email!)...
>>>>
>>>>I'm using a 'header_check' like so:
>>>>
>>>>In main.cf -
>>>>header_checks = pcre:/etc/postfix/header_checks
>>>>
>>>>In header_checks -
>>>>
>>>>/^Received:.*by .*\.lshtm.ac.uk \(Postfix\)/ HOLD
>>>>
>>>>This puts the incoming mail in the 'hold' queue, and then
>>>>I have in MailScanner.conf -
>>>>
>>>>Incoming Queue Dir = /var/spool/postfix/hold
>>>>Outgoing Queue Dir = /var/spool/postfix/incoming
>>>>
>>>
>>>
>>>I think I'll give this a try.
>>>
>>
>>
>>We have been using the hold method here since Julian added postfix
>>support.  Our site is rather large and we use it across a couple of
>>versions of postfix 2.x.  So far, it has worked great.
>>
>>
>>>
>>>
>>>Howard
>>>
>>>
>>>-----Original Message-----
>>>From: Drew Marshall
>>>[<mailto:drew at THEMARSHALLS.CO.UK>mailto:drew at THEMARSHALLS.CO.UK]
>>>Sent: Tuesday, December 16, 2003 5:54 PM
>>>To: <mailto:MAILSCANNER at JISCMAIL.AC.UK>MAILSCANNER at JISCMAIL.AC.UK
>>>Subject: Re: MailScanner/Postfix message duplication - possible fix
>>>
>>>
>>>I wonder if this is only part of the story. Not being a programmer (Or
>>>even someone who 'tinkers with code'!) please forgive me if I am being
>>>stupid or just plain don't understand :-)
>>>
>>>The queue manager runs the queues when it's either called by receipt of
>>>a 1byte message from another part of Postfix or when it's inactivity
>>>timer times out (As set in the master.cf file). I had a play with this
>>>to start with and when I set the idle timer to 28 days I still got
>>>duplicates and the 'skipped' log entry from when MailScanner happened to
>>>be picking up a queued file and the queue runner had been called by
>>>smtpd because it had just received a message. In some instances (One
>>>larger message of 9Mb) this meant on my slow system that I didn't just
>>>get duplicates but I got the damn thing 5 times, in various states of
>>>delivery as it spooled into the deferred queue.
>>>
>>>Now my gamble is that moving your times to 40 seconds or even more will
>>>probably not cure the problem as if your system is fairly busy the queue
>>>runner will be almost continuously running through the deferred queue as
>>>it collects mail and checks for messages that are due for attempted
>>>redelivery (I guess this happens on every visit to the queue to ensure
>>>that ageing messages are not left in deferred for too long). It's that
>>>check that could be the problem. If MS is just about to collect the
>>>message when the queue runner inspects the message for age (Not worth
>>>locking for? Don't know?) then the two paps collide and cause the
>>>situation as seen. It won't matter how long you tell MS to leave the
>>>message there for, the queue runner could still bump into the
>>>collection. On my much quieter system it will probably work more
>>>reliably for longer as the queue runner will be called less by smtpd an
>>>more by the inactivity timer.
>>>
>>>One way round this could be to send the messages to the hold queue as
>>>the queue runner never runs in there. Now just to get the messages
>>>there...
>>>
>>>As I say I could be talking rubbish and I'll go away and keep going with
>>>what ever experiment people want to fix this issue but I though it was
>>>worth knocking some thoughts about.
>>>
>>>Regards
>>>
>>>Drew
>>>
>>
>>--
>>Lindsay Snider <mailto:lindsay at pa.net><lindsay at pa.net>
>>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Postfix.pm.patch
Type: application/octet-stream
Size: 1161 bytes
Desc: not available
Url : http://lists.mailscanner.info/pipermail/mailscanner/attachments/20031222/f5d72d51/Postfix.pm.obj
-------------- next part --------------
--
Julian Field
www.MailScanner.info
MailScanner thanks transtec Computers for their support

PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654