MailScanner/Postfix message duplication - possible fix 2

[Drew Marshall]

I'll have a look at this patch later. Thanks :-)

Interestingly, I had a thought. If the header filter is used to place all
messages in the hold queue and allow MailScanner to collect from there,
you only need to run one Postfix instance. I have this in operation now
since the 19th and not had one duplication (Yet ;-) )

Merry Christmas

Drew
--


Julian Field said:
> I came up with another idea. As you know, I can't reproduce the problem on
> my systems, so you will just have to try this and see what happens.
>
> The attached patch is for Postfix.pm
>          cd /usr/lib/MailScanner/MailScanner
>          patch -p0 < /tmp/Postfix.pm.patch
>
> Please let me know how you get on with it.
>
> At 19:32 19/12/2003, you wrote:
>>Well it's now in place and working so we'll have to wait and see. I'll
>> let
>>you know what happens. Hopefully this could just be the Postfix solution
>> ;-)
>>
>>Drew
>>
>>Lindsay Snider wrote:
>>>
>>>On Thu, 2003-12-18 at 13:21, Kash, Howard (Civ,ARL/CISD) wrote:
>>>
>>>>
>>>>I beginning to think you are correct.  My patch doesn't seem to help.
>>>>Here's is one method suggested by Peter Bates for putting messages in
>>>>the hold queue:
>>>>
>>>>
>>>>
>>>>>
>>>>>I'm using MS with Postfix in a slightly 'non-standard' way, but which
>>>>>is working fine for 13-15K messages we deal with (actually it might be
>>>>>more, I never bothered counting our outgoing email!)...
>>>>>
>>>>>I'm using a 'header_check' like so:
>>>>>
>>>>>In main.cf -
>>>>>header_checks = pcre:/etc/postfix/header_checks
>>>>>
>>>>>In header_checks -
>>>>>
>>>>>/^Received:.*by .*\.lshtm.ac.uk \(Postfix\)/ HOLD
>>>>>
>>>>>This puts the incoming mail in the 'hold' queue, and then
>>>>>I have in MailScanner.conf -
>>>>>
>>>>>Incoming Queue Dir = /var/spool/postfix/hold
>>>>>Outgoing Queue Dir = /var/spool/postfix/incoming
>>>>>
>>>>
>>>>
>>>>I think I'll give this a try.
>>>>
>>>
>>>
>>>We have been using the hold method here since Julian added postfix
>>>support.  Our site is rather large and we use it across a couple of
>>>versions of postfix 2.x.  So far, it has worked great.
>>>
>>>
>>>>
>>>>
>>>>Howard
>>>>
>>>>
>>>>-----Original Message-----
>>>>From: Drew Marshall
>>>>[<mailto:drew at THEMARSHALLS.CO.UK>mailto:drew at THEMARSHALLS.CO.UK]
>>>>Sent: Tuesday, December 16, 2003 5:54 PM
>>>>To: <mailto:MAILSCANNER at JISCMAIL.AC.UK>MAILSCANNER at JISCMAIL.AC.UK
>>>>Subject: Re: MailScanner/Postfix message duplication - possible fix
>>>>
>>>>
>>>>I wonder if this is only part of the story. Not being a programmer (Or
>>>>even someone who 'tinkers with code'!) please forgive me if I am being
>>>>stupid or just plain don't understand :-)
>>>>
>>>>The queue manager runs the queues when it's either called by receipt of
>>>>a 1byte message from another part of Postfix or when it's inactivity
>>>>timer times out (As set in the master.cf file). I had a play with this
>>>>to start with and when I set the idle timer to 28 days I still got
>>>>duplicates and the 'skipped' log entry from when MailScanner happened
>>>> to
>>>>be picking up a queued file and the queue runner had been called by
>>>>smtpd because it had just received a message. In some instances (One
>>>>larger message of 9Mb) this meant on my slow system that I didn't just
>>>>get duplicates but I got the damn thing 5 times, in various states of
>>>>delivery as it spooled into the deferred queue.
>>>>
>>>>Now my gamble is that moving your times to 40 seconds or even more will
>>>>probably not cure the problem as if your system is fairly busy the
>>>> queue
>>>>runner will be almost continuously running through the deferred queue
>>>> as
>>>>it collects mail and checks for messages that are due for attempted
>>>>redelivery (I guess this happens on every visit to the queue to ensure
>>>>that ageing messages are not left in deferred for too long). It's that
>>>>check that could be the problem. If MS is just about to collect the
>>>>message when the queue runner inspects the message for age (Not worth
>>>>locking for? Don't know?) then the two paps collide and cause the
>>>>situation as seen. It won't matter how long you tell MS to leave the
>>>>message there for, the queue runner could still bump into the
>>>>collection. On my much quieter system it will probably work more
>>>>reliably for longer as the queue runner will be called less by smtpd an
>>>>more by the inactivity timer.
>>>>
>>>>One way round this could be to send the messages to the hold queue as
>>>>the queue runner never runs in there. Now just to get the messages
>>>>there...
>>>>
>>>>As I say I could be talking rubbish and I'll go away and keep going
>>>> with
>>>>what ever experiment people want to fix this issue but I though it was
>>>>worth knocking some thoughts about.
>>>>
>>>>Regards
>>>>
>>>>Drew
>>>>
>>>
>>>--
>>>Lindsay Snider <mailto:lindsay at pa.net><lindsay at pa.net>
>>>
> --
> Julian Field
> www.MailScanner.info
> MailScanner thanks transtec Computers for their support
>
> PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654