Mailscanner generated duplicate message

[Cedric Devillers]

Glenn Steen wrote:
> On 14/02/2008, Julian Field <MailScanner at ecs.soton.ac.uk> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>>  Hash: SHA1
>>
>>
>>
>>  Glenn Steen wrote:
>>  > On 08/02/2008, Julian Field <MailScanner at ecs.soton.ac.uk> wrote:
>>  >
>>  >> -----BEGIN PGP SIGNED MESSAGE-----
>>  >> Hash: SHA1
>>  >>
>>  >>
>>  >>
>>  >> Glenn Steen wrote:
>>  >>
>>  >>> On 08/02/2008, Glenn Steen <glenn.steen at gmail.com> wrote:
>>  >>>
>>  >>>
>>  >>>> On 08/02/2008, Glenn Steen <glenn.steen at gmail.com> wrote:
>>  >>>>
>>  >>>>
>>  >>>>> On 07/02/2008, Cedric Devillers <cde at alunys.com> wrote:
>>  >>>>>
>>  >>>>>
>>  >>>>>> Hello,
>>  >>>>>>
>>  >>>>>> I'm trying to revive this thread from the last month because we are
>>  >>>>>> observing the exact same behavior on one of our servers.
>>  >>>>>>
>>  >>>>>>
>>  >>>>> Thanks for doing that, and for providing some more info.
>>  >>>>>
>>  >>>>>
>>  >>>>>
>>  >>>>>> So to remember the facts :
>>  >>>>>>
>>  >>>>>> - We are using mailscanner with postfix, and duplicated messages are
>>  >>>>>> generated by mailscanner.
>>  >>>>>>
>>  >>>>>> - This system is the only one where we are observing this behavior. It
>>  >>>>>> have a little particularity : it mainly act as a mail relay, but
>>  >>>>>> sometimes many mails are generated by the server itself (a script) and
>>  >>>>>> injected in postfix queues via sendmail command. We can always reproduce
>>  >>>>>> some duplicated messages with this script.
>>  >>>>>>
>>  >>>>>> - MailScanner is configured (by ruleset) to bypass scanning for thoses
>>  >>>>>> messages, but they are still entering the mailscanner logic (postix ->
>>  >>>>>> hold queue -> mailscanner (no scan) -> active queue).
>>  >>>>>>
>>  >>>>>>
>>  >>>>> What does the ruleset look like? I'm sure it doesn't matter, but ...
>>  >>>>> just out of curiosity:-)...
>>  >>>>>
>>  >>>>>
>>  >>>>>
>>  >>>>>> - Mailwatch is running on this server, and for each duplicates we see
>>  >>>>>> entries with null size body (2, 3, 4, sometimes 5) then at last a final
>>  >>>>>> entry with the full body. Note that the recipient see the full body on
>>  >>>>>> every duplicate.
>>  >>>>>>
>>  >>>>>> It looks like a locking problem, because all duplicates are with the
>>  >>>>>> same postfix queue ID and different entropy part (ID.xxxx, ID.yyyy,
>>  >>>>>> ID.zzzz, etc). Can it be possible that a mailscanner child "fail" to
>>  >>>>>> lock some queue file when message is marked not to be scanned by
>>  >>>>>> mailscanner ?
>>  >>>>>>
>>  >>>>>>
>>  >>>>> Yes, this seems plausible... Could you provide some log examples? Just
>>  >>>>> to see that it really is separate children reading the same queue
>>  >>>>> file...
>>  >>>>>
>>  >>>>>
>>  >>>>>
>>  >>>>>
>>  >>>>>> I will not be very helpfull to debug perl code, but i can provide any
>>  >>>>>> needed logs to help finding the origin of the problem.
>>  >>>>>>
>>  >>>>>>
>>  >>>>> I'll see what I can do, but... I think this isn't "my" code snippets,
>>  >>>>> but a thing that might have been present for a while... And I have a
>>  >>>>> serious lack of time to spend on this ATM (worse than last time,
>>  >>>>> before Xmas)... So no promises:-).
>>  >>>>>
>>  >>>>>
>>  >>>>>
>>  >>>>>> This is really a serious problem in this particular installation. But i
>>  >>>>>> must say that we have dozens of other servers that are running
>>  >>>>>> mailscanner/postfix, and we are very happy about thems :)
>>  >>>>>>
>>  >>>>>>
>>  >>>>> Does it help if you DO scan with MS, but skip things at the next
>>  >>>>> level, for example:
>>  >>>>> Scan Messages = yes
>>  >>>>> Use SpamAssassin = no
>>  >>>>> Dangerous Content Scanning = no
>>  >>>>> ... and possibly a few more (do them with a ruleset, of course:-)?
>>  >>>>>
>>  >>>>>
>>  >>>>>
>>  >>>> BTW, do you have any milters enabled in Postfix? What version of Postfix?
>>  >>>>
>>  >>>> Cheers
>>  >>>>
>>  >>>>
>>  >>> I think we need Jules on this one, not only feeble lil' me:-).
>>  >>> AFAICS, the locking/unlocking is handled _exactly_ the same regardless
>>  >>> of the scanmail setting... But then, this is a rather complex bit of
>>  >>> code, where the "execution path" isn't always as straightforward as it
>>  >>> seems... Jules, could you spare a moment or two? Just to look at what
>>  >>> could possibly be wrong with the message->scanmail = 0 scenario?
>>  >>>
>>  >>>
>>  >>>
>>  >> Can you *briefly* explain what the problem is, what the symptoms are and
>>  >> where you think the problem might lie? This is a very long thread.... :-)
>>  >>
>>  >> Jules
>>  >>
>>  >>
>>  > In short:
>>  > When using Postfix and setting Scan Messages = no (with a rulset, for
>>  > some....), duplicates are "generated" by several MailScanner children
>>  > picking up and delivering the same message.
>>
>> Is the whole message being delivered multiple times, or are the
>>  duplicates truncated at all?
> AFAIU, the messages are delivered seemingly whole (but seem slightly
> truncated in MW, at least according to Cedric).
> 


That's true, here messages are delivered as complete, but in mailwatch
we can't see the body (only headers) and the size is marked as null
(only a "b" in the size column).


>>  P.S. Sorry for top-posting on this thread a few minutes ago :-(
> It's your list, you are forgiven:-). And with that type of info...
> We'd pretty much forgive anything (even bad language... not that it'd
> ever happen with you:-):-).
> 
> I'll try find some time, but I think that the ones who have reported
> this problem (Cedric in particular) are the ones that need test
> this... So, Cedric... Pretty please try this beta on your production
> host and then report back... So that the fix can be included in the
> next stable release!
> 
> Just out of curiosity (I'll at least DL and read the beta...)... Where
> should I look for the fix?
> 
> Cheers


As is just replied to Jules, i'll try to test it today. But i won't have
much time to play with mailscanner tonight :)
Anyway, if i can't today, i will surely try tommorow.

-- 
AmsterGroup
145 rue Barastraat B -1070 Brussels
T +32(0)2 556 28 11  
F +32(0)2 556 28 10
www.amstergroup.com