Mailscanner generated duplicate message.

[Glenn Steen]

On 07/01/2008, Mark Sapiro <mark at msapiro.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Mark Sapiro wrote:
>
> | Alex Broens wrote:
> |>
> |>
> |>
> |>
> |> On 12/24/2007 3:23 PM, Mark Sapiro wrote:
> |>> Alex Broens wrote:
> |>>> probably totally irrelevant yet got a hunch...
> |>>>
> |>>>
> |>>> what are your settings in MailScanner.conf for
> |>>>
> |>>> Queue Scan Interval
> |>>>
> |>>> Max Unscanned Messages Per Scan
> |>>>
> |>>> Max Unsafe Messages Per Scan
> |>>
> |>>
> |>> Queue Scan Interval = 6
> |>>
> |>> Max Unscanned Messages Per Scan = 30
> |>>
> |>> Max Unsafe Messages Per Scan = 30
> |>>
> |>>
> |>>> Could it be you're seeing a race condition between scanning threads?
> |>>
> |>>
> |>> This is exactly what the problem seems to be, but I don't know what to
> |>> do to prevent it or what I could have done or omitted to cause it.
> |>>
> |>> I suppose I could set
> |>>
> |>> Max Children = 1
> |>>
> |>> but that seems extreme, and it seems if it were necessary, more than
> |>> just me would be seeing this problem.
> |>
> |> Single CPU:
> |>
> |> Max Children = 5
> |
> |
> | This is what I currently have.
> |
> |
> |> Dual:
> |>
> |> Max Children = 8
> |>
> |> (keep the box relaxed till you get the stuff to process)
> |>
> |> Pls try:
> |>
> |> Queue Scan Interval = 15
> |>
> |> Max Unscanned Messages Per Scan = 5
> |> Max Unsafe Messages Per Scan = 5
> |
> |
> | I will try these. Note that I will be offline for the next week, so I
> | won't be able to report much until after the new year.
>
>
> I tried the above settings from Dec 24 through Jan 3. During that
> period, I see three more duplication incidents in my maillog. This is a
> lower rate than previous, but there are still dups.
>
> As of Jan 3, I have reverted the above settings to their default values,
> and set
>
> Max Children = 2
>
> So far, there have been no more dups, but I think it's too early to
> tell. Given the load on my system, Max Children = 2 seems fine. If I do
> see more dups, I may even try setting it to 1 which I'm certain will
> eliminate the problem.
>
>
> |> You may need to tweek "Queue Scan Interval" to your box's perfomance
> |>
> |> my rule of thumb:
> |>
> |> Queue Scan Interval = thread_count + 3
> |>
> |> keep us posted
> |
> |
> | OK.
> |
> | Note that logs indicate that this problem has only occurred on mail
> | which is not actually scanned because of a 'no' in scan.messages.rules.
> | I don't know why this would matter, but it may be significant.
> |
> | All but one of the occurrences were on outgoing mail from localhost. The
> | other one was an incoming message to postmaster. Logs indicate 4 copies
> | of this one were delivered and I undoubtedly saw all four but just
> | deleted them thinking they were multiple spams
> |
> | The nature of the server is that outgoing mail is virtually all Mailman
> | list posts or forwards of mail, all of which was scanned on the way in.
> | I would just as soon not have Postfix hold mail from localhost at all,
> | but I haven't figured out how to do that.
>
>
Hello Mark,

I'm mostly just posting this to tell you that I haven't forgotten
about you and your problem.

I've (finally) hat the time to look a bit at the code, and ... there
simply isn't anything that jumps out as a possible error that would
have this type of behavior.
Perhaps needless to say (so I'll go ahead and say it anyway:-),
detailed study of my logs and quarantine have not turned up any
duplicates of the kind you have. All mine are the ones the extra
"entropy" after the dot (<queue id>.<entropy>:-) is designed to
overcome.
Then again, I seem to recall you mentioning that this only happens for
locally submitted mail... So then the question becomes "what
differentiates them from the normal mail?", and perhaps "How can we
change things so that the locally submitted mails are equivalent with
non-local mail?" as a stopgap solution... While looking at how those
submissiions would be ... mishandled by MS...
In short: I'm in sore need of a repeatable testcase here....:-).

Cheers
-- 
-- Glenn
email: glenn < dot > steen < at > gmail < dot > com
work: glenn < dot > steen < at > ap1 < dot > se