Greylisting (WAS: Re: MailScanner ANNOUNCE: 4.57 released)

[Glenn Steen]

On 11/01/07, Dhawal Doshy <dhawal at netmagicsolutions.com> wrote:
> Glenn Steen wrote:
> > On 10/01/07, Nerijus Baliunas <nerijus at users.sourceforge.net> wrote:
> >> On Wed, 6 Dec 2006 09:14:15 +0100 Glenn Steen <glenn.steen at gmail.com>
> >> wrote:
> >>
> >> > I typed a longish reply to this one yesterday, which gmail then
> >> > promptly swallowed:-).
> >> > Oh well.
> >> >
> >> > The gist of it was "If I get time, I'll look at the code"... and
> >> > "interresting that postcat demangles it correctly (so that the postcat
> >> > of each queue file is ... well, as close to identical as possible)
> >> > ...".
> >>
> >> Could you please tell me where to look in the code? As now I have a real
> >> problem - if the last header is "Content-Transfer-Encoding: base64" it
> >> becomes
> >> Content-Transfer-Encoding: base64
> >>          190324
> >> and so is interpreted as Content-Transfer-Encoding: base64190324, thus
> >> rendering message unparseable...
> >>
> >> Regards,
> >> Nerijus
> > I've chatted a bit with Jules off-list, to bring him up to speed...
> > Lets see what he can do for us, shall we?
> >
> > You should start by familiarizing yourself with the Postfix.pm and
> > PFDiskStore.pm modules in /usr/lib/MailScanner/MailScanner/, if you
> > use the rom install at least (this directory is actually where all the
> > innards of MailScanner is at, so ... reading a bit here and there
> > should give you a few ideas of how it really is come together:-).
>
> Just to give you guys a headstart, there are some newer records that MS
> needs to address.
>
> Quoting Weitse >>
>
> If Mailscanner can speak the Milter protocol, great. Header
> modifications are already supported in Postfix 2.3. Body modifications
> with some luck in Postfix 2.4.
>
> Direct queue file modification no longer works with Mailscanner versions
> that don't recognize the new PTR records that Postfix needs for in-place
> editing.
>
> End Quote <<
>
> - dhawal

Well, after reading some code I at least know what is going on, and
why it doesn't work:-).

Wietse seems to have wanted a way to ... noninvasively... insert
additional data into the Postfix queue file without having to rewrite
the whole thing (as we more or less do in MailScanner), so that
milters could just "tag on" new recipients and headers.

To that effect he made it so that there might be two (or more? I'm not
really clear on this yet, but iut seems there will only be one p
record for additional recipients, but perhaps one per added header)
records of type "p" (for pointer) that has a "value part" of 0 by
default, meaning no "extra info" is to be inserted at that point. If
there is a non-zero value, this is an absolute offset to fseek to for
the actual inserted part, followed by another p record with an
absolute offset to jump back up into the file to the original "jumpoff
point" (after the original p record). The added records are tagged on
after the end record (and a null character denoting end of file?). As
one can guess, much of Wietses patches regarding milter support seem
to focus on detecting and handling missbehaving subsystems that create
p record message loops (wrong "jump-back offsets" leading to endless
loops ("here we GOTO again!":-)).

Currently this only seem to affect milters, and only milters that
would add recipients and/or message headers. Perhaps this will also be
used to ... modify... the body later on, when that type of milter
action is to be supported as well.

This p record thing doesn't sit that well wil how MailScanner
currently (I've been looking at 4.57.6, but will take a look at 4.58.4
too... I don't suppose there are any changes worth mentioning
though:-) decipher a postfix queue file since we both copy any
"unknown" records as is (p records are ... unknown .... to us:), add
in our own headers/recipients/whatnot as needed (making the absolute
offsets plain wrong), as well as (in all likelihood) not copying the
data "after the end record" to the right place (if at all).
Ergo Nerijus problem.

Now, I see at least three somewhat different solutions:
1) Simply enhance the ReadQf function so that it understands and
handles p records (meaning we will copy over the actual data added by
milters, not the p records themselves).
2) recalculate the offsets and make sure the precords ar correct, as
well as the "after end" data being where it is supposed to be.
3) Rewrite MailScanner to use the same p record method of adding
recipients and headers, which of course implies, in a way, that we
would reimplement the Postfix.pm file from scratch... more or less.

I'm hugely in favour of doing number 1), since this will be
a) simplest to implement
b) least amount of code touched
c) probably safest... We'll have to look at doing pretty much the same
sanity checks that Wietse does, but that shouldn't be a huge problem.
d) in keeping with the MailScanner philosophy:-). When we start
decomposing the PF queue files, they stop being queue files. When we
reassemble the queue file, it is a completely new queue file, with a
ne ID ... Why should we preserve a "bandaid like" thing that we don't
need;-).

If I get a little teensy bit of time, I'll make a stab at some code
for handling p records as suggested in #1.

Cheers
-- 
-- Glenn
email: glenn < dot > steen < at > gmail < dot > com
work: glenn < dot > steen < at > ap1 < dot > se