feature request: compress attachments

[Glenn Steen]

On 23/05/07, Julian Field <MailScanner at ecs.soton.ac.uk> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> Glenn Steen wrote:
> > Talking about betas, do you plan on putting one out with some form of
> > "p record handling" for Postfix any time soon? I've been really bogged
> > down lately, so haven't worked on the "PF2.4 version" ... yet:).
> > If you need a new set of "PF2.3" patches, I think I can find them in
> > the archives for you... Or wherever I put them:-). If you want to look
> > at what I have for 2.4/full body edits, I'll liekly need do a new
> > set... Provided I did get around to doing the ugly "spin through
> > security check" in ReadQf... RALM striking again (I think you are
> > familiar with the consept... Random Access Lossy Memory:), mostly
> > since I've been very busy with other stuff.
> How do I make PF generate these "p records"?
By using a milter. The whole construct is to facilitate milters
editing/adding headers (PF2.3) and body content (PF2.4).

> As they can only be generated locally, they shouldn't contain malicious
> content so I probably don't need to merge them into the main text, I
> just need to handle them and be able to put them back in at the end.
> Does that sound likely?
Nope, that will not work. Since they contain fix file offsets (just
like a trusty ol' GOTO statement:) you would have to recalculate them,
and this (although doable) is really _not_ as trivial as one might
think.

> What can p records actually do?
As said they are to facilitate milters editing/adding things (there's
a d record for marking records as deleted too... More on this below).

Now what is done is that when you configure postfix to use a milter,
the queue files will be "prepared" by the insertion of a p record in
each of the three distinct sections of the queue file. These (fixed
size) records of type "p" will have a record value of "0", which
basically means "ignore me".
If a milter adds a header, this will be written to the end of the
queue file, _after_ the "E" record, and the appropriate "p" record
will be updated with the file offset of the beginning of that header.
after the header(s) there will be another "p" record with the offset
just "behind" the "forward pointing" "p" record.
If another header is added (say another milter wants to do that) the
second "p" record, the one pointing "backward", will be updated with a
"forward" offset, to the end of the file, and the header will be added
there along with a new "backward" "p" record pointing to the original
jump-off point.
So you can have several "forward-pointing" "p" records, in a chain,
but only one "backward", making it conceptually a curcular linked list
kind of thing.
In PF2.4, you have this implemented for full body editing (by milters) too.

This means that
a) there is no real sequentiality in the file any more. To read it all
you need to seek to the offsets, read the records and seek on/back.
b) headers and body segments can be interfoliated
c) one simply cannot preserve the records _as is_, and still keep
doing the additions/editing "the MailScanner way" too.
d) Since these edits/additions are done before MailScanner, we cannot
just ignore them. If (for example) someone is to use a greylisting
milter (as Nerijus Baliunas does;-), the queue files generated by
MailScanner will be corrupted.
e) You can not rely on the "E" record to determine that a queue file
is fully written, when full body edits are possible (PF2.4). For the
headers, ReadQf will detect (in a way) that something is up, with my
patches, which covers PF2.3... For the body edits of 2.4, we need do a
little more.

To handle these GOTO-like thingies we have to do one of three possible things:
1) Use them to do the edits/additions that MailScanner needs done.
This would need some changes to PFDiskStore.pm (the Body class) and
probably something for Postfix.pm (ReadQf) to correctly preserve
them... And some major changes (I think) to actually use them for
manipulating the queue files... I haven't looked much at this
possibility, since last we went over this, you liked option #3 (below)
better.
2) Keep doing the MailScanner edits/additions the way we always have,
and try preserve/update the preserved "p" records offsets. This would
probably work too, but would be a PITA logic-wise...
3) Acknowledging the fact that the queue file we actually pass _out_
from ailScanner isn't the same that is passed _into_ MailScanner, that
it in fact is a completely new file (as per Mr V's stipulations some
time ago:-), generated from the message object created from the
incoming queue file, one could as well just remove the "p" records and
add the records pointed to by the "p" record chain into the message
object. So when we generate the new queue file, it'll be a normal
queue file _without_ "p" or "d" records (at least no "d" records in
the body;-).

I've aimed (rather successfully, if I might say so:-) at implementing
#3. My patches (that certainly are too well commented, clutsy here and
there and not final in any way:-) implement this "whizzing around" and
leaves the normal record handling functions to actually read out the
records and put them where they belong.
Since the Body class will now need handle deletions and jumps
correctly (as is now, the deleted chucks will magically reappear when
you transform the "d" records to "N" records, and there will be a
spurious line with a number from where "p"->"N"....), I handle that...
with minimal error checking... I simply have to trust that ReadQf has
validated the _whole_ file... So to make these patches safe for PF2.4,
one would have to add a loop to (as quicjly as possible) read through
the body in search of the "p" record(s) and whatever they may point
too.
This last bit is what is lacking, from the current patches, and I
(unfortunately) have had no time at all to spend on
implementing/testing that part. I started on that, but then two things
happened... You nearly fell off the planet, and my most esteemed
colleague (my Oracle guru:) decided to take early retirement ...
Dumping all his work in my lap. So the "2.4 patch" was never
finished/tested:-(. It should be fairly easy to do though, just one
more loop instead of the "jump to after body" you do now.

> I would be very interested in a set of PF2.3 patches.
I'll find you a link...:
http://article.gmane.org/gmane.mail.virus.mailscanner/51465

Please be gentle, I know it's far from perfect. But it does work.

> What is the difference between 2.3 and 2.4?
See above;)

> Jules
>

Cheers
-- 
-- Glenn
email: glenn < dot > steen < at > gmail < dot > com
work: glenn < dot > steen < at > ap1 < dot > se