duplicate subject lines in headers (again)

[Warwick Brown]

-----Original Message-----
From: MailScanner [mailto:mailscanner-bounces+warwick.x.brown=serco.com at lists.mailscanner.info] On Behalf Of Mark Sapiro
Sent: 03 November 2016 03:23
To: mailscanner at lists.mailscanner.info
Subject: Re: duplicate subject lines in headers (again)

On 11/02/2016 08:22 AM, Warwick Brown wrote:
> 
> I have that historic issue of sending email to yahoo due to duplicate
> subject lines.


Huh? Can you be more specific about what the issue is? If by
"historical" you mean something in the archives, a link to an archived
thread may help.


> I can see there is a work-around of setting "Multiple Headers" from
> "add" to "append", but I was wondering whether there was any appetite to
> get it fixed?


Again, what exactly is the issue. Does the Multiple Headers setting
actually affect Subject: headers.


> Also - what exactly are the implications for using "append" as opposed
> to "add" because there seems to be some anecdotal comments regarding
> breaking DKIM.


If an incoming message has MailScanner headers and those headers are
DKIM signed, changing them in any way or adding additional headers with
the same name will break the DKIM sig. On the other hand, if those
headers have a different org-name in the X-%org-name%-MailScanner
prefix, Multiple Headers = append or replace will modify those headers
and break the DKIM sig, but add will add new headers with a different
org-name and won't affect the existing DKIM sig.

But then, many other transformations will break the DKIM sig anyway -
things like tagging the Subject: or modifying the message body in any
way, e.g. by disarming web bugs.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan


-----

Hi Mark

Thanks for your response. I've hit upon an issue where Yahoo is being funny about RFC compliance and rejecting mail which has two subject lines in the headers.

I notice that one of the headers always has a trailing space, and the other does not. This causes a mismatch between the two subject lines and thus I believe that is where the confusion comes in.

I am using Exim 4.86, ClamAV 0.99, SpamAssassin 3.4.0 with MailScanner 4.85.2 on RHEL7.2.

I have "Multiple Headers" set to "add" as our future plans is to use DKIM, hence why I don't want to use the published work-around of setting "Multiple Headers" to "append" if I can help it. We have set %org-name% appropriately as to not clobber other people's headers.

I have also turned off all of the disarm features as we experienced issues with corporate apps which used the kind of code snippets that disarm acts upon, and changing all the corporate application emails was beyond my control.

I can see from the mailing list archives that this issue has cropped up many times on this list and others, and I can also see there are multiple entries for "$global::MS->{mta}->UniqHeader($this, 'Subject:');" in the Message.pm module, only a mail which passes the checks does not enact any of the functions which would call UniqHeader, so it doesn't get made unique.

One nuance of this bug I noticed is that when MailScanner parses the subject line - it strips off any whitespace at the end of the subject. I suspect this is where the problem may lie. I have considered using an Exim system filter to strip off the trailing space on receiving the mail, but as you rightly state, it would break DKIM, so again something I can't easily do.

So the issue remains - that where whitespace is present in the trailing part of the subject line - MailScanner is discarding that trailing whitespace and adding a second copy of the subject without whitespace, presumably because it believes (due to a difference in the length of a Subject line) that the subject has been changed as part of the scanning process. The only place I can see the length of the subject is considered is within the DeliverFiles function of Message.pm

I'm admittedly not a perl programmer and am reluctant to go hacking the code up in a (now) production environment. We are informing the affected users to check for trailing whitespace before they click send, but it'd be nice if we didn't have to do that.

Thanks again,

Warwick