duplicate subject lines in headers (again)

Warwick Brown Warwick.x.Brown at serco.com
Sat Nov 12 23:44:56 UTC 2016


> However, you shouldn't have to rely on Yahoo to complain about the
> message. If MailScanner is duplicating the Subject: header, this almost
> certainly doesn't depend on the mail being sent to Yahoo nor on Yahoo
> ultimately bouncing it.

Its not just Yahoo, however Yahoo is the only service which makes a big issue out of it citing RFC compliance

>
> I would expect it to occur with all mail that has trailing spaces in the
> Subject:, even a message you just send to yourself.

Yes..that is repeatable

> From the above, it seems that you have both
>
> Use Watermarking = Yes
> Place New Headers At Top Of Message = Yes

Yes - correct

> in your MailScanner config, but even with those settings and testing
> with both messages that do and do not tag the Subject:, I still can't
> duplicate this.

Have you an Exim setup to test? You've got me thinking...i have a personal VM I can burn which I am willing to set-up and provide access for you if you like? It won't be an *exact* match of what I am working on, but at least will  a likeness

> But, your MTA is Exim, and other info (see below) seems to say that this
> may only be an issue when Exim is the MTA.

Most likely, but our poison of choice is Exim because our mail environment is/was horrendously complex and Exim was the only MTA which allowed us the flexibility we required at the time when it was chosen (I would have much preferred sendmail ;-) and was not part of that selection process)

> It seems clear that the second, "stripped" Subject is added by
> MailScanner between adding its normal reporting headers and the
> watermark header, but again, I can't duplicate this.
>
> To test further, I'd like to know everything in your MailScanner config
> that's different from default. Hopefully, you have all your changes in
> /etc/MailScanner/conf.d/* and you can just send me or post those, but if
> not, send me /etc/MailScanner/MailScanner.conf. Also, if you can test
> with a simple message to yourself and find one that reliably triggers
> the problem, I'd like to see that, both as it is sent and as it is
> received after MailScanner duplicates the Subject:.
>
> Also, I finally looked for and found the thread at
> <http://lists.mailscanner.info/pipermail/mailscanner/2014-
> December/101817.html>,
> and while it does contain some additional info, I'm still unable to
> duplicate the issue.

Hence why my subject line includes "again", it seems to be a regressive bug/feature at least in my configuration that uses Exim.

Here is my Mailscanner.conf (I have it installed into /opt/MailScanner and have redacted a few things such as the org name and watermark salt):

# egrep -v '^[     ]*$|^[  ]*\#' /opt/MailScanner/etc/MailScanner.conf
%org-name% = MyCustomOrgName
%org-long-name% = My Custom Organisation Name
%web-site% = www.myorgdomain.com
%etc-dir% = /opt/MailScanner/etc
%report-dir% = /opt/MailScanner/etc/reports/en
%rules-dir% = /opt/MailScanner/etc/rules
%mcp-dir% = /opt/MailScanner/etc/mcp
Max Children = 12
Run As User = exim
Run As Group = exim
Queue Scan Interval = 6
Incoming Queue Dir = /var/spool/exim.in/input
Outgoing Queue Dir = /var/spool/exim.out/input
Incoming Work Dir = /var/spool/MailScanner/incoming
Quarantine Dir = /var/spool/MailScanner/quarantine
PID file = /opt/MailScanner/var/MailScanner.pid
Restart Every = 7200
MTA = exim
Sendmail = /usr/sbin/exim
Sendmail2 = /usr/bin/exim -C /etc/exim/exim_out.conf
Incoming Work User =
Incoming Work Group = clamscan
Incoming Work Permissions = 0640
Quarantine User =
Quarantine Group =
Quarantine Permissions = 0600
Max Unscanned Bytes Per Scan = 100m
Max Unsafe Bytes Per Scan = 50m
Max Unscanned Messages Per Scan = 50
Max Unsafe Messages Per Scan = 50
Max Normal Queue Size = 10000
Scan Messages = yes
Reject Message = no
Maximum Processing Attempts = 6
Processing Attempts Database = /var/spool/MailScanner/incoming/Processing.db
Maximum Attachments Per Message = 200
Expand TNEF = yes
Use TNEF Contents = no
Deliver Unparsable TNEF = no
TNEF Expander = /usr/bin/tnef --maxsize=100000000
TNEF Timeout = 120
File Command = /usr/bin/file
File Timeout = 60
Gunzip Command = /bin/gunzip
Gunzip Timeout = 60
Unrar Command = /usr/local/bin/unrar
Unrar Timeout = 60
Find UU-Encoded Files = yes
Maximum Message Size = %rules-dir%/max.message.size.rules
Maximum Attachment Size = -1
Minimum Attachment Size = -1
Maximum Archive Depth = 8
Find Archives By Content = yes
Unpack Microsoft Documents = yes
Zip Attachments = no
Attachments Zip Filename = MessageAttachments.zip
Attachments Min Total Size To Zip = 100k
Attachment Extensions Not To Zip = .zip .rar .gz .tgz .jpg .jpeg .mpg .mpe .mpeg .mp3 .rpm .htm .html .eml .gz .bz2 .xz
Add Text Of Doc = no
Antiword = /usr/bin/antiword -f
Antiword Timeout = 50
Unzip Maximum Files Per Archive = 0
Unzip Maximum File Size = 50k
Unzip Filenames = *.txt *.ini *.log *.csv
Unzip MimeType = text/plain
Virus Scanning = yes
Virus Scanners = clamd
Virus Scanner Timeout = 600
Deliver Disinfected Files = no
Silent Viruses = HTML-IFrame All-Viruses
Still Deliver Silent Viruses = no
Non-Forging Viruses = Joke/ OF97/ WM97/ W97M/ eicar
Spam-Virus Header = X-%org-name%-MailScanner-SpamVirus-Report:
Virus Names Which Are Spam = Sane*UNOFFICIAL HTML/* *Phish*
Block Encrypted Messages = no
Block Unencrypted Messages = no
Allow Password-Protected Archives = yes
Check Filenames In Password-Protected Archives = yes
Allowed Sophos Error Messages =
Sophos IDE Dir = /opt/sophos-av/lib/sav
Sophos Lib Dir = /opt/sophos-av/lib
Monitors For Sophos Updates = /opt/sophos-av/lib/sav/*.ide
Monitors for ClamAV Updates = /usr/local/share/clamav/*.cld /usr/local/share/clamav/*.cvd /var/lib/clamav/*.inc/* /var/lib/clamav/*.?db /var/lib/clamav/*.cvd
ClamAVmodule Maximum Recursion Level = 8
ClamAVmodule Maximum Files = 1000
ClamAVmodule Maximum File Size = 100000000 # (100 Mbytes)
ClamAVmodule Maximum Compression Ratio = 250
Clamd Port = 3310
Clamd Lock File = # /var/lock/subsys/clamd
Clamd Use Threads = yes
ClamAV Full Message Scan = yes
Fpscand Port = 10200
Dangerous Content Scanning = yes
Allow Partial Messages = no
Allow External Message Bodies = no
Find Phishing Fraud = yes
Also Find Numeric Phishing = yes
Use Stricter Phishing Net = yes
Highlight Phishing Fraud = no
Phishing Safe Sites File = %etc-dir%/phishing.safe.sites.conf %etc-dir%/phishing.safe.sites.custom
Phishing Bad Sites File = %etc-dir%/phishing.bad.sites.conf
Country Sub-Domains List = %etc-dir%/country.domains.conf
Allow IFrame Tags = %rules-dir%/disarm.rules
Allow Form Tags = %rules-dir%/disarm.rules
Allow Script Tags = %rules-dir%/disarm.rules
Allow WebBugs = %rules-dir%/disarm.rules
Ignored Web Bug Filenames = spacer pixel.gif pixel.png gap shim
Known Web Bug Servers = msgtag.com
Web Bug Replacement = http://cdn.mailscanner.info/1x1spacer.gif
Allow Object Codebase Tags = disarm
Convert Dangerous HTML To Text = no
Convert HTML To Text = no
Archives Are = zip rar ole uu tnef
Allow Filenames =
Deny Filenames =
Filename Rules = %etc-dir%/filename.rules.conf
Allow Filetypes =
Allow File MIME Types =
Deny Filetypes =
Deny File MIME Types =
Filetype Rules = %etc-dir%/filetype.rules.conf
Archives: Allow Filenames =
Archives: Deny Filenames =
Archives: Filename Rules = %etc-dir%/archives.filename.rules.conf
Archives: Allow Filetypes =
Archives: Allow File MIME Types =
Archives: Deny Filetypes =
Archives: Deny File MIME Types =
Archives: Filetype Rules = %etc-dir%/archives.filetype.rules.conf
Default Rename Pattern = __FILENAME__.disarmed
Quarantine Infections = yes
Quarantine Silent Viruses = no
Quarantine Modified Body = no
Quarantine Whole Message = yes
Quarantine Whole Messages As Queue Files = yes
Keep Spam And MCP Archive Clean = no
Language Strings = %report-dir%/languages.conf
Rejection Report = %report-dir%/rejection.report.txt
Deleted Bad Content Message Report  = %report-dir%/deleted.content.message.txt
Deleted Bad Filename Message Report = %report-dir%/deleted.filename.message.txt
Deleted Virus Message Report        = %report-dir%/deleted.virus.message.txt
Deleted Size Message Report         = %report-dir%/deleted.size.message.txt
Stored Bad Content Message Report  = %report-dir%/stored.content.message.txt
Stored Bad Filename Message Report = %report-dir%/stored.filename.message.txt
Stored Virus Message Report        = %report-dir%/stored.virus.message.txt
Stored Size Message Report         = %report-dir%/stored.size.message.txt
Disinfected Report = %report-dir%/disinfected.report.txt
Inline HTML Signature = %report-dir%/inline.sig.html
Inline Text Signature = %report-dir%/inline.sig.txt
Signature Image Filename = %report-dir%/sig.jpg
Signature Image <img> Filename = signature.jpg
Inline HTML Warning = %report-dir%/inline.warning.html
Inline Text Warning = %report-dir%/inline.warning.txt
Sender Content Report      = %report-dir%/sender.content.report.txt
Sender Error Report        = %report-dir%/sender.error.report.txt
Sender Bad Filename Report = %report-dir%/sender.filename.report.txt
Sender Virus Report        = %report-dir%/sender.virus.report.txt
Sender Size Report         = %report-dir%/sender.size.report.txt
Hide Incoming Work Dir = yes
Include Scanner Name In Reports = yes
Mail Header = X-%org-name%-MailScanner:
Spam Header = X-%org-name%-MailScanner-SpamCheck:
Spam Score Header = X-%org-name%-MailScanner-SpamScore:
Information Header = X-%org-name%-MailScanner-Information:
Add Envelope From Header = yes
Add Envelope To Header = no
Envelope From Header = X-%org-name%-MailScanner-From:
Envelope To Header = X-%org-name%-MailScanner-To:
ID Header = X-%org-name%-MailScanner-ID:
IP Protocol Version Header = # X-%org-name%-MailScanner-IP-Protocol:
Spam Score Character = s
SpamScore Number Instead Of Stars = yes
Minimum Stars If On Spam List = 0
Clean Header Value       = Found to be clean
Infected Header Value    = Found to be infected
Disinfected Header Value = Disinfected
Information Header Value = Please report any suspicious emails to phishing at myorgdomain.com
Detailed Spam Report = yes
Include Scores In SpamAssassin Report = yes
Always Include SpamAssassin Report = yes
Multiple Headers = add
Place New Headers At Top Of Message = yes
Hostname = %org-name% Core MTA $HOSTNAME
Sign Messages Already Processed = no
Sign Clean Messages = no
Attach Image To Signature = no
Attach Image To HTML Message Only = yes
Allow Multiple HTML Signatures = no
Dont Sign HTML If Headers Exist = # In-Reply-To: References:
Mark Infected Messages = yes
Mark Unscanned Messages = yes
Unscanned Header Value = Not scanned
Remove These Headers = X-Mozilla-Status: X-Mozilla-Status2:
Deliver Cleaned Messages = no
Notify Senders = no
Notify Senders Of Viruses = no
Notify Senders Of Blocked Filenames Or Filetypes = no
Notify Senders Of Blocked Size Attachments = no
Notify Senders Of Other Blocked Content = no
Never Notify Senders Of Precedence = list bulk
Scanned Modify Subject = no # end
Scanned Subject Text = {Scanned}
Virus Modify Subject = start
Virus Subject Text = {Virus?}
Filename Modify Subject = start
Filename Subject Text = {Filename?}
Content Modify Subject = start
Content Subject Text = {Dangerous Content?}
Size Modify Subject = start
Size Subject Text = {Size}
Disarmed Modify Subject = start
Disarmed Subject Text = {Disarmed}
Phishing Modify Subject = start
Phishing Subject Text = {Fraud?}
Spam Modify Subject = start
Spam Subject Text = {Spam?}
High Scoring Spam Modify Subject = start
High Scoring Spam Subject Text = {Spam?}
Warning Is Attachment = yes
Attachment Warning Filename = %org-name%-Attachment-Warning.txt
Attachment Encoding Charset = ISO-8859-1
Archive Mail =
Missing Mail Archive Is = directory
Send Notices = yes
Notices Include Full Headers = yes
Hide Incoming Work Dir in Notices = no
Notice Signature = -- \ Core MTA Service\nwww.myorgdomain.com\n
Notices From = MailScanner
Notices To = phishing at myorg.com
Local Postmaster = phishing at myorg.com
Spam List Definitions = %etc-dir%/spam.lists.conf
Virus Scanner Definitions = %etc-dir%/virus.scanners.conf
Spam Checks = yes
Spam List = # spamhaus-ZEN # You can un-comment this to enable them
Spam Domain List =
Spam Lists To Be Spam = 1
Spam Lists To Reach High Score = 3
Spam List Timeout = 10
Max Spam List Timeouts = 7
Spam List Timeouts History = 10
Is Definitely Not Spam = %rules-dir%/spam.whitelist.rules
Is Definitely Spam = no
Definite Spam Is High Scoring = no
Ignore Spam Whitelist If Recipients Exceed = 20
Max Spam Check Size = 2097152k
Use Watermarking = yes
Add Watermark = yes
Check Watermarks With No Sender = yes
Treat Invalid Watermarks With No Sender as Spam = nothing
Check Watermarks To Skip Spam Checks = yes
Watermark Secret = redacted ;-)
Watermark Lifetime = 604800
Watermark Header = X-%org-name%-MailScanner-Watermark:
Use SpamAssassin = yes
Max SpamAssassin Size = 2097152k
Required SpamAssassin Score = 6
High SpamAssassin Score = 10
SpamAssassin Auto Whitelist = yes
SpamAssassin Timeout = 300
Max SpamAssassin Timeouts = 10
SpamAssassin Timeouts History = 30
Check SpamAssassin If On Spam List = yes
Include Binary Attachments In SpamAssassin = no
Spam Score = yes
Cache SpamAssassin Results = yes
SpamAssassin Cache Database File = /var/spool/MailScanner/incoming/SpamAssassin.cache.db
Rebuild Bayes Every = 0
Wait During Bayes Rebuild = no
Use Custom Spam Scanner = no
Max Custom Spam Scanner Size = 20k
Custom Spam Scanner Timeout = 20
Max Custom Spam Scanner Timeouts = 10
Custom Spam Scanner Timeout History = 20
Spam Actions = deliver header "X-Spam-Status: Yes"
High Scoring Spam Actions = store
Non Spam Actions = deliver header "X-Spam-Status: No"
SpamAssassin Rule Actions =
Sender Spam Report         = %report-dir%/sender.spam.report.txt
Sender Spam List Report    = %report-dir%/sender.spam.rbl.report.txt
Sender SpamAssassin Report = %report-dir%/sender.spam.sa.report.txt
Inline Spam Warning = %report-dir%/inline.spam.warning.txt
Recipient Spam Report = %report-dir%/recipient.spam.report.txt
Enable Spam Bounce = %rules-dir%/bounce.rules
Bounce Spam As Attachment = no
Syslog Facility = mail
Log Speed = yes
Log Spam = yes
Log Non Spam = yes
Log Delivery And Non-Delivery = yes
Log Permitted Filenames = yes
Log Permitted Filetypes = yes
Log Permitted File MIME Types = yes
Log Silent Viruses = yes
Log Dangerous HTML Tags = yes
Log SpamAssassin Rule Actions = yes
SpamAssassin Temporary Dir = /var/spool/MailScanner/incoming/SpamAssassin-Temp
SpamAssassin User State Dir =
SpamAssassin Install Prefix =
SpamAssassin Site Rules Dir = /etc/mail/spamassassin
SpamAssassin Local Rules Dir =
SpamAssassin Local State Dir = # /var/lib/spamassassin
SpamAssassin Default Rules Dir =
DB DSN =
DB Username = sauser
DB Password = pickyourown!
SQL Serial Number =
SQL Quick Peek =
SQL Config =
SQL Ruleset =
SQL SpamAssassin Config =
SQL Debug = no
MCP Checks = no
First Check = spam
MCP Required SpamAssassin Score = 1
MCP High SpamAssassin Score = 10
MCP Error Score = 1
MCP Header = X-%org-name%-MailScanner-MCPCheck:
Non MCP Actions = deliver
MCP Actions = deliver
High Scoring MCP Actions = deliver
Bounce MCP As Attachment = no
MCP Modify Subject = start
MCP Subject Text = {MCP?}
High Scoring MCP Modify Subject = start
High Scoring MCP Subject Text = {MCP?}
Is Definitely MCP = no
Is Definitely Not MCP = no
Definite MCP Is High Scoring = no
Always Include MCP Report = no
Detailed MCP Report = yes
Include Scores In MCP Report = no
Log MCP = no
MCP Max SpamAssassin Timeouts = 20
MCP Max SpamAssassin Size = 100k
MCP SpamAssassin Timeout = 10
MCP SpamAssassin Prefs File = %mcp-dir%/mcp.spam.assassin.prefs.conf
MCP SpamAssassin User State Dir =
MCP SpamAssassin Local Rules Dir = %mcp-dir%
MCP SpamAssassin Default Rules Dir = %mcp-dir%
MCP SpamAssassin Install Prefix = %mcp-dir%
Recipient MCP Report = %report-dir%/recipient.mcp.report.txt
Sender MCP Report = %report-dir%/sender.mcp.report.txt
Use Default Rules With Multiple Recipients = no
Read IP Address From Received Header = no
Spam Score Number Format = %d
MailScanner Version Number = 4.85.2
SpamAssassin Cache Timings = 1800,300,10800,172800,600
Debug = no
Debug SpamAssassin = no
Run In Foreground = no
Always Looked Up Last = no
Always Looked Up Last After Batch = no
Deliver In Background = yes
Delivery Method = queue
Split Exim Spool = no
Lockfile Dir = /var/spool/MailScanner/incoming/Locks
Custom Functions Dir = /opt/MailScanner/lib/MailScanner/CustomFunctions
Lock Type =
Syslog Socket Type =
Automatic Syntax Check = yes
Minimum Code Status = supported
include /opt/MailScanner/etc/conf.d/*

The contents of /opt/MailScanner/etc/conf.d/* is empty.

My rules are as follows:-

# egrep -v '^[     ]*$|^[  ]*\#' /opt/MailScanner/etc/rules/spam.whitelist.rules /opt/MailScanner/etc/rules/disarm.rules /opt/MailScanner/etc/rules/bounce.rules
/opt/MailScanner/etc/rules/spam.whitelist.rules:From:           noreply at redacted.com   yes
/opt/MailScanner/etc/rules/spam.whitelist.rules:From:           /^cmailcampaignname-[0-9a-zA-Z]+ at cmail[0-9]*.com$/        yes
/opt/MailScanner/etc/rules/spam.whitelist.rules:From:           10.1.2.3     yes
<other whitelist rules by IP removed for clarity>
/opt/MailScanner/etc/rules/spam.whitelist.rules:FromOrTo:       default         no

/opt/MailScanner/etc/rules/disarm.rules:From:           cmailcampaignname-*@cmail*\.com   yes
/opt/MailScanner/etc/rules/disarm.rules:FromOrTo:       default                                 yes

/opt/MailScanner/etc/rules/bounce.rules:FromOrTo:       default                 no


I have met all the perl dependencies (except MySQL, which I don't think is material to this issue) and the lint report is as follows:

# ./MailScanner --lint
Trying to setlogsock(unix)

Reading configuration file /opt/MailScanner/etc/MailScanner.conf
Reading configuration file /opt/MailScanner/etc/conf.d/README
Read 870 hostnames from the phishing whitelist
Read 5807 hostnames from the phishing blacklists

Checking version numbers...
Version number in MailScanner.conf (4.85.2) is correct.

Your envelope_sender_header in spam.assassin.prefs.conf is correct.
MailScanner setting GID to  (93)
MailScanner setting UID to  (93)

Checking for SpamAssassin errors (if you use it)...
Using SpamAssassin results cache
Connected to SpamAssassin cache database
plugin: eval failed: install_driver(mysql) failed: Can't locate DBD/mysql.pm in @INC (you may need to install the DBD::mysql module) (@INC contains: lib . ./MailScanner /opt/MailScanner/lib /opt/MailScanner/lib/perl5/site_perl/5.22.1/x86_64-linux-thread-multi /opt/MailScanner/lib/perl5/site_perl/5.22.1 /opt/MailScanner/lib/perl5/5.22.1/x86_64-linux-thread-multi /opt/MailScanner/lib/perl5/5.22.1) at (eval 1199) line 3.
Perhaps the DBD::mysql perl module hasn't been fully installed,
or perhaps the capitalisation of 'mysql' isn't right.
Available drivers: DBM, ExampleP, File, Gofer, Proxy, SQLite, Sponge.
 at /opt/MailScanner/lib/perl5/site_perl/5.22.1/Mail/SpamAssassin/BayesStore/MySQL.pm line 654.
plugin: eval failed: install_driver(mysql) failed: Can't locate DBD/mysql.pm in @INC (you may need to install the DBD::mysql module) (@INC contains: lib . ./MailScanner /opt/MailScanner/lib /opt/MailScanner/lib/perl5/site_perl/5.22.1/x86_64-linux-thread-multi /opt/MailScanner/lib/perl5/site_perl/5.22.1 /opt/MailScanner/lib/perl5/5.22.1/x86_64-linux-thread-multi /opt/MailScanner/lib/perl5/5.22.1) at (eval 1209) line 3.
Perhaps the DBD::mysql perl module hasn't been fully installed,
or perhaps the capitalisation of 'mysql' isn't right.
Available drivers: DBM, ExampleP, File, Gofer, Proxy, SQLite, Sponge.
 at /opt/MailScanner/lib/perl5/site_perl/5.22.1/Mail/SpamAssassin/BayesStore/MySQL.pm line 654.
SpamAssassin reported no errors.
Connected to Processing Attempts Database
Created Processing Attempts Database successfully
There are 2 messages in the Processing Attempts Database
Using locktype = posix
MailScanner.conf says "Virus Scanners = clamd"
Found these virus scanners installed: clamavmodule, clamd
===========================================================================
Filename Checks: Windows/DOS Executable (1 eicar.com)
Filetype Checks: Allowing 1 eicar.com
Other Checks: Found 1 problems
Virus and Content Scanning: Starting
Clamd::INFECTED::Eicar-Test-Signature :: ./1/
Clamd::INFECTED:: Eicar-Test-Signature :: ./1/eicar.com
Virus Scanning: Clamd found 2 infections
Infected message 1 came from 10.1.1.1
Virus Scanning: Found 2 viruses
===========================================================================
Virus Scanner test reports:
Clamd said "eicar.com was infected: Eicar-Test-Signature"

If any of your virus scanners (clamavmodule,clamd)
are not listed there, you should check that they are installed correctly
and that MailScanner is finding them correctly via its virus.scanners.conf.

Let me know if you need any more info/configs...

Thanks again,

Warwick


More information about the MailScanner mailing list