Certain Spamassassin rules do not seem to be firing all of the time

Tue Jun 18 19:40:38 IST 2013

Yeah I know it's very weird and I can't track it down.

Yesterday, I tried removing the  NET:DNS perl module (.65 is what MailScanner (and I believe SpamAssassin use by default) and compiling 0.72 in the hopes that it had something to do with that.  Nope, still happening today.  Fortunately it only seems to be letting a few Spam in overall.    It just happens when there is a black listed domain that is used in a URL that is sent by a non-blacklisted gateway where I get caught by this issue.

I am using Sendmail. Yes there is a .spamassassin directory in root, where the bayes db's are located and autowhitelist db's (I have autowhite list disabled for the moment)  The user_prefs file has no directives set in it, they are all #'ed out.

I don't specify a run as user in my MailScanner.conf, and according to ps all the MailScanner processes are running as root, and my spamassassin -test-mode I have run as root.

I turned on skip_rbl_checks 1 yesterday, since I detect RBL'ed hosts using MailScanner I figured it was kind of pointless to do it again with SpamAssassin..

I also tried tweaking rbl_timeout to 60 seconds instead of 30, what is was before.  Because I did find someone else reporting a similar issue to mine.. back in 2007 someone was reporting this same behavior that rules were not hitting when using Amavis with Spamassassin, but then when you ran them through Spamassassin they worked, and I believe it was the same types of rules I am not hitting on through MailScanner.  And the issue wound up being Net:DNS.

http://www.gossamer-threads.com/lists/spamassassin/users/102307?do=post_view_threaded#102307

If I can't figure this out, I might attempt a fresh install of Cent OS 6.4 and fresh install of MailScanner and SpamAssassin.  Just don't have the time for a full re-install right now.

Yeah I prefer doing the RBL's with MailScanner because it uses such little CPU to perform those tests.  I mark as Spam probably as high as 80% of my incoming mail just on the RBL checks through MailScanner.  Spamasssassin only has to process like 20% of my Spam mail then.  Since we pass everything through and assign the Microsoft SCL score based on if it failed Spam checks, the users can whitelist if someone winds up on an RBL or they want what most consider Spam.

BRIAN M. DUNCAN
Data Security Administrator
Katten Muchin Rosenman LLP
525 W. Monroe Street / Chicago, IL 60661-3693
p / (312) 577-8045 f / (312) 577-4490
brian.duncan at kattenlaw.com / www.kattenlaw.com

From: mailscanner-bounces at lists.mailscanner.info [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf Of Martin Hepworth
Sent: Tuesday, June 18, 2013 10:09 AM
To: MailScanner discussion
Subject: Re: Certain Spamassassin rules do not seem to be firing all of the time

really odd, seems to be suffering with network based rules, not just the URI ones but razor as well.
Personally I always put all the RBL checks into SA rather than letting MailScanner do it by itself. that way no 1 rbl can false postive and email and the RBL just add to the overall score.

What MTA are you running? and is there a .spamassassin directory in root's home dir?

--
Martin Hepworth, CISSP
Oxford, UK

On 15 June 2013 21:08, Duncan, Brian M. <brian.duncan at kattenlaw.com<mailto:brian.duncan at kattenlaw.com>> wrote:
Thanks for the recommendations Martin.

The way I have it setup in Mailscanner is if the sending mail server is on a RBL, (If it on at least 1 of the 4 RBLS we use with MailScanner) It becomes high scoring spam and is tagged and moved on, it does not get scanned by Spamassassin then.

The one thing I never considered before was if Spamassassin is scanning the same sending mail server IP for being listed when it does not get caught by MailScanner as being on any of the 4 RB:'s I use.   Not that it is causing my problem now, but that it is not very efficient if it is doing it again. (I would guess 80% of my mail never gets scanned by SpamAssassin each day because the sending mail gateway is blacklisted and it is marked as Spam and moves on)

When you say: " I normally remove most of the RBL's from being scanned in Spamassassin by giving most of them a zero score (see 50_scores.cf<http://spamassassin.apache.org/dist/rules/50_scores.cf> in the DNSEval section)."

I don't think I follow, are you saying Spamassassin is scanning the sending mail host again against the RBL's? So by giving them a zero score you are avoiding the double effort? This section has nothing to do with the URL/URI scanning that is happening? I assumed the rules that I have that are NOT hitting when it goes through MailScanner/Spamassassin have all been based on the URI/URL's in the body of the message.

When I take these specific Spam messages that make it into my inbox, I am noticing they never hit on the same URIBL hits I get when I move the message locally to the box,  if I take one of these URI based RBL checking rules like for example URIBL_BLACK, I have never seen that rule hit on ANY of these ones making it into my inbox.  If I search my maillog from yesterday for every message that wound up being scanned by Spamassassin, I see that there were 1014 times that rule is listed on detected Spam.

Last night I first tried setting up a caching bind server local to the box.  Made no difference.

I tried upgrading to MailScanner 4.84.5-3 after and updating to SpamAssassin 3.3.2-1 to see if that would make a difference, I even looked at the Perl modules that come with MailScanner, one of them was perl-Net-DNS-0.65-2, I was running
perl-Net-DNS-0.65-1, was hoping that had something to do with this so I updated to .65-2 of that perl modules.. the rest all seemed to be the same version that comes with 4.84.5-3 (I was running 4.83.5-1 before I updated)

I went over all my configs for both MailScanner and SpamAssassin, nothing seems wrong or set to low that would create the situation I am seeing.  I did find I had the pyzor plugin loading in SpamAssassin but no exe, so I just disabled pyzor and verified in the -debug-sa that everything looks fine.

I waited and sure enough it happened again today. We get less mail on the weekends so it took awhile waiting..

I have posted my MailScanner -debug-sa to pastebin if anyone can take a look and give me a recommendation of where to look next.  I am almost out of things to try.

Here is one -debug-sa:

http://pastebin.com/C2XPs7D2

Then I kept running with -debug-sa till it caught one with a DNS based rule like URIBL_* rules.

This one hits on those URIBL rules that are DNS based and it looks like everything is OK as far as I can tell..  This is really the first time I have tried to debug a debug log from MailScanner/Spamassassin before..

http://pastebin.com/iWMnJqf3

Thanks

BRIAN M. DUNCAN
Data Security Administrator
Katten Muchin Rosenman LLP
525 W. Monroe Street / Chicago, IL 60661-3693
p / (312) 577-8045 f / (312) 577-4490
brian.duncan at kattenlaw.com<mailto:brian.duncan at kattenlaw.com> / www.kattenlaw.com<http://www.kattenlaw.com>

From: mailscanner-bounces at lists.mailscanner.info<mailto:mailscanner-bounces at lists.mailscanner.info> [mailto:mailscanner-bounces at lists.mailscanner.info<mailto:mailscanner-bounces at lists.mailscanner.info>] On Behalf Of Martin Hepworth
Sent: Saturday, June 15, 2013 8:46 AM
To: MailScanner discussion
Subject: Re: Certain Spamassassin rules do not seem to be firing all of the time

Ok, you really need to put a local DNS server on the MailScanner box, doesn't matter if the DNS resolver is next to the server in the switch port, DNS is actually quite heavy on network traffic and hitting this all the time makes a huge difference. It can forward to the current machine, but the time this saves is actually quite noticable.
that three seconds for the pass across seems very quick to me, esp as it's got all the DNS requests to process.I normally remove most of the RBL's from being scanned in Spamassassin by giving most of them a zero score (see 50_scores.cf<http://spamassassin.apache.org/dist/rules/50_scores.cf> in the DNSEval section). also make sure you're updating sa rules regularly. In fact its almost as if you've got skip-rbl-checks set to 1 in a spamassassin .cf or mailscanner,conf file somewhere.
I'd double check all the setup to make sure everythings OK, as it's really odd that you're getting DNS based hits in test mode but not in test mode. Check the MailScanner.conf setttings and any site MailScanner.conf, and also get rid of any .spamassassin dirs esp if there's anything in root's home dir (so i presume ther MTA is sendmail?) to make sure that isnt overriding any settings. Check you've got one MailScanner.conf and not multiple ones, sometimes some distributions put the active file in 'non-standard' places.

--
Martin Hepworth, CISSP
Oxford, UK

On 15 June 2013 03:22, Duncan, Brian M. <brian.duncan at kattenlaw.com<mailto:brian.duncan at kattenlaw.com>> wrote:
Thanks, yes I noticed that, they all do seem to be the DNS rules.  I do have a caching DNS server but it is on the local network.  I will try and see if the behavior changes at all by running one locally on the box itself.

When you say "that youre not timing out the network checks in sa too quickly"  I have not changed anything in the defaults of Mailscanner or included any directives that would lower whatever time limits are set by default.

I took a look at the last example I put on pastebin, and it looks like it took 3 seconds to go from my Mailscanner box to my next gateway.

Received: from venus.kattenlaw.com<http://venus.kattenlaw.com> ([10.18.3.33]) by us.kmz.com<http://us.kmz.com>
 ([10.18.16.181]) with ESMTP (TREND IMSS SMTP Service 7.1) id 8e3c2381002025b2
 ; Fri, 14 Jun 2013 14:01:09 -0500
Received: from a.loselit.net<http://a.loselit.net> (a.loselit.net<http://a.loselit.net> [66.96.254.156])    by
 venus.kattenlaw.com<http://venus.kattenlaw.com> (8.13.8/8.13.4) with ESMTP id r5EJ13oK014449       for
 <brian.duncan at kmzr.com<mailto:brian.duncan at kmzr.com>>; Fri, 14 Jun 2013 14:01:06 -0500

I am assuming the 3 seconds going from my incoming mail server Venus, to the next hop in my environment includes the time it took for the Spammer to send me the message.

I also don't see anything in my maillogs related to Spam Assassin timing out for anything.. I recall many years ago when we used to run systems with much less CPU power (10+) seeing Spam Assassin time outs.

Which BTW, at the peak of activity today the lowest idle %idle was 91.00 and that is because I turned off caching of SpamAssassin in Mailscanner to see if that had any impact.

I also looked at the local caching DNS server that is on the same switch as this box, and it was peaking at like 30 Kilobytes per second on UDP 53 requests from anything that uses it locally according to iptraf.

It also seems to be these messages from the same Spammer, as I said before if I take any of these message bodies and send them in myself I seem to get the DNS Spam Assassin hits then.

Really odd one..

Thanks for your help

BRIAN M. DUNCAN
Data Security Administrator
Katten Muchin Rosenman LLP
525 W. Monroe Street / Chicago, IL 60661-3693
p / (312) 577-8045 f / (312) 577-4490
brian.duncan at kattenlaw.com<mailto:brian.duncan at kattenlaw.com> / www.kattenlaw.com<http://www.kattenlaw.com>

From: mailscanner-bounces at lists.mailscanner.info<mailto:mailscanner-bounces at lists.mailscanner.info> [mailto:mailscanner-bounces at lists.mailscanner.info<mailto:mailscanner-bounces at lists.mailscanner.info>] On Behalf Of Martin Hepworth
Sent: Friday, June 14, 2013 4:16 PM
To: MailScanner discussion
Subject: Certain Spamassassin rules do not seem to be firing all of the time

Hmm most if the extra rules youre hitting are dns based
I'd check youre running a local caching dns server on the scanning box and that youre not timing out the network checks in sa too quickly

Martin

On Friday, 14 June 2013, Duncan, Brian M. wrote:
Here is one more that just came in to me and was not tagged as Spam:

http://pastebin.com/w8SJk660

Mailscanner/Spamassassin results:

X-MailScanner-SpamCheck: not spam, SpamAssassin (score=2.999, required 6.5,
                BAYES_60 3.00, RP_MATCHES_RCVD -0.00)

--test-mode results:

Content analysis details:   (10.5 hits, 6.5 required)
 6.6 BAYES_99               BODY: Bayes spam probability is 99 to 100%
                            [score: 1.0000]
-0.0 RP_MATCHES_RCVD        Envelope sender domain matches handover relay domain
 2.5 RAZOR2_CF_RANGE_E8_51_100 Razor2 gives engine 8 confidence level
                            above 50%
                            [cf: 100]
 8.5 RAZOR2_CHECK           Listed in Razor2 (http://razor.sf.net/)
 0.5 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50%
                            [cf: 100]
-7.5 AWL                    AWL: From: address is in the auto white-list

------ End of SpamAssassin results, Original message follows --------

===========================================================

CIRCULAR 230 DISCLOSURE: Pursuant to Regulations Governing Practice Before the Internal Revenue

Service, any tax advice contained herein is not intended or written to be used and cannot be used

by a taxpayer for the purpose of avoiding tax penalties that may be imposed on the taxpayer.

===========================================================

CONFIDENTIALITY NOTICE:

This electronic mail message and any attached files contain information intended for the exclusive

use of the individual or entity to whom it is addressed and may contain information that is

proprietary, privileged, confidential and/or exempt from disclosure under applicable law.  If you

are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or

distribution of this information may be subject to legal restriction or sanction.  Please notify

the sender, by electronic mail or telephone, of any unintended recipients and delete the original

message without making any copies.

===========================================================

NOTIFICATION:  Katten Muchin Rosenman LLP is an Illinois limited liability partnership that has

elected to be governed by the Illinois Uniform Partnership Act (1997).

===========================================================

--
--
Martin Hepworth, CISSP
Oxford, UK

--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/mailman/listinfo/mailscanner

Before posting, read http://wiki.mailscanner.info/posting

Support MailScanner development - buy the book off the website!

--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/mailman/listinfo/mailscanner

Before posting, read http://wiki.mailscanner.info/posting

Support MailScanner development - buy the book off the website!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mailscanner.info/pipermail/mailscanner/attachments/20130618/6c414f14/attachment-0001.html 