FPs and SA 3.2.0

Mon May 21 20:26:50 IST 2007

Calm down Paul ;)

Right lets go back to the beginning.  I noticed that a huge amount of email
was being marked within MailWatch as listed in RBL, yet in the rules
summary no RBLs were actually being reported.  At first I thought it was
down to my list of RBLs I was using within MailScanner.conf.  Therefore I
tried one at a time and got the same result.  Thinking back I even set Spam
List = <null> and got the same result.

At this point I posted my question to the list, and Jules asked me to give
the code change a try.

As soon as I made the change and restarted MS I no longer saw this
situation.

I understand Perl to a degree myself, and have myself seen what can happen
when using a undefined variable to compare against.

At the end of the day it has resolved a issue for us. If I have
misunderstood then so be it, but all I can do is report my personal
findings.

Regards,

On Mon, 21 May 2007 21:03:57 +0200, Paul Bijnens
<paul.bijnens at xplanation.com> wrote:
> --[ UxBoD ]-- wrote:
>> Even with just using one RBL I was getting non-spam items marked as
> SPAM.
>> When I made the change it marked SPAM messages correctly.
> 
> Don't be silly.
> 
> So you added this line;
>     $temp = 0 unless $HitList[0] =~ /a-z/i;
> And you did NOT add this line:
>     $temp = 0 unless $HitList[0] =~ /[a-z]/i;
> 
> and afterwards you get a correct (non-empty, but no FP) list of RBLs?
> What RBL did such a "marked SPAM message" got hit on?
> The list "spamhaus-A-Z"  maybe?
> 
> Or you mean that the SPAM messages were no false positives anymore?
> (Answer: Because SA had marked it like that, not because MailScanner
> found it in a spam list added in MailScanner.conf having the string
> "a-z".)
> 
> Read and understand before you reply!  Please!
> 
> 
> 
>> On Mon, 21 May 2007 19:28:01 +0200, Paul Bijnens
>> <Paul.Bijnens at xplanation.com> wrote:
>>> On 2007-05-21 17:10, Julian Field wrote:
>>>>
>>>> Paul Bijnens wrote:
>>>>> On 2007-05-16 17:41, Julian Field wrote:
>>>>>> I'll put it in the main codebase then. Perl has some very subtle
> bugs
>>> in
>>>>>> it...
>>>>>>
>>>>> I believe I don't need to teach perl to Julian (rather the other way
>>>>> around :-) ), but anyway...
>>>>
>>>>>>>>>
>>>>>>>>>>   # JKF 3/10/2005
>>>>>>>>>>   my $temp = @HitList;
>>>>>>>>>>   $temp = $temp + 0;
>>>>>>>>>>   $temp = 0 unless $HitList[0] =~ /a-z/i;
>>>>>>>>>>   return ($temp, join(', ', @HitList));
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> Let's see if that helps. According to the book, the 2 middle
> lines
>>>>>>>>>> shouldn't be needed at all.
>>>>>>>>>>
>>>>> To me this seems like the array @HitList contains an empty or undef
>>>>> value.  The match against "/[a-z]/i"   (or was that really intended
>>>>> "/a-z/i"??)
>>>> No, your version would match against any string that contained the
>>>> string "a-z" in it (in upper or lower case).
>>> Excuse me :-) but "/a-z/i" is your version and that will search for
>>> a string "a-z" lower or upper case.  My version, "/[a-z]/i", will match
>>> a name with at least one letter in it.  Which is what you're trying to
>>> do, I believe.
>>> You're effectively removing any RBL hits now, which is the main reason
>>> why no more FP's got hit by the current beta tester(-s? -- only one
>>> person as far I see had the problem).
>>>
>>>
> http://lists.mailscanner.info/pipermail/mailscanner/2007-May/073331.html
>>>
>>> I'm still interested in the exact list of RBLs in his config.
>>> Does it happen when 1 list is added? Two?  Some particular list only?
>>>
>>>
>>>>>  just hides the source of the real error: getting an empty
>>>>> value for RBL name.
>>>> If I printed the string of @HitList it turned out to have no contents,
>>> How?    Something like:
>>>
>>>     @HitList = ( "" );	# somehow this ended up in the list
>>>     $temp = @HitList;
>>>     warn("HitList contains $temp entries: '@HitList'\n");
>>>
>>> No (visible) contents, but still one element in the array.
>>>
>>>
>>>> so the scalar of it should have been zero. I have seen the problem of
>>>> "0" not always equaling zero a few other times, hence the addition of
>>>> zero to it to try to fix it, which has normally fixed the problem
>>> You can have that problem with "" or undef, acting as 0 in calculations
>>> but not showing up as a "0" when printed. Indeed fixed by explicitly
>>> converting to number by adding "+ 0".
>>>
>>>> elsewhere. The new modification has only been recently needed, the
> code
>>>> has worked perfectly well for years (the previous version was very old
>>>> code). If it had been needed before, people would have been
> complaining
>>>> loudly about this for the past few years, and they haven't been. So if
>>>> the start of the list doesn't contain a letter (all RBL names must
>>>> contain at least 1 letter or they wouldn't work) then the list must
>>>> actually be empty, so I force it to return zero.
>>> So we have to find out where the list element comes from that does
>>> not contain a letter, but is empty instead.  Instead of covering up the
>>> bug here.  (Still not convinced it is a perl bug.)
>>> Maybe most people use some RBLs at the MTA-level to block the incoming
>>> mail completely and/or use other RBLs in SA for scoring, and let the
>>> spam list entry in MailScanner empty.  Or the bug happens only on
>>> a timeout, like suggested in the OP problem, or only for certain
>>> combinations of timeout values, etc, etc.
>>>
>>>
>>>>> Now finding out where the empty value is coming from, is -- at my
>>>>> current understanding of the code -- not yet successful.
>>>> Yes. I have another demo of a Perl bug which I'll post for you if you
>>>> like. Perl is not bug-free.
>>> Sure not.  But, speaking for myself, it's usually in my own
>>> programs, and not in the perl compiler, that I find the bugs.  :-)
> 
> 
> --
> Paul Bijnens, xplanation Technology Services        Tel  +32 16 397.511
> Technologielaan 21 bus 2, B-3001 Leuven, BELGIUM    Fax  +32 16 397.512
> http://www.xplanation.com/          email:  Paul.Bijnens at xplanation.com
> ***********************************************************************
> * I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, ^^, *
> * F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
> * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
> * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
> * init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
> * ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out          *
> ***********************************************************************
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/mailman/listinfo/mailscanner
> 
> Before posting, read http://wiki.mailscanner.info/posting
> 
> Support MailScanner development - buy the book off the website!
> 
> --
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is
> believed to be clean.
-- 
--[ UxBoD ]--
// PGP Key: "curl -s http://www.splatnix.net/uxbod.asc | gpg --import"
// Fingerprint: 543A E778 7F2D 98F1 3E50 9C1F F190 93E0 E8E8 0CF8
// Keyserver: www.keyserver.net Key-ID: 0xE8E80CF8
// Phone: +44 (0) 845 869 2749  SIP: uxbod at sip.splatnix.net

-- 
This message has been scanned for viruses and dangerous content by MailScanner, and is
believed to be clean.