"Banned Content" question - still getting "Content Check" messages

Quentin Campbell Q.G.Campbell at NEWCASTLE.AC.UK
Thu Jan 13 18:03:10 GMT 2005


Julian

You gave me a version of Message.pm in late December to test out a
phishing net improvement. This already had "$DisarmInsideLink = 0;".

However I have initialised the other variables you suggested. Will let
you know what happens.

Quentin
---
PHONE: +44 191 222 8209    Information Systems and Services (ISS),
                           University of Newcastle,
                           Newcastle upon Tyne,
FAX:   +44 191 222 8765    United Kingdom, NE1 7RU.
------------------------------------------------------------------------
"Any opinion expressed above is mine. The University can get its own."  

>-----Original Message-----
>From: MailScanner mailing list 
>[mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf Of Julian Field
>Sent: 13 January 2005 15:05
>To: MAILSCANNER at JISCMAIL.AC.UK
>Subject: Re: "Banned Content" question - still getting 
>"Content Check" messages
>
>1 more:
>There is a line in Message.pm that says
>
>$DisarmInsideLink = "";
>
>Change it to say
>
>$DisarmInsideLink = 0;
>
>You might be hitting a Perl bug, which would explain the fact you can't
>reproduce the problem reliably.
>
>Julian Field wrote:
>
>> A minor bugfix to Message.pm which probably won't make any 
>difference,
>> but is still worth a try.
>>
>> @@ -3687,6 +3687,13 @@
>>   #print STDERR "Tags to convert are " . 
>$this->{tagstoconvert} . "\n";
>>
>>   # Set the disarm booleans for this message
>> +  $DisarmFormTag     = 0;
>> +  $DisarmScriptTag   = 0;
>> +  $DisarmCodebaseTag = 0;
>> +  $DisarmCodebaseTag = 0;
>> +  $DisarmIframeTag   = 0;
>> +  $DisarmWebBug      = 0;
>> +  $DisarmPhishing    = 0;
>>   $DisarmFormTag     = 1 if $this->{tagstoconvert} =~ /form/i;
>>   $DisarmScriptTag   = 1 if $this->{tagstoconvert} =~ /script/i;
>>   $DisarmCodebaseTag = 1 if $this->{tagstoconvert} =~ /codebase/i;
>>
>> I'm afraid I can't think of much else. All the other variables are
>> initialised, and these few just maintain state for the particular
>> message, they don't hold any message content.
>>
>> Quentin Campbell wrote:
>>
>>> Julian
>>>
>>> I have applied your SMDiskStore.pm changes as well as set 
>"Allow WebBugs
>>> = yes".
>>>
>>> However we are still getting corrupted HTML in multipart/alternative
>>> message. These are all accompanied with "Content Checks: 
>Detected and
>>> will disarm HTML message in ..." messages in the Sendmail 
>log. This is
>>> happening on both 4.35.10 and 4.37.7 systems.
>>>
>>> Unfortunately the problem is INTERMITTENT and it cannot be 
>repeated by
>>> sending the same message again to the same recipient.  :-(
>>>
>>> The "Content Checks:" message is misleading since I am no 
>longer using
>>> the "disarm" content action anywhere, either in 
>MailScanner.conf or in
>>> the MailScanner rules files. Where I specify an action to 
>deal with HTML
>>> content it only uses "striphtml".
>>>
>>> Why then is MailScanner telling me it is "disarming" HTML 
>when I have
>>> not asked it to?
>>>
>>> This is really getting frustrating and more users are 
>complaining.  :-(
>>>
>>> PS I note that I applied two patches from you late last year to the
>>> 4.35.10 system;
>>>   these were to SMDiskStore.pm and SweepContent.pm.
>>>
>>> Quentin
>>> ---
>>> PHONE: +44 191 222 8209    Information Systems and Services (ISS),
>>>                           University of Newcastle,
>>>                           Newcastle upon Tyne,
>>> FAX:   +44 191 222 8765    United Kingdom, NE1 7RU.
>>> 
>---------------------------------------------------------------
>---------
>>> "Any opinion expressed above is mine. The University can 
>get its own."
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: MailScanner mailing list
>>>> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf Of Quentin Campbell
>>>> Sent: 13 January 2005 11:33
>>>> To: MAILSCANNER at JISCMAIL.AC.UK
>>>> Subject: Re: "Banned Content" question - Lock/Unlock does not
>>>> fix problem
>>>>
>>>> Julian
>>>>
>>>> The mods to subs "Lock" and "Unlock" have not fixed the problem.
>>>>
>>>> It appears to be also present on a 4.37.7-1 system. As this
>>>> has the same
>>>> Lock/Unlock code that should not be a surprise.
>>>>
>>>> I will now allow web bugs to see if the corruption stops.
>>>>
>>>> Quentin
>>>> ---
>>>> PHONE: +44 191 222 8209    Information Systems and Services (ISS),
>>>>                          University of Newcastle,
>>>>                          Newcastle upon Tyne,
>>>> FAX:   +44 191 222 8765    United Kingdom, NE1 7RU.
>>>> ---------------------------------------------------------------
>>>> ---------
>>>> "Any opinion expressed above is mine. The University can get
>>>> its own."
>>>>
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: MailScanner mailing list
>>>>> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf Of Quentin Campbell
>>>>> Sent: 13 January 2005 08:50
>>>>> To: MAILSCANNER at JISCMAIL.AC.UK
>>>>> Subject: Re: "Banned Content" question - possibly a Web Bug
>>>>> code problem
>>>>>
>>>>> Julian
>>>>>
>>>>> You had already given me a new SMDiskStore.pm module, dated
>>>>>
>>>>>
>>>> 16 December
>>>>
>>>>
>>>>> to try. The locking code in this differs from the new 
>code you want me
>>>>> to try as follows:
>>>>>
>>>>> <   #JKF MailScanner::Lock::unlockclose($this->{indhandle});
>>>>> <   close($this->{indhandle});
>>>>> ---
>>>>>
>>>>>
>>>>>>  # Now we lock the df file as well, we must unlock it too.
>>>>>>  MailScanner::Lock::unlockclose($this->{indhandle});
>>>>>>  #close($this->{indhandle});
>>>>>>
>>>>>>
>>>>> I have made the change as above and will let you know 
>what happens. I
>>>>> also note that the new code is in the MailScanner-4.37.7-1
>>>>> SMDiskStore.pm which I was planning to move to anyway.
>>>>>
>>>>> I have not touched the "Allow WebBugs = disarm" setting 
>which I assume
>>>>> is an essential part of the test of the changes to SMDiskStore.pm.
>>>>>
>>>>> Quentin
>>>>> ---
>>>>> PHONE: +44 191 222 8209    Information Systems and Services (ISS),
>>>>>                          University of Newcastle,
>>>>>                          Newcastle upon Tyne,
>>>>> FAX:   +44 191 222 8765    United Kingdom, NE1 7RU.
>>>>> ---------------------------------------------------------------
>>>>> ---------
>>>>> "Any opinion expressed above is mine. The University can get
>>>>> its own."
>>>>>
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: MailScanner mailing list
>>>>>> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf Of Julian Field
>>>>>> Sent: 12 January 2005 16:02
>>>>>> To: MAILSCANNER at JISCMAIL.AC.UK
>>>>>> Subject: Re: "Banned Content" question - possibly a Web Bug
>>>>>> code problem
>>>>>>
>>>>>> In which case try editing SMDiskStore.pm and replace the 
>sub Lock and
>>>>>> sub Unlock with this code:
>>>>>>
>>>>>> # Open and lock the message
>>>>>> sub Lock {
>>>>>> my $this = shift;
>>>>>>
>>>>>> #print STDERR "About to lock " . $this->{hpath} . " and " .
>>>>>> #             $this->{dpath} . "\n";
>>>>>> MailScanner::Lock::openlock($this->{inhhandle}, '+<' .
>>>>>> $this->{hpath},
>>>>>> 'w', 'quiet')
>>>>>>   or return undef;
>>>>>> #print STDERR "Got hlock\n";
>>>>>>
>>>>>> # If locking the dfile fails, then must close and unlock the
>>>>>> qffile too
>>>>>> # 14/12/2004 Try putting this back in for now.
>>>>>> unless (MailScanner::Lock::openlock($this->{indhandle},
>>>>>>                    '+<' . $this->{dpath}, 'w', 'quiet')) {
>>>>>>       #JKF 14/12/2004 open($this->{indhandle}, '+<' .
>>>>>> $this->{dpath})) {
>>>>>>   MailScanner::Lock::unlockclose($this->{inhhandle});
>>>>>>   return undef;
>>>>>> }
>>>>>> #print STDERR "Got dlock\n";
>>>>>> return undef unless $this->{inhhandle} && $this->{indhandle};
>>>>>> return 1;
>>>>>> }
>>>>>>
>>>>>>
>>>>>> # Close and unlock the message
>>>>>> sub Unlock {
>>>>>> my $this = shift;
>>>>>>
>>>>>> # Now we lock the df file as well, we must unlock it too.
>>>>>> MailScanner::Lock::unlockclose($this->{indhandle});
>>>>>> #close($this->{indhandle});
>>>>>> MailScanner::Lock::unlockclose($this->{inhhandle});
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> Quentin Campbell wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Julian
>>>>>>>
>>>>>>> The version of MailScanner on which I have seen the problem
>>>>>>>
>>>>>>>
>>>>>> is 4.35.10.
>>>>>>
>>>>>>
>>>>>>> Quentin
>>>>>>> ---
>>>>>>> PHONE: +44 191 222 8209    Information Systems and 
>Services (ISS),
>>>>>>>                          University of Newcastle,
>>>>>>>                          Newcastle upon Tyne,
>>>>>>> FAX:   +44 191 222 8765    United Kingdom, NE1 7RU.
>>>>>>> --------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>> ----------
>>>>>>
>>>>>>
>>>>>>> "Any opinion expressed above is mine. The University can get
>>>>>>>
>>>>>>>
>>>>> its own."
>>>>>
>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: MailScanner mailing list
>>>>>>>> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf Of Julian Field
>>>>>>>> Sent: 12 January 2005 15:30
>>>>>>>> To: MAILSCANNER at JISCMAIL.AC.UK
>>>>>>>> Subject: Re: "Banned Content" question - possibly a Web Bug
>>>>>>>> code problem
>>>>>>>>
>>>>>>>> What version of MailScanner are you using? I slightly 
>improved the
>>>>>>>> locking code (took out an "improvement" I made a long time
>>>>>>>>
>>>>>>>>
>>>>>> ago which I
>>>>>>
>>>>>>
>>>>>>>> only made after lots of people requested it) in 4.37. It now
>>>>>>>>
>>>>>>>>
>>>>>> locks the
>>>>>>
>>>>>>
>>>>>>>> df as well as the qf, which slows down delivery 
>slightly in some
>>>>>>>> situations, but appears to be more reliable than just
>>>>>>>>
>>>>>>>>
>>>>> locking the qf.
>>>>>
>>>>>
>>>>>>>> Quentin Campbell wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: MailScanner mailing list
>>>>>>>>>> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf Of Mike
>>>>>>>>>> Sent: 12 January 2005 11:53
>>>>>>>>>> To: MAILSCANNER at JISCMAIL.AC.UK
>>>>>>>>>> Subject: Re: "Banned Content" question - a related problem
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> From: MailScanner mailing list
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>>> Behalf Of Quentin Campbell
>>>>>>>>>>>
>>>>>>>>>>> All the systems are now up2date as far as RH AS 3 
>patches are
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> concerned.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> All the systems use the Sendmail that comes with these
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> system; the last
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> time they were updated this was Sendmail 8.12.11. I use
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>> the default
>>>>>>
>>>>>>
>>>>>>>>>>> locking in MailScanner.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> I also had this problem on sendmail 8.12.10. After changing
>>>>>>>>>> the locking to posix, the problem was gone. So, although the
>>>>>>>>>> docs state that the locking problem occurs only from 8.13 on,
>>>>>>>>>> it seems that also some 8.12 versions are affected. 
>Please set
>>>>>>>>>> the locking mechanism to "posix" and see if it solves
>>>>>>>>>>
>>>>>>>>>>
>>>>> your problem.
>>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I will do this as a last resort. There are four reasons why
>>>>>>>>>
>>>>>>>>>
>>>>>> I want to
>>>>>>
>>>>>>
>>>>>>>>> investigate other things first. In particular I want 
>to capture a
>>>>>>>>> message before then after it has gone through 
>MailSanner and got
>>>>>>>>> corrupted:
>>>>>>>>>
>>>>>>>>> 1. Locking works OK on RH AS 3 systems with an 
>up-to-date kernel.
>>>>>>>>> 2. The symptoms we are seeing do not appear to be
>>>>>>>>>
>>>>>>>>>
>>>> repeatable so far
>>>>
>>>>
>>>>>>>>> which makes conclusive testing difficult.
>>>>>>>>> 3. I have looked for other evidence of locking problems but
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> cannot find
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> any. For example I can show that all messages tagged 
>as spam by
>>>>>>>>> MailScanner have been tagged once only. If there is a
>>>>>>>>>
>>>>>>>>>
>>>>>> locking problem
>>>>>>
>>>>>>
>>>>>>>>> you will see the same message (ie. same Sendmail QID) being
>>>>>>>>>
>>>>>>>>>
>>>>>> tagged as
>>>>>>
>>>>>>
>>>>>>>>> spam more than once by two or more MS processes.
>>>>>>>>> 4. The problem appears related to the Web Bug check. I will
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> switch that
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> off first. See below for more details of this.
>>>>>>>>>
>>>>>>>>> Having looked further at the problem it appears to be
>>>>>>>>>
>>>>>>>>>
>>>>>> related to MIME
>>>>>>
>>>>>>
>>>>>>>>> multipart/alternative messages having all or part of 
>the HTML part
>>>>>>>>> corrupted. The text part is not being affected.
>>>>>>>>>
>>>>>>>>> In all of the cases the logs show that MailScanner has
>>>>>>>>>
>>>>>>>>>
>>>>>> "disarmed" the
>>>>>>
>>>>>>
>>>>>>>>> HTML content. Since I only "disarm" Web Bugs it appears
>>>>>>>>>
>>>>>>>>>
>>>>>> that there may
>>>>>>
>>>>>>
>>>>>>>>> be a bug in the Web Bugs code that causes an intermittent
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> problem. This
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> suspicion is reinforced by the observation that the problem
>>>>>>>>>
>>>>>>>>>
>>>>>> appears to
>>>>>>
>>>>>>
>>>>>>>>> have started when I enabled the Web Bug check late last
>>>>>>>>>
>>>>>>>>>
>>>>> year. I will
>>>>>
>>>>>
>>>>>>>>> first of all try "Allow WebBugs = yes" and see what happens.
>>>>>>>>>
>>>>>>>>> Quentin
>>>>>>>>>
>>>>>>>>> ------------------------ MailScanner list 
>------------------------
>>>>>>>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>>>>>>>> 'leave mailscanner' in the body of the email.
>>>>>>>>> Before posting, read the MAQ 
>(http://www.mailscanner.biz/maq/) and
>>>>>>>>> the archives 
>(http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>>>>>>>
>>>>>>>>> Support MailScanner development - buy the book off 
>the website!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Julian Field
>>>>>>>> www.MailScanner.info
>>>>>>>> Buy the MailScanner book at www.MailScanner.info/store
>>>>>>>>
>>>>>>>> PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 
>1415 B654
>>>>>>>>
>>>>>>>> ------------------------ MailScanner list 
>------------------------
>>>>>>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>>>>>>> 'leave mailscanner' in the body of the email.
>>>>>>>> Before posting, read the MAQ 
>(http://www.mailscanner.biz/maq/) and
>>>>>>>> the archives 
>(http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>>>>>>
>>>>>>>> Support MailScanner development - buy the book off the website!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ------------------------ MailScanner list 
>------------------------
>>>>>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>>>>>> 'leave mailscanner' in the body of the email.
>>>>>>> Before posting, read the MAQ 
>(http://www.mailscanner.biz/maq/) and
>>>>>>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>>>>>
>>>>>>> Support MailScanner development - buy the book off the website!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Julian Field
>>>>>> www.MailScanner.info
>>>>>> Buy the MailScanner book at www.MailScanner.info/store
>>>>>>
>>>>>> PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
>>>>>>
>>>>>> ------------------------ MailScanner list 
>------------------------
>>>>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>>>>> 'leave mailscanner' in the body of the email.
>>>>>> Before posting, read the MAQ 
>(http://www.mailscanner.biz/maq/) and
>>>>>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>>>>
>>>>>> Support MailScanner development - buy the book off the website!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ------------------------ MailScanner list ------------------------
>>>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>>>> 'leave mailscanner' in the body of the email.
>>>>> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
>>>>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>>>
>>>>> Support MailScanner development - buy the book off the website!
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ------------------------ MailScanner list ------------------------
>>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>>> 'leave mailscanner' in the body of the email.
>>>> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
>>>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>>
>>>> Support MailScanner development - buy the book off the website!
>>>>
>>>>
>>>>
>>>>
>>>
>>> ------------------------ MailScanner list ------------------------
>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>> 'leave mailscanner' in the body of the email.
>>> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
>>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>
>>> Support MailScanner development - buy the book off the website!
>>>
>>>
>>>
>>
>> --
>> Julian Field
>> www.MailScanner.info
>> Buy the MailScanner book at www.MailScanner.info/store
>>
>> PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
>>
>> ------------------------ MailScanner list ------------------------
>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>> 'leave mailscanner' in the body of the email.
>> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>
>> Support MailScanner development - buy the book off the website!
>>
>
>--
>Julian Field
>www.MailScanner.info
>Buy the MailScanner book at www.MailScanner.info/store
>
>PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
>
>------------------------ MailScanner list ------------------------
>To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>'leave mailscanner' in the body of the email.
>Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
>the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>
>Support MailScanner development - buy the book off the website!
>
>

------------------------ MailScanner list ------------------------
To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
'leave mailscanner' in the body of the email.
Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).

Support MailScanner development - buy the book off the website!




More information about the MailScanner mailing list