"Banned Content" question - still getting "Content Check"
messages
Julian Field
MailScanner at ecs.soton.ac.uk
Thu Jan 13 15:04:51 GMT 2005
[ The following text is in the "ISO-8859-1" character set. ]
[ Your display is set for the "US-ASCII" character set. ]
[ Some characters may be displayed incorrectly. ]
1 more:
There is a line in Message.pm that says
$DisarmInsideLink = "";
Change it to say
$DisarmInsideLink = 0;
You might be hitting a Perl bug, which would explain the fact you can't
reproduce the problem reliably.
Julian Field wrote:
> A minor bugfix to Message.pm which probably won't make any difference,
> but is still worth a try.
>
> @@ -3687,6 +3687,13 @@
> #print STDERR "Tags to convert are " . $this->{tagstoconvert} . "\n";
>
> # Set the disarm booleans for this message
> + $DisarmFormTag = 0;
> + $DisarmScriptTag = 0;
> + $DisarmCodebaseTag = 0;
> + $DisarmCodebaseTag = 0;
> + $DisarmIframeTag = 0;
> + $DisarmWebBug = 0;
> + $DisarmPhishing = 0;
> $DisarmFormTag = 1 if $this->{tagstoconvert} =~ /form/i;
> $DisarmScriptTag = 1 if $this->{tagstoconvert} =~ /script/i;
> $DisarmCodebaseTag = 1 if $this->{tagstoconvert} =~ /codebase/i;
>
> I'm afraid I can't think of much else. All the other variables are
> initialised, and these few just maintain state for the particular
> message, they don't hold any message content.
>
> Quentin Campbell wrote:
>
>> Julian
>>
>> I have applied your SMDiskStore.pm changes as well as set "Allow WebBugs
>> = yes".
>>
>> However we are still getting corrupted HTML in multipart/alternative
>> message. These are all accompanied with "Content Checks: Detected and
>> will disarm HTML message in ..." messages in the Sendmail log. This is
>> happening on both 4.35.10 and 4.37.7 systems.
>>
>> Unfortunately the problem is INTERMITTENT and it cannot be repeated by
>> sending the same message again to the same recipient. :-(
>>
>> The "Content Checks:" message is misleading since I am no longer using
>> the "disarm" content action anywhere, either in MailScanner.conf or in
>> the MailScanner rules files. Where I specify an action to deal with HTML
>> content it only uses "striphtml".
>>
>> Why then is MailScanner telling me it is "disarming" HTML when I have
>> not asked it to?
>>
>> This is really getting frustrating and more users are complaining. :-(
>>
>> PS I note that I applied two patches from you late last year to the
>> 4.35.10 system;
>> these were to SMDiskStore.pm and SweepContent.pm.
>>
>> Quentin
>> ---
>> PHONE: +44 191 222 8209 Information Systems and Services (ISS),
>> University of Newcastle,
>> Newcastle upon Tyne,
>> FAX: +44 191 222 8765 United Kingdom, NE1 7RU.
>> ------------------------------------------------------------------------
>> "Any opinion expressed above is mine. The University can get its own."
>>
>>
>>
>>> -----Original Message-----
>>> From: MailScanner mailing list
>>> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf Of Quentin Campbell
>>> Sent: 13 January 2005 11:33
>>> To: MAILSCANNER at JISCMAIL.AC.UK
>>> Subject: Re: "Banned Content" question - Lock/Unlock does not
>>> fix problem
>>>
>>> Julian
>>>
>>> The mods to subs "Lock" and "Unlock" have not fixed the problem.
>>>
>>> It appears to be also present on a 4.37.7-1 system. As this
>>> has the same
>>> Lock/Unlock code that should not be a surprise.
>>>
>>> I will now allow web bugs to see if the corruption stops.
>>>
>>> Quentin
>>> ---
>>> PHONE: +44 191 222 8209 Information Systems and Services (ISS),
>>> University of Newcastle,
>>> Newcastle upon Tyne,
>>> FAX: +44 191 222 8765 United Kingdom, NE1 7RU.
>>> ---------------------------------------------------------------
>>> ---------
>>> "Any opinion expressed above is mine. The University can get
>>> its own."
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: MailScanner mailing list
>>>> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf Of Quentin Campbell
>>>> Sent: 13 January 2005 08:50
>>>> To: MAILSCANNER at JISCMAIL.AC.UK
>>>> Subject: Re: "Banned Content" question - possibly a Web Bug
>>>> code problem
>>>>
>>>> Julian
>>>>
>>>> You had already given me a new SMDiskStore.pm module, dated
>>>>
>>>>
>>> 16 December
>>>
>>>
>>>> to try. The locking code in this differs from the new code you want me
>>>> to try as follows:
>>>>
>>>> < #JKF MailScanner::Lock::unlockclose($this->{indhandle});
>>>> < close($this->{indhandle});
>>>> ---
>>>>
>>>>
>>>>> # Now we lock the df file as well, we must unlock it too.
>>>>> MailScanner::Lock::unlockclose($this->{indhandle});
>>>>> #close($this->{indhandle});
>>>>>
>>>>>
>>>> I have made the change as above and will let you know what happens. I
>>>> also note that the new code is in the MailScanner-4.37.7-1
>>>> SMDiskStore.pm which I was planning to move to anyway.
>>>>
>>>> I have not touched the "Allow WebBugs = disarm" setting which I assume
>>>> is an essential part of the test of the changes to SMDiskStore.pm.
>>>>
>>>> Quentin
>>>> ---
>>>> PHONE: +44 191 222 8209 Information Systems and Services (ISS),
>>>> University of Newcastle,
>>>> Newcastle upon Tyne,
>>>> FAX: +44 191 222 8765 United Kingdom, NE1 7RU.
>>>> ---------------------------------------------------------------
>>>> ---------
>>>> "Any opinion expressed above is mine. The University can get
>>>> its own."
>>>>
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: MailScanner mailing list
>>>>> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf Of Julian Field
>>>>> Sent: 12 January 2005 16:02
>>>>> To: MAILSCANNER at JISCMAIL.AC.UK
>>>>> Subject: Re: "Banned Content" question - possibly a Web Bug
>>>>> code problem
>>>>>
>>>>> In which case try editing SMDiskStore.pm and replace the sub Lock and
>>>>> sub Unlock with this code:
>>>>>
>>>>> # Open and lock the message
>>>>> sub Lock {
>>>>> my $this = shift;
>>>>>
>>>>> #print STDERR "About to lock " . $this->{hpath} . " and " .
>>>>> # $this->{dpath} . "\n";
>>>>> MailScanner::Lock::openlock($this->{inhhandle}, '+<' .
>>>>> $this->{hpath},
>>>>> 'w', 'quiet')
>>>>> or return undef;
>>>>> #print STDERR "Got hlock\n";
>>>>>
>>>>> # If locking the dfile fails, then must close and unlock the
>>>>> qffile too
>>>>> # 14/12/2004 Try putting this back in for now.
>>>>> unless (MailScanner::Lock::openlock($this->{indhandle},
>>>>> '+<' . $this->{dpath}, 'w', 'quiet')) {
>>>>> #JKF 14/12/2004 open($this->{indhandle}, '+<' .
>>>>> $this->{dpath})) {
>>>>> MailScanner::Lock::unlockclose($this->{inhhandle});
>>>>> return undef;
>>>>> }
>>>>> #print STDERR "Got dlock\n";
>>>>> return undef unless $this->{inhhandle} && $this->{indhandle};
>>>>> return 1;
>>>>> }
>>>>>
>>>>>
>>>>> # Close and unlock the message
>>>>> sub Unlock {
>>>>> my $this = shift;
>>>>>
>>>>> # Now we lock the df file as well, we must unlock it too.
>>>>> MailScanner::Lock::unlockclose($this->{indhandle});
>>>>> #close($this->{indhandle});
>>>>> MailScanner::Lock::unlockclose($this->{inhhandle});
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> Quentin Campbell wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Julian
>>>>>>
>>>>>> The version of MailScanner on which I have seen the problem
>>>>>>
>>>>>>
>>>>> is 4.35.10.
>>>>>
>>>>>
>>>>>> Quentin
>>>>>> ---
>>>>>> PHONE: +44 191 222 8209 Information Systems and Services (ISS),
>>>>>> University of Newcastle,
>>>>>> Newcastle upon Tyne,
>>>>>> FAX: +44 191 222 8765 United Kingdom, NE1 7RU.
>>>>>> --------------------------------------------------------------
>>>>>>
>>>>>>
>>>>> ----------
>>>>>
>>>>>
>>>>>> "Any opinion expressed above is mine. The University can get
>>>>>>
>>>>>>
>>>> its own."
>>>>
>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: MailScanner mailing list
>>>>>>> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf Of Julian Field
>>>>>>> Sent: 12 January 2005 15:30
>>>>>>> To: MAILSCANNER at JISCMAIL.AC.UK
>>>>>>> Subject: Re: "Banned Content" question - possibly a Web Bug
>>>>>>> code problem
>>>>>>>
>>>>>>> What version of MailScanner are you using? I slightly improved the
>>>>>>> locking code (took out an "improvement" I made a long time
>>>>>>>
>>>>>>>
>>>>> ago which I
>>>>>
>>>>>
>>>>>>> only made after lots of people requested it) in 4.37. It now
>>>>>>>
>>>>>>>
>>>>> locks the
>>>>>
>>>>>
>>>>>>> df as well as the qf, which slows down delivery slightly in some
>>>>>>> situations, but appears to be more reliable than just
>>>>>>>
>>>>>>>
>>>> locking the qf.
>>>>
>>>>
>>>>>>> Quentin Campbell wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: MailScanner mailing list
>>>>>>>>> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On Behalf Of Mike
>>>>>>>>> Sent: 12 January 2005 11:53
>>>>>>>>> To: MAILSCANNER at JISCMAIL.AC.UK
>>>>>>>>> Subject: Re: "Banned Content" question - a related problem
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> From: MailScanner mailing list
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>> [mailto:MAILSCANNER at JISCMAIL.AC.UK] On
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>>> Behalf Of Quentin Campbell
>>>>>>>>>>
>>>>>>>>>> All the systems are now up2date as far as RH AS 3 patches are
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> concerned.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> All the systems use the Sendmail that comes with these
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> system; the last
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> time they were updated this was Sendmail 8.12.11. I use
>>>>>>>>>>
>>>>>>>>>>
>>>>> the default
>>>>>
>>>>>
>>>>>>>>>> locking in MailScanner.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I also had this problem on sendmail 8.12.10. After changing
>>>>>>>>> the locking to posix, the problem was gone. So, although the
>>>>>>>>> docs state that the locking problem occurs only from 8.13 on,
>>>>>>>>> it seems that also some 8.12 versions are affected. Please set
>>>>>>>>> the locking mechanism to "posix" and see if it solves
>>>>>>>>>
>>>>>>>>>
>>>> your problem.
>>>>
>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I will do this as a last resort. There are four reasons why
>>>>>>>>
>>>>>>>>
>>>>> I want to
>>>>>
>>>>>
>>>>>>>> investigate other things first. In particular I want to capture a
>>>>>>>> message before then after it has gone through MailSanner and got
>>>>>>>> corrupted:
>>>>>>>>
>>>>>>>> 1. Locking works OK on RH AS 3 systems with an up-to-date kernel.
>>>>>>>> 2. The symptoms we are seeing do not appear to be
>>>>>>>>
>>>>>>>>
>>> repeatable so far
>>>
>>>
>>>>>>>> which makes conclusive testing difficult.
>>>>>>>> 3. I have looked for other evidence of locking problems but
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> cannot find
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> any. For example I can show that all messages tagged as spam by
>>>>>>>> MailScanner have been tagged once only. If there is a
>>>>>>>>
>>>>>>>>
>>>>> locking problem
>>>>>
>>>>>
>>>>>>>> you will see the same message (ie. same Sendmail QID) being
>>>>>>>>
>>>>>>>>
>>>>> tagged as
>>>>>
>>>>>
>>>>>>>> spam more than once by two or more MS processes.
>>>>>>>> 4. The problem appears related to the Web Bug check. I will
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> switch that
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> off first. See below for more details of this.
>>>>>>>>
>>>>>>>> Having looked further at the problem it appears to be
>>>>>>>>
>>>>>>>>
>>>>> related to MIME
>>>>>
>>>>>
>>>>>>>> multipart/alternative messages having all or part of the HTML part
>>>>>>>> corrupted. The text part is not being affected.
>>>>>>>>
>>>>>>>> In all of the cases the logs show that MailScanner has
>>>>>>>>
>>>>>>>>
>>>>> "disarmed" the
>>>>>
>>>>>
>>>>>>>> HTML content. Since I only "disarm" Web Bugs it appears
>>>>>>>>
>>>>>>>>
>>>>> that there may
>>>>>
>>>>>
>>>>>>>> be a bug in the Web Bugs code that causes an intermittent
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> problem. This
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> suspicion is reinforced by the observation that the problem
>>>>>>>>
>>>>>>>>
>>>>> appears to
>>>>>
>>>>>
>>>>>>>> have started when I enabled the Web Bug check late last
>>>>>>>>
>>>>>>>>
>>>> year. I will
>>>>
>>>>
>>>>>>>> first of all try "Allow WebBugs = yes" and see what happens.
>>>>>>>>
>>>>>>>> Quentin
>>>>>>>>
>>>>>>>> ------------------------ MailScanner list ------------------------
>>>>>>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>>>>>>> 'leave mailscanner' in the body of the email.
>>>>>>>> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
>>>>>>>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>>>>>>
>>>>>>>> Support MailScanner development - buy the book off the website!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Julian Field
>>>>>>> www.MailScanner.info
>>>>>>> Buy the MailScanner book at www.MailScanner.info/store
>>>>>>>
>>>>>>> PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
>>>>>>>
>>>>>>> ------------------------ MailScanner list ------------------------
>>>>>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>>>>>> 'leave mailscanner' in the body of the email.
>>>>>>> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
>>>>>>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>>>>>
>>>>>>> Support MailScanner development - buy the book off the website!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ------------------------ MailScanner list ------------------------
>>>>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>>>>> 'leave mailscanner' in the body of the email.
>>>>>> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
>>>>>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>>>>
>>>>>> Support MailScanner development - buy the book off the website!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> Julian Field
>>>>> www.MailScanner.info
>>>>> Buy the MailScanner book at www.MailScanner.info/store
>>>>>
>>>>> PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
>>>>>
>>>>> ------------------------ MailScanner list ------------------------
>>>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>>>> 'leave mailscanner' in the body of the email.
>>>>> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
>>>>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>>>
>>>>> Support MailScanner development - buy the book off the website!
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ------------------------ MailScanner list ------------------------
>>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>>> 'leave mailscanner' in the body of the email.
>>>> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
>>>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>>
>>>> Support MailScanner development - buy the book off the website!
>>>>
>>>>
>>>>
>>>>
>>> ------------------------ MailScanner list ------------------------
>>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>>> 'leave mailscanner' in the body of the email.
>>> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
>>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>>
>>> Support MailScanner development - buy the book off the website!
>>>
>>>
>>>
>>>
>>
>> ------------------------ MailScanner list ------------------------
>> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
>> 'leave mailscanner' in the body of the email.
>> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
>> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>>
>> Support MailScanner development - buy the book off the website!
>>
>>
>>
>
> --
> Julian Field
> www.MailScanner.info
> Buy the MailScanner book at www.MailScanner.info/store
>
> PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
>
> ------------------------ MailScanner list ------------------------
> To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
> 'leave mailscanner' in the body of the email.
> Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
> the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
>
> Support MailScanner development - buy the book off the website!
>
--
Julian Field
www.MailScanner.info
Buy the MailScanner book at www.MailScanner.info/store
PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
------------------------ MailScanner list ------------------------
To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
'leave mailscanner' in the body of the email.
Before posting, read the MAQ (http://www.mailscanner.biz/maq/) and
the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).
Support MailScanner development - buy the book off the website!
More information about the MailScanner
mailing list