Max SpamAssassin Size problems

[Scott Silva]

Julian Field spake the following on 8/26/2006 5:59 AM:
> 
> 
> Anthony Peacock wrote:
>>> Ken A wrote:
>>>>
>>>> Logan Shaw wrote:
>>>>> On Thu, 24 Aug 2006, Julian Field wrote:
>>>>>> Anthony Peacock wrote:
>>>>>>> Julian Field wrote:
>>>>>>>> Sounds survivable. After the limit I will keep going until I hit the
>>>>>>>> first line that only contains white space.
>>>>>>> I have been watching this discussion with a growing uneasiness.  I
>>>>>>> could be wrong but doesn't this behaviour open up the system to
>>>>>>> problems with huge image files...
>>>>>> Yes, you are absolutely correct. Non-spam may well include huge images.
>>>>>> The problem with rewinding to the previous boundary is that you may end
>>>>>> up not giving SpamAssassin _anything_ to work with.
>>>>>>
>>>>>> So it's up for a vote:
>>>>>>
>>>>>> do I chop half way through an image?
>>>>>> do I chop at the end of an image?
>>>>>> do I carry on for a max of 100 lines of Base64 data or until the end of
>>>>>> an image, which is earlier?
>>>>> I don't like the last option at all.  It still easily allows
>>>>> a situation where a valid message with a valid image in it
>>>>> gets detected as a corrupt image and hits a rule that scores
>>>>> it as spam.
>>>>>
>>>>> If we assume there are 80 columns of base64 data per line, then
>>>>> we get 60 bytes per line (since each base64 character carries
>>>>> 6 bits of data).  That means 100 lines only holds 6K, maximum.
>>>>>
>>>>> So this option only works if the chop-off point randomly
>>>>> happens to fall within the last 6K (or less) of the image.
>>>>> If the max message size causes the initial chop-off point to
>>>>> fall any earlier, it still creates an invalid image.  If you
>>>>> have a 50K max message size and someone sends a 75K image
>>>>> (which is not out of the ordinary at all), this method will
>>>>> keep going up to 56K and then quit.
>>>>>
>>>>> Basically, adding the 100 extra lines is really not much better
>>>>> than chopping right at the max message size barrier, unless
>>>>> you assume that most images aren't much larger than 6K, which
>>>>> I don't think is a valid assumption at all.  So, this option
>>>>> adds extra complexity and doesn't really give much benefit.
>>>>>
>>>>>   - Logan
>>>> I'm all for #3 and and just set "score FUZZY_OCR_CORRUPT_IMG 0" if you 
>>>> are worried about false positives. Fuzzyocr will get better at sorting 
>>>> this out. And of course in the mean time, don't use outlook, since it 
>>>> will probably render corrupt images just fine. (it's a feature)
>>> This could be controversial here...
>>>
>>> <Evil Grin>
>>> I have another suggestion, why don't we agree to leave the MailScanner 
>>> code alone.  Those people who are experiencing problems with broken 
>>> images can raise the value of "Max SpamAssassin Size" in *THEIR* 
>>> configurations, the rest of us can carry on as normal.
>>>
>>> There is already a way for people to adjust how much information SA gets 
>>> from MailScanner, people who need more information can used that on 
>>> their systems.
>>> </Evil Grin>
>>>
>>> <Ducks and Runs>
> 
> Quack, quack, scamper, scamper....
> 
> In my book, that is a remarkably good idea. It would be much simpler for 
> me to implement than any of the other, increasingly complicated versions.
> 
> What objections to people have to simply letting you set this yourself?
> 
> 
Anything that makes your life easier, Julian, Is OK with me.

I wonder just how many sites, as a percentage of the total installed, would
need this code anyway? If you were trying to please 10 or 20 percent, then I
could see it. But if it is only for a handful of sites, then you are
complicating your code for a small return. I think that the image based plugin
writers should take the possibility of truncated images into account when they
write "their" code, and you shouldn't have to fix MailScanner to make their
code work right. The only other option I could see would be if MailScanner
added a header or some other sort of mark to the "trimmed" mail sent to
spamassassin and the plugins would look for this header and know if a message
was complete or not. But then the spammers would add this to their messages.
Now if someone would spend as much time and money prosecuting spammers as they
do looking for 13 year olds downloading music, maybe we could get ahead of the
game.
I really hate spammers!

-- 

MailScanner is like deodorant...
You hope everybody uses it, and
you notice quickly if they don't!!!!