Scalability of 'spam.whitelist.rules' facility

[Julian Field]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Gareth wrote:
> On Mon, 2007-05-21 at 10:35, Res wrote:
>   
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On Mon, 21 May 2007, --[ UxBoD ]-- wrote:
>>
>>     
>>> Why not migrate them to MySQL and use the SQLBlackWhiteList.pm that is
>>> available from the MailWatch package ?
>>>       
>> This might be fine on low loaded systems, but to do a lookup for every 
>> email on major networks would fail, as sql would not keep up unless you
>> clustered and nobody is going to do that just for whitelisting.
>>
>> I've seen it with vpopmail for instance the cdb file can handle over 6000 
>> concurrent connections with no ill affect, sql spits farts and dies at 
>> 1/5th of that value unless you cluster, so it's a pointless expense.
>>     
>
> It doesn't work like that though. When the mailscanner instance starts
> the code loads the contents of the black/whitelist into memory and then
> checks to see if any changes have been made every 15 minutes.
>   
MailWatch might do that, but MailScanner doesn't.
Since every rule can be an arbitrary regular expression, each and every 
rule must be checked against the addressing information of each message. 
You can't just do a quick lookup.

This is why slurping it into a hash table is faster, so long as you can 
impose the restriction that every rule is only a complete email address 
or a complete domain name. Imposing the artificial restriction enables 
you to greatly optimise the  way you can look up the rules for each 
message. The MailScanner configuration compiler does not (yet) include 
such as optimisation. But it does contain the means for you to be able 
to write your own (Custom Functions).

> This reduces the load on the sql server but of course does increase
> memory usage and the startup time for the mailscanner process.
>
>   

Jules

- -- 
Julian Field MEng CITP
www.MailScanner.info
Buy the MailScanner book at www.MailScanner.info/store

MailScanner customisation, or any advanced system administration help?
Contact me at Jules at Jules.FM

PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
For all your IT requirements visit www.transtec.co.uk



-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.6.1 (Build 1012)
Charset: ISO-8859-1

wj8DBQFGUbL6EfZZRxQVtlQRAs63AJ9NtSVKA6aSLL5sQndvNpjPAMgvrgCgp1rX
ykkla3gC4LlzXN/otuTRZNs=
=SoUv
-----END PGP SIGNATURE-----

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
For all your IT requirements visit www.transtec.co.uk