performance impact of growing rules files

[Drew Marshall]

On Fri, July 7, 2006 16:35, Ken A wrote:
>
>
> Ramprasad wrote:
>> Hi,
>>   We scan mails for quiet a large number of domains ( around 1.5k
>> domains). The scanning happens on multiple identically configured MS
>> +postfix+SA  linux boxes behind load balancers
>>
>>    For every domain that is added there will be entries in
>> spamcheck.rules spamaction.rules etc. Besides the domains will have
>> their own whitelists and blacklists which go into whitelist/blacklist
>> rules files. Already these have more than 10000 lines  each
>>
>> I am not sure how this architecture will scale. Additional hardware is
>> not a problem , but the solution must scale
>>
>>  Assume I have 10x more domains and traffic next year .. will there be a
>> performance hit because Mailscanner has to read such huge rules files.
>> What will be a 100% scalable architecture
>
> You certainly don't want a million rules stuffed into RAM every time
> MailScanner starts up! If the largest number of rules you want
> MailScanner to work with is 10,000 rules, then figure out how many rules
> your average domain has, and divide up your MS boxes into groups based
> on that. Then set the MX for domains [a-c] to MX1, domains [d-f] to MX2
> and so on... Then have your load balancers handle which group of boxes
> those MX's map requests to. This way you don't have an excessive number
> of rules on any one group of MS boxes.

I think (Although couldn't guarantee it) there is a more efficient rules
loading method using custom functions. It was discussed previously (I
think) so a search of the list might give you the details. Other wise I
think Julian would be the best person to confirm this and he is on holiday
at the moment.

Drew


-- 
In line with our policy, this message has 
been scanned for viruses and dangerous 
content by MailScanner, and is believed to be clean.
www.themarshalls.co.uk/policy