performance impact of growing rules files

[Ugo Bellavance]

Drew Marshall wrote:
> On Fri, July 7, 2006 16:35, Ken A wrote:
>>
>> Ramprasad wrote:
>>> Hi,
>>>   We scan mails for quiet a large number of domains ( around 1.5k
>>> domains). The scanning happens on multiple identically configured MS
>>> +postfix+SA  linux boxes behind load balancers
>>>
>>>    For every domain that is added there will be entries in
>>> spamcheck.rules spamaction.rules etc. Besides the domains will have
>>> their own whitelists and blacklists which go into whitelist/blacklist
>>> rules files. Already these have more than 10000 lines  each
>>>
>>> I am not sure how this architecture will scale. Additional hardware is
>>> not a problem , but the solution must scale
>>>
>>>  Assume I have 10x more domains and traffic next year .. will there be a
>>> performance hit because Mailscanner has to read such huge rules files.
>>> What will be a 100% scalable architecture
>> You certainly don't want a million rules stuffed into RAM every time
>> MailScanner starts up! If the largest number of rules you want
>> MailScanner to work with is 10,000 rules, then figure out how many rules
>> your average domain has, and divide up your MS boxes into groups based
>> on that. Then set the MX for domains [a-c] to MX1, domains [d-f] to MX2
>> and so on... Then have your load balancers handle which group of boxes
>> those MX's map requests to. This way you don't have an excessive number
>> of rules on any one group of MS boxes.
> 
> I think (Although couldn't guarantee it) there is a more efficient rules
> loading method using custom functions. It was discussed previously (I
> think) so a search of the list might give you the details. Other wise I
> think Julian would be the best person to confirm this and he is on holiday
> at the moment.
> 
> Drew
> 
> 

Yes, Julian could do something for you using Custom Functions...