performance impact of growing rules files

[Ramprasad]

On Fri, 2006-07-07 at 08:35 -0700, Ken A wrote:
> 
> Ramprasad wrote:
> > Hi,
> >   We scan mails for quiet a large number of domains ( around 1.5k
> > domains). The scanning happens on multiple identically configured MS
> > +postfix+SA  linux boxes behind load balancers
> > 
> >    For every domain that is added there will be entries in
> > spamcheck.rules spamaction.rules etc. Besides the domains will have
> > their own whitelists and blacklists which go into whitelist/blacklist
> > rules files. Already these have more than 10000 lines  each
> > 
> > I am not sure how this architecture will scale. Additional hardware is
> > not a problem , but the solution must scale
> > 
> >  Assume I have 10x more domains and traffic next year .. will there be a
> > performance hit because Mailscanner has to read such huge rules files.
> > What will be a 100% scalable architecture 
> 
> You certainly don't want a million rules stuffed into RAM every time 
> MailScanner starts up! If the largest number of rules you want 
> MailScanner to work with is 10,000 rules, then figure out how many rules 
> your average domain has, and divide up your MS boxes into groups based 
> on that. Then set the MX for domains [a-c] to MX1, domains [d-f] to MX2 
> and so on... Then have your load balancers handle which group of boxes 
> those MX's map requests to. This way you don't have an excessive number 
> of rules on any one group of MS boxes.

I tend to agree. But I would love for a way out without scattering the
MXes. 
 One problem is that would create an enormous maintenance overhead.
Today we have 10 identical MS boxes and maintained comfortably by a team
of 2. When we have differently configured machines , they will require
more attention

Thanks
Ram