Any one else notice...

Sat Jun 1 20:20:05 IST 2002

I've spent the whole day staring at code today...
and I *think* I have found the problem. What changed between the 2 versions
Kelly first reported was that the code was doing 2 extra "-f" checks on
each message. It's quite possible that some OS's don't cache directory
metadata (i.e. the data about the files in the dir) very cleverly when that
data is constantly changing, as it does when I am processing the queue.

So I've optimised out those "-f" checks, and have also been through the
whole of the rest of the code optimising all the "-f" (file existence)
checks and all the "-d" (directory existence) checks.

So if I'm right, it should now run faster than it ever did :-)

All I need now is a willing tester who has got a nice heavily loaded mail
server.
Kelly perhaps?
If Kelly can't then any other offers for testing would be very much
appreciated.

Another thing I've been working on is a much-improved RPM. It will now only
install any needed Perl modules if you don't have a recent-enough version
installed already. And if you find that the TNEF expander isn't coping with
a lot of your Outlook attachments, you now have the option to use a Perl
module TNEF expander instead of the external binary one.

I think there's one or two other things as well, but that's enough to keep
you happy for now :-)

Jules.

At 21:18 31/05/2002, you wrote:
>unfortunatly were already running a name server on the same machine, and
>have 512megs ram in it. Its only a p3 500, but it was keeping up with the
>load great before. I have my sendmail set to do rbl checks, i have spam
>assassin's user_prefs set with skip_rbl_checks 1 and ignore_rbl_checks 1 and
>commented the rbl checks out in mailscanner (this is how i was running
>already) Something changed in the code in the last release or so which
>changed some timing. Julian mentioned to me that spam checking is now done
>on seprate forks (seprate processes) and hes not sure if thats where the
>bottleneck is. Although i did some testing with spam checks = no and it
>improved a little but not as much as it should. In version 3.13-2 witrh spam
>checks off i can process 2000 messages in like 5 minutes, or less. with the
>new version it took well over 15 minutes.
>
>Just some more information for you guys.
>thanks.
>
>----- Original Message -----
>From: "Jeff A. Earickson" <jaearick at COLBY.EDU>
>To: <MAILSCANNER at JISCMAIL.AC.UK>
>Sent: Friday, May 31, 2002 3:49 PM
>Subject: Re: Any one else notice...
>
>
> > Hi,
> >
> >    I'll weigh in on this thread too.  Last Monday, I moved our mail
>service
> > to a Sun E220R, 2 cpus, running Solaris 8.  I'm running mailscanner with
> > spamassassin turned on.  I have all of the "Spam List" options in
> > mailscanner.conf commented out.  I have Spamcop assigned a non-zero score
> > in spamassassin, so I hope/think SA is using spamcop (I'm not sure yet).
> > We use RBL+ in sendmail, and we subscribe to it transfer mode, so RBL+
> > mail gets rejected before getting to mailscanner.  I have the delivery
> > mode in mailscanner set to "queue".  We use Sophos.  We process roughly
> > 20K messages a day.  Result:  the system works great, no slowdowns, no
> > clogged queues, nothing but bliss.  Julian should be knighted, IMHO.
> >
> >    Most of what I see in this thread sounds like DNS slowdowns.  Here's
> > my advice:
> >
> >   * run a modern version bind on your mail server, at least in caching
>mode,
> >     to handle the DNS lookups for you.  If you use RBL+ or other
>zone-transfer
> >     mode DNS blocklists, do the zone transfers to the mail server, so
> >     DNS queries never leave the box for RBL+.  You will probably have to
> >     pay money to get zone transfers.  As a part of running bind on your
>mail
> >     server, make sure /etc/resolv.conf is configured so that the first
>entry
> >     is the external interface (not loopback) of the mail server.  Here is
>my
> >     resolv.conf for my server, emerald:
> >
> > domain colby.edu
> > nameserver 137.146.210.52   # emerald, this host, not loopback -- for RBL+
> > nameserver 137.146.210.46   # opal
> > nameserver 137.146.210.45   # ruby
> > nameserver 139.140.1.1      # polar.bowdoin.edu
> > nameserver 204.70.128.1     # ns.cw.net
> >
> >     Any DNS lookup on emerald goes to the local cache first, then other
> >     local machines, then remotely.
> >
> > *   Have lots of memory in the machine for named to use.  Named is using
> >     about 140 MB of resident memory on my machine right now.  If you are
> >     using bind 9.X (you should be) and have a multi-cpu machine, let bind
> >     run threads on all cpus.
> >
> > *   If you are doing DNS spam-blocking, do it in sendmail.  Reject the
> >     stuff before it gets to mailscanner or spamassassin.
> >
> > *   Comment out some the "Spam List" lookups in mailscanner and see if
> >     that helps.  Fewer DNS lookups, especially to a remote site that is
> >     overloaded (like relays.ordb.org perhaps), may be a bottleneck.
>Likewise,
> >     check the config for SA and try to control DNS lookups there too.
> >
> > *   If you are running Solaris, shut off nscd!!  This code is a real
> >     DNS bottleneck for a system doing beaucoup lookups.  When we first
> >     moved our Apache web server to Sun, we were getting glacial response
> >     times to webpage requests.  I found a technote at the Apache site
>about
> >     nscd problems.  Turned it off, and things ran fast after that.
> >     The same advice applies to other Solaris apps doing massive DNS,
> >     and nscd could appear on other versions of UNIX.  Let bind do the
> >     work instead.
> >
> > Of course our students are gone right now.  The system could blow up when
> > they return next Fall.
> >
> > ** Jeff A. Earickson, Ph.D                         PHONE: 207-872-3659
> > ** Senior UNIX Sysadmin, Information Technology    EMAIL:
>jaearick at colby.edu
> > ** Colby College, 4214 Mayflower Hill,               FAX: 207-872-3076
> > ** Waterville ME, 04901-8842
> > --------------------------------------------------------------------------
>--
> >

--
Julian Field                Teaching Systems Manager
jkf at ecs.soton.ac.uk         Dept. of Electronics & Computer Science
Tel. 023 8059 2817          University of Southampton
                             Southampton SO17 1BJ