Idea for a handy new feature

[Steve Freegard]

-----Original Message-----
From: Julian Field
To: MAILSCANNER at JISCMAIL.AC.UK
Sent: 11/8/03 5:50 PM
Subject: Re: Idea for a handy new feature

At 16:29 08/11/2003, you wrote:
>Hi All,
>
>I've just thought of a new feature for MailScanner that I'd personally
find
>really useful.
>
>In light of the recent problems with Razor, Pyzor, DCC and the RBLs due
to
>DDOS attacks and the like, it is difficult to tell sometimes if
>MailScanner's throughput of messages is as great as it could be due to
these
>outside influences.
>
>A number of people have asked before if it is possible to give a time
to
>process for each message through the various virus/spam/rbl checks -
which
>is a problem as MailScanner batches the messages together.
>
>I think I've come up with a way to achieve this relativly easily that
will
>provide stats for MailScanner-MRTG, Mailstats and MailWatch, and
provide a
>some benchmarks that can be used by a MailScanner admin to tune his/her
>hardware and settings for maximum performance.
>
>Here's my theory:
>
>- MailScanner picks up message(s) from mqueue.in
>maillog: New Batch: Scanning 7 messages, 30160 bytes
>
>Variables to store:
>  batch[Number of messages]
>  batch[Total bytes]
>  batch[Start] = timestamp
>
>- MailScanner goes through the RBL, Spam, Virus, Attachment and Message
>Processing stages
>
>For each process stage record the variables:
>  <<process_name>>[Start] = timestamp
>  <<process_name>>[Finish] = timestamp
>
>- Once processing is finished, and just prior to moving the messages to
>mqueue
>
>Store:
>  batch[Finish] = timestamp
>
>Calculate:
>  batch[Finish] - batch[Start] = batch[Total Time]
>  batch[Total Time] / batch[Total Bytes] = batch[Bytes Per Second]
>
>  For each process:
>  <<process_name>>[Finish] - <<process_name>>[Start] =
<<process_name>>[Total
>Time]
>  <<process_name>>[Total Time] / batch[Total Bytes] =
<<process_name>>[Bytes
>Per Second]
>
>Then for each message in the batch (using the $message object) store:
>   $message->{total_time} = batch[Bytes Per Second] * $message->{size}
>   For each process:
>   $message->{<<process_name>>_time} = <<process_name>>[Bytes Per
Second] *
>$message->{size}

>>> What do you want to do with this? Just store it for MailWatch?

Yes - I would like to be able to store per-message stats in the database if
I can.

>- Move messages to mqueue for delivery
>maillog: Uninfected: Delievered 7 messages, 30160 bytes [Total
tt.ttt at nn.nnn
>bps, RBL tt.ttt at n.nnn bps, SA tt.ttt at nn.nnn bps, Virus tt.ttt at nn.nnn
bps,
>Content tt.ttt at nn.nnn bps] (tt = time, nn = bytes per second)

>>> I would do the times and speeds as a separate log entry. The
>>> calculations
>>> aren't quite as simple as you suggest, due to things like the time taken
>>> to
>>> reconstruct modified messages from their attachments and things like
>>> that.
>>> There are all sorts of "overheads" like that to be considered. So it
>>> might
>>> be good to add an "overheads" time to the log output, otherwise the
>>> figures
>>> won't add up. There will be plenty of rounding errors anyway, which
>>> won't help.

Sounds good - it doesn't have to be perfect ;-)

>Thoughts??
>
>Kind regards,
>Steve.
>
>--
>This email and any files transmitted with it are confidential and
>intended solely for the use of the individual or entity to whom they
>are addressed. If you have received this email in error please notify
>the sender and delete the message from your mailbox.
>
>This footnote also confirms that this email message has been swept by
>MailScanner (www.mailscanner.info) for the presence of computer
viruses.

> --
> Julian Field
> www.MailScanner.info
> Professional Support Services at www.MailScanner.biz
> MailScanner thanks transtec Computers for their support
> PGP footprint: EE81 D763 3DB0 0BFD E1DC  7222 11F6 5947 1415 B654

Cheers,
Steve.

--
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the sender and delete the message from your mailbox.

This footnote also confirms that this email message has been swept by
MailScanner (www.mailscanner.info) for the presence of computer viruses.