MS 4.50: way cool...
Jeff A. Earickson
jaearick at colby.edu
Tue Jan 24 13:40:01 GMT 2006
Gang,
My perl script, or Julian's, or Peter Peters' script is attached.
As you can see from the copyright comment at the top, those two
cooked this script up long ago. Then I've been modifying it for my own
environment over the years. The Batch timing stats were this
month's addition due to HighRes.
Note: syslogging on my systems sends *everything* to a single file:
MailScanner logging, sendmail, the works. The script parses out
sendmail and MS logging and analyzes the results. I hope it is
useful.
I really should get MailWatch or vispan going...
Jeff Earickson
Colby College
On Tue, 24 Jan 2006, Martin Hepworth wrote:
> Date: Tue, 24 Jan 2006 11:10:00 -0000
> From: Martin Hepworth <martinh at solid-state-logic.com>
> Reply-To: MailScanner discussion <mailscanner at lists.mailscanner.info>
> To: 'MailScanner discussion' <mailscanner at lists.mailscanner.info>
> Subject: RE: MS 4.50: way cool...
>
> Hmm
>
> Maybe jeff's got his own stats analysis engine (or runs vispan or
> something..)
>
> --
> Martin Hepworth
> Snr Systems Administrator
> Solid State Logic
> Tel: +44 (0)1865 842300
>
>> -----Original Message-----
>> From: mailscanner-bounces at lists.mailscanner.info [mailto:mailscanner-
>> bounces at lists.mailscanner.info] On Behalf Of Joost Waversveld
>> Sent: 24 January 2006 10:58
>> To: mailscanner at lists.mailscanner.info
>> Subject: RE: MS 4.50: way cool...
>>
>> I ran Version 4.50.8 for a while, I saw the timings in the maillog.
>> When I saw this post,
>> I thought "Cool, that's a nice feature!". I tried it and I did not saw
>> the timings. The
>> output of analyse_SpamAssassin_cache was:
>>
>> =============================================================
>>
>> [root at server MailScanner]# analyse_SpamAssassin_cache --------- TOTALS
>> ---------
>> Total records: 42
>> First seen (oldest): 137447 sec
>> First seen (newest): 309 sec
>> Last seen (oldest): 137447 sec
>> Last seen (newest): 309 sec
>> Cache Hit Rate 4%
>> -------- NON-SPAM --------
>> Total records: 4
>> First seen (oldest): 1587 sec
>> First seen (newest): 1345 sec
>> Last seen (oldest): 1587 sec
>> Last seen (newest): 1345 sec
>> -------- LOW-SPAM --------
>> Total records: 0
>> First seen (oldest): 0 sec
>> First seen (newest): 0 sec
>> Last seen (oldest): 0 sec
>> Last seen (newest): 0 sec
>> ------- HIGH-SPAM --------
>> Total records: 27
>> First seen (oldest): 10793 sec
>> First seen (newest): 309 sec
>> Last seen (oldest): 10793 sec
>> Last seen (newest): 309 sec
>> -------- VIRUSES --------
>> Total records: 11
>> First seen (oldest): 137447 sec
>> First seen (newest): 3896 sec
>> Last seen (oldest): 137447 sec
>> Last seen (newest): 3896 sec
>> ----- TOP 5 HASHES -------
>> MD5 COUNT FIRST LAST
>> 4241bc4eef8c5c2ed34c112b2401397d 2 8005 1754
>> 12bc9faf120bea4712776cabfbeca4a5 2 5363 5356
>> =============================================================
>>
>> I was disappointed, but decided to install the latest BETA available on
>> mailscanner.info.
>> Maybe it was an newer version. But after the upgrade I still see the
>> same output from
>> this command. I do not see the timings.
>>
>> What am I doing wrong??
>>
>> Best regards,
>>
>> Joost Waversveld
>>
>>
>> ----- Message from martinh at solid-state-logic.com ---------
>> Date: Tue, 24 Jan 2006 08:53:41 -0000
>> From: Martin Hepworth <martinh at solid-state-logic.com>
>> Reply-To: MailScanner discussion <mailscanner at lists.mailscanner.info>
>> Subject: RE: MS 4.50: way cool...
>> To: 'MailScanner discussion' <mailscanner at lists.mailscanner.info>
>>
>>
>>> Peter
>>>
>>> In 4.50.5 (I think it first appeared in that beta) there's a script in
>> the
>>> bin directory called "analyse_spamassassin_cache"
>>>
>>> --
>>> Martin Hepworth
>>> Snr Systems Administrator
>>> Solid State Logic
>>> Tel: +44 (0)1865 842300
>>>
>>>> -----Original Message-----
>>>> From: mailscanner-bounces at lists.mailscanner.info [mailto:mailscanner-
>>>> bounces at lists.mailscanner.info] On Behalf Of Peter Russell
>>>> Sent: 24 January 2006 01:50
>>>> To: MailScanner discussion
>>>> Subject: Re: MS 4.50: way cool...
>>>>
>>>> How did you do this report?
>>>>
>>>>
>>>>
>>>>
>>>> Jeff A. Earickson wrote:
>>>>> Julian,
>>>>> In my nightly report at 4 AM last night, the cache hit rate
>>>>> was 72%. Wowee!
>>>>>
>>>>> With the HighRes timings, I use that information to compute how
>>>>> long batches take, and some statistics. From yesterday:
>>>>>
>>>>> ===Mailscanner Summaries:
>>>>> Total messages scanned: 28180
>>>>> Total Message Batches: 20368
>>>>> Average Messages per Batch: 1.38
>>>>> Minimum Batch Time (sec): 2.57
>>>>> Maximum Batch Time (sec): 185.12
>>>>> Average Batch Time (sec): 8.45
>>>>> Total MBytes scanned: 1011.47
>>>>> Total virii detected: 31
>>>>> Total spams tagged: 4702
>>>>> Total spams delivered: 1679
>>>>> Total spams deleted: 3274
>>>>>
>>>>> The batch timing gives a good overall clue as to the speed/efficiency
>>>>> of one's system. Thanks!
>>>>>
>>>>> Jeff Earickson
>>>>> Colby College
>>>> --
>>>> MailScanner mailing list
>>>> MailScanner at lists.mailscanner.info
>>>> http://lists.mailscanner.info/mailman/listinfo/mailscanner
>>>>
>>>> Before posting, read http://wiki.mailscanner.info/posting
>>>>
>>>> Support MailScanner development - buy the book off the website!
>>>
>>>
>>> **********************************************************************
>>>
>>> This email and any files transmitted with it are confidential and
>>> intended solely for the use of the individual or entity to whom they
>>> are addressed. If you have received this email in error please notify
>>> the system manager.
>>>
>>> This footnote confirms that this email message has been swept
>>> for the presence of computer viruses and is believed to be clean.
>>>
>>> **********************************************************************
>>>
>>> --
>>> MailScanner mailing list
>>> MailScanner at lists.mailscanner.info
>>> http://lists.mailscanner.info/mailman/listinfo/mailscanner
>>>
>>> Before posting, read http://wiki.mailscanner.info/posting
>>>
>>> Support MailScanner development - buy the book off the website!
>>>
>>
>>
>> ----- End message from martinh at solid-state-logic.com -----
>>
>>
>> --
>> MailScanner mailing list
>> MailScanner at lists.mailscanner.info
>> http://lists.mailscanner.info/mailman/listinfo/mailscanner
>>
>> Before posting, read http://wiki.mailscanner.info/posting
>>
>> Support MailScanner development - buy the book off the website!
>
>
> **********************************************************************
>
> This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they
> are addressed. If you have received this email in error please notify
> the system manager.
>
> This footnote confirms that this email message has been swept
> for the presence of computer viruses and is believed to be clean.
>
> **********************************************************************
>
> --
> MailScanner mailing list
> MailScanner at lists.mailscanner.info
> http://lists.mailscanner.info/mailman/listinfo/mailscanner
>
> Before posting, read http://wiki.mailscanner.info/posting
>
> Support MailScanner development - buy the book off the website!
>
-------------- next part --------------
#!/usr/bin/perl
#
# Analog4MailScanner - Log Analyzer for MailScanner from Julian Field
# Copyright (C) 2002 Peter Peters, universiteit Twente,
# Enschede, The Netherlands
#
# Main program....
#unshift @INC, "/opt/mailscanner/bin";
#show only this number of viruses, 0 = all
$limit = 20;
$Logfile = "/var/adm/syslog/0";
$Logfile = $ARGV[0] if defined $ARGV[0];
# counters for sendmail
$TotalMails = 0;
$Discarded = 0;
$REFUSED_mail_abuse = 0;
$REFUSED_spamcop = 0;
$REFUSED_spamhaus_sbl = 0;
$REFUSED_spamhaus_xbl = 0;
$REFUSED_relays_ordb = 0;
$Connection_rate_limit = 0;
$Too_many_connections = 0;
$Banned_spam_domain = 0;
$Banned_spam_IP = 0;
$Banned_spammer = 0;
$Pregreeting_punts = 0;
#$Fix_reverse_DNS = 0;
#$Does_not_resolve = 0;
#$Possibly_forged = 0;
#---counters for mailscanner
$BatchCounter = 0;
$MinBatchTime = 999999.0;
$MaxBatchTime = 0.0;
$TotalBatchTime = 0.0;
$TotalMsgsScanned = 0;
$TotalBytesScanned = 0;
#$TotalSecondsScanned = 0;
$TotalViruses = 0;
$TotalSpam = 0;
$TotalSpamsDeleted = 0;
$TotalSpamsDelivered = 0;
$TotalAssassin = 0;
$TotalAssassinTimeouts = 0;
$TotalAssassinScore = 0;
$TotalSpamCop = 0;
$TotalSpamHaus = 0;
$TotalORDB_RBL = 0;
$TotalCBL = 0;
$TotalDSBL = 0;
$TotalNJABL = 0;
$TotalSBL_XBL = 0;
if ($Logfile =~ /\.gz$/)
{
open(LOG, "zcat $Logfile |")
or (warn("Cannot access log file $file, skipping, $!"), next);
}
else
{
open(LOG, $Logfile)
or (warn("Cannot access log file $file, skipping, $!"), next);
}
while(<LOG>)
{
chomp;
if(/sendmail/)
{
$TotalMails += $1 if /nrcpts=(\d+),/;
if(/ruleset=check_rcpt/)
{
$Discarded++ if /discard$/;
$REFUSED_mail_abuse++ if/See http:\/\/mail-abuse.com\/cgi-bin\/lookup/;
#$REFUSED_spamcop++ if/spamcop.net/;
$REFUSED_spamcop++ if/See http:\/\/spamcop.net\/bl.shtml for further information/;
$REFUSED_spamhaus_sbl++ if/http:\/\/www.abuse.net\/sbl.phtml/;
$REFUSED_spamhaus_xbl++ if/http:\/\/cbl.abuseat.org/;
$REFUSED_relays_ordb++ if/ordb.org/;
$Banned_spam_domain++ if/Domain banned because of SPAM/;
$Banned_spam_IP++ if/IP number banned because of SPAM/;
$Banned_spammer++ if/Mail from SPAMMERs rejected/;
}
if(/\[(\S+)\] due to pre-greeting traffic/)
{
$pregreet{$1}++;
$Pregreeting_punts++;
}
if(/relay=\[(\S+)\], discard$/)
{
$Discarded++;
$discard{$1}++;
}
if(/\[(\S+)\], reject=421 4.3.2 Connection rate limit exceeded/)
{
$Connection_rate_limit++;
$ratelimit{$1}++;
}
if(/\[(\S+)\], reject=421 4.3.2 Too many open connections/)
{
$Too_many_connections++;
$connlimit{$1}++;
}
#---from require_rdns.m4
# if(/Fix reverse DNS for (\S+),/)
# {
# $Fix_reverse_DNS++;
# $fixdns{$1}++;
# }
if(/Client IP address (\S+) does not resolve/)
{
$Does_not_resolve++;
}
if(/Possibly forged hostname for (\S+)/)
{
$Possibly_forged++;
}
next;
}
if(/mailscanner/i)
{
#---v4
if(/New Batch: Scanning (\d+) messages, (\d+) bytes/)
{
$BatchCounter++;
$TotalMsgsScanned += $1;
$TotalBytesScanned += $2;
}
if(/Batch processed in (\d+).(\d+) seconds/)
{
$batchtime = sprintf("%d.%d", $1, $2);
$MinBatchTime = $batchtime if ($batchtime < $MinBatchTime);
$MaxBatchTime = $batchtime if ($batchtime > $MaxBatchTime);
$TotalBatchTime += $batchtime;
}
$TotalViruses++ if/>>> Virus/;
$TotalViruses++ if/INFECTED::/;
$TotalSpamsDeleted++ if/actions are delete/;
$TotalSpamsDelivered++ if/actions are deliver/;
$TotalSpamsDelivered++ if/actions are store,deliver/;
$TotalAssassinTimeouts++ if/SpamAssassin timed out/;
if(/is spam/)
{
$TotalSpam++;
$TotalSpamCop++ if/spamcop.net/;
$TotalSpamHaus++ if/spamhaus.org/;
$TotalORDB_RBL++ if/ORDB-RBL/;
$TotalCBL++ if/CBL/;
$TotalDSBL++ if/DSBL/;
$TotalNJABL++ if/NJABL/;
$TotalSBL_XBL++ if/SBL+XBL/;
if (/SpamAssassin/)
{
$TotalAssassinScore += $1+($2/100) if /score=(\d+)\.(\d+),/;
$TotalAssassin++;
}
}
#---phishing fraud
if(/Found phishing fraud from (\S+) claiming to be (\S+)/)
{
$tag = $1 . "\t" . $2;
$Phishing{$tag}++;
}
if(/ClamAV: (\S+) contains (\S+) $/)
{
$tag = "ClamAV" . "\t" . $2;
$Virus{$tag}++;
}
#---sophos or clamavmodule output (perl)
if(/(\S+)::INFECTED:: (\S+)::/)
{
$tag = $1 . "\t" . $2;
$Virus{$tag}++;
}
$Virusfrom{$2}++ if /Infected message (\S+) came from (\S+)/;
#--- from mailscanner filename.rules.conf
$Rule{$1}++ if / Possible (.*)/;
# $Virus{$1}++ if / in email in (\S+)/;
# $Virus{$1}++ if / attack in (\S+)/;
# $Virus{$1}++ if / often mailicious in (\S+)/;
# $Virus{$1}++ if / extension in (\S+)/;
# $Virus{$1}++ if / part of it (\S+)/;
}
}
close LOG;
print "===Sendmail Summaries:\n";
print "Total recipients: $TotalMails\n";
print "Total Discards: $Discarded\n";
print "Total Refused by MAPS: $REFUSED_mail_abuse\n";
print "Total Refused by Spamcop: $REFUSED_spamcop\n";
print "Total Refused by SpamHaus SBL: $REFUSED_spamhaus_sbl\n";
print "Total Refused by SpamHaus XBL: $REFUSED_spamhaus_xbl\n";
print "Total Refused by ORDB: $REFUSED_relays_ordb\n";
print "Total Connection rate limit: $Connection_rate_limit\n";
print "Total Too many connections: $Too_many_connections\n";
print "Total Banned by Domain: $Banned_spam_domain\n";
print "Total Banned by IP: $Banned_spam_IP\n";
print "Total Banned Spammers: $Banned_spammer\n";
print "Total Pre-Greeting Punts: $Pregreeting_punts\n";
#print "Total RDNS Fix Reverse DNS: $Fix_reverse_DNS\n";
#print "Total RDNS no resolve: $Does_not_resolve\n";
#print "Total RDNS Possible forgery: $Possibly_forged\n";
print "\n===Mailscanner Summaries:\n";
print "Total messages scanned: $TotalMsgsScanned\n";
print "Total Message Batches: $BatchCounter\n";
$AveMessageBatch = $TotalMsgsScanned/$BatchCounter;
printf("Average Messages per Batch: %.2f\n",$AveMessageBatch);
print "Minimum Batch Time (sec): $MinBatchTime\n";
print "Maximum Batch Time (sec): $MaxBatchTime\n";
$AveBatchTime = $TotalBatchTime/$BatchCounter;
printf("Average Batch Time (sec): %.2f\n",$AveBatchTime);
$MBytes = $TotalBytesScanned/(1024 * 1024);
printf "Total MBytes scanned: %-6.2f\n", $MBytes;
#print "Total Seconds scanned: $TotalSecondsScanned\n";
print "Total virii detected: $TotalViruses\n";
print "Total spams tagged: $TotalSpam\n";
print "Total spams delivered: $TotalSpamsDelivered\n";
print "Total spams deleted: $TotalSpamsDeleted\n";
print "\n";
print "Total SpamAssassin: $TotalAssassin\n";
print "Total SpamAssassin Timeouts: $TotalAssassinTimeouts\n";
#printf "Total SpamAssassin score: %-8.2f\n", $TotalAssassinScore;
$AverageScore = $TotalAssassinScore/$TotalAssassin;
printf "Avg SpamAssassin score: %-6.2f\n", $AverageScore;
#print "\n";
print "Total MailScanner SpamCop: $TotalSpamCop\n";
print "Total MailScanner SpamHaus: $TotalSpamHaus\n";
print "Total MailScanner ORDB-RBL: $TotalORDB_RBL\n";
print "Total MailScanner CBL: $TotalCBL\n";
print "Total MailScanner DSBL: $TotalDSBL\n";
print "Total MailScanner NJABL: $TotalNJABL\n";
print "Total MailScanner SBL+XBL: $TotalSBL_XBL\n";
print "\n=== Pre-greeting Rejections";
print " (top $limit)" if $limit;
print ":\n";
@pregreets = sort {$pregreet{$b} <=> $pregreet{$a}} keys(%pregreet);
@pregreets = splice(@pregreets,0,$limit) if $limit;
for $pregid (@pregreets)
{
printf ("%6d: %-s\n",$pregreet{$pregid},$pregid);
}
print "\n=== Connection rate limit exceeded";
print " (top $limit)" if $limit;
print ":\n";
@ratelimits = sort {$ratelimit{$b} <=> $ratelimit{$a}} keys(%ratelimit);
@ratelimits = splice(@ratelimits,0,$limit) if $limit;
for $rateid (@ratelimits)
{
printf ("%6d: %-s\n",$ratelimit{$rateid},$rateid);
}
print "\n=== Open Connections limit exceeded";
print " (top $limit)" if $limit;
print ":\n";
@connlimits = sort {$connlimit{$b} <=> $connlimit{$a}} keys(%connlimit);
@connlimits = splice(@connlimits,0,$limit) if $limit;
for $connid (@connlimits)
{
printf ("%6d: %-s\n",$connlimit{$connid},$connid);
}
#print "\n=== Fix Reverse DNS rejections";
#print " (top $limit)" if $limit;
#print ":\n";
#@fixdnss = sort {$fixdns{$b} <=> $fixdns{$a}} keys(%fixdns);
#@fixdnss = splice(@fixdnss,0,$limit) if $limit;
#for $fixid (@fixdnss)
#{
# printf ("%6d: %-s\n",$fixdns{$fixid},$fixid);
#}
print "\n=== Discards";
print " (top $limit)" if $limit;
print ":\n";
@discards = sort {$discard{$b} <=> $discard{$a}} keys(%discard);
@discards = splice(@discards,0,$limit) if $limit;
for $discardid (@discards)
{
printf ("%6d: %-s\n",$discard{$discardid},$discardid);
}
print "\n=== Virus Senders";
print " (top $limit)" if $limit;
print ":\n";
@viruses = sort {$Virusfrom{$b} <=> $Virusfrom{$a}} keys(%Virusfrom);
@viruses = splice(@viruses,0,$limit) if $limit;
for $virusid (@viruses)
{
#printf ("%29s: %d\n",$virusid,$Virus{$virusid});
printf ("%6d: %-s\n",$Virusfrom{$virusid},$virusid);
}
print "\n=== Viruses found";
print " (top $limit)" if $limit;
print ":\n";
@viruses = sort {$Virus{$b} <=> $Virus{$a}} keys(%Virus);
@viruses = splice(@viruses,0,$limit) if $limit;
for $virusid (@viruses)
{
#printf ("%29s: %d\n",$virusid,$Virus{$virusid});
printf ("%6d: %-s\n",$Virus{$virusid},$virusid);
}
print "\n=== Phishing fraud URLs";
print " (top $limit)" if $limit;
print ":\n";
@phish = sort {$Phishing{$b} <=> $Phishing{$a}} keys(%Phishing);
@phish = splice(@phish,0,$limit) if $limit;
for $phishid (@phish)
{
printf ("%6d: %-s\n",$Phishing{$phishid},$phishid);
}
# print "\n=== Mailscanner Rules Complaints Found";
# print " (top $limit)" if $limit;
# print ":\n";
# @rules = sort {$Rule{$b} <=> $Rule{$a}} keys(%Rule);
# @rules = splice(@rules,0,$limit) if $limit;
# for $ruleid (@rules)
# {
# #printf ("%29s: %d\n",$ruleid,$Rule{$ruleid});
# printf ("%6d: %-s\n",$Rule{$ruleid},$ruleid);
# }
More information about the MailScanner
mailing list