Building a log gathering agent

[Hugo van der Kooij]

On Mon, 5 Mar 2007, Denis Beauchemin wrote:

> Hugo van der Kooij a écrit :
>>  Hi,
>>
>>  I am attempting to build a more concise overview of infections based on
>>  MailScanner logs.
>>
>>  The first stage is to write an agent to gather the logs. I wrote one that
>>  can understand ClamAV, F-Prot and McAfee output as far as I could test it
>>  on my logs.
>>
>>  If you are willing to assist me I would appreciate it if you can get the
>>  perl script from: http://hugo.vanderkooij.org/email/stats/maillog-virus.pl
>>
>>  You need the following perl modules:
>>      File::Basename;
>>      Getopt::Std;
>>      Parse::Syslog;
>>      Time::Local;
>>  (Centos users should be able to get all of the through `yum install`
>>  commands. But I will not document it at this time.)
>>
>>  Please run it against 1 of your logfiles and store the output. If you get
>>  anything other than a overview of the number of hits on scanners I would
>>  very appriciate it if you could send me the output file along with a
>>  filtered output of your logfile by email so I can anticipate other
>>  scanners and other detection strings.
>>
>>  For example:
>>  ./maillog-virus.pl -l /var/log/maillog.1 > /tmp/hvdkooij-output
>>  grep "Viruses marked as silent" > /tmp/hvdkooij-syslog
>>  tar -tzf /tmp/hvdkooij.tar.gz /tmp/hvdkooij-output /tmp/hvdkooij-syslog
>>
>>  If you use your own initials instead of mine I can keep the seperated more
>>  easily.
>>
>>  I will try to update the script based on your feedback the upcoming week.
>
> It's awfully slow on my 471,455 lines maillog: 3m36.936s;  I have a similar 
> script that goes through the same file in 0m0.196s!

If you prefilter the file with grep it propably is a bit faster. I suspect 
it is the added syslog parser that is the main killer. At present I don't 
need it to gather the details to learn to parse all those scanners. But 
later on I want to use the timestamps to build the accuracy into the 
system. And I don't trust myself (yet) to write a good timestamp parser.

> Besides it doesn't seems to know about "ClamAV Module:" nor "Bitdefender:" 
> (but you didn't mention this one as supported).

If you are willing to share a log file you could gzip it and send it of 
to me. Preferably after the grep shown above.

Hugo.

-- 
 	hvdkooij at vanderkooij.org	http://hugo.vanderkooij.org/
 	    This message is using 100% recycled electrons.

 	Some men see computers as they are and say "Windows"
 	I use computers with Linux and say "Why Windows?"
 		(Thanks JFK, for the insight.)