Postfix and MailScanner logger

[Steve Freegard]

Alvaro Marín wrote:
> Hi,
> 
> Jason Ede escribió:
>> I've been thinking of writing something very similar for use here to bring together from all our mail servers with an additional field of receiving server name...  Any chance can post to the list if you manage to do this?
>>
>>> -----Original Message-----
>>> From: mailscanner-bounces at lists.mailscanner.info [mailto:mailscanner-
>>> bounces at lists.mailscanner.info] On Behalf Of Alvaro Marín
>>> Sent: 06 November 2009 10:37
>>> To: MailScanner discussion
>>> Subject: Re: Postfix and MailScanner logger
>>>
>>> Alex Broens escribió:
>>>> On 11/6/2009 11:21 AM, Antony Stone wrote:
>>>>> On Friday 06 November 2009 09:53, Alvaro Marín wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm searching for a logger tool to store in a MySQL table Postfix
>>> and
>>>>>> MailScanner's logs.
>>>>>> The idea is to save in a database values like:
>>>>>>
>>>>>> - sender IP
>>>>>> - sender address
>>>>>> - destination address
>>>>>> - if the mail was blocked by RBLs, rate-limit...
>>>>>> - the result of SpamAssassin test (from MailScanner)
>>>>>> - the result of MailScanner process (deliver, deliver+header or
>>> delete)
>>>>>> Anyone knows a tool like this or I've to do it myself? :)
>>>>> Have you considered http://mailwatch.sourceforge.net/ ?
>>>> mailwatch only stores data processed by MailScanner and not MTA data,
>>>> Sendmail, Postfix,etc data.
>>>>
>>>> rsyslog or syslog-ng seem to closer to what Alvaro is looking for.
>>> Yes, as you've said, Mailwatch is only for MailScanner.
>>>
>>> Anyway, I think that I have soo much traffic to have a solution like
>>> rsyslog/syslog-ng (in real time) or MW. I'm searching for a script that
>>> parses the logs (each hour, for example) and stores SMTP/MailScanner
>>> data in MySQL; then a web interface in PHP.
>>>
>>> I think that I'll have to program it myself :)
>>>
>>> Thanks!
>>>
>>> Regards,
>>>
>>> --
>>> Alvaro Marín Illera
>>> Hostalia Internet
>>> www.hostalia.com
> 
> The developer of the project has updated the link:
> 
> http://white-box.us/wp-content/uploads/2009/06/Maillog_Logger-0.2.0alpha1.zip
> 
> but I need something with more options. I'll try to do something and if
> I can, share it.
> 

Having written something very similar myself - from experience I can say 
that PHP + MySQL/PostgreSQL is very limited to scalability; even a low 
volume system will generate 300,000 records per day from a single server 
if you record every connection and status.  Scaling this for even a 
medium volume site with 20-30 connections per second over a couple of 
machines is extremely difficult.

My personal recommendation would be to look at Splunk - 
http://www.splunk.com/ as it's designed with exactly this in mind (it's 
also free for < 500Mb of index volume per day - so you can try it and see).

Simply add a single CustomFunction to MailScanner to log the extra lines 
that you'll need to syslog and this should be able to do everything you 
require.

Regards,
Steve.