maillog stops logging?

[Alan Dobkin]

On 1/28/2016 8:32 AM, Christophe GRENIER wrote:
> On Thu, 28 Jan 2016, Walt Thiessen wrote:
>> CentOS 7
>
> Have you updated systemd a few days ago ?
> https://bugzilla.redhat.com/show_bug.cgi?id=1292447

I'm glad to hear that I'm not the only one having this problem. I have 
been grappling with this issue for the past few months with some 
workarounds but not a permanent solution. I didn't make the connection 
with MailScanner until now. I have several systems running CentOS 7, but 
the one running MailScanner is the only one that regularly stops logging.

The crux of the problem is that RHEL/CentOS 7 uses systemd, which 
replaces the legacy System V startup scripts and runlevels. It also uses 
a new service called journald for event logging, which replaces syslog. 
There are hooks to maintain backward compatibility with rsyslogd using a 
socket so messages are still written to the standard /var/log/messages 
and /var/log/maillog files. Rsyslogd is still used by default even if 
this is a standalone system, so it is not only for remote logging.

It all works fairly well in most cases, but the journald log files use a 
binary format which is susceptible to corruption. Once they get corrupt, 
all logging stops, not just maillog. The "fix" is to delete the corrupt 
journal file and restart both journald and rsyslogd. I've enabled the 
following setting in my journald.conf file (under /etc/systemd) to force 
it to create a new file every hour as a precaution:

MaxFileSec=1h

(See man journald.conf for other settings.)

This does a pretty good job of making sure I never miss more than an 
hour of events due to a corrupt log file, but it is far from an ideal 
solution. To see if you have corrupt log files, use this command:

journalctl --verify

As Christophe pointed out, there is a documented bug and many reports of 
similar corruption issues with journald,
but there is no permanent fix as far as I am aware. Since this is only 
happening with MailScanner in my environment, I suspect it is caused by 
the Perl Syslog module somehow conflicting with journald. There is a 
another Perl module for journald, which may solve this problem. It is 
currently alpha code, and I'm not sure how much work is involved to make 
this work with MailScanner:

http://search.cpan.org/~lkundrak/Log-Journald-0.10/lib/Log/Journald.pm

This is a pretty serious issue, as consistent logging is critical for 
troubleshooting and identifying security issues. As it stands currently, 
I would consider RHEL/CentOS 7 an unsupported OS for new MailScanner 
installations until this problem is resolved.

Alan