Hard Lock

[Dhawal Doshy]

    [ The following text is in the "utf-8" character set. ]
    [ Your display is set for the "US-ASCII" character set.  ]
    [ Some characters may be displayed incorrectly. ]

Information Services writes: 

> I posted this on the CentOS forumn, and received no responses.  I am really
> not sure where the issue is coming from, but my mailscanner systems
> occassionally lock up, and I have been unable to resolve why this is
> happening.  can anyone shed some light on this for me. 
> 
> here is what I have on the centos forumn 
> 
> ----------------------------------------------------------------------------
> I have two CentOS 4.1 mailscanner servers that like to lock up for some
> unknown reason. 
> 
> Both machines are Dell Optiplex G1's with powerleaps installed to make them
> 1.1Ghz processors.
> 512 RAM 
> 
> As I said above, running 4.1 
> 
> Other information: 
> 
> mailscanner-4.47.4-2
> clamav-0.87
> inoculate
> mailwatch-1.0.1
> sendmail-8.13.4-1
> webmin-1.210
> phpmyadmin-2.6.3-pl1
> spamassassin-3.1.0 
> 
> 
> Both machines are setup this way. I have two issues with both, and cannot
> figure them out. 
> 
> First, 
> 
> When I reboot the systems, it takes about 20 minutes before the login screen
> appears. I am able to shell into the systems themselves and work on them,
> but I would like to resolve why they don't bring the login screen up right
> away after the boot process. The GUI either sets at the blank screen with
> the black curser outlined in white and is an 'X' symbol, or at the progress
> bar at 100 percent until it finally shows the login screen. 
> 
> Issue 2: 
> 
> I have been havin problems with both servers locking up. One server more
> than the other. 
> 
> Here is information I have from the logs and am not sure what do to about
> them. Today I downloaded chkrootkit to see if my systems have been tampered
> with, but not sure what to make of a line of information. Below is
> information from my var/log/messages, var/log/maillog, and piped info from
> running the chkrootkit. 
> 
> /VAR/LOG/MAILLOG
> ------------------------------
> Dec 23 00:37:55 wks-lin9 MailScanner[24300]: Started SQL Logging child
> Dec 23 00:38:00 wks-lin9 MailScanner[24300]: Logging message jBN6bgKn030181
> to SQL
> Dec 23 00:38:00 wks-lin9 MailScanner[24300]: Logging message jBN6bfED030180
> to SQL
> Dec 23 00:38:00 wks-lin9 MailScanner[30202]: jBN6bgKn030181: Logged to
> MailWatch SQL
> Dec 23 00:38:00 wks-lin9 MailScanner[30202]: jBN6bfED030180: Logged to
> MailWatch SQL
> Dec 23 00:40:01 wks-lin9 sendmail[30089]: jBMLmqie001220: timeout waiting
> for input from jacobson-fw.jacobsonco.com. during client greeting
> Dec 23 00:40:01 wks-lin9 sendmail[30089]: jBMLmqie001220: to=<
> Jarrod.Carley at jacobsonco.com>,<mark.fincham at jacobsonco.com>, delay=08:51:09,
> xdelay=00:05:00, mailer=esmtp, pri=4906016, relay=jacobson-fw.jacobsonco.com.
> [65.201.33.146], dsn=4.0.0, stat=Deferred: Connection timed out with
> jacobson-fw.jacobsonco.com.
> Dec 23 14:28:27 wks-lin9 sendmail[2295]: alias database /etc/aliases rebuilt
> by root
> ------------------------------ 
> 
> /VAR/LOG/MESSAGES
> ------------------------------
> Dec 23 00:35:01 wks-lin9 crond(pam_unix)[30112]: session opened for user
> root by (uid=0)
> Dec 23 00:35:01 wks-lin9 crond(pam_unix)[30114]: session opened for user
> root by (uid=0)
> Dec 23 00:35:02 wks-lin9 crond(pam_unix)[30112]: session closed for user
> root
> Dec 23 00:35:09 wks-lin9 crond(pam_unix)[30114]: session closed for user
> root
> Dec 23 00:40:01 wks-lin9 crond(pam_unix)[30204]: session opened for user
> root by (uid=0)
> Dec 23 00:40:01 wks-lin9 crond(pam_unix)[30207]: session opened for user
> root by (uid=0)
> Dec 23 00:40:01 wks-lin9 crond(pam_unix)[30205]: session opened for user
> root by (uid=0)
> Dec 23 00:40:02 wks-lin9 crond(pam_unix)[30207]: session closed for user
> root
> Dec 23 00:40:03 wks-lin9 crond(pam_unix)[30204]: session closed for user
> root
> Dec 23 00:40:10 wks-lin9 crond(pam_unix)[30205]: session closed for user
> root
> Dec 23 14:28:22 wks-lin9 syslogd 1.4.1: restart.
> Dec 23 14:28:22 wks-lin9 syslog: syslogd startup succeeded
> Dec 23 14:28:22 wks-lin9 kernel: klogd 1.4.1, log source = /proc/kmsg
> started.
> ------------------------------ 
> 
> OUTPUT FROM RUNNING CHKROOTKIT
> ------------------------------
> [snip] 
> 
> Searching for suspicious files and dirs, it may take a while...
> /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/auto/Gaim/.packlist
> /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/auto/mod_perl/.packlist
> /usr/lib/perl5/5.8.5/i386-linux-thread-multi/.packlist
> /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/razor-agents/.packlist
> /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/Mail/SpamAssassin/.packlist
> /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/razor-agents-sdk/.packlist
> /usr/lib/qt-3.3/etc/settings/.qt_plugins_3.3rc.lock /usr/lib/qt-3.3
> /etc/settings/.qtrc.lock 
> 
> [snip]
> ------------------------------ 
> 
> I am stuck at the moment. It does not appear to me that my systems have been
> 'hijacked' or any other meaningful information has been given so I can
> narrow down the cause of my problems. Of course, that is from my knowledge
> level, maybe someone else can tell me what they see from this information,
> or I could provide even further information to figure this out. I checked my
> cron jobs to see if there would be anything that would be causing this
> lockup, but I have the nothing out of the ordinary, and I am not too
> concerned with it because I think it would be causing a problem at a set
> time if it was something in cron causing the issue. 
> 
> Any help is appreciated.

Nothing really odd from what you've posted.. here's a few things to try. 

An alternate to chkrootkit. 

wget http://downloads.rootkit.nl/rkhunter-1.2.7.tar.gz
md5sum -b rkhunter-1.2.7.tar.gz # this ought to be 
288ba8a87352716384823c9ea1958fa7
rpmbuild -tb rkhunter-1.2.7.tar.gz
rkhunter --update
rkhunter -c 

Next check the output of dmesg carefully (and slowly) 

Also why do your servers need to run in 'init 5'? i would change it to '3' 

Also install sysstat (yum -y install sysstat) and monitor the output of 
'iostat -x 5' for some time. Could be a bad Disk causing IO contention. 

Check the output of 'chkconfig --list | grep 3:on | sort' and shutdown 
unnecessary services (spamd,clamd etc.) 

Finally, install some utilities from dell (omsa or something) to check the 
physical state of the machine. 

All i can think of now, HTH.
 - dhawal

------------------------ MailScanner list ------------------------
To unsubscribe, email jiscmail at jiscmail.ac.uk with the words:
'leave mailscanner' in the body of the email.
Before posting, read the Wiki (http://wiki.mailscanner.info/) and
the archives (http://www.jiscmail.ac.uk/lists/mailscanner.html).

Support MailScanner development - buy the book off the website!