Random messages stuck in mqueue

[Julian Field]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Corey McFadden wrote:
>
> It does seem as though the problem begins when MailScanner disposes of 
> a message.  Maybe someone could clarify the process whereby Sendmail 
> picks up the message and decides to deliver it (locally or relay or 
> whatever.)
See my explanation below. The normal setting to use is "Delivery Method 
= batch". Though for debugging MailScanner, I run no outgoing queue 
runner process at all, and have "Delivery Method = queue". The result is 
that processed messages are placed in the outgoing mqueue and no attempt 
to deliver them is made at all. I can then look at the messages, check 
whether they are correct and that the development code is working, and 
then just delete them from the mqueue by hand. I don't want to ever 
actually deliver these messages, all the information I want is in the 
df+qf files.
>
> It doesn't look like a DNS issue.  We've got a local resolver (that 
> has a PTR record for the machine's IP) and a backup resolver.  I've 
> done a number of DNS tests and haven't been able to produce an error 
> there.
It's usually fairly easy to test DNS speed problems with a few "dig" 
commands for some random domains (most English words .com exist, so 
picking a random word is usually pretty reliable, and probably won't be 
in your DNS cache already).
>
> On other systems everything happens so quickly that it's hard to see 
> what happens but does it resemble this:
>  - MailScanner finds message in mqueue.in <http://mqueue.in>
>  - MailScanner processes message and dumps qf/df into mqueue
>  - Sendmail daemon periodically scans mqueue
>
> or:
>  - MailScanner finds message in mqueue.in <http://mqueue.in>
>  - MailScanner processes message and spawns a sendmail process for 
> delivery
It always put the qf/df in the mqueue first. It doesn't directly pass 
the message text to sendmail, that would involve an extra copy of the 
message being written down a pipe which slows things down. I try quite 
hard to make MailScanner as fast and efficient as I can. Which is why I 
have a full MailScanner+SpamAssassin+ClamAV setup on one box in my 
office which can process 2.2 million messages per day, on 1 server.
> What are the implications of the "Delivery Method" queue vs. batch 
> option on the above?
The explanation below is described for sendmail, but the same basic 
design applies to all the other supported MTAs as well. They all work in 
pretty much the same way, more or less.

With "Delivery Method = queue" the processed message(s) is placed in the 
mqueue. That's it. You will then have a sendmail queue runner regularly 
attempting delivery of everything in the queue that's due for a delivery 
attempt. So if the outgoing sendmail process is running (ie the queue 
runner) then delivery will be delayed until the next regular queue run. 
You can start and stop this bit with "service MailScanner startout" and 
"service MailScanner stopout".

With "Delivery Method = batch" the processed message(s) are placed in 
the mqueue. Then a "sendmail -qI....." command is then executed, with 
"....." set to the IDs of the message files just placed in the mqueue. 
This tells sendmail to immediately make 1 delivery attempt of each 
message. If this succeeds, the net result is instant message delivery. 
If it fails, then it is left to be retried by the outgoing sendmail 
queue runner process, as described above. In cases where the batch is 
large, measures are taken to ensure that the "sendmail -qI....." command 
is not too long for the operating system to be able to handle. When 
there are multiple mqueues in use, a separate "sendmail -qI....." 
command is issued for the messages in each outgoing queue, so that 
delivery of all messages is attempted, regardless of the number of 
outgoing queues they are spread across. This is needed as the mqueue 
directory can be specified by a ruleset or Custom Function, allowing you 
to use different queue runner parameters for each of several queues 
(e.g. a fast queue runner for local or small messages, with a slower one 
for remote or large messages, for example).

Hopefully that explains what happens in full detail.

>
> Thanks for the input!
>
> -Corey
>
>
>
>
> On 8/1/07, * Scott Silva* <ssilva at sgvwater.com 
> <mailto:ssilva at sgvwater.com>> wrote:
>
>     Corey McFadden spake the following on 7/31/2007 9:46 PM:
>     >
>     > Guys,
>     >
>     > We're experiencing an issue on one of our boxes where messages
>     (a good
>     > percentage) are hung in /var/spool/mqueue after being processed by
>     > MailScanner.  Even those to be delivered locally will sit for
>     > (sometimes) hours until they're finally delivered.
>     >
>     > I've never seen this behavior before and something is definitely
>     awry.
>     >
>     > Here's the environment:
>     >
>     > This is CentOS release 4.5 (Final)
>     > This is Perl version 5.008008 (5.8.8)
>     >
>     > This is MailScanner version 4.61.7
>     >
>     > Executing 'sendmail -q -v' WILL process the stuck messages but
>     can take
>     > quite a long time to run because it will try to deliver everything
>     > queued in a linear fashion.
>     >
>     Since it is in mqueue and not mqueue.in <http://mqueue.in>,
>     mailscanner is pretty much out of the
>     loop. Check if you have a FQDN that resolves properly as this will
>     choke
>     sendmail quickly. Also try a local caching nameserver, as sendmail
>     might be
>     taking a long time to resolve delivery addresses.
>
>     --
>
>     MailScanner is like deodorant...
>     You hope everybody uses it, and
>     you notice quickly if they don't!!!!
>
>     --
>     MailScanner mailing list
>     mailscanner at lists.mailscanner.info
>     <mailto:mailscanner at lists.mailscanner.info>
>     http://lists.mailscanner.info/mailman/listinfo/mailscanner
>
>     Before posting, read http://wiki.mailscanner.info/posting
>     <http://wiki.mailscanner.info/posting>
>
>     Support MailScanner development - buy the book off the website!
>
>

Jules

- -- 
Julian Field MEng CITP
www.MailScanner.info
Buy the MailScanner book at www.MailScanner.info/store

MailScanner customisation, or any advanced system administration help?
Contact me at Jules at Jules.FM

PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
For all your IT requirements visit www.transtec.co.uk


-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.6.2 (Build 2014)
Charset: ISO-8859-1

wj8DBQFGsNm8EfZZRxQVtlQRAuAWAJ9bWzKisXCEcEeJGZ5TrFdMcBFAKgCcCMjh
gmylU66oi8Mi4/pxd+5q+wg=
=Sru3
-----END PGP SIGNATURE-----

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
For all your IT requirements visit www.transtec.co.uk