On the 'antistarving' patch

[Mariano Absatz]

Hi,

a while ago, I made a patch to MailScanner to prevent 'message
starving' in the queue directories because of a predetermination of
the order in which multiple queue directories are chosen (see
http://tinyurl.com/5xjy7).

Julian kindly added it in the latest beta...

Today I had the first opportunity to see it working 'in the real
world' and the results are mixed... that is. The patch DOES what it is
supposed to do and does it right.

The problem is that the problem persits 'intra-directory'.

I guess the problem is as follows. I have 26 different input
directories. I have 8 concurrent MS processes and batches are made up
of up to 100 messages.

When the input directories have MORE than 100 messages each, MS does
the following.

It chooses the directories in random order and reads each filename in
all of them in one array (@SortedFiles).

Then, as it notices there are too many files, it takes the first 100
(it doesn't sort them by mtime).

But, as every directory has MORE than 100 messages, it just chooses
the first 100 messages from one (random) directory.

After it finishes processing them, it deletes the 100 messages from
the input directory and starts over.

In the meantime, the incoming mta process keeps delivering new files
into these directories at random.

I guess that (at least on fedora/ext3) the directory entries
corresponding to erased files must be reused when a new file is added
to the directory so that the new files, after a batch is erased, get
'before' older files in the directory.

This is a total guess, anyone that knows better than I about the ext3
filesystem can confirm/deny this.

The net result is that, while the messages keep coming in at the same
or higher rate than current, I keep having this 'starving' problem,
only a little alleviated.

After the tide passes its peak (that is messages start to arrive
slower than MS peak processing rate), the starving starts to slowly go
away (that is, as every incoming directory has less than 100 messages
on average).




At http://baby.com.ar/MailScanner/NoStarveQueue/new/peak.html I left
graphs of the actual data. The first one is the size of all queues
combined (each color is a different server). The second one shows the
age of the oldest file in the incoming queue.

The 45° ascending diagonals in the second graph show that at least one
message is 'starving'.

And, yes... that is 40K messages waiting to be delivered with messages
that waited for 2 1/2 hours :-(



After seeing this I think that maybe we could replace the old 'while'
and shuffle the @SortedFiles array before using it.

That is replace the following code:

    # SortedFiles is array of full pathnames now, not just filenames
    if ($UnsortedBatchesLeft>0) {
      $UnsortedBatchesLeft--;
    } else {
      @SortedFiles = sort { $ModDate{$a} <=> $ModDate{$b} } keys %ModDate;
    }

with something like:
    # SortedFiles is array of full pathnames now, not just filenames
    if ($UnsortedBatchesLeft>0) {
      $UnsortedBatchesLeft--;
      array_shuffle(\@SortedFiles);
    } else {
      @SortedFiles = sort { $ModDate{$a} <=> $ModDate{$b} } keys %ModDate;
    }

Of course, someone has to write 'array_shuffle()'. I don't have any
time right now to write this (or worse, to test it), but I'd use the
'fisher_yates_shuffle()' function in recipe 4.17 (Randomizing an
Array) of the Perl Cookbook (1st. edition, pg 121-122).

A stupid question for Julian: 
where should we put that function?

It is a 'generic' array shuffling function, but I don't see any
'MailScanner::Util' or equivalent package ... we could put it in the
same package as where we use it (so we don't have to export or qualify
the function), but this is MailScanner::Sendmail that is MTA
dependent, and all MTAs could use the same function...

If no one can think of a better place, it could be in
'MailScanner::MessageBatch' since we're using it in a batch context...

Regards.

-- 
Mariano Absatz - El Baby
el (dot) baby (AT) gmail (dot) com
el (punto) baby (ARROBA:@) gmail (punto) com

-------------------------- MailScanner list ----------------------
To leave, send    leave mailscanner    to jiscmail at jiscmail.ac.uk
Before posting, please see the Most Asked Questions at
http://www.mailscanner.biz/maq/     and the archives at
http://www.jiscmail.ac.uk/lists/mailscanner.html