Maximum Processing Attempts

[Julian Field]

On 28/04/2009 11:06, David Lee wrote:
> On Tue, 28 Apr 2009, Kai Schaetzl wrote:
>
>> Julian Field wrote on Mon, 27 Apr 2009 22:40:16 +0100:
>>
>>> Should I leave it switched on by default, so heavily loaded sites can
>>> switch it off to gain that extra bit of performance, at the cost of a
>>> potential reliability hit?
>>
>>> From the past discussion I think it is mainly those heavy-traffic sites
>> that would benefit from it. I think we had exactly two folks 
>> mentioning a
>> problem with occasional messages getting not processed somehow. Both 
>> with
>> high-volume sites. So, the majority does not need this feature to be on.
>> On the other hand, exactly those people who need it will also be hit 
>> most
>> by the performance hit (as small as it may be).
>> Unfortunately none of them participated in the recent discussion.
>
> Kai, the feature isn't primarily about "occasional messages getting 
> not processed somehow".  Rather it is about the severe knock-on 
> effects that can have.
>
> (Reminder: If there is some characteristic with a number of emails 
> that causes perl/MS to crash, then that causes the whole "batch", 
> including many innocent emails, not to get processed.  Subsequent runs 
> keep tripping over the same thing.  If there are a few more such rogue 
> emails than there are MS children, then almost all the innocent email 
> gets held up indefinitely.  Such rogue emails are typically spam, and 
> so there tend to be many instances of it.  Yes, the problem is rare.  
> But when it hits, it can be very severe: all email blocked; massive 
> inbound queue build-up; load average through the roof.)
>
> We've been running it in production ever since Julian first put it 
> into a beta.  (I was honour-bound to do so; I had suggested it!)  I 
> haven't noticed a performance impact.
>
>
> My vote is to leave it on.
>
> Email sys.admins have varying ability.  When this problem hits, even 
> experienced sys.admins struggle.  (Been there; more than once.)  The 
> less experienced would struggle even more.  The default of this option 
> should be biased in favour of the inexperienced sys.admin.  So the 
> default (I suggest) should be "on".  (A sys.admin. who is experienced 
> enough to know they don't need it is, by definition, experienced 
> enough to find their way to switching it off.)
>
I am convinced by these arguments. I have improved the code since the 
last beta, so that it pre-prepares all the SQL statements once when the 
database is opened, so the SQL code should be even faster than it was. I 
have also switched off the fsync() call that was done at the end of 
every database write operation, so that should speed it up too.

I will release a new beta right now for you to test what is hopefully 
the final version of the code, and the feature will remain enabled in 
the production stable release.

Those who know enough that they can dig themselves out of the hole 
without the assistance of this feature can switch it off.

Jules

-- 
Julian Field MEng CITP CEng
www.MailScanner.info
Buy the MailScanner book at www.MailScanner.info/store

Need help customising MailScanner?
Contact me!
Need help fixing or optimising your systems?
Contact me!
Need help getting you started solving new requirements from your boss?
Contact me!

PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
Follow me at twitter.com/JulesFM and twitter.com/MailScanner


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.