Found nn messages in the processing-messages database
Julian Field
MailScanner at ecs.soton.ac.uk
Thu Apr 23 09:31:45 IST 2009
On 23/04/2009 02:57, Mark Sapiro wrote:
> Mark Sapiro wrote:
>
>
>> Julian Field wrote:
>>
>>> On 22/4/09 15:55, Mark Sapiro wrote:
>>>
>>>> Julian Field wrote:
>>>>
>>>>
>>>>> On 22/4/09 00:43, Mark Sapiro wrote:
>>>>>
>>>>>
>>>>>> Julian Fieldwrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 21/4/09 21:11, Mark Sapiro wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Tue, Apr 21, 2009 at 06:58:30PM +0100, Julian Field wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Please also try MailScanner 4.76.15 and let me know how you get on. I
>>>>>>>>> have neatened up the code quite a bit.
>>>>>>>>>
>>>>>>>>> Jules.
>>>>>>>>>
>>>>>>>>> On 21/4/09 18:04, Julian Field wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> In /usr/lib/MailScanner/MailScanner/Message.pm, around line 1243, you
>>>>>>>>>> will find a line that says
>>>>>>>>>> next if $message->{abandoned};
>>>>>>>>>> Please try commenting out that line by putting a '#' at the start of it.
>>>>>>>>>>
>>>>>>>>>> Then restart MailScanner, and let me know if that solves the problem.
>>>>>>>>>>
>>>>>>>>>> Thanks for your patience in helping me resolve this issue!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> I installed 4.76.15. Am I also supposed to comment the above line in
>>>>>>>> 4.76.15 or was that just for 4.76.14? I did not comment the line, and
>>>>>>>> I am still seeing messages left in the database.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> It was just for .14. I am really running out of ideas here. There is no
>>>>>>> way the thing would fail unless the DELETE was failing.
>>>>>>> If it doesn't manage to find the complete message, it doesn't add it to
>>>>>>> the database. I made extra doubly sure of that in .15, so I really can't
>>>>>>> see how it is getting left in there. If it was doubly added to the
>>>>>>> database, it would still get deleted, unless the delete itself was
>>>>>>> failing. I wonder if the DBI parameter substitution is handling the
>>>>>>> parameter string incorrectly if it's all decimal digits, thinking it's a
>>>>>>> number and not a string?
>>>>>>>
>>>>>>> Here's a patch for 4.76.15 for you to try that will test this hypothesis.
>>>>>>>
>>>>>>> ---PATCH START---
>>>>>>> --- MessageBatch.pm.old 2009-04-21 21:33:37.000000000 +0100
>>>>>>> +++ MessageBatch.pm 2009-04-21 21:35:03.000000000 +0100
>>>>>>> @@ -1259,7 +1259,8 @@
>>>>>>>
>>>>>>> foreach $id (keys %gotridof) {
>>>>>>> next unless $id;
>>>>>>> - $sth->execute($id);
>>>>>>> + #$sth->execute($id);
>>>>>>> + $MailScanner::ProcDBH->do("DELETE FROM processing WHERE (id='$id')");
>>>>>>> }
>>>>>>> $sth->finish;
>>>>>>>
>>>>>>> ---PATCH END---
>>>>>>>
>>>>>>> As you can see, it's a simple one-liner. Please let me know if it makes
>>>>>>> any difference.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> I have installed the patch and it is encouraging so far. While it is
>>>>>> too soon to say for sure that there won't be any residual entries in
>>>>>> the database, I've been running for over 2 hours and processed at
>>>>>> least one large batch of outgoing messages, and don't have any yet.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> If it proves that solves the problem, I have another alternative I would
>>>>> like you to try as it will be faster, if it works around the Perl bug.
>>>>> Instead of
>>>>> $MailScanner::ProcDBH->do("DELETE FROM processing WHERE (id='$id')");
>>>>> make the line say
>>>>> $sth->execute("$id");
>>>>> and give that a go. Notice it is *almost* the same as the line was
>>>>> before, but with quotes around the $id. That may be enough to circumvent
>>>>> this bug.
>>>>>
>>>>>
>>>> I am certain the first patch fixed the problem.
>>>>
>>> Wonderful!
>>> In that case, a big "Well done" to whomever noticed that all the
>>> messages stuck in the database were all numeric before the decimal
>>> point, as that's what finally alerted me to the possibility of this
>>> being a Perl bug.
>>>
>
> It was Kai Schaetzl who first remarked on this. Thanks Kai.
>
> I have looked a bit more clusely at the messages that were left in the
> database. and it is not only that the id is all-decimal; the entropy
> fragment must also be all-decimal or contain only a single 'E', so in
> every case, they are being interpred as a floating point number rather
> than a string.
>
Which is a Perl bug. Perl should carry them around as both a number and
a string, and the final destination type is a string (that's the type of
the field in the SQL query), so someone is squashing it to just a number
before checking the destination type. That's either a bug in Perl or in
the implementation of DBD::SQLite.
> E.g., the latest list from running for a while with the
>
> $sth->execute("$id");
>
> patch is
>
> 74221690410.052E4
> 36888690435.06105
> 73061690441.05915
> 15461690460.09210
>
> all of which are valid floating point number representations. If you
> look at the lists in the messages I previously posted such as at
> <http://lists.mailscanner.info/pipermail/mailscanner/2009-April/091022.html>,
> you will see that those too are all decimal to the left of the period
> and all decimal or contain at most one 'E' to the right of the period
> so they are all valid floating point number representations, and were
> apparently treated as floating point numbers when passed to the
> statement handler method with perl-DBD-SQLite-1.21, but apparently not
> with the prior perl-DBD-SQLite-1.13.
>
Well diagnosed, sir :-)
So an alternative solution would be to put a "P" (for Postfix, but is
arbitrary) on the front of the message id.
Would you like me to do that, or just live with the slight inefficiency
in the database deletion?
Jules
--
Julian Field MEng CITP CEng
www.MailScanner.info
Buy the MailScanner book at www.MailScanner.info/store
Need help customising MailScanner?
Contact me!
Need help fixing or optimising your systems?
Contact me!
Need help getting you started solving new requirements from your boss?
Contact me!
PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
More information about the MailScanner
mailing list