Found nn messages in the processing-messages database

[Mark Sapiro]

Mark Sapiro wrote:

>Julian Field wrote:
>>
>>On 22/4/09 15:55, Mark Sapiro wrote:
>>> Julian Field wrote:
>>>    
>>>> On 22/4/09 00:43, Mark Sapiro wrote:
>>>>      
>>>>> Julian Fieldwrote:
>>>>>
>>>>>        
>>>>>> On 21/4/09 21:11, Mark Sapiro wrote:
>>>>>>
>>>>>>          
>>>>>>> On Tue, Apr 21, 2009 at 06:58:30PM +0100, Julian Field wrote:
>>>>>>>
>>>>>>>
>>>>>>>            
>>>>>>>> Please also try MailScanner 4.76.15 and let me know how you get on. I
>>>>>>>> have neatened up the code quite a bit.
>>>>>>>>
>>>>>>>> Jules.
>>>>>>>>
>>>>>>>> On 21/4/09 18:04, Julian Field wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>              
>>>>>>>>> In /usr/lib/MailScanner/MailScanner/Message.pm, around line 1243, you
>>>>>>>>> will find a line that says
>>>>>>>>> next if $message->{abandoned};
>>>>>>>>> Please try commenting out that line by putting a '#' at the start of it.
>>>>>>>>>
>>>>>>>>> Then restart MailScanner, and let me know if that solves the problem.
>>>>>>>>>
>>>>>>>>> Thanks for your patience in helping me resolve this issue!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                
>>>>>>> I installed 4.76.15. Am I also supposed to comment the above line in
>>>>>>> 4.76.15 or was that just for 4.76.14? I did not comment the line, and
>>>>>>> I am still seeing messages left in the database.
>>>>>>>
>>>>>>>
>>>>>>>            
>>>>>> It was just for .14. I am really running out of ideas here. There is no
>>>>>> way the thing would fail unless the DELETE was failing.
>>>>>> If it doesn't manage to find the complete message, it doesn't add it to
>>>>>> the database. I made extra doubly sure of that in .15, so I really can't
>>>>>> see how it is getting left in there. If it was doubly added to the
>>>>>> database, it would still get deleted, unless the delete itself was
>>>>>> failing. I wonder if the DBI parameter substitution is handling the
>>>>>> parameter string incorrectly if it's all decimal digits, thinking it's a
>>>>>> number and not a string?
>>>>>>
>>>>>> Here's a patch for 4.76.15 for you to try that will test this hypothesis.
>>>>>>
>>>>>> ---PATCH START---
>>>>>> --- MessageBatch.pm.old    2009-04-21 21:33:37.000000000 +0100
>>>>>> +++ MessageBatch.pm    2009-04-21 21:35:03.000000000 +0100
>>>>>> @@ -1259,7 +1259,8 @@
>>>>>>
>>>>>>      foreach $id (keys %gotridof) {
>>>>>>        next unless $id;
>>>>>> -    $sth->execute($id);
>>>>>> +    #$sth->execute($id);
>>>>>> +    $MailScanner::ProcDBH->do("DELETE FROM processing WHERE (id='$id')");
>>>>>>      }
>>>>>>      $sth->finish;
>>>>>>
>>>>>> ---PATCH END---
>>>>>>
>>>>>> As you can see, it's a simple one-liner. Please let me know if it makes
>>>>>> any difference.
>>>>>>
>>>>>>          
>>>>> I have installed the patch and it is encouraging so far. While it is
>>>>> too soon to say for sure that there won't be any residual entries in
>>>>> the database, I've been running for over 2 hours and processed at
>>>>> least one large batch of outgoing messages, and don't have any yet.
>>>>>
>>>>>
>>>>>        
>>>> If it proves that solves the problem, I have another alternative I would
>>>> like you to try as it will be faster, if it works around the Perl bug.
>>>> Instead of
>>>>      $MailScanner::ProcDBH->do("DELETE FROM processing WHERE (id='$id')");
>>>> make the line say
>>>>      $sth->execute("$id");
>>>> and give that a go. Notice it is *almost* the same as the line was
>>>> before, but with quotes around the $id. That may be enough to circumvent
>>>> this bug.
>>>>      
>>>
>>> I am certain the first patch fixed the problem.
>>Wonderful!
>>In that case, a big "Well done" to whomever noticed that all the 
>>messages stuck in the database were all numeric before the decimal 
>>point, as that's what finally alerted me to the possibility of this 
>>being a Perl bug.


It was Kai Schaetzl who first remarked on this. Thanks Kai.

I have looked a bit more clusely at the messages that were left in the
database. and it is not only that the id is all-decimal; the entropy
fragment must also be all-decimal or contain only a single 'E', so in
every case, they are being interpred as a floating point number rather
than a string.

E.g., the latest list from running for a while with the

    $sth->execute("$id");

patch is

74221690410.052E4
36888690435.06105
73061690441.05915
15461690460.09210

all of which are valid floating point number representations. If you
look at the lists in the messages I previously posted such as at
<http://lists.mailscanner.info/pipermail/mailscanner/2009-April/091022.html>,
you will see that those too are all decimal to the left of the period
and all decimal or contain at most one 'E' to the right of the period
so they are all valid floating point number representations, and were
apparently treated as floating point numbers when passed to the
statement handler method with perl-DBD-SQLite-1.21, but apparently not
with the prior perl-DBD-SQLite-1.13.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan