Duplicated messages

Wed Jul 29 13:27:29 UTC 2015

Hi Jeremy,

No I haven't yet - I'm short on space on my root partition still because of
the large mysql DB so I want to clean that up first. Can you advise how
best to do this? Is it safe enough to do delete from maillog where
id='XXX';  for all the ID's with the dupes? is there any other tables that
need to be cleared?

Regards,
Mark

On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:

>  Log won't show taint issues. Setup log rotation.
>
>  Have you enabled debug in mailscanner config like I stated yesterday ?
>
>  --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
> Web - http://www.fluxlabs.net
>
>
> On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
>   Hi all,
>
>  So I have resolved getting the missing mails delivered from the
> quarantine. The main problem stopping this from being easy from the command
> line was the fact that "Quarantine Whole Messages As Queue Files = no" was
> set, whilst the MTA in use is exim. I've changed that setting to "yes"
> now...
>
>  I've read that if its postfix you can just send that "message" file back
> to the queue, I guess the headers are kept with the message in the
> quarantine with postfix. With exim they seem to be split between the
> database and the message file.
>
>  I first put the message ID's in to a file "idlist.txt" that had been put
> in to the quarantine with the "Other Bad Content Detected" error (every
> single email after a certain time on that day), then pulled the header from
> the db and combined them with the following simple loop;
>
>  -------
> #!/bin/bash
>  for msgid in `cat idlist.txt`;
> do
>     /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from
> maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *"
> >> with-headers/$msgid &&
>     /bin/cat 20150724/$msgid/message >> with-headers/$msgid
> done
>  -------
>
>  now I'm sending them out slowly (every 30 secs) with another simple
> loop...
>
>  -------
>  #!/bin/bash
> for msgs in with-headers/*;
> do
>     cat $msgs | exim -ti
>     mv $msgs with-headers-processed/
>     sleep 30
> done
>  -------
>
>  So at least the missing mail is now going to users.. but I'm no closer
> to knowing exactly why this happened in the first place. Jeremy mentioned a
> known "taint" issue? Can anyone elaborate on that?
>
>  I've also found now that Archive is enabled, and is set to "Archive Mail
> = " which I guess just defaults to the quarantine dir, as they seem to go
> to the "nonspam" folder in there (interestingly in a exim usable format!!)
> That couldn't have anything to do with the loop that appears to have killed
> my mailcleaner DB? I wouldn't think so as this has been running for years
> like this and not had this issue before but thought it worth mentioning.
>
>  Any other theories or places to check for clues? unfortunately the
> mail.log of the day got removed by the first person looking at the issue to
> try to free up space as it was over 4GB.
>
>  Regards,
> Mark
>
>
>
>
>
>
> On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk> wrote:
>
>> Hi Jerry,
>>
>>  If you wanted to pull a bunch of items from the quarantine from the
>> command line and re-process them through Mailcleaner, how would you do that?
>>
>>  Regards,
>> Mark
>>
>> On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com>
>> wrote:
>>
>>> I am not sure on what parameters Mailwatch calls and logs “other bad
>>> content”.  The MailScanner setting is "Notify Senders of Other Blocked
>>> Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked
>>> content” for all we know. You are going to have to follow the below
>>> suggestion and enable debug or see if you can get an idea from
>>> /var/log/maillog.
>>>
>>> -
>>>  Jerry Benton
>>>  www.mailborder.com
>>>
>>>
>>>
>>>   On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk>
>>> wrote:
>>>
>>>  Of course, apologies - I'm using Mailwatch. Any advice on how to most
>>> efficiently pull things out of quarantine via command-line? (note they are
>>> stored as "message" rather than queue items, that would be too easy..)
>>>
>>>  I don't have Archive enabled, everything has gone in to the quarantine
>>> because of this "Other Bad Content Detected"
>>>
>>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com>
>>> wrote:
>>>
>>>> By the way, there is no web interface in the MailScanner package. There
>>>> are 3rd party products of course (I created one myself) but those questions
>>>> would need to be directed to those support forums or mailing lists.
>>>>
>>>> -
>>>>  Jerry Benton
>>>>  www.mailborder.com
>>>>
>>>>
>>>>
>>>>   On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk>
>>>> wrote:
>>>>
>>>>  How do I try send them through again? At the moment they are just
>>>> "message" in the quarantine, and if I try open them through the web
>>>> interface it times out, I guess because its trying to open each one of the
>>>> dupes?
>>>>
>>>>  "Fatal error: Maximum execution time of 30 seconds exceeded in
>>>> /var/www/html/mailscanner/functions.php on line 1022"
>>>>
>>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>>>
>>>>>  Yup. Turn on debug and watch it pass through. Last time I saw these
>>>>> it was a taint issue .. Which I am assuming has been fixed by now.
>>>>>
>>>>>  --
>>>>> Jeremy McSpadden | Flux Labs
>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>> Web - http://www.fluxlabs.net
>>>>>
>>>>>
>>>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk>
>>>>> wrote:
>>>>>
>>>>>   Hi Jeremy,
>>>>>
>>>>>  Are you saying that something in these messages is crashing
>>>>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>>>>> (all different types and from different servers) are now in the quarantine
>>>>> because of "Other Bad Content Detected" with the report "MailScanner:
>>>>> Message attempted to kill MailScanner". It seems it succeeded...
>>>>>
>>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>> wrote:
>>>>>
>>>>>>  It's probably looping/crashing mailscanner. Drop MS into debug mode
>>>>>> and watch logs.
>>>>>>
>>>>>>  --
>>>>>> Jeremy McSpadden | Flux Labs
>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>> Web - http://www.fluxlabs.net
>>>>>>
>>>>>>
>>>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>> wrote:
>>>>>>
>>>>>>   An update to this, the "2 or 4" duplicates showing in the exim log
>>>>>> look like they are actually just separate deliveries to other addresses, so
>>>>>> not duplicates. In 1 example there is a single email with 2 recipients (2
>>>>>> entries in exim log) that has over 1500+ entries in the mailcleaner DB. It
>>>>>> looks like this email hasn't been delivered to the recipient at all either.
>>>>>>
>>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>>  If anyone could provide advice that would be great. Running Debian
>>>>>>> Wheezy Mailscanner 4.79.11-2.2
>>>>>>>
>>>>>>>  Our incoming dir filled up just before the weekend so we didn't
>>>>>>> see the issue for a couple of days. Normally we would just shut down
>>>>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>>>>> However on this occasion, the root partition also become full because of
>>>>>>> the mysql DB (it got to 14G in 2 days..).
>>>>>>>
>>>>>>>  For some reason everything started duplicating. I can see lots of
>>>>>>> incoming messages in the exim logs with duplication (2 or 4 of what looks
>>>>>>> like the same email) but in the mailscanner database there is hundreds of
>>>>>>> each email listed (apparently there was over 9 million messages delivered
>>>>>>> on 1 day compared with the server average of about 1500!)
>>>>>>>
>>>>>>>  It seems like some sort of loop, but afaik nothing specific was
>>>>>>> changed in the config apart from the fact incoming became full. Space has
>>>>>>> been cleared on the root partition and incoming, and everything appears to
>>>>>>> be running as normal right now.
>>>>>>>
>>>>>>>  Any advice on debugging this would be much appreciated, also, how
>>>>>>> best should I clear out the DB of all the dupes?
>>>>>>>
>>>>>>>  Thanks!
>>>>>>>
>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/921b1470/attachment.html>