Duplicated messages

Wed Jul 29 13:59:59 UTC 2015

I agree with you completely, however that doesn't help my immediate
situation. Can you provide advice on deleting from the mailscanner DB? is
there any other tables I need to remove the offending ID's entries from?

Regards,
Mark

On 29 July 2015 at 14:34, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:

>  It's 2015 .. That shouldn't be an excuse. It's like 10 cents per 100gb
> of drive... Upgrade
>
>  --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
> Web - http://www.fluxlabs.net
>
>
> On Jul 29, 2015, at 8:27 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
>   Hi Jeremy,
>
>  No I haven't yet - I'm short on space on my root partition still because
> of the large mysql DB so I want to clean that up first. Can you advise how
> best to do this? Is it safe enough to do delete from maillog where
> id='XXX';  for all the ID's with the dupes? is there any other tables that
> need to be cleared?
>
>  Regards,
> Mark
>
> On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>
>>  Log won't show taint issues. Setup log rotation.
>>
>>  Have you enabled debug in mailscanner config like I stated yesterday ?
>>
>>  --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>> Web - http://www.fluxlabs.net
>>
>>
>> On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>>   Hi all,
>>
>>  So I have resolved getting the missing mails delivered from the
>> quarantine. The main problem stopping this from being easy from the command
>> line was the fact that "Quarantine Whole Messages As Queue Files = no" was
>> set, whilst the MTA in use is exim. I've changed that setting to "yes"
>> now...
>>
>>  I've read that if its postfix you can just send that "message" file
>> back to the queue, I guess the headers are kept with the message in the
>> quarantine with postfix. With exim they seem to be split between the
>> database and the message file.
>>
>>  I first put the message ID's in to a file "idlist.txt" that had been
>> put in to the quarantine with the "Other Bad Content Detected" error (every
>> single email after a certain time on that day), then pulled the header from
>> the db and combined them with the following simple loop;
>>
>>  -------
>> #!/bin/bash
>>  for msgid in `cat idlist.txt`;
>> do
>>     /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from
>> maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *"
>> >> with-headers/$msgid &&
>>     /bin/cat 20150724/$msgid/message >> with-headers/$msgid
>> done
>>  -------
>>
>>  now I'm sending them out slowly (every 30 secs) with another simple
>> loop...
>>
>>  -------
>>  #!/bin/bash
>> for msgs in with-headers/*;
>> do
>>     cat $msgs | exim -ti
>>     mv $msgs with-headers-processed/
>>     sleep 30
>> done
>>  -------
>>
>>  So at least the missing mail is now going to users.. but I'm no closer
>> to knowing exactly why this happened in the first place. Jeremy mentioned a
>> known "taint" issue? Can anyone elaborate on that?
>>
>>  I've also found now that Archive is enabled, and is set to "Archive
>> Mail = " which I guess just defaults to the quarantine dir, as they seem to
>> go to the "nonspam" folder in there (interestingly in a exim usable
>> format!!) That couldn't have anything to do with the loop that appears to
>> have killed my mailcleaner DB? I wouldn't think so as this has been running
>> for years like this and not had this issue before but thought it worth
>> mentioning.
>>
>>  Any other theories or places to check for clues? unfortunately the
>> mail.log of the day got removed by the first person looking at the issue to
>> try to free up space as it was over 4GB.
>>
>>  Regards,
>> Mark
>>
>>
>>
>>
>>
>>
>> On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>>> Hi Jerry,
>>>
>>>  If you wanted to pull a bunch of items from the quarantine from the
>>> command line and re-process them through Mailcleaner, how would you do that?
>>>
>>>  Regards,
>>> Mark
>>>
>>> On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com>
>>> wrote:
>>>
>>>> I am not sure on what parameters Mailwatch calls and logs “other bad
>>>> content”.  The MailScanner setting is "Notify Senders of Other Blocked
>>>> Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked
>>>> content” for all we know. You are going to have to follow the below
>>>> suggestion and enable debug or see if you can get an idea from
>>>> /var/log/maillog.
>>>>
>>>> -
>>>>  Jerry Benton
>>>>  www.mailborder.com
>>>>
>>>>
>>>>
>>>>   On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk>
>>>> wrote:
>>>>
>>>>  Of course, apologies - I'm using Mailwatch. Any advice on how to most
>>>> efficiently pull things out of quarantine via command-line? (note they are
>>>> stored as "message" rather than queue items, that would be too easy..)
>>>>
>>>>  I don't have Archive enabled, everything has gone in to the
>>>> quarantine because of this "Other Bad Content Detected"
>>>>
>>>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com>
>>>> wrote:
>>>>
>>>>> By the way, there is no web interface in the MailScanner package.
>>>>> There are 3rd party products of course (I created one myself) but those
>>>>> questions would need to be directed to those support forums or mailing
>>>>> lists.
>>>>>
>>>>> -
>>>>>  Jerry Benton
>>>>>  www.mailborder.com
>>>>>
>>>>>
>>>>>
>>>>>   On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk>
>>>>> wrote:
>>>>>
>>>>>  How do I try send them through again? At the moment they are just
>>>>> "message" in the quarantine, and if I try open them through the web
>>>>> interface it times out, I guess because its trying to open each one of the
>>>>> dupes?
>>>>>
>>>>>  "Fatal error: Maximum execution time of 30 seconds exceeded in
>>>>> /var/www/html/mailscanner/functions.php on line 1022"
>>>>>
>>>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>> wrote:
>>>>>
>>>>>>  Yup. Turn on debug and watch it pass through. Last time I saw these
>>>>>> it was a taint issue .. Which I am assuming has been fixed by now.
>>>>>>
>>>>>>  --
>>>>>> Jeremy McSpadden | Flux Labs
>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>> Web - http://www.fluxlabs.net
>>>>>>
>>>>>>
>>>>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>> wrote:
>>>>>>
>>>>>>   Hi Jeremy,
>>>>>>
>>>>>>  Are you saying that something in these messages is crashing
>>>>>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>>>>>> (all different types and from different servers) are now in the quarantine
>>>>>> because of "Other Bad Content Detected" with the report "MailScanner:
>>>>>> Message attempted to kill MailScanner". It seems it succeeded...
>>>>>>
>>>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>>> wrote:
>>>>>>
>>>>>>>  It's probably looping/crashing mailscanner. Drop MS into debug
>>>>>>> mode and watch logs.
>>>>>>>
>>>>>>>  --
>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>>> Web - http://www.fluxlabs.net
>>>>>>>
>>>>>>>
>>>>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>> wrote:
>>>>>>>
>>>>>>>   An update to this, the "2 or 4" duplicates showing in the exim
>>>>>>> log look like they are actually just separate deliveries to other
>>>>>>> addresses, so not duplicates. In 1 example there is a single email with 2
>>>>>>> recipients (2 entries in exim log) that has over 1500+ entries in the
>>>>>>> mailcleaner DB. It looks like this email hasn't been delivered to the
>>>>>>> recipient at all either.
>>>>>>>
>>>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>>  If anyone could provide advice that would be great. Running Debian
>>>>>>>> Wheezy Mailscanner 4.79.11-2.2
>>>>>>>>
>>>>>>>>  Our incoming dir filled up just before the weekend so we didn't
>>>>>>>> see the issue for a couple of days. Normally we would just shut down
>>>>>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>>>>>> However on this occasion, the root partition also become full because of
>>>>>>>> the mysql DB (it got to 14G in 2 days..).
>>>>>>>>
>>>>>>>>  For some reason everything started duplicating. I can see lots of
>>>>>>>> incoming messages in the exim logs with duplication (2 or 4 of what looks
>>>>>>>> like the same email) but in the mailscanner database there is hundreds of
>>>>>>>> each email listed (apparently there was over 9 million messages delivered
>>>>>>>> on 1 day compared with the server average of about 1500!)
>>>>>>>>
>>>>>>>>  It seems like some sort of loop, but afaik nothing specific was
>>>>>>>> changed in the config apart from the fact incoming became full. Space has
>>>>>>>> been cleared on the root partition and incoming, and everything appears to
>>>>>>>> be running as normal right now.
>>>>>>>>
>>>>>>>>  Any advice on debugging this would be much appreciated, also, how
>>>>>>>> best should I clear out the DB of all the dupes?
>>>>>>>>
>>>>>>>>  Thanks!
>>>>>>>>
>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/372ce321/attachment.html>