Duplicated messages
Mark Adams
mark at workshopit.co.uk
Wed Jul 29 14:36:03 UTC 2015
Good point, I'll just keep a dump of the DB elsewhere in case I need to
look for something on that day in the future. So just "delete from maillog
where date='2015-07-24';" is the best way to clean this out?
Regards,
Mark
On 29 July 2015 at 15:17, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
> Sure. But your looping through 9 million entries. Your better off to
> drop where date=
>
> --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
> Web - http://www.fluxlabs.net
>
>
> On Jul 29, 2015, at 9:15 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
> Very small, and I can't argue with you about that....
>
> Yes it grew pretty much the whole 14G in one day. Ideally I'd like to
> keep the stuff from before the problem occurred on that day, is it not ok
> to delete from maillog where id="XX" ?
>
> On 29 July 2015 at 15:12, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>
>> How small is that drive ? Pretty bad planning on someone's part .. Then
>> to be reliant on a single MX for mail flow.
>>
>> Either way .. Dump all entries from that day. Did I read it grew 10gb
>> in a day ?
>>
>> --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>> Web - http://www.fluxlabs.net
>>
>>
>> On Jul 29, 2015, at 9:08 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>> It's not 1000 entries, its about that many (some id's with a huge
>> amount more) per message ID. The Mailscanner DB has over 9,000,000 entries
>> for that day. I was planning on looping through and deleting all the
>> offending entries for the 300+ emails that seemed to get stuck.
>>
>> The mailscanner DB is currently 14G (maillog.MYD)
>>
>> On 29 July 2015 at 15:04, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>
>>> 1000 entries from a sql DB may be about 5mb of space.
>>> I would do a little more digging on the / partition to see what's eating
>>> space. Probably better off dropping some older archive dir
>>>
>>> --
>>> Jeremy McSpadden | Flux Labs
>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>> Web - http://www.fluxlabs.net
>>>
>>>
>>> On Jul 29, 2015, at 9:00 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>
>>> I agree with you completely, however that doesn't help my immediate
>>> situation. Can you provide advice on deleting from the mailscanner DB? is
>>> there any other tables I need to remove the offending ID's entries from?
>>>
>>> Regards,
>>> Mark
>>>
>>> On 29 July 2015 at 14:34, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>>
>>>> It's 2015 .. That shouldn't be an excuse. It's like 10 cents per
>>>> 100gb of drive... Upgrade
>>>>
>>>> --
>>>> Jeremy McSpadden | Flux Labs
>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>> Web - http://www.fluxlabs.net
>>>>
>>>>
>>>> On Jul 29, 2015, at 8:27 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>
>>>> Hi Jeremy,
>>>>
>>>> No I haven't yet - I'm short on space on my root partition still
>>>> because of the large mysql DB so I want to clean that up first. Can you
>>>> advise how best to do this? Is it safe enough to do delete from maillog
>>>> where id='XXX'; for all the ID's with the dupes? is there any other tables
>>>> that need to be cleared?
>>>>
>>>> Regards,
>>>> Mark
>>>>
>>>> On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>>>
>>>>> Log won't show taint issues. Setup log rotation.
>>>>>
>>>>> Have you enabled debug in mailscanner config like I stated yesterday
>>>>> ?
>>>>>
>>>>> --
>>>>> Jeremy McSpadden | Flux Labs
>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>> Web - http://www.fluxlabs.net
>>>>>
>>>>>
>>>>> On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> So I have resolved getting the missing mails delivered from the
>>>>> quarantine. The main problem stopping this from being easy from the command
>>>>> line was the fact that "Quarantine Whole Messages As Queue Files = no" was
>>>>> set, whilst the MTA in use is exim. I've changed that setting to "yes"
>>>>> now...
>>>>>
>>>>> I've read that if its postfix you can just send that "message" file
>>>>> back to the queue, I guess the headers are kept with the message in the
>>>>> quarantine with postfix. With exim they seem to be split between the
>>>>> database and the message file.
>>>>>
>>>>> I first put the message ID's in to a file "idlist.txt" that had been
>>>>> put in to the quarantine with the "Other Bad Content Detected" error (every
>>>>> single email after a certain time on that day), then pulled the header from
>>>>> the db and combined them with the following simple loop;
>>>>>
>>>>> -------
>>>>> #!/bin/bash
>>>>> for msgid in `cat idlist.txt`;
>>>>> do
>>>>> /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from
>>>>> maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *"
>>>>> >> with-headers/$msgid &&
>>>>> /bin/cat 20150724/$msgid/message >> with-headers/$msgid
>>>>> done
>>>>> -------
>>>>>
>>>>> now I'm sending them out slowly (every 30 secs) with another simple
>>>>> loop...
>>>>>
>>>>> -------
>>>>> #!/bin/bash
>>>>> for msgs in with-headers/*;
>>>>> do
>>>>> cat $msgs | exim -ti
>>>>> mv $msgs with-headers-processed/
>>>>> sleep 30
>>>>> done
>>>>> -------
>>>>>
>>>>> So at least the missing mail is now going to users.. but I'm no
>>>>> closer to knowing exactly why this happened in the first place. Jeremy
>>>>> mentioned a known "taint" issue? Can anyone elaborate on that?
>>>>>
>>>>> I've also found now that Archive is enabled, and is set to "Archive
>>>>> Mail = " which I guess just defaults to the quarantine dir, as they seem to
>>>>> go to the "nonspam" folder in there (interestingly in a exim usable
>>>>> format!!) That couldn't have anything to do with the loop that appears to
>>>>> have killed my mailcleaner DB? I wouldn't think so as this has been running
>>>>> for years like this and not had this issue before but thought it worth
>>>>> mentioning.
>>>>>
>>>>> Any other theories or places to check for clues? unfortunately the
>>>>> mail.log of the day got removed by the first person looking at the issue to
>>>>> try to free up space as it was over 4GB.
>>>>>
>>>>> Regards,
>>>>> Mark
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>
>>>>>> Hi Jerry,
>>>>>>
>>>>>> If you wanted to pull a bunch of items from the quarantine from the
>>>>>> command line and re-process them through Mailcleaner, how would you do that?
>>>>>>
>>>>>> Regards,
>>>>>> Mark
>>>>>>
>>>>>> On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I am not sure on what parameters Mailwatch calls and logs “other bad
>>>>>>> content”. The MailScanner setting is "Notify Senders of Other Blocked
>>>>>>> Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked
>>>>>>> content” for all we know. You are going to have to follow the below
>>>>>>> suggestion and enable debug or see if you can get an idea from
>>>>>>> /var/log/maillog.
>>>>>>>
>>>>>>> -
>>>>>>> Jerry Benton
>>>>>>> www.mailborder.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Of course, apologies - I'm using Mailwatch. Any advice on how to
>>>>>>> most efficiently pull things out of quarantine via command-line? (note they
>>>>>>> are stored as "message" rather than queue items, that would be too easy..)
>>>>>>>
>>>>>>> I don't have Archive enabled, everything has gone in to the
>>>>>>> quarantine because of this "Other Bad Content Detected"
>>>>>>>
>>>>>>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> By the way, there is no web interface in the MailScanner package.
>>>>>>>> There are 3rd party products of course (I created one myself) but those
>>>>>>>> questions would need to be directed to those support forums or mailing
>>>>>>>> lists.
>>>>>>>>
>>>>>>>> -
>>>>>>>> Jerry Benton
>>>>>>>> www.mailborder.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> How do I try send them through again? At the moment they are just
>>>>>>>> "message" in the quarantine, and if I try open them through the web
>>>>>>>> interface it times out, I guess because its trying to open each one of the
>>>>>>>> dupes?
>>>>>>>>
>>>>>>>> "Fatal error: Maximum execution time of 30 seconds exceeded in
>>>>>>>> /var/www/html/mailscanner/functions.php on line 1022"
>>>>>>>>
>>>>>>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yup. Turn on debug and watch it pass through. Last time I saw
>>>>>>>>> these it was a taint issue .. Which I am assuming has been fixed by now.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile -
>>>>>>>>> 850-890-2543
>>>>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>>>>> Web - http://www.fluxlabs.net
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Jeremy,
>>>>>>>>>
>>>>>>>>> Are you saying that something in these messages is crashing
>>>>>>>>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>>>>>>>>> (all different types and from different servers) are now in the quarantine
>>>>>>>>> because of "Other Bad Content Detected" with the report "MailScanner:
>>>>>>>>> Message attempted to kill MailScanner". It seems it succeeded...
>>>>>>>>>
>>>>>>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> It's probably looping/crashing mailscanner. Drop MS into debug
>>>>>>>>>> mode and watch logs.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile -
>>>>>>>>>> 850-890-2543
>>>>>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>>>>>> Web - http://www.fluxlabs.net
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> An update to this, the "2 or 4" duplicates showing in the exim
>>>>>>>>>> log look like they are actually just separate deliveries to other
>>>>>>>>>> addresses, so not duplicates. In 1 example there is a single email with 2
>>>>>>>>>> recipients (2 entries in exim log) that has over 1500+ entries in the
>>>>>>>>>> mailcleaner DB. It looks like this email hasn't been delivered to the
>>>>>>>>>> recipient at all either.
>>>>>>>>>>
>>>>>>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> If anyone could provide advice that would be great. Running
>>>>>>>>>>> Debian Wheezy Mailscanner 4.79.11-2.2
>>>>>>>>>>>
>>>>>>>>>>> Our incoming dir filled up just before the weekend so we
>>>>>>>>>>> didn't see the issue for a couple of days. Normally we would just shut down
>>>>>>>>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>>>>>>>>> However on this occasion, the root partition also become full because of
>>>>>>>>>>> the mysql DB (it got to 14G in 2 days..).
>>>>>>>>>>>
>>>>>>>>>>> For some reason everything started duplicating. I can see lots
>>>>>>>>>>> of incoming messages in the exim logs with duplication (2 or 4 of what
>>>>>>>>>>> looks like the same email) but in the mailscanner database there is
>>>>>>>>>>> hundreds of each email listed (apparently there was over 9 million messages
>>>>>>>>>>> delivered on 1 day compared with the server average of about 1500!)
>>>>>>>>>>>
>>>>>>>>>>> It seems like some sort of loop, but afaik nothing specific
>>>>>>>>>>> was changed in the config apart from the fact incoming became full. Space
>>>>>>>>>>> has been cleared on the root partition and incoming, and everything appears
>>>>>>>>>>> to be running as normal right now.
>>>>>>>>>>>
>>>>>>>>>>> Any advice on debugging this would be much appreciated, also,
>>>>>>>>>>> how best should I clear out the DB of all the dupes?
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/7ec84142/attachment-0001.html>
More information about the MailScanner
mailing list