<div dir="ltr">It's not 1000 entries, its about that many (some id's with a huge amount more) per message ID. The Mailscanner DB has over 9,000,000 entries for that day. I was planning on looping through and deleting all the offending entries for the 300+ emails that seemed to get stuck.<div><br></div><div>The mailscanner DB is currently 14G (maillog.MYD)</div><div class="gmail_extra"><br><div class="gmail_quote">On 29 July 2015 at 15:04, Jeremy McSpadden <span dir="ltr"><<a href="mailto:jeremy@fluxlabs.net" target="_blank">jeremy@fluxlabs.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>1000 entries from a sql DB may be about 5mb of space. </div>
<div>I would do a little more digging on the / partition to see what's eating space. Probably better off dropping some older archive dir</div><span class="">
<div><br>
<div>
<div><span style="background-color:rgba(255,255,255,0)">--<br>
Jeremy McSpadden | Flux Labs<br>
Local - <a href="tel:850-250-5590;501" target="_blank">850-250-5590x501</a> | Mobile - <a href="tel:850-890-2543" target="_blank">850-890-2543</a> <br>
Fax - <a href="tel:850-254-2955" target="_blank">850-254-2955</a> | Toll Free - <a href="tel:877-699-FLUX" target="_blank">877-699-FLUX</a><br>
Web - <a href="http://www.fluxlabs.net/" target="_blank">http://www.fluxlabs.net</a></span></div>
<div><br>
</div>
</div>
</div>
</span><div><div class="h5"><div><br>
On Jul 29, 2015, at 9:00 AM, Mark Adams <<a href="mailto:mark@workshopit.co.uk" target="_blank">mark@workshopit.co.uk</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">I agree with you completely, however that doesn't help my immediate situation. Can you provide advice on deleting from the mailscanner DB? is there any other tables I need to remove the offending ID's entries from?
<div><br>
Regards,</div>
<div>Mark<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 29 July 2015 at 14:34, Jeremy McSpadden <span dir="ltr">
<<a href="mailto:jeremy@fluxlabs.net" target="_blank">jeremy@fluxlabs.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>It's 2015 .. That shouldn't be an excuse. It's like 10 cents per 100gb of drive... Upgrade <span><br>
<br>
<div>
<div><span style="background-color:rgba(255,255,255,0)">--<br>
Jeremy McSpadden | Flux Labs<br>
Local - <a href="tel:850-250-5590;501" target="_blank">850-250-5590x501</a> | Mobile - <a href="tel:850-890-2543" target="_blank">850-890-2543</a> <br>
Fax - <a href="tel:850-254-2955" target="_blank">850-254-2955</a> | Toll Free - <a href="tel:877-699-FLUX" target="_blank">877-699-FLUX</a><br>
Web - <a href="http://www.fluxlabs.net/" target="_blank">http://www.fluxlabs.net</a></span></div>
<div><br>
</div>
</div>
</span></div>
<div>
<div>
<div><br>
On Jul 29, 2015, at 8:27 AM, Mark Adams <<a href="mailto:mark@workshopit.co.uk" target="_blank">mark@workshopit.co.uk</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">Hi Jeremy,
<div><br>
</div>
<div>No I haven't yet - I'm short on space on my root partition still because of the large mysql DB so I want to clean that up first. Can you advise how best to do this? Is it safe enough to do delete from maillog where id='XXX'; for all the ID's with the
dupes? is there any other tables that need to be cleared?</div>
<div><br>
</div>
<div>Regards,<br>
Mark<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 29 July 2015 at 14:16, Jeremy McSpadden <span dir="ltr">
<<a href="mailto:jeremy@fluxlabs.net" target="_blank">jeremy@fluxlabs.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>Log won't show taint issues. Setup log rotation. </div>
<div><br>
</div>
<div>Have you enabled debug in mailscanner config like I stated yesterday ? <span><br>
<br>
<div>
<div><span style="background-color:rgba(255,255,255,0)">--<br>
Jeremy McSpadden | Flux Labs<br>
Local - <a href="tel:850-250-5590;501" target="_blank">850-250-5590x501</a> | Mobile - <a href="tel:850-890-2543" target="_blank">850-890-2543</a> <br>
Fax - <a href="tel:850-254-2955" target="_blank">850-254-2955</a> | Toll Free - <a href="tel:877-699-FLUX" target="_blank">877-699-FLUX</a><br>
Web - <a href="http://www.fluxlabs.net/" target="_blank">http://www.fluxlabs.net</a></span></div>
<div><br>
</div>
</div>
</span></div>
<div>
<div>
<div><br>
On Jul 29, 2015, at 8:13 AM, Mark Adams <<a href="mailto:mark@workshopit.co.uk" target="_blank">mark@workshopit.co.uk</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">Hi all,
<div><br>
</div>
<div>So I have resolved getting the missing mails delivered from the quarantine. The main problem stopping this from being easy from the command line was the fact that "Quarantine Whole Messages As Queue Files = no" was set, whilst the MTA in use is exim. I've
changed that setting to "yes" now...</div>
<div><br>
</div>
<div>I've read that if its postfix you can just send that "message" file back to the queue, I guess the headers are kept with the message in the quarantine with postfix. With exim they seem to be split between the database and the message file.</div>
<div><br>
</div>
<div>I first put the message ID's in to a file "idlist.txt" that had been put in to the quarantine with the "Other Bad Content Detected" error (every single email after a certain time on that day), then pulled the header from the db and combined them with the
following simple loop;</div>
<div><br>
</div>
<div>-------</div>
<div>#!/bin/bash<br>
</div>
<div>
<div>for msgid in `cat idlist.txt`;</div>
<div>do</div>
<div> /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *" >> with-headers/$msgid &&</div>
<div> /bin/cat 20150724/$msgid/message >> with-headers/$msgid</div>
<div>done</div>
</div>
<div>-------<br>
</div>
<div><br>
</div>
<div>now I'm sending them out slowly (every 30 secs) with another simple loop...</div>
<div><br>
</div>
<div>-------<br>
</div>
<div>
<div>#!/bin/bash</div>
<div>for msgs in with-headers/*;</div>
<div>do</div>
<div> cat $msgs | exim -ti</div>
<div> mv $msgs with-headers-processed/</div>
<div> sleep 30</div>
<div>done</div>
</div>
<div>-------<br>
</div>
<div><br>
</div>
<div>So at least the missing mail is now going to users.. but I'm no closer to knowing exactly why this happened in the first place. Jeremy mentioned a known "taint" issue? Can anyone elaborate on that?</div>
<div><br>
</div>
<div>I've also found now that Archive is enabled, and is set to "Archive Mail = " which I guess just defaults to the quarantine dir, as they seem to go to the "nonspam" folder in there (interestingly in a exim usable format!!) That couldn't have anything to
do with the loop that appears to have killed my mailcleaner DB? I wouldn't think so as this has been running for years like this and not had this issue before but thought it worth mentioning.</div>
<div><br>
</div>
<div>Any other theories or places to check for clues? unfortunately the mail.log of the day got removed by the first person looking at the issue to try to free up space as it was over 4GB.</div>
<div><br>
</div>
<div>Regards,<br>
Mark</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 28 July 2015 at 17:07, Mark Adams <span dir="ltr"><<a href="mailto:mark@workshopit.co.uk" target="_blank">mark@workshopit.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Jerry,
<div><br>
</div>
<div>If you wanted to pull a bunch of items from the quarantine from the command line and re-process them through Mailcleaner, how would you do that?</div>
<div><br>
</div>
<div>Regards,<br>
Mark</div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 28 July 2015 at 17:00, Jerry Benton <span dir="ltr"><<a href="mailto:jerry.benton@mailborder.com" target="_blank">jerry.benton@mailborder.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">I am not sure on what parameters Mailwatch calls and logs “other bad content”. The MailScanner setting is "Notify Senders of Other Blocked Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked content”
for all we know. You are going to have to follow the below suggestion and enable debug or see if you can get an idea from /var/log/maillog.<span><br>
<div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br>
-</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Jerry Benton</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<a href="http://www.mailborder.com" target="_blank">www.mailborder.com</a></div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br>
</div>
<br>
</div>
<br>
</span>
<div>
<blockquote type="cite"><span>
<div>On Jul 28, 2015, at 11:49 AM, Mark Adams <<a href="mailto:mark@workshopit.co.uk" target="_blank">mark@workshopit.co.uk</a>> wrote:</div>
<br>
</span>
<div>
<div dir="ltr"><span>Of course, apologies - I'm using Mailwatch. Any advice on how to most efficiently pull things out of quarantine via command-line? (note they are stored as "message" rather than queue items, that would be too easy..)
<div><br>
</div>
</span>
<div><span>I don't have Archive enabled, everything has gone in to the quarantine because of this "Other Bad Content Detected"<br>
</span>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span>On 28 July 2015 at 16:43, Jerry Benton <span dir="ltr">
<<a href="mailto:jerry.benton@mailborder.com" target="_blank">jerry.benton@mailborder.com</a>></span> wrote:<br>
</span>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word"><span>By the way, there is no web interface in the MailScanner package. There are 3rd party products of course (I created one myself) but those questions would need to be directed to those support forums or mailing lists.</span><span><br>
<div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br>
-</div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Jerry Benton</div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<a href="http://www.mailborder.com/" target="_blank">www.mailborder.com</a></div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br>
</div>
<br>
</div>
<br>
</span>
<div>
<blockquote type="cite">
<div>On Jul 28, 2015, at 11:34 AM, Mark Adams <<a href="mailto:mark@workshopit.co.uk" target="_blank">mark@workshopit.co.uk</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div class="gmail_extra">How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">"Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 28 July 2015 at 16:31, Jeremy McSpadden <span dir="ltr">
<<a href="mailto:jeremy@fluxlabs.net" target="_blank">jeremy@fluxlabs.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now. <span><br>
<br>
<div>
<div><span style="background-color:rgba(255,255,255,0)">--<br>
Jeremy McSpadden | Flux Labs<br>
Local - <a href="tel:850-250-5590;501" target="_blank">850-250-5590x501</a> | Mobile - <a href="tel:850-890-2543" target="_blank">850-890-2543</a> <br>
Fax - <a href="tel:850-254-2955" target="_blank">850-254-2955</a> | Toll Free - <a href="tel:877-699-FLUX" target="_blank">877-699-FLUX</a><br>
Web - <a href="http://www.fluxlabs.net/" target="_blank">http://www.fluxlabs.net</a></span></div>
<div><br>
</div>
</div>
</span></div>
<div>
<div><br>
On Jul 28, 2015, at 10:20 AM, Mark Adams <<a href="mailto:mark@workshopit.co.uk" target="_blank">mark@workshopit.co.uk</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">Hi Jeremy,
<div><br>
</div>
<div>Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with
the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 28 July 2015 at 15:59, Jeremy McSpadden <span dir="ltr">
<<a href="mailto:jeremy@fluxlabs.net" target="_blank">jeremy@fluxlabs.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs. </div>
<div><br>
<div>
<div><span style="background-color:rgba(255,255,255,0)">--<br>
Jeremy McSpadden | Flux Labs<br>
Local - <a href="tel:850-250-5590;501" target="_blank">850-250-5590x501</a> | Mobile - <a href="tel:850-890-2543" target="_blank">850-890-2543</a> <br>
Fax - <a href="tel:850-254-2955" target="_blank">850-254-2955</a> | Toll Free - <a href="tel:877-699-FLUX" target="_blank">877-699-FLUX</a><br>
Web - <a href="http://www.fluxlabs.net/" target="_blank">http://www.fluxlabs.net</a></span></div>
<div><br>
</div>
</div>
</div>
<div>
<div>
<div><br>
On Jul 28, 2015, at 9:54 AM, Mark Adams <<a href="mailto:mark@workshopit.co.uk" target="_blank">mark@workshopit.co.uk</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that
has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 28 July 2015 at 15:14, Mark Adams <span dir="ltr"><<a href="mailto:mark@workshopit.co.uk" target="_blank">mark@workshopit.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi All,<br clear="all">
<div><br>
</div>
If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2
<div><br>
</div>
<div>Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition
also become full because of the mysql DB (it got to 14G in 2 days..).</div>
<div><br>
</div>
<div>For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was
over 9 million messages delivered on 1 day compared with the server average of about 1500!)</div>
<div><br>
</div>
<div>It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.</div>
<div><br>
</div>
<div>Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?</div>
<div><br>
</div>
<div>Thanks!</div></div></blockquote></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></blockquote></div></div></div></div></blockquote></div></div></blockquote></div></div></div></div></div></blockquote></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></blockquote></div>
</div></div>