From greminn at gmail.com  Thu Jul  2 21:12:46 2015
From: greminn at gmail.com (Simon)
Date: Fri, 3 Jul 2015 09:12:46 +1200
Subject: Stored Bad Filename Message Report sending when HIGH SPAM
In-Reply-To: <CAM+OHScAiTmYLVbyLj+_kqyYhdL+Z_U5+8uJ9qn=qprOiL6=3w@mail.gmail.com>
References: <CAM+OHScAiTmYLVbyLj+_kqyYhdL+Z_U5+8uJ9qn=qprOiL6=3w@mail.gmail.com>
Message-ID: <CAM+OHSd-VOfDfbQ_MSEdUYahrmYF9XQ5G4u67DEJ1U4L6Z9vAQ@mail.gmail.com>

Hi There,

Is there any further info i can provide on this? Would we use a filter for
this?

Thanks

Simon

On Mon, Jun 29, 2015 at 9:43 AM, Simon <greminn at gmail.com> wrote:

> Hi There,
>
> We have just started trialling MailScanner 4.85.2 on Centos 6.6 and its
> working really well.
>
> We have "Notify Senders Of Blocked Filenames Or Filetypes" = 'yes' so our
> clients get notified when an attachment has been blocked.. and this works
> well for legitimate senders. However in one case the message is clearly
> SPAM (e.g. its SA score is 12.37) - is there any way to
> stop MailScanner sending these reports in these instances?
>
> Thanks
>
> Simon
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150703/18f36d56/attachment.html>

From greminn at gmail.com  Thu Jul  2 21:40:59 2015
From: greminn at gmail.com (Simon)
Date: Fri, 3 Jul 2015 09:40:59 +1200
Subject: Blacklisted from addresses triggering SPAM notification
Message-ID: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>

Hi There,

We have incoming email from a domain that we have blacklisted for the
client. In mailscanner.conf we have set:

Spam Actions = store notify header "X-Spam-Status: Yes"
High Scoring Spam Actions = store

What is happening is that the blacklisted domain is triggering the "notify"
to the client. I would have thought that if you blacklist something thats
it.. its gone burgers!

Is there any way we can stop notify to the client in this case?

Thanks

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150703/a40deaa4/attachment.html>

From jeremy at fluxlabs.net  Thu Jul  2 21:45:55 2015
From: jeremy at fluxlabs.net (Jeremy McSpadden)
Date: Thu, 2 Jul 2015 21:45:55 +0000
Subject: Blacklisted from addresses triggering SPAM notification
In-Reply-To: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>
Message-ID: <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>

Remove notify

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 2, 2015, at 4:41 PM, Simon <greminn at gmail.com<mailto:greminn at gmail.com>> wrote:

Hi There,

We have incoming email from a domain that we have blacklisted for the client. In mailscanner.conf we have set:

Spam Actions = store notify header "X-Spam-Status: Yes"
High Scoring Spam Actions = store

What is happening is that the blacklisted domain is triggering the "notify" to the client. I would have thought that if you blacklist something thats it.. its gone burgers!

Is there any way we can stop notify to the client in this case?

Thanks

Simon


--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/listinfo/mailscanner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150702/b67786ab/attachment.html>

From greminn at gmail.com  Thu Jul  2 21:57:46 2015
From: greminn at gmail.com (Simon)
Date: Fri, 3 Jul 2015 09:57:46 +1200
Subject: Blacklisted from addresses triggering SPAM notification
In-Reply-To: <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>
 <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>
Message-ID: <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>

Hi Jeremy,

OK... but wouldn't that also stop mailscanner notifying clients of
(non-blacklisted) SPAM?

I guess from what i'm thinking if someone adds something to a blacklist,
then they are making a choice not to get anything from that email/domain
anymore, where as email that comes in that is (low?) spam they might want
to be notified to then take action to release/train etc...

Simon

On Fri, Jul 3, 2015 at 9:45 AM, Jeremy McSpadden <jeremy at fluxlabs.net>
wrote:

>  Remove notify
>
>  --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
> Web - http://www.fluxlabs.net
>
>
> On Jul 2, 2015, at 4:41 PM, Simon <greminn at gmail.com> wrote:
>
>   Hi There,
>
>  We have incoming email from a domain that we have blacklisted for the
> client. In mailscanner.conf we have set:
>
>  Spam Actions = store notify header "X-Spam-Status: Yes"
> High Scoring Spam Actions = store
>
>  What is happening is that the blacklisted domain is triggering the
> "notify" to the client. I would have thought that if you blacklist
> something thats it.. its gone burgers!
>
>  Is there any way we can stop notify to the client in this case?
>
>  Thanks
>
>  Simon
>
>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
>
>
>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150703/4928095c/attachment.html>

From greminn at gmail.com  Thu Jul  2 22:37:09 2015
From: greminn at gmail.com (Simon)
Date: Fri, 3 Jul 2015 10:37:09 +1200
Subject: Blacklisted from addresses triggering SPAM notification
In-Reply-To: <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>
 <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>
 <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>
Message-ID: <CAM+OHScHkRt+k154N+mjK92VkRroHiyF+oQcaaFTNjG2zijg3A@mail.gmail.com>

An update here.. it looks like we didn't have mailscanner configured
correctly. As we don't 'notify' high spam.. this setting worked perfectly
for us:

# Setting this to yes means that spam found in the blacklist is treated
# as "High Scoring Spam" in the "Spam Actions" section below. Setting it
# to no means that it will be treated as "normal" spam.
# This can also be the filename of a ruleset.
Definite Spam Is High Scoring = yes

Thanks!

On Fri, Jul 3, 2015 at 9:57 AM, Simon <greminn at gmail.com> wrote:

> Hi Jeremy,
>
> OK... but wouldn't that also stop mailscanner notifying clients of
> (non-blacklisted) SPAM?
>
> I guess from what i'm thinking if someone adds something to a blacklist,
> then they are making a choice not to get anything from that email/domain
> anymore, where as email that comes in that is (low?) spam they might want
> to be notified to then take action to release/train etc...
>
> Simon
>
> On Fri, Jul 3, 2015 at 9:45 AM, Jeremy McSpadden <jeremy at fluxlabs.net>
> wrote:
>
>>  Remove notify
>>
>>  --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>> Web - http://www.fluxlabs.net
>>
>>
>> On Jul 2, 2015, at 4:41 PM, Simon <greminn at gmail.com> wrote:
>>
>>   Hi There,
>>
>>  We have incoming email from a domain that we have blacklisted for the
>> client. In mailscanner.conf we have set:
>>
>>  Spam Actions = store notify header "X-Spam-Status: Yes"
>> High Scoring Spam Actions = store
>>
>>  What is happening is that the blacklisted domain is triggering the
>> "notify" to the client. I would have thought that if you blacklist
>> something thats it.. its gone burgers!
>>
>>  Is there any way we can stop notify to the client in this case?
>>
>>  Thanks
>>
>>  Simon
>>
>>
>>
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>>
>>
>>
>>
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150703/f668c36e/attachment.html>

From levnagdimunov0 at hotmail.com  Fri Jul 10 01:20:42 2015
From: levnagdimunov0 at hotmail.com (Lev Nagdimunov)
Date: Thu, 9 Jul 2015 21:20:42 -0400
Subject: Mailscanner and Base64
In-Reply-To: <CAM+OHScHkRt+k154N+mjK92VkRroHiyF+oQcaaFTNjG2zijg3A@mail.gmail.com>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>,
 <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>,
 <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>,
 <CAM+OHScHkRt+k154N+mjK92VkRroHiyF+oQcaaFTNjG2zijg3A@mail.gmail.com>
Message-ID: <BLU180-W44DBBEF9FF4933A2D9126E8A9F0@phx.gbl>

Hello,

I reported this issue earlier and was told there was not time to fix it then. Wondering if this is still the case, or if someone can point me at least approximately to where code dealing with base64 would be located in Mailscanner's source. 

The issue is that Mailscanner will change the length of 
encoded (at least base64 encoded) email on anything greater than 60 
characters down to 60 characters in the spool file for Exim. For any DKIM 
signed email, this will break the body hash. Normally it's not a problem
 since it's already been processed by the MTA at that point, but if you 
do a blind forward afterward then it is a problem. 

Thank you.  		 	   		  
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150709/73e4bc3f/attachment.html>

From jerry.benton at mailborder.com  Fri Jul 10 01:25:19 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Thu, 9 Jul 2015 21:25:19 -0400
Subject: Mailscanner and Base64
In-Reply-To: <BLU180-W44DBBEF9FF4933A2D9126E8A9F0@phx.gbl>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>
 <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>
 <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>
 <CAM+OHScHkRt+k154N+mjK92VkRroHiyF+oQcaaFTNjG2zijg3A@mail.gmail.com>
 <BLU180-W44DBBEF9FF4933A2D9126E8A9F0@phx.gbl>
Message-ID: <1615AA20-A36A-4756-A49A-9CA0E947D493@mailborder.com>

Have you tested the latest version of MailScanner?

https://www.mailscanner.info/downloads/ <https://www.mailscanner.info/downloads/>

-
Jerry Benton
www.mailborder.com



> On Jul 9, 2015, at 9:20 PM, Lev Nagdimunov <levnagdimunov0 at hotmail.com> wrote:
> 
> Hello,
> 
> I reported this issue earlier and was told there was not time to fix it then. Wondering if this is still the case, or if someone can point me at least approximately to where code dealing with base64 would be located in Mailscanner's source. 
> 
> The issue is that Mailscanner will change the length of encoded (at least base64 encoded) email on anything greater than 60 characters down to 60 characters in the spool file for Exim. For any DKIM signed email, this will break the body hash. Normally it's not a problem since it's already been processed by the MTA at that point, but if you do a blind forward afterward then it is a problem. 
> 
> Thank you.  
> 
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150709/25161a19/attachment.html>

From levnagdimunov0 at hotmail.com  Fri Jul 10 15:54:58 2015
From: levnagdimunov0 at hotmail.com (Lev Nagdimunov)
Date: Fri, 10 Jul 2015 11:54:58 -0400
Subject: Mailscanner and Base64
In-Reply-To: <1615AA20-A36A-4756-A49A-9CA0E947D493@mailborder.com>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>,
 <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>,
 <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>,
 <CAM+OHScHkRt+k154N+mjK92VkRroHiyF+oQcaaFTNjG2zijg3A@mail.gmail.com>,
 <BLU180-W44DBBEF9FF4933A2D9126E8A9F0@phx.gbl>,
 <1615AA20-A36A-4756-A49A-9CA0E947D493@mailborder.com>
Message-ID: <BLU180-W45325BE0D9618CAEE278DA8A9F0@phx.gbl>

Hello Jerry,

Thanks for the prompt response. I haven't tested any new version since you took over; I will test it tonight or tomorrow night.

   Lev 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150710/8994de7c/attachment.html>

From jerry.benton at mailborder.com  Fri Jul 10 15:57:37 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Fri, 10 Jul 2015 11:57:37 -0400
Subject: Mailscanner and Base64
In-Reply-To: <BLU180-W45325BE0D9618CAEE278DA8A9F0@phx.gbl>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>
 <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>
 <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>
 <CAM+OHScHkRt+k154N+mjK92VkRroHiyF+oQcaaFTNjG2zijg3A@mail.gmail.com>
 <BLU180-W44DBBEF9FF4933A2D9126E8A9F0@phx.gbl>
 <1615AA20-A36A-4756-A49A-9CA0E947D493@mailborder.com>
 <BLU180-W45325BE0D9618CAEE278DA8A9F0@phx.gbl>
Message-ID: <E84D3AB6-9D05-4B26-9BC1-37083697BA38@mailborder.com>

I would suggest doing a fresh install using the included installer for your test.

-
Jerry Benton
www.mailborder.com
Sent from my iPhone

> On Jul 10, 2015, at 11:54, Lev Nagdimunov <levnagdimunov0 at hotmail.com> wrote:
> 
> Hello Jerry,
> 
> Thanks for the prompt response. I haven't tested any new version since you took over; I will test it tonight or tomorrow night.
> 
>    Lev
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150710/1736531f/attachment.html>

From levnagdimunov0 at hotmail.com  Sat Jul 11 03:08:13 2015
From: levnagdimunov0 at hotmail.com (Lev Nagdimunov)
Date: Fri, 10 Jul 2015 23:08:13 -0400
Subject: Mailscanner and Base64
In-Reply-To: <E84D3AB6-9D05-4B26-9BC1-37083697BA38@mailborder.com>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>,
 <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>,
 <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>,
 <CAM+OHScHkRt+k154N+mjK92VkRroHiyF+oQcaaFTNjG2zijg3A@mail.gmail.com>,
 <BLU180-W44DBBEF9FF4933A2D9126E8A9F0@phx.gbl>,
 <1615AA20-A36A-4756-A49A-9CA0E947D493@mailborder.com>,
 <BLU180-W45325BE0D9618CAEE278DA8A9F0@phx.gbl>,
 <E84D3AB6-9D05-4B26-9BC1-37083697BA38@mailborder.com>
Message-ID: <BLU180-W7467719FD4A24353011EE8A9E0@phx.gbl>

Hello Jerry,

I downloaded the v4.85.2-3 RPM for Redhat/CentOS and used its included installer. However, I still get the same issue. Its very clear: in the spool prior to MailScanner scanning the file, the exim -D file will have the original length of each base64 encoded line. Once MailScanner scans the file, the version of the exim -D file copied to the outgoing spool changes each line to be 60 characters long (all the data is preserved so the file has more lines). 

Thank you,

   Lev

From: jerry.benton at mailborder.com
Subject: Re: Mailscanner and Base64
Date: Fri, 10 Jul 2015 11:57:37 -0400
To: mailscanner at lists.mailscanner.info

I would suggest doing a fresh install using the included installer for your test.

-Jerry Bentonwww.mailborder.comSent from my iPhone 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150710/968d4a6d/attachment.html>

From jerry.benton at mailborder.com  Sat Jul 11 04:29:44 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Sat, 11 Jul 2015 00:29:44 -0400
Subject: Mailscanner and Base64
In-Reply-To: <BLU180-W7467719FD4A24353011EE8A9E0@phx.gbl>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>
 <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>
 <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>
 <CAM+OHScHkRt+k154N+mjK92VkRroHiyF+oQcaaFTNjG2zijg3A@mail.gmail.com>
 <BLU180-W44DBBEF9FF4933A2D9126E8A9F0@phx.gbl>
 <1615AA20-A36A-4756-A49A-9CA0E947D493@mailborder.com>
 <BLU180-W45325BE0D9618CAEE278DA8A9F0@phx.gbl>
 <E84D3AB6-9D05-4B26-9BC1-37083697BA38@mailborder.com>
 <BLU180-W7467719FD4A24353011EE8A9E0@phx.gbl>
Message-ID: <A0113234-AD0E-4C5F-862E-2020205A20B2@mailborder.com>

Andrew is the Exim expert. Let’s see if he has an opinion. Thanks for testing this by the way.

-
Jerry Benton
www.mailborder.com



> On Jul 10, 2015, at 11:08 PM, Lev Nagdimunov <levnagdimunov0 at hotmail.com> wrote:
> 
> Hello Jerry,
> 
> I downloaded the v4.85.2-3 RPM for Redhat/CentOS and used its included installer. However, I still get the same issue. Its very clear: in the spool prior to MailScanner scanning the file, the exim -D file will have the original length of each base64 encoded line. Once MailScanner scans the file, the version of the exim -D file copied to the outgoing spool changes each line to be 60 characters long (all the data is preserved so the file has more lines). 
> 
> Thank you,
> 
>    Lev
> 
> From: jerry.benton at mailborder.com
> Subject: Re: Mailscanner and Base64
> Date: Fri, 10 Jul 2015 11:57:37 -0400
> To: mailscanner at lists.mailscanner.info
> 
> I would suggest doing a fresh install using the included installer for your test.
> 
> -
> Jerry Benton
> www.mailborder.com <http://www.mailborder.com/>Sent from my iPhone
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150711/484aa71e/attachment.html>

From levnagdimunov0 at hotmail.com  Sat Jul 11 21:55:17 2015
From: levnagdimunov0 at hotmail.com (Lev Nagdimunov)
Date: Sat, 11 Jul 2015 17:55:17 -0400
Subject: Mailscanner and Base64
In-Reply-To: <A0113234-AD0E-4C5F-862E-2020205A20B2@mailborder.com>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>,
 <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>,
 <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>,
 <CAM+OHScHkRt+k154N+mjK92VkRroHiyF+oQcaaFTNjG2zijg3A@mail.gmail.com>,
 <BLU180-W44DBBEF9FF4933A2D9126E8A9F0@phx.gbl>,
 <1615AA20-A36A-4756-A49A-9CA0E947D493@mailborder.com>,
 <BLU180-W45325BE0D9618CAEE278DA8A9F0@phx.gbl>,
 <E84D3AB6-9D05-4B26-9BC1-37083697BA38@mailborder.com>,
 <BLU180-W7467719FD4A24353011EE8A9E0@phx.gbl>,
 <A0113234-AD0E-4C5F-862E-2020205A20B2@mailborder.com>
Message-ID: <BLU180-W172845367AC9963B4166EA8A9E0@phx.gbl>

Jerry,

Found the issue, mostly its between the chair and the keyboard. Turns out the specific emails I was using as a test are not only base64 but they contain something MailScanner was disarming. Disarming anything breaks DKIM anyway since it changes the content, therefore the base64 issue becomes irrelevant. And it looks like base64 line length is changed only when the body is modified (e.g. disarmed) by mailscanner. Probably should add a note in the configuration about DKIM and disarming similar to the one that already exists for DKIM and headers. An option to not disarm only when the email is DKIM signed would be great but that's a feature request and not a bug like I thought. 

Thank you,

   Lev

Subject: Re: Mailscanner and Base64
From: jerry.benton at mailborder.com
Date: Sat, 11 Jul 2015 00:29:44 -0400
To: mailscanner at lists.mailscanner.info

Andrew is the Exim expert. Let’s see if he has an opinion. Thanks for testing this by the way.

-Jerry Bentonwww.mailborder.com



On Jul 10, 2015, at 11:08 PM, Lev Nagdimunov <levnagdimunov0 at hotmail.com> wrote:Hello Jerry,

I downloaded the v4.85.2-3 RPM for Redhat/CentOS and used its included installer. However, I still get the same issue. Its very clear: in the spool prior to MailScanner scanning the file, the exim -D file will have the original length of each base64 encoded line. Once MailScanner scans the file, the version of the exim -D file copied to the outgoing spool changes each line to be 60 characters long (all the data is preserved so the file has more lines). 

Thank you,

   Lev

From: jerry.benton at mailborder.com
Subject: Re: Mailscanner and Base64
Date: Fri, 10 Jul 2015 11:57:37 -0400
To: mailscanner at lists.mailscanner.info

I would suggest doing a fresh install using the included installer for your test.

-Jerry Bentonwww.mailborder.comSent from my iPhone-- MailScanner mailing listmailscanner at lists.mailscanner.infohttp://lists.mailscanner.info/listinfo/mailscanner

-- 
MailScanner mailing list
mailscanner at lists.mailscanner.info
http://lists.mailscanner.info/listinfo/mailscanner 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150711/4defc74f/attachment.html>

From andrew at topdog.za.net  Sat Jul 11 22:53:43 2015
From: andrew at topdog.za.net (Andrew Colin Kissa)
Date: Sun, 12 Jul 2015 00:53:43 +0200
Subject: Mailscanner and Base64
In-Reply-To: <BLU180-W172845367AC9963B4166EA8A9E0@phx.gbl>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>,
 <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>,
 <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>,
 <CAM+OHScHkRt+k154N+mjK92VkRroHiyF+oQcaaFTNjG2zijg3A@mail.gmail.com>,
 <BLU180-W44DBBEF9FF4933A2D9126E8A9F0@phx.gbl>,
 <1615AA20-A36A-4756-A49A-9CA0E947D493@mailborder.com>,
 <BLU180-W45325BE0D9618CAEE278DA8A9F0@phx.gbl>,
 <E84D3AB6-9D05-4B26-9BC1-37083697BA38@mailborder.com>,
 <BLU180-W7467719FD4A24353011EE8A9E0@phx.gbl>,
 <A0113234-AD0E-4C5F-862E-2020205A20B2@mailborder.com>
 <BLU180-W172845367AC9963B4166EA8A9E0@phx.gbl>
Message-ID: <3F87F5AF-7095-40D4-8A3D-E2D46109DA1D@topdog.za.net>


On 11 Jul 2015, at 11:55 PM, Lev Nagdimunov <levnagdimunov0 at hotmail.com> wrote:

> An option to not disarm only when the email is DKIM signed would be great but that's a feature request and not a bug like I thought.

I would think disarming still trumps signing, your destination server should not be performing
DKIM checks in any case.

For outbound messages you should not have an issue as messages should only be signed
after they have been processed by MailScanner

- Andrew

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150712/afadcf76/attachment.sig>

From levnagdimunov0 at hotmail.com  Sun Jul 12 04:15:40 2015
From: levnagdimunov0 at hotmail.com (Lev Nagdimunov)
Date: Sun, 12 Jul 2015 00:15:40 -0400
Subject: Mailscanner and Base64
In-Reply-To: <3F87F5AF-7095-40D4-8A3D-E2D46109DA1D@topdog.za.net>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>, ,
 <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>, ,
 <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>, ,
 <CAM+OHScHkRt+k154N+mjK92VkRroHiyF+oQcaaFTNjG2zijg3A@mail.gmail.com>, ,
 <BLU180-W44DBBEF9FF4933A2D9126E8A9F0@phx.gbl>, ,
 <1615AA20-A36A-4756-A49A-9CA0E947D493@mailborder.com>, ,
 <BLU180-W45325BE0D9618CAEE278DA8A9F0@phx.gbl>, ,
 <E84D3AB6-9D05-4B26-9BC1-37083697BA38@mailborder.com>, ,
 <BLU180-W7467719FD4A24353011EE8A9E0@phx.gbl>, ,
 <A0113234-AD0E-4C5F-862E-2020205A20B2@mailborder.com>,
 <BLU180-W172845367AC9963B4166EA8A9E0@phx.gbl>,
 <3F87F5AF-7095-40D4-8A3D-E2D46109DA1D@topdog.za.net>
Message-ID: <BLU180-W68057064ED4235528A08A28A9D0@phx.gbl>

Hello Andrew,

The issue comes up when doing immediate forwarding. That is, a people setup an email address to forward to some other address that they actually check (whether that be because the other address is old and rarely used anymore, or because they don't receive enough traffic to check that address very often or some other reason). Therefore, they receive email and have immediate forwarding setup via virtual aliases. This immediate forwarding, if disarmed, will get rejected if the email is DKIM signed. The only solution I can see currently is to disable disarming, but it would be nice to have the ability to disable it only in cases where there is a DKIM signature present.

    Lev

From: andrew at topdog.za.net
Subject: Re: Mailscanner and Base64
Date: Sun, 12 Jul 2015 00:53:43 +0200
To: mailscanner at lists.mailscanner.info

 
On 11 Jul 2015, at 11:55 PM, Lev Nagdimunov <levnagdimunov0 at hotmail.com> wrote:
 
> An option to not disarm only when the email is DKIM signed would be great but that's a feature request and not a bug like I thought.
 
I would think disarming still trumps signing, your destination server should not be performing
DKIM checks in any case.
 
For outbound messages you should not have an issue as messages should only be signed
after they have been processed by MailScanner
 
- Andrew
 

-- 
MailScanner mailing list
mailscanner at lists.mailscanner.info
http://lists.mailscanner.info/listinfo/mailscanner 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150712/3ad5da8e/attachment.html>

From andrew at topdog.za.net  Sun Jul 12 06:16:35 2015
From: andrew at topdog.za.net (Andrew Colin Kissa)
Date: Sun, 12 Jul 2015 08:16:35 +0200
Subject: Mailscanner and Base64
In-Reply-To: <BLU180-W68057064ED4235528A08A28A9D0@phx.gbl>
References: <CAM+OHScA-8oN9EGevxNUdFpKEuriNZf5+Gvis3chwkFd-c4EWg@mail.gmail.com>, ,
 <BFC14FA6-E248-46E3-B9B8-CC56A9DF2F9D@fluxlabs.net>, ,
 <CAM+OHSeLBj-CzzN5u97vKPXAKucihhbS9gOJ78bTpkynxcfr1A@mail.gmail.com>, ,
 <CAM+OHScHkRt+k154N+mjK92VkRroHiyF+oQcaaFTNjG2zijg3A@mail.gmail.com>, ,
 <BLU180-W44DBBEF9FF4933A2D9126E8A9F0@phx.gbl>, ,
 <1615AA20-A36A-4756-A49A-9CA0E947D493@mailborder.com>, ,
 <BLU180-W45325BE0D9618CAEE278DA8A9F0@phx.gbl>, ,
 <E84D3AB6-9D05-4B26-9BC1-37083697BA38@mailborder.com>, ,
 <BLU180-W7467719FD4A24353011EE8A9E0@phx.gbl>, ,
 <A0113234-AD0E-4C5F-862E-2020205A20B2@mailborder.com>,
 <BLU180-W172845367AC9963B4166EA8A9E0@phx.gbl>,
 <3F87F5AF-7095-40D4-8A3D-E2D46109DA1D@topdog.za.net>
 <BLU180-W68057064ED4235528A08A28A9D0@phx.gbl>
Message-ID: <3F646051-7595-4EF9-B796-1E9DFD6C1AB9@topdog.za.net>

Hi Lev,

On 12 Jul 2015, at 6:15 AM, Lev Nagdimunov <levnagdimunov0 at hotmail.com> wrote:

> The issue comes up when doing immediate forwarding. That is, a people setup an email address to forward to some other address that they actually check (whether that be because the other address is old and rarely used anymore, or because they don't receive enough traffic to check that address very often or some other reason). Therefore, they receive email and have immediate forwarding setup via virtual aliases. This immediate forwarding, if disarmed, will get rejected if the email is DKIM signed. The only solution I can see currently is to disable disarming, but it would be nice to have the ability to disable it only in cases where there is a DKIM signature present.

The solution to that is to use SRS for all forwarded email.

- Andrew

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150712/10c20514/attachment.sig>

From wt at dld2000.com  Tue Jul 14 13:17:53 2015
From: wt at dld2000.com (Walt Thiessen)
Date: Tue, 14 Jul 2015 09:17:53 -0400
Subject: individual white lists
Message-ID: <55A50C01.7040204@dld2000.com>

Hello,

I'm a new mailscanner user, and I have a question about white lists.

I’ve been thinking a lot lately about the problem of email spam, and I 
want to try something.

I want to create a white list system where the white list is 
individualized per email account and where it can be updated via a 
php/mysql script.

Can anyone advise me whether there’s a way to accomplish this with 
mailscanner?

Walt

From mike at sentinelbox.net  Tue Jul 14 13:47:39 2015
From: mike at sentinelbox.net (michael pap)
Date: Tue, 14 Jul 2015 09:47:39 -0400
Subject: individual white lists
In-Reply-To: <55A50C01.7040204@dld2000.com>
References: <55A50C01.7040204@dld2000.com>
Message-ID: <B6ED4CF6F501A44C8A063DB139649A984C0A078A@ORION.emfabox.local>

Hi Walt,

Take a look at  MailScanner ->CustomFunctions ->  SpamWhitelist.pm 

Or check out MailWatch -> MailScanner_perl_scripts -> SQLBlackWhiteList.pm & SQLSpamSettings.pm & MailWatch.pm

Michael
www.emfabox.com




-----Original Message-----
From: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] On Behalf Of Walt Thiessen
Sent: Tuesday, July 14, 2015 9:18 AM
To: mailscanner at lists.mailscanner.info
Subject: individual white lists

Hello,

I'm a new mailscanner user, and I have a question about white lists.

I’ve been thinking a lot lately about the problem of email spam, and I want to try something.

I want to create a white list system where the white list is individualized per email account and where it can be updated via a php/mysql script.

Can anyone advise me whether there’s a way to accomplish this with mailscanner?

Walt


--
MailScanner mailing list
mailscanner at lists.mailscanner.info
http://lists.mailscanner.info/listinfo/mailscanner


--
This message has been checked by EMFABox and is found to be clean.

Follow this link to mark it as spam:
http://mail.sentinelbox.net/cgi-bin/learn-msg.cgi?id=D8992E2548.AD644
Follow this link to blacklist sender:
http://mail.sentinelbox.net/cgi-bin/learn-msg.cgi?blacklist=1&id=D8992E2548.AD644


--
This email has been scanned by the EMFABox eMail service.
ID: 558E542977.A72F7


From jerry.benton at mailborder.com  Tue Jul 14 15:43:16 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Tue, 14 Jul 2015 11:43:16 -0400
Subject: individual white lists
In-Reply-To: <55A50C01.7040204@dld2000.com>
References: <55A50C01.7040204@dld2000.com>
Message-ID: <41EF5653-F15D-4D1F-B64B-6781E87292AC@mailborder.com>

Mailborder can also easily do this. (GUI for MailScanner) There is a free version. 

-
Jerry Benton
www.mailborder.com



> On Jul 14, 2015, at 9:17 AM, Walt Thiessen <wt at dld2000.com> wrote:
> 
> Hello,
> 
> I'm a new mailscanner user, and I have a question about white lists.
> 
> I’ve been thinking a lot lately about the problem of email spam, and I want to try something.
> 
> I want to create a white list system where the white list is individualized per email account and where it can be updated via a php/mysql script.
> 
> Can anyone advise me whether there’s a way to accomplish this with mailscanner?
> 
> Walt
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 


From greminn at gmail.com  Tue Jul 14 22:01:34 2015
From: greminn at gmail.com (Simon)
Date: Wed, 15 Jul 2015 10:01:34 +1200
Subject: Recipient.spam.report from variable
Message-ID: <CAM+OHSfm7EVpCSJ1nBO6N-jGgy6stnfdpMh8WAug2wrm7W6CRA@mail.gmail.com>

Hi There,

Using postfix on Centos 6.6 and latest mailscanner...

We are sending our users the recipient.spam.report.txt alert on spam (not
high-spam). A question that has come up quite often is that the $from does
not contain the actual "From" address. e.g:

From: bounce.491fc2.c7b9ec1.sally=blabla.co.nz at multi262.postfix.bmsend.com

Rather than:

From: Emily.Jones at whateverdomain.com

The issue our users find is that they dont actually know who
bounce.491fc2.c7b9ec1.sally=blabla.co.nz at multi262.postfix.bmsend.com is so
they cant make a decision on if to get it released or not.

Is there any way to show the From address rather than the return path? or
is this the way postfix works with MailScanner?

Thanks

Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150715/7ada2500/attachment.html>

From mark at msapiro.net  Wed Jul 15 00:51:31 2015
From: mark at msapiro.net (Mark Sapiro)
Date: Tue, 14 Jul 2015 17:51:31 -0700
Subject: Recipient.spam.report from variable
In-Reply-To: <CAM+OHSfm7EVpCSJ1nBO6N-jGgy6stnfdpMh8WAug2wrm7W6CRA@mail.gmail.com>
References: <CAM+OHSfm7EVpCSJ1nBO6N-jGgy6stnfdpMh8WAug2wrm7W6CRA@mail.gmail.com>
Message-ID: <55A5AE93.9070105@msapiro.net>

On 7/14/15 3:01 PM, Simon wrote:
> Hi There,
> 
> Using postfix on Centos 6.6 and latest mailscanner...
> 
> We are sending our users the recipient.spam.report.txt alert on spam
> (not high-spam). A question that has come up quite often is that
> the $from does not contain the actual "From" address. e.g:
> 
> From:
> bounce.491fc2.c7b9ec1.sally=blabla.co.nz at multi262.postfix.bmsend.com
> 
> Rather than:
> 
> From: Emily.Jones at whateverdomain.com


Right. The $from replacement in report templates is the SMTP envelope
from (return path), not the From: header.


> The issue our users find is that they dont actually know who
> bounce.491fc2.c7b9ec1.sally=blabla.co.nz at multi262.postfix.bmsend.com
> is so they cant make a
> decision on if to get it released or not. 


One could argue that the envelope from is a better indicator of whether
or not you want the message than is the From: header. From: headers are
often spoofed in spam. An envelope sender such as the above says to me
that the message is bulk mail from some kind of advertizing service or
mail list, thus the local part which returns to some bounce processor
with an encoding of the recipient address. If I don't recognize the
domain as that of a list I subscribe to, it's spam.

The bottom line however is that if a message appears to be From: or the
envelope is from an address whose mail you don't want, then you don't
want it, but even if it appears to be from your best friend, if it was
scored as spam, it's probably spam.

I know that doesn't address your issue, but it's something to think about.


> Is there any way to show the From address rather than the return path?
> or is this the way postfix works with MailScanner?


It's not just Postfix; it's all MTAs. MTAs deal with SMTP envelope
senders and recipients, not addresses in message headers which are just
part of the message payload as far as the MTA is concerned.

That said, MailScanner does look at the message and extracts, e.g. the
Subject: as $subject and Date: as $date and certainly could extract the
From: and assign it to a replacement variable, but a cursory look at the
code says it doesn't currently do that.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

From greminn at gmail.com  Wed Jul 15 02:14:53 2015
From: greminn at gmail.com (Simon)
Date: Wed, 15 Jul 2015 14:14:53 +1200
Subject: Recipient.spam.report from variable
In-Reply-To: <55A5AE93.9070105@msapiro.net>
References: <CAM+OHSfm7EVpCSJ1nBO6N-jGgy6stnfdpMh8WAug2wrm7W6CRA@mail.gmail.com>
 <55A5AE93.9070105@msapiro.net>
Message-ID: <288601AF-D728-4EE8-BB7F-25DF63DB43D0@gmail.com>


> On 15/07/2015, at 12:51 pm, Mark Sapiro <mark at msapiro.net> wrote:
> 
> One could argue that the envelope from is a better indicator of whether
> or not you want the message than is the From: header. From: headers are
> often spoofed in spam. An envelope sender such as the above says to me
> that the message is bulk mail from some kind of advertizing service or
> mail list, thus the local part which returns to some bounce processor
> with an encoding of the recipient address. If I don't recognize the
> domain as that of a list I subscribe to, it's spam.

Oh i agree with you here :)

> That said, MailScanner does look at the message and extracts, e.g. the
> Subject: as $subject and Date: as $date and certainly could extract the
> From: and assign it to a replacement variable, but a cursory look at the
> code says it doesn't currently do that.

It would be *nice* to be able to provide both to a client (which allows them to make a better informed decision). Maybe, as a suggestion, Mailscanner could provide both $from and $envelope_from  as variables in the reports?

Thanks

Simon

From vpdose at kirchenweg.de  Wed Jul 22 12:21:47 2015
From: vpdose at kirchenweg.de (Volker Dose)
Date: Wed, 22 Jul 2015 14:21:47 +0200
Subject: MailScanner: allowing attachments identified as text/plain by file -i
Message-ID: <C7EB6B94-8471-4282-9B2F-FC68948401B6@kirchenweg.de>

Hi list,
 
I am struggling with the ”magic”  fifth field in filetype.rules.conf – as so many others in the past, as far as I understand old posting.
 
Let me explain my settings:
 
I have a list of attachments, I do allow in filetype.rules.conf (like text, pics, html, pdf and other stuff) and  the last line is a deny for every other attachment. I did this, because I do not want to get anything to my mailserver, where I am not 100% sure of the filetype – so executables are banned and also every unknown  filetype.
 
This file looks like this:
 
 
-------
allow   ASCII text      ASCII text      ASCII text
allow   PC bitmap       PC bitmap       PC bitmap
allow   Emacs v18       Emacs v18       Emacs v18
allow   C++ source      C++ source      C++ source
allow   source          diverse source  diverse source
[…]
deny            .*      Deny unidentified attachments                   Deny unidentified attachments
----------
 
 
But  from time to time I get a false positive, often non-english text-parts are not very good identified, like Finnish or east-European languages.  Often the pdf attachment is identified fine and mailscanner processes it,  but txt and html-parts are too often blocked.
 

 
But using the file –I command I have a much higher rate of messages identified as text or html mail-part.
 
So I wanted to use this  feature Julian implemented 2008:
 
 
------------
This 5th field is optional, and specifies a regular expression which is
matched against the MIME type as determined by the "file -i" command.
 
If it is never specified, then the "file -i" command will never be run
on your message attachments so there is no appreciable overhead on the
speed of MailScanner caused by this new feature.
 
If the "mime type" *and* the filetype fields are both specified (and are
not "-") then either matching will cause the rule to fire. In a "deny"
rule like the example above, then *either* test firing will cause the
attachment to be blocked. In an "allow" rule then *both* of the tests
must pass to cause the attachment to be allowed and hence no more rules
to be checked. This sounds a bit odd but actually ends up doing pretty
much what you expect it to. I'm sure you'll let me know if I'm wrong
there :-)
---------
 
I added a line like this in my filetype.rules.conf:
 
allow         -                            text/plain                      -                       -
 
But the message mentioned above still triggered my last line
 
deny            .*      Deny unidentified attachments                   Deny unidentified attachments
 

For example: Yesterday I realized, the text-message of an email (starting with the string “THX!”) war identified as “AHX version” from my file (version 5.14) command but as text/plain with „file -i"

I understand the text from Julian, that both the “file” and the “file -i”-field has to match  and added a line like this:
 
allow   AHX version     text/plain      -       -
 
Which works – but only because I  have added the “file”-regex to that line, too.
 
I am looking for a “match all” at that point – the dash “-“ did not work for me.
 
I wonder if there is a  way to allow  any attachments, that give you a “text/plain” when using “file –i”.
 
 
Any help appreciated!
 
I am using MS-4.84.6-1 on a CentOS 6.6 32 bit.
 
And by the way: I love MailScanner – thanks to all of you helping make the software work.
 
Best regards
Volker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150722/4bb50ecb/attachment.html>

From wilson.galafassi at gmail.com  Tue Jul 28 13:34:21 2015
From: wilson.galafassi at gmail.com (Wilson A. Galafassi Jr. [Gmail])
Date: Tue, 28 Jul 2015 10:34:21 -0300
Subject: pdf corruption
Message-ID: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>

Hi,

I'm currently using 4.84.5 to store all my messages. I have a problem with
some PDF files been corrupted after mailscanner process the files. 

Some ide ato fix this issue or tell mailscanner to don't process pdf files?

Thanks,
Wilson
 




From mark at workshopit.co.uk  Tue Jul 28 14:14:17 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Tue, 28 Jul 2015 15:14:17 +0100
Subject: Duplicated messages
Message-ID: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>

Hi All,

If anyone could provide advice that would be great. Running Debian Wheezy
Mailscanner 4.79.11-2.2

Our incoming dir filled up just before the weekend so we didn't see the
issue for a couple of days. Normally we would just shut down mailcleaner
and delete the dir then start it up again and all would be ok. However on
this occasion, the root partition also become full because of the mysql DB
(it got to 14G in 2 days..).

For some reason everything started duplicating. I can see lots of incoming
messages in the exim logs with duplication (2 or 4 of what looks like the
same email) but in the mailscanner database there is hundreds of each email
listed (apparently there was over 9 million messages delivered on 1 day
compared with the server average of about 1500!)

It seems like some sort of loop, but afaik nothing specific was changed in
the config apart from the fact incoming became full. Space has been cleared
on the root partition and incoming, and everything appears to be running as
normal right now.

Any advice on debugging this would be much appreciated, also, how best
should I clear out the DB of all the dupes?

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/886e8b40/attachment.html>

From mark at workshopit.co.uk  Tue Jul 28 14:54:15 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Tue, 28 Jul 2015 15:54:15 +0100
Subject: Duplicated messages
In-Reply-To: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
Message-ID: <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>

An update to this, the "2 or 4" duplicates showing in the exim log look
like they are actually just separate deliveries to other addresses, so not
duplicates. In 1 example there is a single email with 2 recipients (2
entries in exim log) that has over 1500+ entries in the mailcleaner DB. It
looks like this email hasn't been delivered to the recipient at all either.

On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk> wrote:

> Hi All,
>
> If anyone could provide advice that would be great. Running Debian Wheezy
> Mailscanner 4.79.11-2.2
>
> Our incoming dir filled up just before the weekend so we didn't see the
> issue for a couple of days. Normally we would just shut down mailcleaner
> and delete the dir then start it up again and all would be ok. However on
> this occasion, the root partition also become full because of the mysql DB
> (it got to 14G in 2 days..).
>
> For some reason everything started duplicating. I can see lots of incoming
> messages in the exim logs with duplication (2 or 4 of what looks like the
> same email) but in the mailscanner database there is hundreds of each email
> listed (apparently there was over 9 million messages delivered on 1 day
> compared with the server average of about 1500!)
>
> It seems like some sort of loop, but afaik nothing specific was changed in
> the config apart from the fact incoming became full. Space has been cleared
> on the root partition and incoming, and everything appears to be running as
> normal right now.
>
> Any advice on debugging this would be much appreciated, also, how best
> should I clear out the DB of all the dupes?
>
> Thanks!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/339fe5ff/attachment.html>

From jeremy at fluxlabs.net  Tue Jul 28 14:59:01 2015
From: jeremy at fluxlabs.net (Jeremy McSpadden)
Date: Tue, 28 Jul 2015 14:59:01 +0000
Subject: Duplicated messages
In-Reply-To: <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>,
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
Message-ID: <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>

It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.

On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:
Hi All,

If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2

Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).

For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)

It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.

Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?

Thanks!



--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/listinfo/mailscanner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/7eb46cf9/attachment.html>

From mark at workshopit.co.uk  Tue Jul 28 15:20:29 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Tue, 28 Jul 2015 16:20:29 +0100
Subject: Duplicated messages
In-Reply-To: <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
Message-ID: <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>

Hi Jeremy,

Are you saying that something in these messages is crashing Mailscanner?
Everything seems to be OK right now, but all 70 of the emails (all
different types and from different servers) are now in the quarantine
because of "Other Bad Content Detected" with the report "MailScanner:
Message attempted to kill MailScanner". It seems it succeeded...

On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:

>  It's probably looping/crashing mailscanner. Drop MS into debug mode and
> watch logs.
>
>  --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
> Web - http://www.fluxlabs.net
>
>
> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
>   An update to this, the "2 or 4" duplicates showing in the exim log look
> like they are actually just separate deliveries to other addresses, so not
> duplicates. In 1 example there is a single email with 2 recipients (2
> entries in exim log) that has over 1500+ entries in the mailcleaner DB. It
> looks like this email hasn't been delivered to the recipient at all either.
>
> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk> wrote:
>
>> Hi All,
>>
>>  If anyone could provide advice that would be great. Running Debian
>> Wheezy Mailscanner 4.79.11-2.2
>>
>>  Our incoming dir filled up just before the weekend so we didn't see the
>> issue for a couple of days. Normally we would just shut down mailcleaner
>> and delete the dir then start it up again and all would be ok. However on
>> this occasion, the root partition also become full because of the mysql DB
>> (it got to 14G in 2 days..).
>>
>>  For some reason everything started duplicating. I can see lots of
>> incoming messages in the exim logs with duplication (2 or 4 of what looks
>> like the same email) but in the mailscanner database there is hundreds of
>> each email listed (apparently there was over 9 million messages delivered
>> on 1 day compared with the server average of about 1500!)
>>
>>  It seems like some sort of loop, but afaik nothing specific was changed
>> in the config apart from the fact incoming became full. Space has been
>> cleared on the root partition and incoming, and everything appears to be
>> running as normal right now.
>>
>>  Any advice on debugging this would be much appreciated, also, how best
>> should I clear out the DB of all the dupes?
>>
>>  Thanks!
>>
>
>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
>
>
>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/b7c87ad0/attachment.html>

From jeremy at fluxlabs.net  Tue Jul 28 15:31:42 2015
From: jeremy at fluxlabs.net (Jeremy McSpadden)
Date: Tue, 28 Jul 2015 15:31:42 +0000
Subject: Duplicated messages
In-Reply-To: <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>,
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
Message-ID: <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>

Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi Jeremy,

Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...

On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.

On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:
Hi All,

If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2

Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).

For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)

It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.

Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?

Thanks!



--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/listinfo/mailscanner




--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/listinfo/mailscanner



--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/listinfo/mailscanner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/890d5cc7/attachment.html>

From mark at workshopit.co.uk  Tue Jul 28 15:34:15 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Tue, 28 Jul 2015 16:34:15 +0100
Subject: Duplicated messages
In-Reply-To: <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
Message-ID: <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>

How do I try send them through again? At the moment they are just "message"
in the quarantine, and if I try open them through the web interface it
times out, I guess because its trying to open each one of the dupes?

"Fatal error: Maximum execution time of 30 seconds exceeded in
/var/www/html/mailscanner/functions.php on line 1022"

On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:

>  Yup. Turn on debug and watch it pass through. Last time I saw these it
> was a taint issue .. Which I am assuming has been fixed by now.
>
>  --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
> Web - http://www.fluxlabs.net
>
>
> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
>   Hi Jeremy,
>
>  Are you saying that something in these messages is crashing Mailscanner?
> Everything seems to be OK right now, but all 70 of the emails (all
> different types and from different servers) are now in the quarantine
> because of "Other Bad Content Detected" with the report "MailScanner:
> Message attempted to kill MailScanner". It seems it succeeded...
>
> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>
>>  It's probably looping/crashing mailscanner. Drop MS into debug mode and
>> watch logs.
>>
>>  --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>> Web - http://www.fluxlabs.net
>>
>>
>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>>   An update to this, the "2 or 4" duplicates showing in the exim log
>> look like they are actually just separate deliveries to other addresses, so
>> not duplicates. In 1 example there is a single email with 2 recipients (2
>> entries in exim log) that has over 1500+ entries in the mailcleaner DB. It
>> looks like this email hasn't been delivered to the recipient at all either.
>>
>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>>> Hi All,
>>>
>>>  If anyone could provide advice that would be great. Running Debian
>>> Wheezy Mailscanner 4.79.11-2.2
>>>
>>>  Our incoming dir filled up just before the weekend so we didn't see
>>> the issue for a couple of days. Normally we would just shut down
>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>> However on this occasion, the root partition also become full because of
>>> the mysql DB (it got to 14G in 2 days..).
>>>
>>>  For some reason everything started duplicating. I can see lots of
>>> incoming messages in the exim logs with duplication (2 or 4 of what looks
>>> like the same email) but in the mailscanner database there is hundreds of
>>> each email listed (apparently there was over 9 million messages delivered
>>> on 1 day compared with the server average of about 1500!)
>>>
>>>  It seems like some sort of loop, but afaik nothing specific was
>>> changed in the config apart from the fact incoming became full. Space has
>>> been cleared on the root partition and incoming, and everything appears to
>>> be running as normal right now.
>>>
>>>  Any advice on debugging this would be much appreciated, also, how best
>>> should I clear out the DB of all the dupes?
>>>
>>>  Thanks!
>>>
>>
>>
>>
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>>
>>
>>
>>
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>>
>>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
>
>
>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/c4093b3b/attachment.html>

From jerry.benton at mailborder.com  Tue Jul 28 15:36:50 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Tue, 28 Jul 2015 11:36:50 -0400
Subject: Duplicated messages
In-Reply-To: <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
Message-ID: <B5EC6B58-0C19-4029-8464-6A35EE592692@mailborder.com>

It is not crashing. It is splitting the email into separate recipients per your settings. If you have archiving enabled, then you are doubling your fun. 

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk> wrote:
> 
> How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?
> 
> "Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"
> 
> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
> Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now. 
> 
> --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
> 
> 
> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
> 
>> Hi Jeremy,
>> 
>> Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...
>> 
>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>> It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs. 
>> 
>> --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>> 
>> 
>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>> 
>>> An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.
>>> 
>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>> Hi All,
>>> 
>>> If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2
>>> 
>>> Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).
>>> 
>>> For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)
>>> 
>>> It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.
>>> 
>>> Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?
>>> 
>>> Thanks!
>>> 
>>> 
>>> 
>>> -- 
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>> 
>> 
>> 
>> -- 
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>> 
> 
> 
> 
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
> 
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/ce8903b8/attachment.html>

From jerry.benton at mailborder.com  Tue Jul 28 15:37:33 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Tue, 28 Jul 2015 11:37:33 -0400
Subject: pdf corruption
In-Reply-To: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>
References: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>
Message-ID: <0958DB54-40A4-4F51-AED8-DECF644B9CC1@mailborder.com>

Upgrade to 4.85.2?

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 9:34 AM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
> 
> Hi,
> 
> I'm currently using 4.84.5 to store all my messages. I have a problem with
> some PDF files been corrupted after mailscanner process the files. 
> 
> Some ide ato fix this issue or tell mailscanner to don't process pdf files?
> 
> Thanks,
> Wilson
> 
> 
> 
> 
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 


From jerry.benton at mailborder.com  Tue Jul 28 15:43:33 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Tue, 28 Jul 2015 11:43:33 -0400
Subject: Duplicated messages
In-Reply-To: <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
Message-ID: <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>

By the way, there is no web interface in the MailScanner package. There are 3rd party products of course (I created one myself) but those questions would need to be directed to those support forums or mailing lists.

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk> wrote:
> 
> How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?
> 
> "Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"
> 
> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
> Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now. 
> 
> --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
> 
> 
> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
> 
>> Hi Jeremy,
>> 
>> Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...
>> 
>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>> It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs. 
>> 
>> --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>> 
>> 
>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>> 
>>> An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.
>>> 
>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>> Hi All,
>>> 
>>> If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2
>>> 
>>> Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).
>>> 
>>> For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)
>>> 
>>> It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.
>>> 
>>> Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?
>>> 
>>> Thanks!
>>> 
>>> 
>>> 
>>> -- 
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>> 
>> 
>> 
>> -- 
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>> 
> 
> 
> 
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
> 
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/82022a32/attachment-0001.html>

From mark at workshopit.co.uk  Tue Jul 28 15:43:52 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Tue, 28 Jul 2015 16:43:52 +0100
Subject: Duplicated messages
In-Reply-To: <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
Message-ID: <CABF=4+ze3MKhFp=p8U79tbB08s0qkMyDnK_-A9itHT04bzx2tw@mail.gmail.com>

OK, I take that comment about 70 messages back. It looks like everything in
the problem time period has gone to the quarantine, around 1100 messages
(good and spam). Is there any more info on what triggers this "Other Bad
Content Detected"?

Any suggestions on how to most efficiently get the good ones out of the
quarantine?

Thanks

On 28 July 2015 at 16:34, Mark Adams <mark at workshopit.co.uk> wrote:

> How do I try send them through again? At the moment they are just
> "message" in the quarantine, and if I try open them through the web
> interface it times out, I guess because its trying to open each one of the
> dupes?
>
> "Fatal error: Maximum execution time of 30 seconds exceeded in
> /var/www/html/mailscanner/functions.php on line 1022"
>
> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>
>>  Yup. Turn on debug and watch it pass through. Last time I saw these it
>> was a taint issue .. Which I am assuming has been fixed by now.
>>
>>  --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>> Web - http://www.fluxlabs.net
>>
>>
>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>>   Hi Jeremy,
>>
>>  Are you saying that something in these messages is crashing
>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>> (all different types and from different servers) are now in the quarantine
>> because of "Other Bad Content Detected" with the report "MailScanner:
>> Message attempted to kill MailScanner". It seems it succeeded...
>>
>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>
>>>  It's probably looping/crashing mailscanner. Drop MS into debug mode
>>> and watch logs.
>>>
>>>  --
>>> Jeremy McSpadden | Flux Labs
>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>> Web - http://www.fluxlabs.net
>>>
>>>
>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>
>>>   An update to this, the "2 or 4" duplicates showing in the exim log
>>> look like they are actually just separate deliveries to other addresses, so
>>> not duplicates. In 1 example there is a single email with 2 recipients (2
>>> entries in exim log) that has over 1500+ entries in the mailcleaner DB. It
>>> looks like this email hasn't been delivered to the recipient at all either.
>>>
>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk> wrote:
>>>
>>>> Hi All,
>>>>
>>>>  If anyone could provide advice that would be great. Running Debian
>>>> Wheezy Mailscanner 4.79.11-2.2
>>>>
>>>>  Our incoming dir filled up just before the weekend so we didn't see
>>>> the issue for a couple of days. Normally we would just shut down
>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>> However on this occasion, the root partition also become full because of
>>>> the mysql DB (it got to 14G in 2 days..).
>>>>
>>>>  For some reason everything started duplicating. I can see lots of
>>>> incoming messages in the exim logs with duplication (2 or 4 of what looks
>>>> like the same email) but in the mailscanner database there is hundreds of
>>>> each email listed (apparently there was over 9 million messages delivered
>>>> on 1 day compared with the server average of about 1500!)
>>>>
>>>>  It seems like some sort of loop, but afaik nothing specific was
>>>> changed in the config apart from the fact incoming became full. Space has
>>>> been cleared on the root partition and incoming, and everything appears to
>>>> be running as normal right now.
>>>>
>>>>  Any advice on debugging this would be much appreciated, also, how
>>>> best should I clear out the DB of all the dupes?
>>>>
>>>>  Thanks!
>>>>
>>>
>>>
>>>
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info
>>> http://lists.mailscanner.info/listinfo/mailscanner
>>>
>>>
>>>
>>>
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info
>>> http://lists.mailscanner.info/listinfo/mailscanner
>>>
>>>
>>
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>>
>>
>>
>>
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>>
>>


-- 
Mark Adams
*Workshop IT:*

5 Cowcross Street
London EC1M 6DW
020 7183 0498
www.workshopit.co.uk
Registered in England and Wales: 8366747
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/f8b27507/attachment.html>

From jerry.benton at mailborder.com  Tue Jul 28 15:44:49 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Tue, 28 Jul 2015 11:44:49 -0400
Subject: Duplicated messages
In-Reply-To: <CABF=4+ze3MKhFp=p8U79tbB08s0qkMyDnK_-A9itHT04bzx2tw@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <CABF=4+ze3MKhFp=p8U79tbB08s0qkMyDnK_-A9itHT04bzx2tw@mail.gmail.com>
Message-ID: <09C86762-9B8F-49C9-8728-13083CA2B604@mailborder.com>

You appear to have archiving enabled. 

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 11:43 AM, Mark Adams <mark at workshopit.co.uk> wrote:
> 
> OK, I take that comment about 70 messages back. It looks like everything in the problem time period has gone to the quarantine, around 1100 messages (good and spam). Is there any more info on what triggers this "Other Bad Content Detected"? 
> 
> Any suggestions on how to most efficiently get the good ones out of the quarantine?
> 
> Thanks
> 
> On 28 July 2015 at 16:34, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
> How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?
> 
> "Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"
> 
> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
> Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now. 
> 
> --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
> 
> 
> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
> 
>> Hi Jeremy,
>> 
>> Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...
>> 
>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>> It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs. 
>> 
>> --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>> 
>> 
>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>> 
>>> An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.
>>> 
>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>> Hi All,
>>> 
>>> If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2
>>> 
>>> Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).
>>> 
>>> For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)
>>> 
>>> It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.
>>> 
>>> Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?
>>> 
>>> Thanks!
>>> 
>>> 
>>> 
>>> -- 
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>> 
>> 
>> 
>> -- 
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>> 
> 
> 
> 
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
> 
> 
> 
> 
> -- 
> Mark Adams
> Workshop IT: 
> 
> 5 Cowcross Street
> London EC1M 6DW
> 020 7183 0498
> www.workshopit.co.uk <http://www.workshopit.co.uk/>
> Registered in England and Wales: 8366747
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/ce3e5f32/attachment.html>

From mark at workshopit.co.uk  Tue Jul 28 15:49:10 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Tue, 28 Jul 2015 16:49:10 +0100
Subject: Duplicated messages
In-Reply-To: <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
Message-ID: <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>

Of course, apologies - I'm using Mailwatch. Any advice on how to most
efficiently pull things out of quarantine via command-line? (note they are
stored as "message" rather than queue items, that would be too easy..)

I don't have Archive enabled, everything has gone in to the quarantine
because of this "Other Bad Content Detected"

On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com> wrote:

> By the way, there is no web interface in the MailScanner package. There
> are 3rd party products of course (I created one myself) but those questions
> would need to be directed to those support forums or mailing lists.
>
> -
> Jerry Benton
> www.mailborder.com
>
>
>
> On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
> How do I try send them through again? At the moment they are just
> "message" in the quarantine, and if I try open them through the web
> interface it times out, I guess because its trying to open each one of the
> dupes?
>
> "Fatal error: Maximum execution time of 30 seconds exceeded in
> /var/www/html/mailscanner/functions.php on line 1022"
>
> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>
>>  Yup. Turn on debug and watch it pass through. Last time I saw these it
>> was a taint issue .. Which I am assuming has been fixed by now.
>>
>>  --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>> Web - http://www.fluxlabs.net
>>
>>
>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>>   Hi Jeremy,
>>
>>  Are you saying that something in these messages is crashing
>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>> (all different types and from different servers) are now in the quarantine
>> because of "Other Bad Content Detected" with the report "MailScanner:
>> Message attempted to kill MailScanner". It seems it succeeded...
>>
>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>
>>>  It's probably looping/crashing mailscanner. Drop MS into debug mode
>>> and watch logs.
>>>
>>>  --
>>> Jeremy McSpadden | Flux Labs
>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>> Web - http://www.fluxlabs.net
>>>
>>>
>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>
>>>   An update to this, the "2 or 4" duplicates showing in the exim log
>>> look like they are actually just separate deliveries to other addresses, so
>>> not duplicates. In 1 example there is a single email with 2 recipients (2
>>> entries in exim log) that has over 1500+ entries in the mailcleaner DB. It
>>> looks like this email hasn't been delivered to the recipient at all either.
>>>
>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk> wrote:
>>>
>>>> Hi All,
>>>>
>>>>  If anyone could provide advice that would be great. Running Debian
>>>> Wheezy Mailscanner 4.79.11-2.2
>>>>
>>>>  Our incoming dir filled up just before the weekend so we didn't see
>>>> the issue for a couple of days. Normally we would just shut down
>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>> However on this occasion, the root partition also become full because of
>>>> the mysql DB (it got to 14G in 2 days..).
>>>>
>>>>  For some reason everything started duplicating. I can see lots of
>>>> incoming messages in the exim logs with duplication (2 or 4 of what looks
>>>> like the same email) but in the mailscanner database there is hundreds of
>>>> each email listed (apparently there was over 9 million messages delivered
>>>> on 1 day compared with the server average of about 1500!)
>>>>
>>>>  It seems like some sort of loop, but afaik nothing specific was
>>>> changed in the config apart from the fact incoming became full. Space has
>>>> been cleared on the root partition and incoming, and everything appears to
>>>> be running as normal right now.
>>>>
>>>>  Any advice on debugging this would be much appreciated, also, how
>>>> best should I clear out the DB of all the dupes?
>>>>
>>>>  Thanks!
>>>>
>>>
>>>
>>>
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info
>>> http://lists.mailscanner.info/listinfo/mailscanner
>>>
>>>
>>>
>>>
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info
>>> http://lists.mailscanner.info/listinfo/mailscanner
>>>
>>>
>>
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>>
>>
>>
>>
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>>
>>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
>
>
>
>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/c60e8e88/attachment.html>

From jerry.benton at mailborder.com  Tue Jul 28 16:00:27 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Tue, 28 Jul 2015 12:00:27 -0400
Subject: Duplicated messages
In-Reply-To: <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
Message-ID: <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>

I am not sure on what parameters Mailwatch calls and logs “other bad content”.  The MailScanner setting is "Notify Senders of Other Blocked Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked content” for all we know. You are going to have to follow the below suggestion and enable debug or see if you can get an idea from /var/log/maillog.

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk> wrote:
> 
> Of course, apologies - I'm using Mailwatch. Any advice on how to most efficiently pull things out of quarantine via command-line? (note they are stored as "message" rather than queue items, that would be too easy..) 
> 
> I don't have Archive enabled, everything has gone in to the quarantine because of this "Other Bad Content Detected"
> 
> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com <mailto:jerry.benton at mailborder.com>> wrote:
> By the way, there is no web interface in the MailScanner package. There are 3rd party products of course (I created one myself) but those questions would need to be directed to those support forums or mailing lists.
> 
> -
> Jerry Benton
> www.mailborder.com <http://www.mailborder.com/>
> 
> 
> 
>> On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>> 
>> How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?
>> 
>> "Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"
>> 
>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>> Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now. 
>> 
>> --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>> 
>> 
>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>> 
>>> Hi Jeremy,
>>> 
>>> Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...
>>> 
>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>>> It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs. 
>>> 
>>> --
>>> Jeremy McSpadden | Flux Labs
>>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>>> 
>>> 
>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>> 
>>>> An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.
>>>> 
>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>> Hi All,
>>>> 
>>>> If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2
>>>> 
>>>> Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).
>>>> 
>>>> For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)
>>>> 
>>>> It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.
>>>> 
>>>> Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?
>>>> 
>>>> Thanks!
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> MailScanner mailing list
>>>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>>>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>>> 
>>> 
>>> 
>>> -- 
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>> 
>> 
>> 
>> -- 
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>> 
> 
> 
> 
> 
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
> 
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/b8a52b74/attachment.html>

From jerry.benton at mailborder.com  Tue Jul 28 16:02:05 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Tue, 28 Jul 2015 12:02:05 -0400
Subject: Duplicated messages
In-Reply-To: <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
Message-ID: <B07DA3EC-3E62-4DC8-9A47-94317F8CCCD6@mailborder.com>

I forgot to mention, you really should upgrade your MailScanner installation. It is as easy as typing “install” now. 

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 12:00 PM, Jerry Benton <jerry.benton at mailborder.com> wrote:
> 
> I am not sure on what parameters Mailwatch calls and logs “other bad content”.  The MailScanner setting is "Notify Senders of Other Blocked Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked content” for all we know. You are going to have to follow the below suggestion and enable debug or see if you can get an idea from /var/log/maillog.
> 
> -
> Jerry Benton
> www.mailborder.com <http://www.mailborder.com/>
> 
> 
> 
>> On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>> 
>> Of course, apologies - I'm using Mailwatch. Any advice on how to most efficiently pull things out of quarantine via command-line? (note they are stored as "message" rather than queue items, that would be too easy..) 
>> 
>> I don't have Archive enabled, everything has gone in to the quarantine because of this "Other Bad Content Detected"
>> 
>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com <mailto:jerry.benton at mailborder.com>> wrote:
>> By the way, there is no web interface in the MailScanner package. There are 3rd party products of course (I created one myself) but those questions would need to be directed to those support forums or mailing lists.
>> 
>> -
>> Jerry Benton
>> www.mailborder.com <http://www.mailborder.com/>
>> 
>> 
>> 
>>> On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>> 
>>> How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?
>>> 
>>> "Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"
>>> 
>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>>> Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now. 
>>> 
>>> --
>>> Jeremy McSpadden | Flux Labs
>>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>>> 
>>> 
>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>> 
>>>> Hi Jeremy,
>>>> 
>>>> Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...
>>>> 
>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>>>> It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs. 
>>>> 
>>>> --
>>>> Jeremy McSpadden | Flux Labs
>>>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>>>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>>>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>>>> 
>>>> 
>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>> 
>>>>> An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.
>>>>> 
>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>>> Hi All,
>>>>> 
>>>>> If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2
>>>>> 
>>>>> Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).
>>>>> 
>>>>> For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)
>>>>> 
>>>>> It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.
>>>>> 
>>>>> Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?
>>>>> 
>>>>> Thanks!
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> MailScanner mailing list
>>>>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>>>>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> MailScanner mailing list
>>>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>>>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> MailScanner mailing list
>>>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>>>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>>> 
>>> 
>>> 
>>> -- 
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>>> 
>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
>> 
>> 
>> 
>> -- 
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/cf164eb7/attachment-0001.html>

From mark at workshopit.co.uk  Tue Jul 28 16:07:18 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Tue, 28 Jul 2015 17:07:18 +0100
Subject: Duplicated messages
In-Reply-To: <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
Message-ID: <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>

Hi Jerry,

If you wanted to pull a bunch of items from the quarantine from the command
line and re-process them through Mailcleaner, how would you do that?

Regards,
Mark

On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com> wrote:

> I am not sure on what parameters Mailwatch calls and logs “other bad
> content”.  The MailScanner setting is "Notify Senders of Other Blocked
> Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked
> content” for all we know. You are going to have to follow the below
> suggestion and enable debug or see if you can get an idea from
> /var/log/maillog.
>
> -
> Jerry Benton
> www.mailborder.com
>
>
>
> On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
> Of course, apologies - I'm using Mailwatch. Any advice on how to most
> efficiently pull things out of quarantine via command-line? (note they are
> stored as "message" rather than queue items, that would be too easy..)
>
> I don't have Archive enabled, everything has gone in to the quarantine
> because of this "Other Bad Content Detected"
>
> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com>
> wrote:
>
>> By the way, there is no web interface in the MailScanner package. There
>> are 3rd party products of course (I created one myself) but those questions
>> would need to be directed to those support forums or mailing lists.
>>
>> -
>> Jerry Benton
>> www.mailborder.com
>>
>>
>>
>> On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>> How do I try send them through again? At the moment they are just
>> "message" in the quarantine, and if I try open them through the web
>> interface it times out, I guess because its trying to open each one of the
>> dupes?
>>
>> "Fatal error: Maximum execution time of 30 seconds exceeded in
>> /var/www/html/mailscanner/functions.php on line 1022"
>>
>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>
>>>  Yup. Turn on debug and watch it pass through. Last time I saw these it
>>> was a taint issue .. Which I am assuming has been fixed by now.
>>>
>>>  --
>>> Jeremy McSpadden | Flux Labs
>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>> Web - http://www.fluxlabs.net
>>>
>>>
>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>
>>>   Hi Jeremy,
>>>
>>>  Are you saying that something in these messages is crashing
>>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>>> (all different types and from different servers) are now in the quarantine
>>> because of "Other Bad Content Detected" with the report "MailScanner:
>>> Message attempted to kill MailScanner". It seems it succeeded...
>>>
>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>>
>>>>  It's probably looping/crashing mailscanner. Drop MS into debug mode
>>>> and watch logs.
>>>>
>>>>  --
>>>> Jeremy McSpadden | Flux Labs
>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>> Web - http://www.fluxlabs.net
>>>>
>>>>
>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>
>>>>   An update to this, the "2 or 4" duplicates showing in the exim log
>>>> look like they are actually just separate deliveries to other addresses, so
>>>> not duplicates. In 1 example there is a single email with 2 recipients (2
>>>> entries in exim log) that has over 1500+ entries in the mailcleaner DB. It
>>>> looks like this email hasn't been delivered to the recipient at all either.
>>>>
>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>>  If anyone could provide advice that would be great. Running Debian
>>>>> Wheezy Mailscanner 4.79.11-2.2
>>>>>
>>>>>  Our incoming dir filled up just before the weekend so we didn't see
>>>>> the issue for a couple of days. Normally we would just shut down
>>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>>> However on this occasion, the root partition also become full because of
>>>>> the mysql DB (it got to 14G in 2 days..).
>>>>>
>>>>>  For some reason everything started duplicating. I can see lots of
>>>>> incoming messages in the exim logs with duplication (2 or 4 of what looks
>>>>> like the same email) but in the mailscanner database there is hundreds of
>>>>> each email listed (apparently there was over 9 million messages delivered
>>>>> on 1 day compared with the server average of about 1500!)
>>>>>
>>>>>  It seems like some sort of loop, but afaik nothing specific was
>>>>> changed in the config apart from the fact incoming became full. Space has
>>>>> been cleared on the root partition and incoming, and everything appears to
>>>>> be running as normal right now.
>>>>>
>>>>>  Any advice on debugging this would be much appreciated, also, how
>>>>> best should I clear out the DB of all the dupes?
>>>>>
>>>>>  Thanks!
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> MailScanner mailing list
>>>> mailscanner at lists.mailscanner.info
>>>> http://lists.mailscanner.info/listinfo/mailscanne
>>>> <http://lists.mailscanner.info/listinfo/mailscanner>r
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/559b2064/attachment.html>

From jerry.benton at mailborder.com  Tue Jul 28 16:15:02 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Tue, 28 Jul 2015 12:15:02 -0400
Subject: Duplicated messages
In-Reply-To: <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
Message-ID: <12918F3D-5E50-4F75-9E43-01892E5577F5@mailborder.com>

Shameless jab at Mailwatch: in Mailborder I would check the box and click Resend  :)

If they are quarantined as queue files and you are using Postfix, because that is what I know really well, you simply copy the message to /var/spool/postfix/incoming and the MTA will pick it up. If you have like 1100 of them, you will probably need to run a query on the DB to get the message IDs and create a script that cycles through them. 

If you are using sendmail or exim, I have no idea. Someone else on the list would have to offer their expertise. Also, if you did not quarantine the items as queue files (MailScanner setting) then you are going to have to research that as well. 



-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 12:07 PM, Mark Adams <mark at workshopit.co.uk> wrote:
> 
> Hi Jerry,
> 
> If you wanted to pull a bunch of items from the quarantine from the command line and re-process them through Mailcleaner, how would you do that?
> 
> Regards,
> Mark
> 
> On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com <mailto:jerry.benton at mailborder.com>> wrote:
> I am not sure on what parameters Mailwatch calls and logs “other bad content”.  The MailScanner setting is "Notify Senders of Other Blocked Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked content” for all we know. You are going to have to follow the below suggestion and enable debug or see if you can get an idea from /var/log/maillog.
> 
> -
> Jerry Benton
> www.mailborder.com <http://www.mailborder.com/>
> 
> 
> 
>> On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>> 
>> Of course, apologies - I'm using Mailwatch. Any advice on how to most efficiently pull things out of quarantine via command-line? (note they are stored as "message" rather than queue items, that would be too easy..) 
>> 
>> I don't have Archive enabled, everything has gone in to the quarantine because of this "Other Bad Content Detected"
>> 
>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com <mailto:jerry.benton at mailborder.com>> wrote:
>> By the way, there is no web interface in the MailScanner package. There are 3rd party products of course (I created one myself) but those questions would need to be directed to those support forums or mailing lists.
>> 
>> -
>> Jerry Benton
>> www.mailborder.com <http://www.mailborder.com/>
>> 
>> 
>> 
>>> On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>> 
>>> How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?
>>> 
>>> "Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"
>>> 
>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>>> Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now. 
>>> 
>>> --
>>> Jeremy McSpadden | Flux Labs
>>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>>> 
>>> 
>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>> 
>>>> Hi Jeremy,
>>>> 
>>>> Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...
>>>> 
>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>>>> It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs. 
>>>> 
>>>> --
>>>> Jeremy McSpadden | Flux Labs
>>>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>>>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>>>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>>>> 
>>>> 
>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>> 
>>>>> An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.
>>>>> 
>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>>> Hi All,
>>>>> 
>>>>> If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2
>>>>> 
>>>>> Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).
>>>>> 
>>>>> For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)
>>>>> 
>>>>> It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.
>>>>> 
>>>>> Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?
>>>>> 
>>>>> Thanks!
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> MailScanner mailing list
>>>>> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
>>>>> http://lists.mailscanner.info/listinfo/mailscanne <http://lists.mailscanner.info/listinfo/mailscanner>r
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/5d129ff1/attachment.html>

From wilson.galafassi at gmail.com  Tue Jul 28 16:21:40 2015
From: wilson.galafassi at gmail.com (Wilson A. Galafassi Jr. [Gmail])
Date: Tue, 28 Jul 2015 13:21:40 -0300
Subject: RES: pdf corruption
In-Reply-To: <0958DB54-40A4-4F51-AED8-DECF644B9CC1@mailborder.com>
References: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>
 <0958DB54-40A4-4F51-AED8-DECF644B9CC1@mailborder.com>
Message-ID: <18cd01d0c951$7d565c10$78031430$@gmail.com>

Same problem...

-----Mensagem original-----
De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em nome de Jerry Benton
Enviada em: terça-feira, 28 de julho de 2015 12:38
Para: MailScanner Discussion
Assunto: Re: pdf corruption

Upgrade to 4.85.2?

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 9:34 AM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
> 
> Hi,
> 
> I'm currently using 4.84.5 to store all my messages. I have a problem 
> with some PDF files been corrupted after mailscanner process the files.
> 
> Some ide ato fix this issue or tell mailscanner to don't process pdf files?
> 
> Thanks,
> Wilson
> 
> 
> 
> 
> 
> 
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 



-- 
MailScanner mailing list
mailscanner at lists.mailscanner.info
http://lists.mailscanner.info/listinfo/mailscanner



From jerry.benton at mailborder.com  Tue Jul 28 16:29:01 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Tue, 28 Jul 2015 12:29:01 -0400
Subject: RES: pdf corruption
In-Reply-To: <18cd01d0c951$7d565c10$78031430$@gmail.com>
References: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>
 <0958DB54-40A4-4F51-AED8-DECF644B9CC1@mailborder.com>
 <18cd01d0c951$7d565c10$78031430$@gmail.com>
Message-ID: <4C1B0338-A96A-47B1-B169-E1DD5987F431@mailborder.com>

I’ve never heard a report of this issue. What virus scanner are you using? Aside from running the “file” command to check MIME types and extracting a copy of archives for scanning, MailScanner doesn’t change file attachments. (Unless you have zip attachments enabled.)

We can test this if you like. Send one of these PDFs to support at linuxref.com. This is a domain in my lab. If it comes through ok, I would make the educated guess that you probably have something somewhere else corrupting them. 

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 12:21 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
> 
> Same problem...
> 
> -----Mensagem original-----
> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em nome de Jerry Benton
> Enviada em: terça-feira, 28 de julho de 2015 12:38
> Para: MailScanner Discussion
> Assunto: Re: pdf corruption
> 
> Upgrade to 4.85.2?
> 
> -
> Jerry Benton
> www.mailborder.com
> 
> 
> 
>> On Jul 28, 2015, at 9:34 AM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>> 
>> Hi,
>> 
>> I'm currently using 4.84.5 to store all my messages. I have a problem 
>> with some PDF files been corrupted after mailscanner process the files.
>> 
>> Some ide ato fix this issue or tell mailscanner to don't process pdf files?
>> 
>> Thanks,
>> Wilson
>> 
>> 
>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
> 
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 
> 
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 


From mailscanner at replies.cyways.com  Tue Jul 28 16:49:52 2015
From: mailscanner at replies.cyways.com (Peter Lemieux)
Date: Tue, 28 Jul 2015 12:49:52 -0400
Subject: Duplicated messages
In-Reply-To: <CABF=4+ze3MKhFp=p8U79tbB08s0qkMyDnK_-A9itHT04bzx2tw@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <CABF=4+ze3MKhFp=p8U79tbB08s0qkMyDnK_-A9itHT04bzx2tw@mail.gmail.com>
Message-ID: <55B7B2B0.7070008@replies.cyways.com>

I solved a somewhat different but related problem today.  A client has 
stopped accepting .doc|.docx files; they are now quarantined with a notice 
to the sender requesting a PDF instead.  They asked me to make a list of the 
quarantined documents.  I used this:

cd /var/spool/MailScanner/quarantine
for f in 2015*; do echo $f; for g in $f/t*; do ls $g | grep \.doc; done; done

That iterated through the quarantine and listed all the documents matching 
\.doc.  I then replaced "
ls $g | grep \.doc
with
cp -f $g/*.doc* /var/spool/MailScanner/quarantine/docs
to copies the quarantined files in a directory /docs/ that I created.  This 
method does overwrite files with identical filenames, but I didn't need to 
go to the trouble of dealing with that for this particular project.

I'm using sendmail, so the quarantined files are stored in subdirectories of 
/var/spool/MailScanner/quarantine/2015xxxx/ using the message IDs to name 
the subdirectories.  That's where the "$f/t*" comes from.  I don't know 
where messages are quarantined if you use a different SMTP exchanger like 
Postfix or Exim.

Peter


On 07/28/2015 11:43 AM, Mark Adams wrote:
> Any suggestions on how to most efficiently get the good ones out of the
> quarantine?
>
> Thanks
>
> On 28 July 2015 at 16:34, Mark Adams <mark at workshopit.co.uk
> <mailto:mark at workshopit.co.uk>> wrote:
>
>     How do I try send them through again? At the moment they are just
>     "message" in the quarantine, and if I try open them through the web
>     interface it times out, I guess because its trying to open each one of
>     the dupes?
>
>     "Fatal error: Maximum execution time of 30 seconds exceeded in
>     /var/www/html/mailscanner/functions.php on line 1022"
>
>     On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net
>     <mailto:jeremy at fluxlabs.net>> wrote:
>
>         Yup. Turn on debug and watch it pass through. Last time I saw these
>         it was a taint issue .. Which I am assuming has been fixed by now.
>
>         --
>         Jeremy McSpadden | Flux Labs
>         Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile -
>         850-890-2543 <tel:850-890-2543>
>         Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX
>         <tel:877-699-FLUX>
>         Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>
>
>         On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk
>         <mailto:mark at workshopit.co.uk>> wrote:
>
>>         Hi Jeremy,
>>
>>         Are you saying that something in these messages is crashing
>>         Mailscanner? Everything seems to be OK right now, but all 70 of
>>         the emails (all different types and from different servers) are
>>         now in the quarantine because of "Other Bad Content Detected" with
>>         the report "MailScanner: Message attempted to kill MailScanner".
>>         It seems it succeeded...
>>
>>         On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net
>>         <mailto:jeremy at fluxlabs.net>> wrote:
>>
>>             It's probably looping/crashing mailscanner. Drop MS into debug
>>             mode and watch logs.
>>
>>             --
>>             Jeremy McSpadden | Flux Labs
>>             Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile -
>>             850-890-2543 <tel:850-890-2543>
>>             Fax - 850-254-2955 <tel:850-254-2955> | Toll Free -
>>             877-699-FLUX <tel:877-699-FLUX>
>>             Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>>
>>
>>             On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk
>>             <mailto:mark at workshopit.co.uk>> wrote:
>>
>>>             An update to this, the "2 or 4" duplicates showing in the
>>>             exim log look like they are actually just separate deliveries
>>>             to other addresses, so not duplicates. In 1 example there is
>>>             a single email with 2 recipients (2 entries in exim log) that
>>>             has over 1500+ entries in the mailcleaner DB. It looks like
>>>             this email hasn't been delivered to the recipient at all either.
>>>
>>>             On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk
>>>             <mailto:mark at workshopit.co.uk>> wrote:
>>>
>>>                 Hi All,
>>>
>>>                 If anyone could provide advice that would be great.
>>>                 Running Debian Wheezy Mailscanner 4.79.11-2.2
>>>
>>>                 Our incoming dir filled up just before the weekend so we
>>>                 didn't see the issue for a couple of days. Normally we
>>>                 would just shut down mailcleaner and delete the dir then
>>>                 start it up again and all would be ok. However on this
>>>                 occasion, the root partition also become full because of
>>>                 the mysql DB (it got to 14G in 2 days..).
>>>
>>>                 For some reason everything started duplicating. I can see
>>>                 lots of incoming messages in the exim logs with
>>>                 duplication (2 or 4 of what looks like the same email)
>>>                 but in the mailscanner database there is hundreds of each
>>>                 email listed (apparently there was over 9 million
>>>                 messages delivered on 1 day compared with the server
>>>                 average of about 1500!)
>>>
>>>                 It seems like some sort of loop, but afaik nothing
>>>                 specific was changed in the config apart from the fact
>>>                 incoming became full. Space has been cleared on the root
>>>                 partition and incoming, and everything appears to be
>>>                 running as normal right now.
>>>
>>>                 Any advice on debugging this would be much appreciated,
>>>                 also, how best should I clear out the DB of all the dupes?
>>>
>>>                 Thanks!
>>>
>>>
>>>
>>>
>>>             --
>>>             MailScanner mailing list
>>>             mailscanner at lists.mailscanner.info
>>>             <mailto:mailscanner at lists.mailscanner.info>
>>>             http://lists.mailscanner.info/listinfo/mailscanner
>>>
>>
>>
>>
>>             --
>>             MailScanner mailing list
>>             mailscanner at lists.mailscanner.info
>>             <mailto:mailscanner at lists.mailscanner.info>
>>             http://lists.mailscanner.info/listinfo/mailscanner
>>
>>
>>
>>         --
>>         MailScanner mailing list
>>         mailscanner at lists.mailscanner.info
>>         <mailto:mailscanner at lists.mailscanner.info>
>>         http://lists.mailscanner.info/listinfo/mailscanner
>>
>
>
>
>         --
>         MailScanner mailing list
>         mailscanner at lists.mailscanner.info
>         <mailto:mailscanner at lists.mailscanner.info>
>         http://lists.mailscanner.info/listinfo/mailscanner
>
>
>
>
> --
> Mark Adams
> *Workshop IT:*
>
> 5 Cowcross Street
> London EC1M 6DW
> 020 7183 0498
> www.workshopit.co.uk <http://www.workshopit.co.uk/>
> Registered in England and Wales: 8366747
>
>
>
>

From wilson.galafassi at gmail.com  Tue Jul 28 17:02:21 2015
From: wilson.galafassi at gmail.com (Wilson A. Galafassi Jr. [Gmail])
Date: Tue, 28 Jul 2015 14:02:21 -0300
Subject: RES: RES: pdf corruption
In-Reply-To: <4C1B0338-A96A-47B1-B169-E1DD5987F431@mailborder.com>
References: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>
 <0958DB54-40A4-4F51-AED8-DECF644B9CC1@mailborder.com>
 <18cd01d0c951$7d565c10$78031430$@gmail.com>
 <4C1B0338-A96A-47B1-B169-E1DD5987F431@mailborder.com>
Message-ID: <190d01d0c957$2c744720$855cd560$@gmail.com>

I don't use any antivirus. Mailscanner is used only to archiving purposes.

I have sent to you 2 files (the same)  1 corrupted (after processed by mailscanner ) and 1 ok.

Thanks,
Wilson


-----Mensagem original-----
De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em nome de Jerry Benton
Enviada em: terça-feira, 28 de julho de 2015 13:29
Para: MailScanner Discussion
Assunto: Re: RES: pdf corruption

I’ve never heard a report of this issue. What virus scanner are you using? Aside from running the “file” command to check MIME types and extracting a copy of archives for scanning, MailScanner doesn’t change file attachments. (Unless you have zip attachments enabled.)

We can test this if you like. Send one of these PDFs to support at linuxref.com. This is a domain in my lab. If it comes through ok, I would make the educated guess that you probably have something somewhere else corrupting them. 

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 12:21 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
> 
> Same problem...
> 
> -----Mensagem original-----
> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em 
> nome de Jerry Benton Enviada em: terça-feira, 28 de julho de 2015 
> 12:38
> Para: MailScanner Discussion
> Assunto: Re: pdf corruption
> 
> Upgrade to 4.85.2?
> 
> -
> Jerry Benton
> www.mailborder.com
> 
> 
> 
>> On Jul 28, 2015, at 9:34 AM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>> 
>> Hi,
>> 
>> I'm currently using 4.84.5 to store all my messages. I have a problem 
>> with some PDF files been corrupted after mailscanner process the files.
>> 
>> Some ide ato fix this issue or tell mailscanner to don't process pdf files?
>> 
>> Thanks,
>> Wilson
>> 
>> 
>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
> 
> 
> 
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 
> 
> 
> 
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 



-- 
MailScanner mailing list
mailscanner at lists.mailscanner.info
http://lists.mailscanner.info/listinfo/mailscanner



From wilson.galafassi at gmail.com  Tue Jul 28 17:04:35 2015
From: wilson.galafassi at gmail.com (Wilson A. Galafassi Jr. [Gmail])
Date: Tue, 28 Jul 2015 14:04:35 -0300
Subject: RES: RES: pdf corruption
In-Reply-To: <4C1B0338-A96A-47B1-B169-E1DD5987F431@mailborder.com>
References: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>
 <0958DB54-40A4-4F51-AED8-DECF644B9CC1@mailborder.com>
 <18cd01d0c951$7d565c10$78031430$@gmail.com>
 <4C1B0338-A96A-47B1-B169-E1DD5987F431@mailborder.com>
Message-ID: <191401d0c957$7c363750$74a2a5f0$@gmail.com>

Another thing: if i generate the files using adobe acrobat the corruption doesn't occour. The corruption occour only on pdf generated by an ERP used on my company.

-----Mensagem original-----
De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em nome de Jerry Benton
Enviada em: terça-feira, 28 de julho de 2015 13:29
Para: MailScanner Discussion
Assunto: Re: RES: pdf corruption

I’ve never heard a report of this issue. What virus scanner are you using? Aside from running the “file” command to check MIME types and extracting a copy of archives for scanning, MailScanner doesn’t change file attachments. (Unless you have zip attachments enabled.)

We can test this if you like. Send one of these PDFs to support at linuxref.com. This is a domain in my lab. If it comes through ok, I would make the educated guess that you probably have something somewhere else corrupting them. 

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 12:21 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
> 
> Same problem...
> 
> -----Mensagem original-----
> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em 
> nome de Jerry Benton Enviada em: terça-feira, 28 de julho de 2015 
> 12:38
> Para: MailScanner Discussion
> Assunto: Re: pdf corruption
> 
> Upgrade to 4.85.2?
> 
> -
> Jerry Benton
> www.mailborder.com
> 
> 
> 
>> On Jul 28, 2015, at 9:34 AM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>> 
>> Hi,
>> 
>> I'm currently using 4.84.5 to store all my messages. I have a problem 
>> with some PDF files been corrupted after mailscanner process the files.
>> 
>> Some ide ato fix this issue or tell mailscanner to don't process pdf files?
>> 
>> Thanks,
>> Wilson
>> 
>> 
>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
> 
> 
> 
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 
> 
> 
> 
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 



-- 
MailScanner mailing list
mailscanner at lists.mailscanner.info
http://lists.mailscanner.info/listinfo/mailscanner



From jerry.benton at mailborder.com  Tue Jul 28 17:09:49 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Tue, 28 Jul 2015 13:09:49 -0400
Subject: RES: RES: pdf corruption
In-Reply-To: <190d01d0c957$2c744720$855cd560$@gmail.com>
References: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>
 <0958DB54-40A4-4F51-AED8-DECF644B9CC1@mailborder.com>
 <18cd01d0c951$7d565c10$78031430$@gmail.com>
 <4C1B0338-A96A-47B1-B169-E1DD5987F431@mailborder.com>
 <190d01d0c957$2c744720$855cd560$@gmail.com>
Message-ID: <CC2973D1-8765-4092-B367-21E906625790@mailborder.com>

The file came through with no problems. (See screenshot.) Are the emails passing through something else before MailScanner or before hitting the user’s inbox?

-
Jerry Benton
www.mailborder.com


> On Jul 28, 2015, at 1:02 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
> 
> I don't use any antivirus. Mailscanner is used only to archiving purposes.
> 
> I have sent to you 2 files (the same)  1 corrupted (after processed by mailscanner ) and 1 ok.
> 
> Thanks,
> Wilson
> 
> 
> -----Mensagem original-----
> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em nome de Jerry Benton
> Enviada em: terça-feira, 28 de julho de 2015 13:29
> Para: MailScanner Discussion
> Assunto: Re: RES: pdf corruption
> 
> I’ve never heard a report of this issue. What virus scanner are you using? Aside from running the “file” command to check MIME types and extracting a copy of archives for scanning, MailScanner doesn’t change file attachments. (Unless you have zip attachments enabled.)
> 
> We can test this if you like. Send one of these PDFs to support at linuxref.com. This is a domain in my lab. If it comes through ok, I would make the educated guess that you probably have something somewhere else corrupting them. 
> 
> -
> Jerry Benton
> www.mailborder.com
> 
> 
> 
>> On Jul 28, 2015, at 12:21 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>> 
>> Same problem...
>> 
>> -----Mensagem original-----
>> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em 
>> nome de Jerry Benton Enviada em: terça-feira, 28 de julho de 2015 
>> 12:38
>> Para: MailScanner Discussion
>> Assunto: Re: pdf corruption
>> 
>> Upgrade to 4.85.2?
>> 
>> -
>> Jerry Benton
>> www.mailborder.com
>> 
>> 
>> 
>>> On Jul 28, 2015, at 9:34 AM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> I'm currently using 4.84.5 to store all my messages. I have a problem 
>>> with some PDF files been corrupted after mailscanner process the files.
>>> 
>>> Some ide ato fix this issue or tell mailscanner to don't process pdf files?
>>> 
>>> Thanks,
>>> Wilson
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info
>>> http://lists.mailscanner.info/listinfo/mailscanner
>>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
> 
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 
> 
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/6d9522f6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2015-07-28 at 1.07.53 PM.png
Type: image/png
Size: 203538 bytes
Desc: not available
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/6d9522f6/attachment-0001.png>

From jerry.benton at mailborder.com  Tue Jul 28 17:11:14 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Tue, 28 Jul 2015 13:11:14 -0400
Subject: RES: RES: pdf corruption
In-Reply-To: <191401d0c957$7c363750$74a2a5f0$@gmail.com>
References: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>
 <0958DB54-40A4-4F51-AED8-DECF644B9CC1@mailborder.com>
 <18cd01d0c951$7d565c10$78031430$@gmail.com>
 <4C1B0338-A96A-47B1-B169-E1DD5987F431@mailborder.com>
 <191401d0c957$7c363750$74a2a5f0$@gmail.com>
Message-ID: <BBEF7C34-D106-456F-9D66-BF7FB2FCEEF1@mailborder.com>

A test in the lab with the uncorrupted file you sent remained uncorrupted after passing through MailScanner. See the private email I sent for details.

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 1:04 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
> 
> Another thing: if i generate the files using adobe acrobat the corruption doesn't occour. The corruption occour only on pdf generated by an ERP used on my company.
> 
> -----Mensagem original-----
> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em nome de Jerry Benton
> Enviada em: terça-feira, 28 de julho de 2015 13:29
> Para: MailScanner Discussion
> Assunto: Re: RES: pdf corruption
> 
> I’ve never heard a report of this issue. What virus scanner are you using? Aside from running the “file” command to check MIME types and extracting a copy of archives for scanning, MailScanner doesn’t change file attachments. (Unless you have zip attachments enabled.)
> 
> We can test this if you like. Send one of these PDFs to support at linuxref.com. This is a domain in my lab. If it comes through ok, I would make the educated guess that you probably have something somewhere else corrupting them. 
> 
> -
> Jerry Benton
> www.mailborder.com
> 
> 
> 
>> On Jul 28, 2015, at 12:21 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>> 
>> Same problem...
>> 
>> -----Mensagem original-----
>> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em 
>> nome de Jerry Benton Enviada em: terça-feira, 28 de julho de 2015 
>> 12:38
>> Para: MailScanner Discussion
>> Assunto: Re: pdf corruption
>> 
>> Upgrade to 4.85.2?
>> 
>> -
>> Jerry Benton
>> www.mailborder.com
>> 
>> 
>> 
>>> On Jul 28, 2015, at 9:34 AM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> I'm currently using 4.84.5 to store all my messages. I have a problem 
>>> with some PDF files been corrupted after mailscanner process the files.
>>> 
>>> Some ide ato fix this issue or tell mailscanner to don't process pdf files?
>>> 
>>> Thanks,
>>> Wilson
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info
>>> http://lists.mailscanner.info/listinfo/mailscanner
>>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
> 
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 
> 
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 


From jerry.benton at mailborder.com  Tue Jul 28 17:17:29 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Tue, 28 Jul 2015 13:17:29 -0400
Subject: RES: RES: pdf corruption
In-Reply-To: <BBEF7C34-D106-456F-9D66-BF7FB2FCEEF1@mailborder.com>
References: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>
 <0958DB54-40A4-4F51-AED8-DECF644B9CC1@mailborder.com>
 <18cd01d0c951$7d565c10$78031430$@gmail.com>
 <4C1B0338-A96A-47B1-B169-E1DD5987F431@mailborder.com>
 <191401d0c957$7c363750$74a2a5f0$@gmail.com>
 <BBEF7C34-D106-456F-9D66-BF7FB2FCEEF1@mailborder.com>
Message-ID: <487A91EF-EB52-44CC-9EB8-7E2298B08418@mailborder.com>

Apologies for sending the screenshot to the list. I have deleted the png file from the list server. 

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 1:11 PM, Jerry Benton <jerry.benton at mailborder.com> wrote:
> 
> A test in the lab with the uncorrupted file you sent remained uncorrupted after passing through MailScanner. See the private email I sent for details.
> 
> -
> Jerry Benton
> www.mailborder.com
> 
> 
> 
>> On Jul 28, 2015, at 1:04 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>> 
>> Another thing: if i generate the files using adobe acrobat the corruption doesn't occour. The corruption occour only on pdf generated by an ERP used on my company.
>> 
>> -----Mensagem original-----
>> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em nome de Jerry Benton
>> Enviada em: terça-feira, 28 de julho de 2015 13:29
>> Para: MailScanner Discussion
>> Assunto: Re: RES: pdf corruption
>> 
>> I’ve never heard a report of this issue. What virus scanner are you using? Aside from running the “file” command to check MIME types and extracting a copy of archives for scanning, MailScanner doesn’t change file attachments. (Unless you have zip attachments enabled.)
>> 
>> We can test this if you like. Send one of these PDFs to support at linuxref.com. This is a domain in my lab. If it comes through ok, I would make the educated guess that you probably have something somewhere else corrupting them. 
>> 
>> -
>> Jerry Benton
>> www.mailborder.com
>> 
>> 
>> 
>>> On Jul 28, 2015, at 12:21 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>>> 
>>> Same problem...
>>> 
>>> -----Mensagem original-----
>>> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em 
>>> nome de Jerry Benton Enviada em: terça-feira, 28 de julho de 2015 
>>> 12:38
>>> Para: MailScanner Discussion
>>> Assunto: Re: pdf corruption
>>> 
>>> Upgrade to 4.85.2?
>>> 
>>> -
>>> Jerry Benton
>>> www.mailborder.com
>>> 
>>> 
>>> 
>>>> On Jul 28, 2015, at 9:34 AM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I'm currently using 4.84.5 to store all my messages. I have a problem 
>>>> with some PDF files been corrupted after mailscanner process the files.
>>>> 
>>>> Some ide ato fix this issue or tell mailscanner to don't process pdf files?
>>>> 
>>>> Thanks,
>>>> Wilson
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> MailScanner mailing list
>>>> mailscanner at lists.mailscanner.info
>>>> http://lists.mailscanner.info/listinfo/mailscanner
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info
>>> http://lists.mailscanner.info/listinfo/mailscanner
>>> 
>>> 
>>> 
>>> 
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info
>>> http://lists.mailscanner.info/listinfo/mailscanner
>>> 
>> 
>> 
>> 
>> -- 
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
>> 
>> 
>> 
>> -- 
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
> 


From endelwar at aregar.it  Tue Jul 28 17:57:10 2015
From: endelwar at aregar.it (Manuel Dalla Lana)
Date: Tue, 28 Jul 2015 19:57:10 +0200
Subject: Duplicated messages
In-Reply-To: <12918F3D-5E50-4F75-9E43-01892E5577F5@mailborder.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
 <12918F3D-5E50-4F75-9E43-01892E5577F5@mailborder.com>
Message-ID: <55B7C276.7090000@aregar.it>

Il 28/07/15 18:15, Jerry Benton ha scritto:
> Shameless jab at Mailwatch: in Mailborder I would check the box and 
> click Resend  :)
exactly as in MailWatch, even before Mailborder existed :) (reflected 
jab :P)

Manuel
(the guy behind MailWatch ATM)

From wilson.galafassi at gmail.com  Tue Jul 28 19:55:59 2015
From: wilson.galafassi at gmail.com (Wilson A. Galafassi Jr. [Gmail])
Date: Tue, 28 Jul 2015 16:55:59 -0300
Subject: RES: RES: RES: pdf corruption
In-Reply-To: <487A91EF-EB52-44CC-9EB8-7E2298B08418@mailborder.com>
References: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>
 <0958DB54-40A4-4F51-AED8-DECF644B9CC1@mailborder.com>
 <18cd01d0c951$7d565c10$78031430$@gmail.com>
 <4C1B0338-A96A-47B1-B169-E1DD5987F431@mailborder.com>
 <191401d0c957$7c363750$74a2a5f0$@gmail.com>
 <BBEF7C34-D106-456F-9D66-BF7FB2FCEEF1@mailborder.com>
 <487A91EF-EB52-44CC-9EB8-7E2298B08418@mailborder.com>
Message-ID: <19a701d0c96f$6e5d5e70$4b181b50$@gmail.com>

No problem.

-----Mensagem original-----
De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em nome de Jerry Benton
Enviada em: terça-feira, 28 de julho de 2015 14:17
Para: MailScanner Discussion
Assunto: Re: RES: RES: pdf corruption

Apologies for sending the screenshot to the list. I have deleted the png file from the list server. 

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 1:11 PM, Jerry Benton <jerry.benton at mailborder.com> wrote:
> 
> A test in the lab with the uncorrupted file you sent remained uncorrupted after passing through MailScanner. See the private email I sent for details.
> 
> -
> Jerry Benton
> www.mailborder.com
> 
> 
> 
>> On Jul 28, 2015, at 1:04 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>> 
>> Another thing: if i generate the files using adobe acrobat the corruption doesn't occour. The corruption occour only on pdf generated by an ERP used on my company.
>> 
>> -----Mensagem original-----
>> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] 
>> Em nome de Jerry Benton Enviada em: terça-feira, 28 de julho de 2015 
>> 13:29
>> Para: MailScanner Discussion
>> Assunto: Re: RES: pdf corruption
>> 
>> I’ve never heard a report of this issue. What virus scanner are you 
>> using? Aside from running the “file” command to check MIME types and 
>> extracting a copy of archives for scanning, MailScanner doesn’t 
>> change file attachments. (Unless you have zip attachments enabled.)
>> 
>> We can test this if you like. Send one of these PDFs to support at linuxref.com. This is a domain in my lab. If it comes through ok, I would make the educated guess that you probably have something somewhere else corrupting them. 
>> 
>> -
>> Jerry Benton
>> www.mailborder.com
>> 
>> 
>> 
>>> On Jul 28, 2015, at 12:21 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>>> 
>>> Same problem...
>>> 
>>> -----Mensagem original-----
>>> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] 
>>> Em nome de Jerry Benton Enviada em: terça-feira, 28 de julho de 2015
>>> 12:38
>>> Para: MailScanner Discussion
>>> Assunto: Re: pdf corruption
>>> 
>>> Upgrade to 4.85.2?
>>> 
>>> -
>>> Jerry Benton
>>> www.mailborder.com
>>> 
>>> 
>>> 
>>>> On Jul 28, 2015, at 9:34 AM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I'm currently using 4.84.5 to store all my messages. I have a 
>>>> problem with some PDF files been corrupted after mailscanner process the files.
>>>> 
>>>> Some ide ato fix this issue or tell mailscanner to don't process pdf files?
>>>> 
>>>> Thanks,
>>>> Wilson
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> MailScanner mailing list
>>>> mailscanner at lists.mailscanner.info
>>>> http://lists.mailscanner.info/listinfo/mailscanner
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info
>>> http://lists.mailscanner.info/listinfo/mailscanner
>>> 
>>> 
>>> 
>>> 
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info
>>> http://lists.mailscanner.info/listinfo/mailscanner
>>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
> 



--
MailScanner mailing list
mailscanner at lists.mailscanner.info
http://lists.mailscanner.info/listinfo/mailscanner



From wilson.galafassi at gmail.com  Tue Jul 28 19:56:18 2015
From: wilson.galafassi at gmail.com (Wilson A. Galafassi Jr. [Gmail])
Date: Tue, 28 Jul 2015 16:56:18 -0300
Subject: RES: RES: RES: pdf corruption
In-Reply-To: <BBEF7C34-D106-456F-9D66-BF7FB2FCEEF1@mailborder.com>
References: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>
 <0958DB54-40A4-4F51-AED8-DECF644B9CC1@mailborder.com>
 <18cd01d0c951$7d565c10$78031430$@gmail.com>
 <4C1B0338-A96A-47B1-B169-E1DD5987F431@mailborder.com>
 <191401d0c957$7c363750$74a2a5f0$@gmail.com>
 <BBEF7C34-D106-456F-9D66-BF7FB2FCEEF1@mailborder.com>
Message-ID: <19ac01d0c96f$7935b4a0$6ba11de0$@gmail.com>

Thanks. Any other tip?

-----Mensagem original-----
De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em nome de Jerry Benton
Enviada em: terça-feira, 28 de julho de 2015 14:11
Para: MailScanner Discussion
Assunto: Re: RES: RES: pdf corruption

A test in the lab with the uncorrupted file you sent remained uncorrupted after passing through MailScanner. See the private email I sent for details.

-
Jerry Benton
www.mailborder.com



> On Jul 28, 2015, at 1:04 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
> 
> Another thing: if i generate the files using adobe acrobat the corruption doesn't occour. The corruption occour only on pdf generated by an ERP used on my company.
> 
> -----Mensagem original-----
> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em 
> nome de Jerry Benton Enviada em: terça-feira, 28 de julho de 2015 
> 13:29
> Para: MailScanner Discussion
> Assunto: Re: RES: pdf corruption
> 
> I’ve never heard a report of this issue. What virus scanner are you 
> using? Aside from running the “file” command to check MIME types and 
> extracting a copy of archives for scanning, MailScanner doesn’t change 
> file attachments. (Unless you have zip attachments enabled.)
> 
> We can test this if you like. Send one of these PDFs to support at linuxref.com. This is a domain in my lab. If it comes through ok, I would make the educated guess that you probably have something somewhere else corrupting them. 
> 
> -
> Jerry Benton
> www.mailborder.com
> 
> 
> 
>> On Jul 28, 2015, at 12:21 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>> 
>> Same problem...
>> 
>> -----Mensagem original-----
>> De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] 
>> Em nome de Jerry Benton Enviada em: terça-feira, 28 de julho de 2015
>> 12:38
>> Para: MailScanner Discussion
>> Assunto: Re: pdf corruption
>> 
>> Upgrade to 4.85.2?
>> 
>> -
>> Jerry Benton
>> www.mailborder.com
>> 
>> 
>> 
>>> On Jul 28, 2015, at 9:34 AM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> I'm currently using 4.84.5 to store all my messages. I have a 
>>> problem with some PDF files been corrupted after mailscanner process the files.
>>> 
>>> Some ide ato fix this issue or tell mailscanner to don't process pdf files?
>>> 
>>> Thanks,
>>> Wilson
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> --
>>> MailScanner mailing list
>>> mailscanner at lists.mailscanner.info
>>> http://lists.mailscanner.info/listinfo/mailscanner
>>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
>> 
>> 
>> 
>> --
>> MailScanner mailing list
>> mailscanner at lists.mailscanner.info
>> http://lists.mailscanner.info/listinfo/mailscanner
>> 
> 
> 
> 
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 
> 
> 
> 
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 



-- 
MailScanner mailing list
mailscanner at lists.mailscanner.info
http://lists.mailscanner.info/listinfo/mailscanner



From wilson.galafassi at gmail.com  Tue Jul 28 19:56:27 2015
From: wilson.galafassi at gmail.com (Wilson A. Galafassi Jr. [Gmail])
Date: Tue, 28 Jul 2015 16:56:27 -0300
Subject: RES: RES: RES: pdf corruption
In-Reply-To: <CC2973D1-8765-4092-B367-21E906625790@mailborder.com>
References: <174f01d0c93a$1ddf3d40$599db7c0$@gmail.com>
 <0958DB54-40A4-4F51-AED8-DECF644B9CC1@mailborder.com>
 <18cd01d0c951$7d565c10$78031430$@gmail.com>
 <4C1B0338-A96A-47B1-B169-E1DD5987F431@mailborder.com>
 <190d01d0c957$2c744720$855cd560$@gmail.com>
 <CC2973D1-8765-4092-B367-21E906625790@mailborder.com>
Message-ID: <19ad01d0c96f$7eee6590$7ccb30b0$@gmail.com>

thanks

 

De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em nome de Jerry Benton
Enviada em: terça-feira, 28 de julho de 2015 14:10
Para: MailScanner Discussion
Assunto: Re: RES: RES: pdf corruption

 

The file came through with no problems. (See screenshot.) Are the emails passing through something else before MailScanner or before hitting the user’s inbox?


-
Jerry Benton
www.mailborder.com <http://www.mailborder.com> 







On Jul 28, 2015, at 1:02 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com <mailto:wilson.galafassi at gmail.com> > wrote:

I don't use any antivirus. Mailscanner is used only to archiving purposes.

I have sent to you 2 files (the same)  1 corrupted (after processed by mailscanner ) and 1 ok.

Thanks,
Wilson


-----Mensagem original-----
De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em nome de Jerry Benton
Enviada em: terça-feira, 28 de julho de 2015 13:29
Para: MailScanner Discussion
Assunto: Re: RES: pdf corruption

I’ve never heard a report of this issue. What virus scanner are you using? Aside from running the “file” command to check MIME types and extracting a copy of archives for scanning, MailScanner doesn’t change file attachments. (Unless you have zip attachments enabled.)

We can test this if you like. Send one of these PDFs to support at linuxref.com <mailto:support at linuxref.com> . This is a domain in my lab. If it comes through ok, I would make the educated guess that you probably have something somewhere else corrupting them. 

-
Jerry Benton
www.mailborder.com <http://www.mailborder.com> 






On Jul 28, 2015, at 12:21 PM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com <mailto:wilson.galafassi at gmail.com> > wrote:

Same problem...

-----Mensagem original-----
De: MailScanner [mailto:mailscanner-bounces at lists.mailscanner.info] Em 
nome de Jerry Benton Enviada em: terça-feira, 28 de julho de 2015 
12:38
Para: MailScanner Discussion
Assunto: Re: pdf corruption

Upgrade to 4.85.2?

-
Jerry Benton
www.mailborder.com <http://www.mailborder.com> 






On Jul 28, 2015, at 9:34 AM, Wilson A. Galafassi Jr. [Gmail] <wilson.galafassi at gmail.com <mailto:wilson.galafassi at gmail.com> > wrote:

Hi,

I'm currently using 4.84.5 to store all my messages. I have a problem 
with some PDF files been corrupted after mailscanner process the files.

Some ide ato fix this issue or tell mailscanner to don't process pdf files?

Thanks,
Wilson






--
MailScanner mailing list
mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> 
http://lists.mailscanner.info/listinfo/mailscanner




--
MailScanner mailing list
mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> 
http://lists.mailscanner.info/listinfo/mailscanner




--
MailScanner mailing list
mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> 
http://lists.mailscanner.info/listinfo/mailscanner




-- 
MailScanner mailing list
mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> 
http://lists.mailscanner.info/listinfo/mailscanner




-- 
MailScanner mailing list
mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> 
http://lists.mailscanner.info/listinfo/mailscanner

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/5c997eba/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 203538 bytes
Desc: not available
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150728/5c997eba/attachment-0001.png>

From mark at workshopit.co.uk  Wed Jul 29 13:13:11 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Wed, 29 Jul 2015 14:13:11 +0100
Subject: Duplicated messages
In-Reply-To: <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
Message-ID: <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>

Hi all,

So I have resolved getting the missing mails delivered from the quarantine.
The main problem stopping this from being easy from the command line was
the fact that "Quarantine Whole Messages As Queue Files = no" was set,
whilst the MTA in use is exim. I've changed that setting to "yes" now...

I've read that if its postfix you can just send that "message" file back to
the queue, I guess the headers are kept with the message in the quarantine
with postfix. With exim they seem to be split between the database and the
message file.

I first put the message ID's in to a file "idlist.txt" that had been put in
to the quarantine with the "Other Bad Content Detected" error (every single
email after a certain time on that day), then pulled the header from the db
and combined them with the following simple loop;

-------
#!/bin/bash
for msgid in `cat idlist.txt`;
do
    /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from
maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *"
>> with-headers/$msgid &&
    /bin/cat 20150724/$msgid/message >> with-headers/$msgid
done
-------

now I'm sending them out slowly (every 30 secs) with another simple loop...

-------
#!/bin/bash
for msgs in with-headers/*;
do
    cat $msgs | exim -ti
    mv $msgs with-headers-processed/
    sleep 30
done
-------

So at least the missing mail is now going to users.. but I'm no closer to
knowing exactly why this happened in the first place. Jeremy mentioned a
known "taint" issue? Can anyone elaborate on that?

I've also found now that Archive is enabled, and is set to "Archive Mail =
" which I guess just defaults to the quarantine dir, as they seem to go to
the "nonspam" folder in there (interestingly in a exim usable format!!)
That couldn't have anything to do with the loop that appears to have killed
my mailcleaner DB? I wouldn't think so as this has been running for years
like this and not had this issue before but thought it worth mentioning.

Any other theories or places to check for clues? unfortunately the mail.log
of the day got removed by the first person looking at the issue to try to
free up space as it was over 4GB.

Regards,
Mark






On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk> wrote:

> Hi Jerry,
>
> If you wanted to pull a bunch of items from the quarantine from the
> command line and re-process them through Mailcleaner, how would you do that?
>
> Regards,
> Mark
>
> On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com>
> wrote:
>
>> I am not sure on what parameters Mailwatch calls and logs “other bad
>> content”.  The MailScanner setting is "Notify Senders of Other Blocked
>> Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked
>> content” for all we know. You are going to have to follow the below
>> suggestion and enable debug or see if you can get an idea from
>> /var/log/maillog.
>>
>> -
>> Jerry Benton
>> www.mailborder.com
>>
>>
>>
>> On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>> Of course, apologies - I'm using Mailwatch. Any advice on how to most
>> efficiently pull things out of quarantine via command-line? (note they are
>> stored as "message" rather than queue items, that would be too easy..)
>>
>> I don't have Archive enabled, everything has gone in to the quarantine
>> because of this "Other Bad Content Detected"
>>
>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com>
>> wrote:
>>
>>> By the way, there is no web interface in the MailScanner package. There
>>> are 3rd party products of course (I created one myself) but those questions
>>> would need to be directed to those support forums or mailing lists.
>>>
>>> -
>>> Jerry Benton
>>> www.mailborder.com
>>>
>>>
>>>
>>> On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>
>>> How do I try send them through again? At the moment they are just
>>> "message" in the quarantine, and if I try open them through the web
>>> interface it times out, I guess because its trying to open each one of the
>>> dupes?
>>>
>>> "Fatal error: Maximum execution time of 30 seconds exceeded in
>>> /var/www/html/mailscanner/functions.php on line 1022"
>>>
>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>>
>>>>  Yup. Turn on debug and watch it pass through. Last time I saw these
>>>> it was a taint issue .. Which I am assuming has been fixed by now.
>>>>
>>>>  --
>>>> Jeremy McSpadden | Flux Labs
>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>> Web - http://www.fluxlabs.net
>>>>
>>>>
>>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>
>>>>   Hi Jeremy,
>>>>
>>>>  Are you saying that something in these messages is crashing
>>>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>>>> (all different types and from different servers) are now in the quarantine
>>>> because of "Other Bad Content Detected" with the report "MailScanner:
>>>> Message attempted to kill MailScanner". It seems it succeeded...
>>>>
>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>>>
>>>>>  It's probably looping/crashing mailscanner. Drop MS into debug mode
>>>>> and watch logs.
>>>>>
>>>>>  --
>>>>> Jeremy McSpadden | Flux Labs
>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>> Web - http://www.fluxlabs.net
>>>>>
>>>>>
>>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>
>>>>>   An update to this, the "2 or 4" duplicates showing in the exim log
>>>>> look like they are actually just separate deliveries to other addresses, so
>>>>> not duplicates. In 1 example there is a single email with 2 recipients (2
>>>>> entries in exim log) that has over 1500+ entries in the mailcleaner DB. It
>>>>> looks like this email hasn't been delivered to the recipient at all either.
>>>>>
>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>>  If anyone could provide advice that would be great. Running Debian
>>>>>> Wheezy Mailscanner 4.79.11-2.2
>>>>>>
>>>>>>  Our incoming dir filled up just before the weekend so we didn't see
>>>>>> the issue for a couple of days. Normally we would just shut down
>>>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>>>> However on this occasion, the root partition also become full because of
>>>>>> the mysql DB (it got to 14G in 2 days..).
>>>>>>
>>>>>>  For some reason everything started duplicating. I can see lots of
>>>>>> incoming messages in the exim logs with duplication (2 or 4 of what looks
>>>>>> like the same email) but in the mailscanner database there is hundreds of
>>>>>> each email listed (apparently there was over 9 million messages delivered
>>>>>> on 1 day compared with the server average of about 1500!)
>>>>>>
>>>>>>  It seems like some sort of loop, but afaik nothing specific was
>>>>>> changed in the config apart from the fact incoming became full. Space has
>>>>>> been cleared on the root partition and incoming, and everything appears to
>>>>>> be running as normal right now.
>>>>>>
>>>>>>  Any advice on debugging this would be much appreciated, also, how
>>>>>> best should I clear out the DB of all the dupes?
>>>>>>
>>>>>>  Thanks!
>>>>>>
>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/a59428dd/attachment.html>

From jeremy at fluxlabs.net  Wed Jul 29 13:16:08 2015
From: jeremy at fluxlabs.net (Jeremy McSpadden)
Date: Wed, 29 Jul 2015 13:16:08 +0000
Subject: Duplicated messages
In-Reply-To: <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>,
 <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
Message-ID: <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>

Log won't show taint issues. Setup log rotation.

Have you enabled debug in mailscanner config like I stated yesterday ?

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi all,

So I have resolved getting the missing mails delivered from the quarantine. The main problem stopping this from being easy from the command line was the fact that "Quarantine Whole Messages As Queue Files = no" was set, whilst the MTA in use is exim. I've changed that setting to "yes" now...

I've read that if its postfix you can just send that "message" file back to the queue, I guess the headers are kept with the message in the quarantine with postfix. With exim they seem to be split between the database and the message file.

I first put the message ID's in to a file "idlist.txt" that had been put in to the quarantine with the "Other Bad Content Detected" error (every single email after a certain time on that day), then pulled the header from the db and combined them with the following simple loop;

-------
#!/bin/bash
for msgid in `cat idlist.txt`;
do
    /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *" >> with-headers/$msgid &&
    /bin/cat 20150724/$msgid/message >> with-headers/$msgid
done
-------

now I'm sending them out slowly (every 30 secs) with another simple loop...

-------
#!/bin/bash
for msgs in with-headers/*;
do
    cat $msgs | exim -ti
    mv $msgs with-headers-processed/
    sleep 30
done
-------

So at least the missing mail is now going to users.. but I'm no closer to knowing exactly why this happened in the first place. Jeremy mentioned a known "taint" issue? Can anyone elaborate on that?

I've also found now that Archive is enabled, and is set to "Archive Mail = " which I guess just defaults to the quarantine dir, as they seem to go to the "nonspam" folder in there (interestingly in a exim usable format!!) That couldn't have anything to do with the loop that appears to have killed my mailcleaner DB? I wouldn't think so as this has been running for years like this and not had this issue before but thought it worth mentioning.

Any other theories or places to check for clues? unfortunately the mail.log of the day got removed by the first person looking at the issue to try to free up space as it was over 4GB.

Regards,
Mark






On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:
Hi Jerry,

If you wanted to pull a bunch of items from the quarantine from the command line and re-process them through Mailcleaner, how would you do that?

Regards,
Mark

On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com<mailto:jerry.benton at mailborder.com>> wrote:
I am not sure on what parameters Mailwatch calls and logs “other bad content”.  The MailScanner setting is "Notify Senders of Other Blocked Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked content” for all we know. You are going to have to follow the below suggestion and enable debug or see if you can get an idea from /var/log/maillog.

-
Jerry Benton
www.mailborder.com<http://www.mailborder.com>



On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Of course, apologies - I'm using Mailwatch. Any advice on how to most efficiently pull things out of quarantine via command-line? (note they are stored as "message" rather than queue items, that would be too easy..)

I don't have Archive enabled, everything has gone in to the quarantine because of this "Other Bad Content Detected"

On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com<mailto:jerry.benton at mailborder.com>> wrote:
By the way, there is no web interface in the MailScanner package. There are 3rd party products of course (I created one myself) but those questions would need to be directed to those support forums or mailing lists.

-
Jerry Benton
www.mailborder.com<http://www.mailborder.com/>



On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?

"Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"

On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi Jeremy,

Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...

On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.

On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:
Hi All,

If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2

Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).

For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)

It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.

Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?

Thanks!



--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/listinfo/mailscanner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/c6f220f3/attachment.html>

From mark at workshopit.co.uk  Wed Jul 29 13:27:29 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Wed, 29 Jul 2015 14:27:29 +0100
Subject: Duplicated messages
In-Reply-To: <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
 <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
 <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>
Message-ID: <CABF=4+w8EEbJM1d2=rEXCwEoVn8hhdn+MJW7ypdT-PnpE8Gt6g@mail.gmail.com>

Hi Jeremy,

No I haven't yet - I'm short on space on my root partition still because of
the large mysql DB so I want to clean that up first. Can you advise how
best to do this? Is it safe enough to do delete from maillog where
id='XXX';  for all the ID's with the dupes? is there any other tables that
need to be cleared?

Regards,
Mark

On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:

>  Log won't show taint issues. Setup log rotation.
>
>  Have you enabled debug in mailscanner config like I stated yesterday ?
>
>  --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
> Web - http://www.fluxlabs.net
>
>
> On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
>   Hi all,
>
>  So I have resolved getting the missing mails delivered from the
> quarantine. The main problem stopping this from being easy from the command
> line was the fact that "Quarantine Whole Messages As Queue Files = no" was
> set, whilst the MTA in use is exim. I've changed that setting to "yes"
> now...
>
>  I've read that if its postfix you can just send that "message" file back
> to the queue, I guess the headers are kept with the message in the
> quarantine with postfix. With exim they seem to be split between the
> database and the message file.
>
>  I first put the message ID's in to a file "idlist.txt" that had been put
> in to the quarantine with the "Other Bad Content Detected" error (every
> single email after a certain time on that day), then pulled the header from
> the db and combined them with the following simple loop;
>
>  -------
> #!/bin/bash
>  for msgid in `cat idlist.txt`;
> do
>     /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from
> maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *"
> >> with-headers/$msgid &&
>     /bin/cat 20150724/$msgid/message >> with-headers/$msgid
> done
>  -------
>
>  now I'm sending them out slowly (every 30 secs) with another simple
> loop...
>
>  -------
>  #!/bin/bash
> for msgs in with-headers/*;
> do
>     cat $msgs | exim -ti
>     mv $msgs with-headers-processed/
>     sleep 30
> done
>  -------
>
>  So at least the missing mail is now going to users.. but I'm no closer
> to knowing exactly why this happened in the first place. Jeremy mentioned a
> known "taint" issue? Can anyone elaborate on that?
>
>  I've also found now that Archive is enabled, and is set to "Archive Mail
> = " which I guess just defaults to the quarantine dir, as they seem to go
> to the "nonspam" folder in there (interestingly in a exim usable format!!)
> That couldn't have anything to do with the loop that appears to have killed
> my mailcleaner DB? I wouldn't think so as this has been running for years
> like this and not had this issue before but thought it worth mentioning.
>
>  Any other theories or places to check for clues? unfortunately the
> mail.log of the day got removed by the first person looking at the issue to
> try to free up space as it was over 4GB.
>
>  Regards,
> Mark
>
>
>
>
>
>
> On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk> wrote:
>
>> Hi Jerry,
>>
>>  If you wanted to pull a bunch of items from the quarantine from the
>> command line and re-process them through Mailcleaner, how would you do that?
>>
>>  Regards,
>> Mark
>>
>> On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com>
>> wrote:
>>
>>> I am not sure on what parameters Mailwatch calls and logs “other bad
>>> content”.  The MailScanner setting is "Notify Senders of Other Blocked
>>> Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked
>>> content” for all we know. You are going to have to follow the below
>>> suggestion and enable debug or see if you can get an idea from
>>> /var/log/maillog.
>>>
>>> -
>>>  Jerry Benton
>>>  www.mailborder.com
>>>
>>>
>>>
>>>   On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk>
>>> wrote:
>>>
>>>  Of course, apologies - I'm using Mailwatch. Any advice on how to most
>>> efficiently pull things out of quarantine via command-line? (note they are
>>> stored as "message" rather than queue items, that would be too easy..)
>>>
>>>  I don't have Archive enabled, everything has gone in to the quarantine
>>> because of this "Other Bad Content Detected"
>>>
>>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com>
>>> wrote:
>>>
>>>> By the way, there is no web interface in the MailScanner package. There
>>>> are 3rd party products of course (I created one myself) but those questions
>>>> would need to be directed to those support forums or mailing lists.
>>>>
>>>> -
>>>>  Jerry Benton
>>>>  www.mailborder.com
>>>>
>>>>
>>>>
>>>>   On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk>
>>>> wrote:
>>>>
>>>>  How do I try send them through again? At the moment they are just
>>>> "message" in the quarantine, and if I try open them through the web
>>>> interface it times out, I guess because its trying to open each one of the
>>>> dupes?
>>>>
>>>>  "Fatal error: Maximum execution time of 30 seconds exceeded in
>>>> /var/www/html/mailscanner/functions.php on line 1022"
>>>>
>>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>>>
>>>>>  Yup. Turn on debug and watch it pass through. Last time I saw these
>>>>> it was a taint issue .. Which I am assuming has been fixed by now.
>>>>>
>>>>>  --
>>>>> Jeremy McSpadden | Flux Labs
>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>> Web - http://www.fluxlabs.net
>>>>>
>>>>>
>>>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk>
>>>>> wrote:
>>>>>
>>>>>   Hi Jeremy,
>>>>>
>>>>>  Are you saying that something in these messages is crashing
>>>>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>>>>> (all different types and from different servers) are now in the quarantine
>>>>> because of "Other Bad Content Detected" with the report "MailScanner:
>>>>> Message attempted to kill MailScanner". It seems it succeeded...
>>>>>
>>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>> wrote:
>>>>>
>>>>>>  It's probably looping/crashing mailscanner. Drop MS into debug mode
>>>>>> and watch logs.
>>>>>>
>>>>>>  --
>>>>>> Jeremy McSpadden | Flux Labs
>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>> Web - http://www.fluxlabs.net
>>>>>>
>>>>>>
>>>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>> wrote:
>>>>>>
>>>>>>   An update to this, the "2 or 4" duplicates showing in the exim log
>>>>>> look like they are actually just separate deliveries to other addresses, so
>>>>>> not duplicates. In 1 example there is a single email with 2 recipients (2
>>>>>> entries in exim log) that has over 1500+ entries in the mailcleaner DB. It
>>>>>> looks like this email hasn't been delivered to the recipient at all either.
>>>>>>
>>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>>  If anyone could provide advice that would be great. Running Debian
>>>>>>> Wheezy Mailscanner 4.79.11-2.2
>>>>>>>
>>>>>>>  Our incoming dir filled up just before the weekend so we didn't
>>>>>>> see the issue for a couple of days. Normally we would just shut down
>>>>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>>>>> However on this occasion, the root partition also become full because of
>>>>>>> the mysql DB (it got to 14G in 2 days..).
>>>>>>>
>>>>>>>  For some reason everything started duplicating. I can see lots of
>>>>>>> incoming messages in the exim logs with duplication (2 or 4 of what looks
>>>>>>> like the same email) but in the mailscanner database there is hundreds of
>>>>>>> each email listed (apparently there was over 9 million messages delivered
>>>>>>> on 1 day compared with the server average of about 1500!)
>>>>>>>
>>>>>>>  It seems like some sort of loop, but afaik nothing specific was
>>>>>>> changed in the config apart from the fact incoming became full. Space has
>>>>>>> been cleared on the root partition and incoming, and everything appears to
>>>>>>> be running as normal right now.
>>>>>>>
>>>>>>>  Any advice on debugging this would be much appreciated, also, how
>>>>>>> best should I clear out the DB of all the dupes?
>>>>>>>
>>>>>>>  Thanks!
>>>>>>>
>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/921b1470/attachment.html>

From jeremy at fluxlabs.net  Wed Jul 29 13:34:02 2015
From: jeremy at fluxlabs.net (Jeremy McSpadden)
Date: Wed, 29 Jul 2015 13:34:02 +0000
Subject: Duplicated messages
In-Reply-To: <CABF=4+w8EEbJM1d2=rEXCwEoVn8hhdn+MJW7ypdT-PnpE8Gt6g@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
 <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
 <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>,
 <CABF=4+w8EEbJM1d2=rEXCwEoVn8hhdn+MJW7ypdT-PnpE8Gt6g@mail.gmail.com>
Message-ID: <11C72614-623D-4E8C-931F-484CDC8A2BE7@fluxlabs.net>

It's 2015 .. That shouldn't be an excuse. It's like 10 cents per 100gb of drive... Upgrade

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 8:27 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi Jeremy,

No I haven't yet - I'm short on space on my root partition still because of the large mysql DB so I want to clean that up first. Can you advise how best to do this? Is it safe enough to do delete from maillog where id='XXX';  for all the ID's with the dupes? is there any other tables that need to be cleared?

Regards,
Mark

On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
Log won't show taint issues. Setup log rotation.

Have you enabled debug in mailscanner config like I stated yesterday ?

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi all,

So I have resolved getting the missing mails delivered from the quarantine. The main problem stopping this from being easy from the command line was the fact that "Quarantine Whole Messages As Queue Files = no" was set, whilst the MTA in use is exim. I've changed that setting to "yes" now...

I've read that if its postfix you can just send that "message" file back to the queue, I guess the headers are kept with the message in the quarantine with postfix. With exim they seem to be split between the database and the message file.

I first put the message ID's in to a file "idlist.txt" that had been put in to the quarantine with the "Other Bad Content Detected" error (every single email after a certain time on that day), then pulled the header from the db and combined them with the following simple loop;

-------
#!/bin/bash
for msgid in `cat idlist.txt`;
do
    /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *" >> with-headers/$msgid &&
    /bin/cat 20150724/$msgid/message >> with-headers/$msgid
done
-------

now I'm sending them out slowly (every 30 secs) with another simple loop...

-------
#!/bin/bash
for msgs in with-headers/*;
do
    cat $msgs | exim -ti
    mv $msgs with-headers-processed/
    sleep 30
done
-------

So at least the missing mail is now going to users.. but I'm no closer to knowing exactly why this happened in the first place. Jeremy mentioned a known "taint" issue? Can anyone elaborate on that?

I've also found now that Archive is enabled, and is set to "Archive Mail = " which I guess just defaults to the quarantine dir, as they seem to go to the "nonspam" folder in there (interestingly in a exim usable format!!) That couldn't have anything to do with the loop that appears to have killed my mailcleaner DB? I wouldn't think so as this has been running for years like this and not had this issue before but thought it worth mentioning.

Any other theories or places to check for clues? unfortunately the mail.log of the day got removed by the first person looking at the issue to try to free up space as it was over 4GB.

Regards,
Mark






On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:
Hi Jerry,

If you wanted to pull a bunch of items from the quarantine from the command line and re-process them through Mailcleaner, how would you do that?

Regards,
Mark

On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com<mailto:jerry.benton at mailborder.com>> wrote:
I am not sure on what parameters Mailwatch calls and logs “other bad content”.  The MailScanner setting is "Notify Senders of Other Blocked Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked content” for all we know. You are going to have to follow the below suggestion and enable debug or see if you can get an idea from /var/log/maillog.

-
Jerry Benton
www.mailborder.com<http://www.mailborder.com>



On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Of course, apologies - I'm using Mailwatch. Any advice on how to most efficiently pull things out of quarantine via command-line? (note they are stored as "message" rather than queue items, that would be too easy..)

I don't have Archive enabled, everything has gone in to the quarantine because of this "Other Bad Content Detected"

On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com<mailto:jerry.benton at mailborder.com>> wrote:
By the way, there is no web interface in the MailScanner package. There are 3rd party products of course (I created one myself) but those questions would need to be directed to those support forums or mailing lists.

-
Jerry Benton
www.mailborder.com<http://www.mailborder.com/>



On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?

"Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"

On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi Jeremy,

Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...

On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.

On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:
Hi All,

If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2

Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).

For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)

It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.

Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?

Thanks!



--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/listinfo/mailscanner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/976a4b4d/attachment-0001.html>

From mark at workshopit.co.uk  Wed Jul 29 13:59:59 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Wed, 29 Jul 2015 14:59:59 +0100
Subject: Duplicated messages
In-Reply-To: <11C72614-623D-4E8C-931F-484CDC8A2BE7@fluxlabs.net>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
 <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
 <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>
 <CABF=4+w8EEbJM1d2=rEXCwEoVn8hhdn+MJW7ypdT-PnpE8Gt6g@mail.gmail.com>
 <11C72614-623D-4E8C-931F-484CDC8A2BE7@fluxlabs.net>
Message-ID: <CABF=4+ySoMqGpu6CfhfHmVUhKEcsyZx3jsyuFmib5D6i5h9wWg@mail.gmail.com>

I agree with you completely, however that doesn't help my immediate
situation. Can you provide advice on deleting from the mailscanner DB? is
there any other tables I need to remove the offending ID's entries from?

Regards,
Mark

On 29 July 2015 at 14:34, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:

>  It's 2015 .. That shouldn't be an excuse. It's like 10 cents per 100gb
> of drive... Upgrade
>
>  --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
> Web - http://www.fluxlabs.net
>
>
> On Jul 29, 2015, at 8:27 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
>   Hi Jeremy,
>
>  No I haven't yet - I'm short on space on my root partition still because
> of the large mysql DB so I want to clean that up first. Can you advise how
> best to do this? Is it safe enough to do delete from maillog where
> id='XXX';  for all the ID's with the dupes? is there any other tables that
> need to be cleared?
>
>  Regards,
> Mark
>
> On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>
>>  Log won't show taint issues. Setup log rotation.
>>
>>  Have you enabled debug in mailscanner config like I stated yesterday ?
>>
>>  --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>> Web - http://www.fluxlabs.net
>>
>>
>> On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>>   Hi all,
>>
>>  So I have resolved getting the missing mails delivered from the
>> quarantine. The main problem stopping this from being easy from the command
>> line was the fact that "Quarantine Whole Messages As Queue Files = no" was
>> set, whilst the MTA in use is exim. I've changed that setting to "yes"
>> now...
>>
>>  I've read that if its postfix you can just send that "message" file
>> back to the queue, I guess the headers are kept with the message in the
>> quarantine with postfix. With exim they seem to be split between the
>> database and the message file.
>>
>>  I first put the message ID's in to a file "idlist.txt" that had been
>> put in to the quarantine with the "Other Bad Content Detected" error (every
>> single email after a certain time on that day), then pulled the header from
>> the db and combined them with the following simple loop;
>>
>>  -------
>> #!/bin/bash
>>  for msgid in `cat idlist.txt`;
>> do
>>     /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from
>> maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *"
>> >> with-headers/$msgid &&
>>     /bin/cat 20150724/$msgid/message >> with-headers/$msgid
>> done
>>  -------
>>
>>  now I'm sending them out slowly (every 30 secs) with another simple
>> loop...
>>
>>  -------
>>  #!/bin/bash
>> for msgs in with-headers/*;
>> do
>>     cat $msgs | exim -ti
>>     mv $msgs with-headers-processed/
>>     sleep 30
>> done
>>  -------
>>
>>  So at least the missing mail is now going to users.. but I'm no closer
>> to knowing exactly why this happened in the first place. Jeremy mentioned a
>> known "taint" issue? Can anyone elaborate on that?
>>
>>  I've also found now that Archive is enabled, and is set to "Archive
>> Mail = " which I guess just defaults to the quarantine dir, as they seem to
>> go to the "nonspam" folder in there (interestingly in a exim usable
>> format!!) That couldn't have anything to do with the loop that appears to
>> have killed my mailcleaner DB? I wouldn't think so as this has been running
>> for years like this and not had this issue before but thought it worth
>> mentioning.
>>
>>  Any other theories or places to check for clues? unfortunately the
>> mail.log of the day got removed by the first person looking at the issue to
>> try to free up space as it was over 4GB.
>>
>>  Regards,
>> Mark
>>
>>
>>
>>
>>
>>
>> On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>>> Hi Jerry,
>>>
>>>  If you wanted to pull a bunch of items from the quarantine from the
>>> command line and re-process them through Mailcleaner, how would you do that?
>>>
>>>  Regards,
>>> Mark
>>>
>>> On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com>
>>> wrote:
>>>
>>>> I am not sure on what parameters Mailwatch calls and logs “other bad
>>>> content”.  The MailScanner setting is "Notify Senders of Other Blocked
>>>> Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked
>>>> content” for all we know. You are going to have to follow the below
>>>> suggestion and enable debug or see if you can get an idea from
>>>> /var/log/maillog.
>>>>
>>>> -
>>>>  Jerry Benton
>>>>  www.mailborder.com
>>>>
>>>>
>>>>
>>>>   On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk>
>>>> wrote:
>>>>
>>>>  Of course, apologies - I'm using Mailwatch. Any advice on how to most
>>>> efficiently pull things out of quarantine via command-line? (note they are
>>>> stored as "message" rather than queue items, that would be too easy..)
>>>>
>>>>  I don't have Archive enabled, everything has gone in to the
>>>> quarantine because of this "Other Bad Content Detected"
>>>>
>>>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com>
>>>> wrote:
>>>>
>>>>> By the way, there is no web interface in the MailScanner package.
>>>>> There are 3rd party products of course (I created one myself) but those
>>>>> questions would need to be directed to those support forums or mailing
>>>>> lists.
>>>>>
>>>>> -
>>>>>  Jerry Benton
>>>>>  www.mailborder.com
>>>>>
>>>>>
>>>>>
>>>>>   On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk>
>>>>> wrote:
>>>>>
>>>>>  How do I try send them through again? At the moment they are just
>>>>> "message" in the quarantine, and if I try open them through the web
>>>>> interface it times out, I guess because its trying to open each one of the
>>>>> dupes?
>>>>>
>>>>>  "Fatal error: Maximum execution time of 30 seconds exceeded in
>>>>> /var/www/html/mailscanner/functions.php on line 1022"
>>>>>
>>>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>> wrote:
>>>>>
>>>>>>  Yup. Turn on debug and watch it pass through. Last time I saw these
>>>>>> it was a taint issue .. Which I am assuming has been fixed by now.
>>>>>>
>>>>>>  --
>>>>>> Jeremy McSpadden | Flux Labs
>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>> Web - http://www.fluxlabs.net
>>>>>>
>>>>>>
>>>>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>> wrote:
>>>>>>
>>>>>>   Hi Jeremy,
>>>>>>
>>>>>>  Are you saying that something in these messages is crashing
>>>>>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>>>>>> (all different types and from different servers) are now in the quarantine
>>>>>> because of "Other Bad Content Detected" with the report "MailScanner:
>>>>>> Message attempted to kill MailScanner". It seems it succeeded...
>>>>>>
>>>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>>> wrote:
>>>>>>
>>>>>>>  It's probably looping/crashing mailscanner. Drop MS into debug
>>>>>>> mode and watch logs.
>>>>>>>
>>>>>>>  --
>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>>> Web - http://www.fluxlabs.net
>>>>>>>
>>>>>>>
>>>>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>> wrote:
>>>>>>>
>>>>>>>   An update to this, the "2 or 4" duplicates showing in the exim
>>>>>>> log look like they are actually just separate deliveries to other
>>>>>>> addresses, so not duplicates. In 1 example there is a single email with 2
>>>>>>> recipients (2 entries in exim log) that has over 1500+ entries in the
>>>>>>> mailcleaner DB. It looks like this email hasn't been delivered to the
>>>>>>> recipient at all either.
>>>>>>>
>>>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>>  If anyone could provide advice that would be great. Running Debian
>>>>>>>> Wheezy Mailscanner 4.79.11-2.2
>>>>>>>>
>>>>>>>>  Our incoming dir filled up just before the weekend so we didn't
>>>>>>>> see the issue for a couple of days. Normally we would just shut down
>>>>>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>>>>>> However on this occasion, the root partition also become full because of
>>>>>>>> the mysql DB (it got to 14G in 2 days..).
>>>>>>>>
>>>>>>>>  For some reason everything started duplicating. I can see lots of
>>>>>>>> incoming messages in the exim logs with duplication (2 or 4 of what looks
>>>>>>>> like the same email) but in the mailscanner database there is hundreds of
>>>>>>>> each email listed (apparently there was over 9 million messages delivered
>>>>>>>> on 1 day compared with the server average of about 1500!)
>>>>>>>>
>>>>>>>>  It seems like some sort of loop, but afaik nothing specific was
>>>>>>>> changed in the config apart from the fact incoming became full. Space has
>>>>>>>> been cleared on the root partition and incoming, and everything appears to
>>>>>>>> be running as normal right now.
>>>>>>>>
>>>>>>>>  Any advice on debugging this would be much appreciated, also, how
>>>>>>>> best should I clear out the DB of all the dupes?
>>>>>>>>
>>>>>>>>  Thanks!
>>>>>>>>
>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/372ce321/attachment.html>

From jeremy at fluxlabs.net  Wed Jul 29 14:04:59 2015
From: jeremy at fluxlabs.net (Jeremy McSpadden)
Date: Wed, 29 Jul 2015 14:04:59 +0000
Subject: Duplicated messages
In-Reply-To: <CABF=4+ySoMqGpu6CfhfHmVUhKEcsyZx3jsyuFmib5D6i5h9wWg@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
 <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
 <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>
 <CABF=4+w8EEbJM1d2=rEXCwEoVn8hhdn+MJW7ypdT-PnpE8Gt6g@mail.gmail.com>
 <11C72614-623D-4E8C-931F-484CDC8A2BE7@fluxlabs.net>,
 <CABF=4+ySoMqGpu6CfhfHmVUhKEcsyZx3jsyuFmib5D6i5h9wWg@mail.gmail.com>
Message-ID: <4C8BEE26-3CA7-450A-BD4E-226C0A461202@fluxlabs.net>

1000 entries from a sql DB may be about 5mb of space.
I would do a little more digging on the / partition to see what's eating space. Probably better off dropping some older archive dir

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 9:00 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

I agree with you completely, however that doesn't help my immediate situation. Can you provide advice on deleting from the mailscanner DB? is there any other tables I need to remove the offending ID's entries from?

Regards,
Mark

On 29 July 2015 at 14:34, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
It's 2015 .. That shouldn't be an excuse. It's like 10 cents per 100gb of drive... Upgrade

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 8:27 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi Jeremy,

No I haven't yet - I'm short on space on my root partition still because of the large mysql DB so I want to clean that up first. Can you advise how best to do this? Is it safe enough to do delete from maillog where id='XXX';  for all the ID's with the dupes? is there any other tables that need to be cleared?

Regards,
Mark

On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
Log won't show taint issues. Setup log rotation.

Have you enabled debug in mailscanner config like I stated yesterday ?

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi all,

So I have resolved getting the missing mails delivered from the quarantine. The main problem stopping this from being easy from the command line was the fact that "Quarantine Whole Messages As Queue Files = no" was set, whilst the MTA in use is exim. I've changed that setting to "yes" now...

I've read that if its postfix you can just send that "message" file back to the queue, I guess the headers are kept with the message in the quarantine with postfix. With exim they seem to be split between the database and the message file.

I first put the message ID's in to a file "idlist.txt" that had been put in to the quarantine with the "Other Bad Content Detected" error (every single email after a certain time on that day), then pulled the header from the db and combined them with the following simple loop;

-------
#!/bin/bash
for msgid in `cat idlist.txt`;
do
    /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *" >> with-headers/$msgid &&
    /bin/cat 20150724/$msgid/message >> with-headers/$msgid
done
-------

now I'm sending them out slowly (every 30 secs) with another simple loop...

-------
#!/bin/bash
for msgs in with-headers/*;
do
    cat $msgs | exim -ti
    mv $msgs with-headers-processed/
    sleep 30
done
-------

So at least the missing mail is now going to users.. but I'm no closer to knowing exactly why this happened in the first place. Jeremy mentioned a known "taint" issue? Can anyone elaborate on that?

I've also found now that Archive is enabled, and is set to "Archive Mail = " which I guess just defaults to the quarantine dir, as they seem to go to the "nonspam" folder in there (interestingly in a exim usable format!!) That couldn't have anything to do with the loop that appears to have killed my mailcleaner DB? I wouldn't think so as this has been running for years like this and not had this issue before but thought it worth mentioning.

Any other theories or places to check for clues? unfortunately the mail.log of the day got removed by the first person looking at the issue to try to free up space as it was over 4GB.

Regards,
Mark






On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:
Hi Jerry,

If you wanted to pull a bunch of items from the quarantine from the command line and re-process them through Mailcleaner, how would you do that?

Regards,
Mark

On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com<mailto:jerry.benton at mailborder.com>> wrote:
I am not sure on what parameters Mailwatch calls and logs "other bad content".  The MailScanner setting is "Notify Senders of Other Blocked Content". Mailwatch could be calling a trigger of a spam RBL "other blocked content" for all we know. You are going to have to follow the below suggestion and enable debug or see if you can get an idea from /var/log/maillog.

-
Jerry Benton
www.mailborder.com<http://www.mailborder.com>



On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Of course, apologies - I'm using Mailwatch. Any advice on how to most efficiently pull things out of quarantine via command-line? (note they are stored as "message" rather than queue items, that would be too easy..)

I don't have Archive enabled, everything has gone in to the quarantine because of this "Other Bad Content Detected"

On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com<mailto:jerry.benton at mailborder.com>> wrote:
By the way, there is no web interface in the MailScanner package. There are 3rd party products of course (I created one myself) but those questions would need to be directed to those support forums or mailing lists.

-
Jerry Benton
www.mailborder.com<http://www.mailborder.com/>



On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?

"Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"

On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi Jeremy,

Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...

On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.

On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:
Hi All,

If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2

Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).

For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)

It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.

Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?

Thanks!


--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/listinfo/mailscanner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/cb14179e/attachment.html>

From mark at workshopit.co.uk  Wed Jul 29 14:08:21 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Wed, 29 Jul 2015 15:08:21 +0100
Subject: Duplicated messages
In-Reply-To: <4C8BEE26-3CA7-450A-BD4E-226C0A461202@fluxlabs.net>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
 <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
 <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>
 <CABF=4+w8EEbJM1d2=rEXCwEoVn8hhdn+MJW7ypdT-PnpE8Gt6g@mail.gmail.com>
 <11C72614-623D-4E8C-931F-484CDC8A2BE7@fluxlabs.net>
 <CABF=4+ySoMqGpu6CfhfHmVUhKEcsyZx3jsyuFmib5D6i5h9wWg@mail.gmail.com>
 <4C8BEE26-3CA7-450A-BD4E-226C0A461202@fluxlabs.net>
Message-ID: <CABF=4+z8LopwFLOzBmE=Md8r+L0SVs+m+cRmKxaO7V0_OmiWwQ@mail.gmail.com>

It's not 1000 entries, its about that many (some id's with a huge amount
more) per message ID. The Mailscanner DB has over 9,000,000 entries for
that day. I was planning on looping through and deleting all the offending
entries for the 300+ emails that seemed to get stuck.

The mailscanner DB is currently 14G (maillog.MYD)

On 29 July 2015 at 15:04, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:

>  1000 entries from a sql DB may be about 5mb of space.
> I would do a little more digging on the / partition to see what's eating
> space. Probably better off dropping some older archive dir
>
>  --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
> Web - http://www.fluxlabs.net
>
>
> On Jul 29, 2015, at 9:00 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
>   I agree with you completely, however that doesn't help my immediate
> situation. Can you provide advice on deleting from the mailscanner DB? is
> there any other tables I need to remove the offending ID's entries from?
>
> Regards,
> Mark
>
> On 29 July 2015 at 14:34, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>
>>  It's 2015 .. That shouldn't be an excuse. It's like 10 cents per 100gb
>> of drive... Upgrade
>>
>>  --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>> Web - http://www.fluxlabs.net
>>
>>
>> On Jul 29, 2015, at 8:27 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>>   Hi Jeremy,
>>
>>  No I haven't yet - I'm short on space on my root partition still
>> because of the large mysql DB so I want to clean that up first. Can you
>> advise how best to do this? Is it safe enough to do delete from maillog
>> where id='XXX';  for all the ID's with the dupes? is there any other tables
>> that need to be cleared?
>>
>>  Regards,
>> Mark
>>
>> On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>
>>>  Log won't show taint issues. Setup log rotation.
>>>
>>>  Have you enabled debug in mailscanner config like I stated yesterday ?
>>>
>>>  --
>>> Jeremy McSpadden | Flux Labs
>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>> Web - http://www.fluxlabs.net
>>>
>>>
>>> On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>
>>>   Hi all,
>>>
>>>  So I have resolved getting the missing mails delivered from the
>>> quarantine. The main problem stopping this from being easy from the command
>>> line was the fact that "Quarantine Whole Messages As Queue Files = no" was
>>> set, whilst the MTA in use is exim. I've changed that setting to "yes"
>>> now...
>>>
>>>  I've read that if its postfix you can just send that "message" file
>>> back to the queue, I guess the headers are kept with the message in the
>>> quarantine with postfix. With exim they seem to be split between the
>>> database and the message file.
>>>
>>>  I first put the message ID's in to a file "idlist.txt" that had been
>>> put in to the quarantine with the "Other Bad Content Detected" error (every
>>> single email after a certain time on that day), then pulled the header from
>>> the db and combined them with the following simple loop;
>>>
>>>  -------
>>> #!/bin/bash
>>>  for msgid in `cat idlist.txt`;
>>> do
>>>     /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from
>>> maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *"
>>> >> with-headers/$msgid &&
>>>     /bin/cat 20150724/$msgid/message >> with-headers/$msgid
>>> done
>>>  -------
>>>
>>>  now I'm sending them out slowly (every 30 secs) with another simple
>>> loop...
>>>
>>>  -------
>>>  #!/bin/bash
>>> for msgs in with-headers/*;
>>> do
>>>     cat $msgs | exim -ti
>>>     mv $msgs with-headers-processed/
>>>     sleep 30
>>> done
>>>  -------
>>>
>>>  So at least the missing mail is now going to users.. but I'm no closer
>>> to knowing exactly why this happened in the first place. Jeremy mentioned a
>>> known "taint" issue? Can anyone elaborate on that?
>>>
>>>  I've also found now that Archive is enabled, and is set to "Archive
>>> Mail = " which I guess just defaults to the quarantine dir, as they seem to
>>> go to the "nonspam" folder in there (interestingly in a exim usable
>>> format!!) That couldn't have anything to do with the loop that appears to
>>> have killed my mailcleaner DB? I wouldn't think so as this has been running
>>> for years like this and not had this issue before but thought it worth
>>> mentioning.
>>>
>>>  Any other theories or places to check for clues? unfortunately the
>>> mail.log of the day got removed by the first person looking at the issue to
>>> try to free up space as it was over 4GB.
>>>
>>>  Regards,
>>> Mark
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk> wrote:
>>>
>>>> Hi Jerry,
>>>>
>>>>  If you wanted to pull a bunch of items from the quarantine from the
>>>> command line and re-process them through Mailcleaner, how would you do that?
>>>>
>>>>  Regards,
>>>> Mark
>>>>
>>>> On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com>
>>>> wrote:
>>>>
>>>>> I am not sure on what parameters Mailwatch calls and logs “other bad
>>>>> content”.  The MailScanner setting is "Notify Senders of Other Blocked
>>>>> Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked
>>>>> content” for all we know. You are going to have to follow the below
>>>>> suggestion and enable debug or see if you can get an idea from
>>>>> /var/log/maillog.
>>>>>
>>>>> -
>>>>>  Jerry Benton
>>>>>  www.mailborder.com
>>>>>
>>>>>
>>>>>
>>>>>   On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk>
>>>>> wrote:
>>>>>
>>>>>  Of course, apologies - I'm using Mailwatch. Any advice on how to
>>>>> most efficiently pull things out of quarantine via command-line? (note they
>>>>> are stored as "message" rather than queue items, that would be too easy..)
>>>>>
>>>>>  I don't have Archive enabled, everything has gone in to the
>>>>> quarantine because of this "Other Bad Content Detected"
>>>>>
>>>>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com>
>>>>> wrote:
>>>>>
>>>>>> By the way, there is no web interface in the MailScanner package.
>>>>>> There are 3rd party products of course (I created one myself) but those
>>>>>> questions would need to be directed to those support forums or mailing
>>>>>> lists.
>>>>>>
>>>>>> -
>>>>>>  Jerry Benton
>>>>>>  www.mailborder.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>   On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>> wrote:
>>>>>>
>>>>>>  How do I try send them through again? At the moment they are just
>>>>>> "message" in the quarantine, and if I try open them through the web
>>>>>> interface it times out, I guess because its trying to open each one of the
>>>>>> dupes?
>>>>>>
>>>>>>  "Fatal error: Maximum execution time of 30 seconds exceeded in
>>>>>> /var/www/html/mailscanner/functions.php on line 1022"
>>>>>>
>>>>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>>> wrote:
>>>>>>
>>>>>>>  Yup. Turn on debug and watch it pass through. Last time I saw
>>>>>>> these it was a taint issue .. Which I am assuming has been fixed by now.
>>>>>>>
>>>>>>>  --
>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>>> Web - http://www.fluxlabs.net
>>>>>>>
>>>>>>>
>>>>>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>> wrote:
>>>>>>>
>>>>>>>   Hi Jeremy,
>>>>>>>
>>>>>>>  Are you saying that something in these messages is crashing
>>>>>>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>>>>>>> (all different types and from different servers) are now in the quarantine
>>>>>>> because of "Other Bad Content Detected" with the report "MailScanner:
>>>>>>> Message attempted to kill MailScanner". It seems it succeeded...
>>>>>>>
>>>>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>  It's probably looping/crashing mailscanner. Drop MS into debug
>>>>>>>> mode and watch logs.
>>>>>>>>
>>>>>>>>  --
>>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>>>>>
>>>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>>>> Web - http://www.fluxlabs.net
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>   An update to this, the "2 or 4" duplicates showing in the exim
>>>>>>>> log look like they are actually just separate deliveries to other
>>>>>>>> addresses, so not duplicates. In 1 example there is a single email with 2
>>>>>>>> recipients (2 entries in exim log) that has over 1500+ entries in the
>>>>>>>> mailcleaner DB. It looks like this email hasn't been delivered to the
>>>>>>>> recipient at all either.
>>>>>>>>
>>>>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>>  If anyone could provide advice that would be great. Running
>>>>>>>>> Debian Wheezy Mailscanner 4.79.11-2.2
>>>>>>>>>
>>>>>>>>>  Our incoming dir filled up just before the weekend so we didn't
>>>>>>>>> see the issue for a couple of days. Normally we would just shut down
>>>>>>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>>>>>>> However on this occasion, the root partition also become full because of
>>>>>>>>> the mysql DB (it got to 14G in 2 days..).
>>>>>>>>>
>>>>>>>>>  For some reason everything started duplicating. I can see lots
>>>>>>>>> of incoming messages in the exim logs with duplication (2 or 4 of what
>>>>>>>>> looks like the same email) but in the mailscanner database there is
>>>>>>>>> hundreds of each email listed (apparently there was over 9 million messages
>>>>>>>>> delivered on 1 day compared with the server average of about 1500!)
>>>>>>>>>
>>>>>>>>>  It seems like some sort of loop, but afaik nothing specific was
>>>>>>>>> changed in the config apart from the fact incoming became full. Space has
>>>>>>>>> been cleared on the root partition and incoming, and everything appears to
>>>>>>>>> be running as normal right now.
>>>>>>>>>
>>>>>>>>>  Any advice on debugging this would be much appreciated, also,
>>>>>>>>> how best should I clear out the DB of all the dupes?
>>>>>>>>>
>>>>>>>>>  Thanks!
>>>>>>>>>
>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/d3100932/attachment.html>

From jeremy at fluxlabs.net  Wed Jul 29 14:12:29 2015
From: jeremy at fluxlabs.net (Jeremy McSpadden)
Date: Wed, 29 Jul 2015 14:12:29 +0000
Subject: Duplicated messages
In-Reply-To: <CABF=4+z8LopwFLOzBmE=Md8r+L0SVs+m+cRmKxaO7V0_OmiWwQ@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
 <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
 <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>
 <CABF=4+w8EEbJM1d2=rEXCwEoVn8hhdn+MJW7ypdT-PnpE8Gt6g@mail.gmail.com>
 <11C72614-623D-4E8C-931F-484CDC8A2BE7@fluxlabs.net>
 <CABF=4+ySoMqGpu6CfhfHmVUhKEcsyZx3jsyuFmib5D6i5h9wWg@mail.gmail.com>
 <4C8BEE26-3CA7-450A-BD4E-226C0A461202@fluxlabs.net>,
 <CABF=4+z8LopwFLOzBmE=Md8r+L0SVs+m+cRmKxaO7V0_OmiWwQ@mail.gmail.com>
Message-ID: <288C4909-64BE-428B-BC76-B095BF2E63A7@fluxlabs.net>

How small is that drive ? Pretty bad planning on someone's part .. Then to be reliant on a single MX for mail flow.

Either way .. Dump all entries from that day. Did I read it grew 10gb in a day ?

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 9:08 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

It's not 1000 entries, its about that many (some id's with a huge amount more) per message ID. The Mailscanner DB has over 9,000,000 entries for that day. I was planning on looping through and deleting all the offending entries for the 300+ emails that seemed to get stuck.

The mailscanner DB is currently 14G (maillog.MYD)

On 29 July 2015 at 15:04, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
1000 entries from a sql DB may be about 5mb of space.
I would do a little more digging on the / partition to see what's eating space. Probably better off dropping some older archive dir

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 9:00 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

I agree with you completely, however that doesn't help my immediate situation. Can you provide advice on deleting from the mailscanner DB? is there any other tables I need to remove the offending ID's entries from?

Regards,
Mark

On 29 July 2015 at 14:34, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
It's 2015 .. That shouldn't be an excuse. It's like 10 cents per 100gb of drive... Upgrade

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 8:27 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi Jeremy,

No I haven't yet - I'm short on space on my root partition still because of the large mysql DB so I want to clean that up first. Can you advise how best to do this? Is it safe enough to do delete from maillog where id='XXX';  for all the ID's with the dupes? is there any other tables that need to be cleared?

Regards,
Mark

On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
Log won't show taint issues. Setup log rotation.

Have you enabled debug in mailscanner config like I stated yesterday ?

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi all,

So I have resolved getting the missing mails delivered from the quarantine. The main problem stopping this from being easy from the command line was the fact that "Quarantine Whole Messages As Queue Files = no" was set, whilst the MTA in use is exim. I've changed that setting to "yes" now...

I've read that if its postfix you can just send that "message" file back to the queue, I guess the headers are kept with the message in the quarantine with postfix. With exim they seem to be split between the database and the message file.

I first put the message ID's in to a file "idlist.txt" that had been put in to the quarantine with the "Other Bad Content Detected" error (every single email after a certain time on that day), then pulled the header from the db and combined them with the following simple loop;

-------
#!/bin/bash
for msgid in `cat idlist.txt`;
do
    /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *" >> with-headers/$msgid &&
    /bin/cat 20150724/$msgid/message >> with-headers/$msgid
done
-------

now I'm sending them out slowly (every 30 secs) with another simple loop...

-------
#!/bin/bash
for msgs in with-headers/*;
do
    cat $msgs | exim -ti
    mv $msgs with-headers-processed/
    sleep 30
done
-------

So at least the missing mail is now going to users.. but I'm no closer to knowing exactly why this happened in the first place. Jeremy mentioned a known "taint" issue? Can anyone elaborate on that?

I've also found now that Archive is enabled, and is set to "Archive Mail = " which I guess just defaults to the quarantine dir, as they seem to go to the "nonspam" folder in there (interestingly in a exim usable format!!) That couldn't have anything to do with the loop that appears to have killed my mailcleaner DB? I wouldn't think so as this has been running for years like this and not had this issue before but thought it worth mentioning.

Any other theories or places to check for clues? unfortunately the mail.log of the day got removed by the first person looking at the issue to try to free up space as it was over 4GB.

Regards,
Mark






On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:
Hi Jerry,

If you wanted to pull a bunch of items from the quarantine from the command line and re-process them through Mailcleaner, how would you do that?

Regards,
Mark

On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com<mailto:jerry.benton at mailborder.com>> wrote:
I am not sure on what parameters Mailwatch calls and logs “other bad content”.  The MailScanner setting is "Notify Senders of Other Blocked Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked content” for all we know. You are going to have to follow the below suggestion and enable debug or see if you can get an idea from /var/log/maillog.

-
Jerry Benton
www.mailborder.com<http://www.mailborder.com>



On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Of course, apologies - I'm using Mailwatch. Any advice on how to most efficiently pull things out of quarantine via command-line? (note they are stored as "message" rather than queue items, that would be too easy..)

I don't have Archive enabled, everything has gone in to the quarantine because of this "Other Bad Content Detected"

On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com<mailto:jerry.benton at mailborder.com>> wrote:
By the way, there is no web interface in the MailScanner package. There are 3rd party products of course (I created one myself) but those questions would need to be directed to those support forums or mailing lists.

-
Jerry Benton
www.mailborder.com<http://www.mailborder.com/>



On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?

"Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"

On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi Jeremy,

Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...

On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.

On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:
Hi All,

If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2

Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).

For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)

It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.

Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?

Thanks!


--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/listinfo/mailscanner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/9e0ca257/attachment-0001.html>

From mark at workshopit.co.uk  Wed Jul 29 14:15:43 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Wed, 29 Jul 2015 15:15:43 +0100
Subject: Duplicated messages
In-Reply-To: <288C4909-64BE-428B-BC76-B095BF2E63A7@fluxlabs.net>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
 <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
 <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>
 <CABF=4+w8EEbJM1d2=rEXCwEoVn8hhdn+MJW7ypdT-PnpE8Gt6g@mail.gmail.com>
 <11C72614-623D-4E8C-931F-484CDC8A2BE7@fluxlabs.net>
 <CABF=4+ySoMqGpu6CfhfHmVUhKEcsyZx3jsyuFmib5D6i5h9wWg@mail.gmail.com>
 <4C8BEE26-3CA7-450A-BD4E-226C0A461202@fluxlabs.net>
 <CABF=4+z8LopwFLOzBmE=Md8r+L0SVs+m+cRmKxaO7V0_OmiWwQ@mail.gmail.com>
 <288C4909-64BE-428B-BC76-B095BF2E63A7@fluxlabs.net>
Message-ID: <CABF=4+ze+FC7JCpofK5sGMQ2QoBE6pSbEtWtNNiOkoE6mYSu2A@mail.gmail.com>

Very small, and I can't argue with you about that....

Yes it grew pretty much the whole 14G in one day. Ideally I'd like to keep
the stuff from before the problem occurred on that day, is it not ok to
delete from maillog where id="XX" ?

On 29 July 2015 at 15:12, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:

>  How small is that drive ? Pretty bad planning on someone's part .. Then
> to be reliant on a single MX for mail flow.
>
>  Either way .. Dump all entries from that day. Did I read it grew 10gb in
> a day ?
>
>  --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
> Web - http://www.fluxlabs.net
>
>
> On Jul 29, 2015, at 9:08 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
>   It's not 1000 entries, its about that many (some id's with a huge
> amount more) per message ID. The Mailscanner DB has over 9,000,000 entries
> for that day. I was planning on looping through and deleting all the
> offending entries for the 300+ emails that seemed to get stuck.
>
>  The mailscanner DB is currently 14G (maillog.MYD)
>
> On 29 July 2015 at 15:04, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>
>>  1000 entries from a sql DB may be about 5mb of space.
>> I would do a little more digging on the / partition to see what's eating
>> space. Probably better off dropping some older archive dir
>>
>>  --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>> Web - http://www.fluxlabs.net
>>
>>
>> On Jul 29, 2015, at 9:00 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>>   I agree with you completely, however that doesn't help my immediate
>> situation. Can you provide advice on deleting from the mailscanner DB? is
>> there any other tables I need to remove the offending ID's entries from?
>>
>> Regards,
>> Mark
>>
>> On 29 July 2015 at 14:34, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>
>>>  It's 2015 .. That shouldn't be an excuse. It's like 10 cents per 100gb
>>> of drive... Upgrade
>>>
>>>  --
>>> Jeremy McSpadden | Flux Labs
>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>> Web - http://www.fluxlabs.net
>>>
>>>
>>> On Jul 29, 2015, at 8:27 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>
>>>   Hi Jeremy,
>>>
>>>  No I haven't yet - I'm short on space on my root partition still
>>> because of the large mysql DB so I want to clean that up first. Can you
>>> advise how best to do this? Is it safe enough to do delete from maillog
>>> where id='XXX';  for all the ID's with the dupes? is there any other tables
>>> that need to be cleared?
>>>
>>>  Regards,
>>> Mark
>>>
>>> On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>>
>>>>  Log won't show taint issues. Setup log rotation.
>>>>
>>>>  Have you enabled debug in mailscanner config like I stated yesterday
>>>> ?
>>>>
>>>>  --
>>>> Jeremy McSpadden | Flux Labs
>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>> Web - http://www.fluxlabs.net
>>>>
>>>>
>>>> On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>
>>>>   Hi all,
>>>>
>>>>  So I have resolved getting the missing mails delivered from the
>>>> quarantine. The main problem stopping this from being easy from the command
>>>> line was the fact that "Quarantine Whole Messages As Queue Files = no" was
>>>> set, whilst the MTA in use is exim. I've changed that setting to "yes"
>>>> now...
>>>>
>>>>  I've read that if its postfix you can just send that "message" file
>>>> back to the queue, I guess the headers are kept with the message in the
>>>> quarantine with postfix. With exim they seem to be split between the
>>>> database and the message file.
>>>>
>>>>  I first put the message ID's in to a file "idlist.txt" that had been
>>>> put in to the quarantine with the "Other Bad Content Detected" error (every
>>>> single email after a certain time on that day), then pulled the header from
>>>> the db and combined them with the following simple loop;
>>>>
>>>>  -------
>>>> #!/bin/bash
>>>>  for msgid in `cat idlist.txt`;
>>>> do
>>>>     /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from
>>>> maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *"
>>>> >> with-headers/$msgid &&
>>>>     /bin/cat 20150724/$msgid/message >> with-headers/$msgid
>>>> done
>>>>  -------
>>>>
>>>>  now I'm sending them out slowly (every 30 secs) with another simple
>>>> loop...
>>>>
>>>>  -------
>>>>  #!/bin/bash
>>>> for msgs in with-headers/*;
>>>> do
>>>>     cat $msgs | exim -ti
>>>>     mv $msgs with-headers-processed/
>>>>     sleep 30
>>>> done
>>>>  -------
>>>>
>>>>  So at least the missing mail is now going to users.. but I'm no
>>>> closer to knowing exactly why this happened in the first place. Jeremy
>>>> mentioned a known "taint" issue? Can anyone elaborate on that?
>>>>
>>>>  I've also found now that Archive is enabled, and is set to "Archive
>>>> Mail = " which I guess just defaults to the quarantine dir, as they seem to
>>>> go to the "nonspam" folder in there (interestingly in a exim usable
>>>> format!!) That couldn't have anything to do with the loop that appears to
>>>> have killed my mailcleaner DB? I wouldn't think so as this has been running
>>>> for years like this and not had this issue before but thought it worth
>>>> mentioning.
>>>>
>>>>  Any other theories or places to check for clues? unfortunately the
>>>> mail.log of the day got removed by the first person looking at the issue to
>>>> try to free up space as it was over 4GB.
>>>>
>>>>  Regards,
>>>> Mark
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>
>>>>> Hi Jerry,
>>>>>
>>>>>  If you wanted to pull a bunch of items from the quarantine from the
>>>>> command line and re-process them through Mailcleaner, how would you do that?
>>>>>
>>>>>  Regards,
>>>>> Mark
>>>>>
>>>>> On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com>
>>>>> wrote:
>>>>>
>>>>>> I am not sure on what parameters Mailwatch calls and logs “other bad
>>>>>> content”.  The MailScanner setting is "Notify Senders of Other Blocked
>>>>>> Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked
>>>>>> content” for all we know. You are going to have to follow the below
>>>>>> suggestion and enable debug or see if you can get an idea from
>>>>>> /var/log/maillog.
>>>>>>
>>>>>> -
>>>>>>  Jerry Benton
>>>>>>  www.mailborder.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>   On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>> wrote:
>>>>>>
>>>>>>  Of course, apologies - I'm using Mailwatch. Any advice on how to
>>>>>> most efficiently pull things out of quarantine via command-line? (note they
>>>>>> are stored as "message" rather than queue items, that would be too easy..)
>>>>>>
>>>>>>  I don't have Archive enabled, everything has gone in to the
>>>>>> quarantine because of this "Other Bad Content Detected"
>>>>>>
>>>>>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com>
>>>>>> wrote:
>>>>>>
>>>>>>> By the way, there is no web interface in the MailScanner package.
>>>>>>> There are 3rd party products of course (I created one myself) but those
>>>>>>> questions would need to be directed to those support forums or mailing
>>>>>>> lists.
>>>>>>>
>>>>>>> -
>>>>>>>  Jerry Benton
>>>>>>>  www.mailborder.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>   On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>> wrote:
>>>>>>>
>>>>>>>  How do I try send them through again? At the moment they are just
>>>>>>> "message" in the quarantine, and if I try open them through the web
>>>>>>> interface it times out, I guess because its trying to open each one of the
>>>>>>> dupes?
>>>>>>>
>>>>>>>  "Fatal error: Maximum execution time of 30 seconds exceeded in
>>>>>>> /var/www/html/mailscanner/functions.php on line 1022"
>>>>>>>
>>>>>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>  Yup. Turn on debug and watch it pass through. Last time I saw
>>>>>>>> these it was a taint issue .. Which I am assuming has been fixed by now.
>>>>>>>>
>>>>>>>>  --
>>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>>>>>
>>>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>>>> Web - http://www.fluxlabs.net
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>   Hi Jeremy,
>>>>>>>>
>>>>>>>>  Are you saying that something in these messages is crashing
>>>>>>>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>>>>>>>> (all different types and from different servers) are now in the quarantine
>>>>>>>> because of "Other Bad Content Detected" with the report "MailScanner:
>>>>>>>> Message attempted to kill MailScanner". It seems it succeeded...
>>>>>>>>
>>>>>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>  It's probably looping/crashing mailscanner. Drop MS into debug
>>>>>>>>> mode and watch logs.
>>>>>>>>>
>>>>>>>>>  --
>>>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile -
>>>>>>>>> 850-890-2543
>>>>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>>>>> Web - http://www.fluxlabs.net
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>   An update to this, the "2 or 4" duplicates showing in the exim
>>>>>>>>> log look like they are actually just separate deliveries to other
>>>>>>>>> addresses, so not duplicates. In 1 example there is a single email with 2
>>>>>>>>> recipients (2 entries in exim log) that has over 1500+ entries in the
>>>>>>>>> mailcleaner DB. It looks like this email hasn't been delivered to the
>>>>>>>>> recipient at all either.
>>>>>>>>>
>>>>>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>>  If anyone could provide advice that would be great. Running
>>>>>>>>>> Debian Wheezy Mailscanner 4.79.11-2.2
>>>>>>>>>>
>>>>>>>>>>  Our incoming dir filled up just before the weekend so we didn't
>>>>>>>>>> see the issue for a couple of days. Normally we would just shut down
>>>>>>>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>>>>>>>> However on this occasion, the root partition also become full because of
>>>>>>>>>> the mysql DB (it got to 14G in 2 days..).
>>>>>>>>>>
>>>>>>>>>>  For some reason everything started duplicating. I can see lots
>>>>>>>>>> of incoming messages in the exim logs with duplication (2 or 4 of what
>>>>>>>>>> looks like the same email) but in the mailscanner database there is
>>>>>>>>>> hundreds of each email listed (apparently there was over 9 million messages
>>>>>>>>>> delivered on 1 day compared with the server average of about 1500!)
>>>>>>>>>>
>>>>>>>>>>  It seems like some sort of loop, but afaik nothing specific was
>>>>>>>>>> changed in the config apart from the fact incoming became full. Space has
>>>>>>>>>> been cleared on the root partition and incoming, and everything appears to
>>>>>>>>>> be running as normal right now.
>>>>>>>>>>
>>>>>>>>>>  Any advice on debugging this would be much appreciated, also,
>>>>>>>>>> how best should I clear out the DB of all the dupes?
>>>>>>>>>>
>>>>>>>>>>  Thanks!
>>>>>>>>>>
>>>>>>>>>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
>
>
>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
>
>
>


-- 
Mark Adams
*Workshop IT:*

5 Cowcross Street
London EC1M 6DW
020 7183 0498
www.workshopit.co.uk
Registered in England and Wales: 8366747
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/d01164e9/attachment.html>

From jeremy at fluxlabs.net  Wed Jul 29 14:17:58 2015
From: jeremy at fluxlabs.net (Jeremy McSpadden)
Date: Wed, 29 Jul 2015 14:17:58 +0000
Subject: Duplicated messages
In-Reply-To: <CABF=4+ze+FC7JCpofK5sGMQ2QoBE6pSbEtWtNNiOkoE6mYSu2A@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
 <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
 <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>
 <CABF=4+w8EEbJM1d2=rEXCwEoVn8hhdn+MJW7ypdT-PnpE8Gt6g@mail.gmail.com>
 <11C72614-623D-4E8C-931F-484CDC8A2BE7@fluxlabs.net>
 <CABF=4+ySoMqGpu6CfhfHmVUhKEcsyZx3jsyuFmib5D6i5h9wWg@mail.gmail.com>
 <4C8BEE26-3CA7-450A-BD4E-226C0A461202@fluxlabs.net>
 <CABF=4+z8LopwFLOzBmE=Md8r+L0SVs+m+cRmKxaO7V0_OmiWwQ@mail.gmail.com>
 <288C4909-64BE-428B-BC76-B095BF2E63A7@fluxlabs.net>,
 <CABF=4+ze+FC7JCpofK5sGMQ2QoBE6pSbEtWtNNiOkoE6mYSu2A@mail.gmail.com>
Message-ID: <0721F5F8-9A09-4D55-9683-ECC9A384B5C7@fluxlabs.net>

Sure. But your looping through 9 million entries. Your better off to drop where date=

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 9:15 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Very small, and I can't argue with you about that....

Yes it grew pretty much the whole 14G in one day. Ideally I'd like to keep the stuff from before the problem occurred on that day, is it not ok to delete from maillog where id="XX" ?

On 29 July 2015 at 15:12, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
How small is that drive ? Pretty bad planning on someone's part .. Then to be reliant on a single MX for mail flow.

Either way .. Dump all entries from that day. Did I read it grew 10gb in a day ?

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 9:08 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

It's not 1000 entries, its about that many (some id's with a huge amount more) per message ID. The Mailscanner DB has over 9,000,000 entries for that day. I was planning on looping through and deleting all the offending entries for the 300+ emails that seemed to get stuck.

The mailscanner DB is currently 14G (maillog.MYD)

On 29 July 2015 at 15:04, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
1000 entries from a sql DB may be about 5mb of space.
I would do a little more digging on the / partition to see what's eating space. Probably better off dropping some older archive dir

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 9:00 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

I agree with you completely, however that doesn't help my immediate situation. Can you provide advice on deleting from the mailscanner DB? is there any other tables I need to remove the offending ID's entries from?

Regards,
Mark

On 29 July 2015 at 14:34, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
It's 2015 .. That shouldn't be an excuse. It's like 10 cents per 100gb of drive... Upgrade

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 8:27 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi Jeremy,

No I haven't yet - I'm short on space on my root partition still because of the large mysql DB so I want to clean that up first. Can you advise how best to do this? Is it safe enough to do delete from maillog where id='XXX';  for all the ID's with the dupes? is there any other tables that need to be cleared?

Regards,
Mark

On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
Log won't show taint issues. Setup log rotation.

Have you enabled debug in mailscanner config like I stated yesterday ?

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi all,

So I have resolved getting the missing mails delivered from the quarantine. The main problem stopping this from being easy from the command line was the fact that "Quarantine Whole Messages As Queue Files = no" was set, whilst the MTA in use is exim. I've changed that setting to "yes" now...

I've read that if its postfix you can just send that "message" file back to the queue, I guess the headers are kept with the message in the quarantine with postfix. With exim they seem to be split between the database and the message file.

I first put the message ID's in to a file "idlist.txt" that had been put in to the quarantine with the "Other Bad Content Detected" error (every single email after a certain time on that day), then pulled the header from the db and combined them with the following simple loop;

-------
#!/bin/bash
for msgid in `cat idlist.txt`;
do
    /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *" >> with-headers/$msgid &&
    /bin/cat 20150724/$msgid/message >> with-headers/$msgid
done
-------

now I'm sending them out slowly (every 30 secs) with another simple loop...

-------
#!/bin/bash
for msgs in with-headers/*;
do
    cat $msgs | exim -ti
    mv $msgs with-headers-processed/
    sleep 30
done
-------

So at least the missing mail is now going to users.. but I'm no closer to knowing exactly why this happened in the first place. Jeremy mentioned a known "taint" issue? Can anyone elaborate on that?

I've also found now that Archive is enabled, and is set to "Archive Mail = " which I guess just defaults to the quarantine dir, as they seem to go to the "nonspam" folder in there (interestingly in a exim usable format!!) That couldn't have anything to do with the loop that appears to have killed my mailcleaner DB? I wouldn't think so as this has been running for years like this and not had this issue before but thought it worth mentioning.

Any other theories or places to check for clues? unfortunately the mail.log of the day got removed by the first person looking at the issue to try to free up space as it was over 4GB.

Regards,
Mark






On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:
Hi Jerry,

If you wanted to pull a bunch of items from the quarantine from the command line and re-process them through Mailcleaner, how would you do that?

Regards,
Mark

On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com<mailto:jerry.benton at mailborder.com>> wrote:
I am not sure on what parameters Mailwatch calls and logs “other bad content”.  The MailScanner setting is "Notify Senders of Other Blocked Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked content” for all we know. You are going to have to follow the below suggestion and enable debug or see if you can get an idea from /var/log/maillog.

-
Jerry Benton
www.mailborder.com<http://www.mailborder.com>



On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Of course, apologies - I'm using Mailwatch. Any advice on how to most efficiently pull things out of quarantine via command-line? (note they are stored as "message" rather than queue items, that would be too easy..)

I don't have Archive enabled, everything has gone in to the quarantine because of this "Other Bad Content Detected"

On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com<mailto:jerry.benton at mailborder.com>> wrote:
By the way, there is no web interface in the MailScanner package. There are 3rd party products of course (I created one myself) but those questions would need to be directed to those support forums or mailing lists.

-
Jerry Benton
www.mailborder.com<http://www.mailborder.com/>



On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?

"Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"

On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

Hi Jeremy,

Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...

On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net<mailto:jeremy at fluxlabs.net>> wrote:
It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs.

--
Jeremy McSpadden | Flux Labs
Local - 850-250-5590x501<tel:850-250-5590;501> | Mobile - 850-890-2543<tel:850-890-2543>
Fax - 850-254-2955<tel:850-254-2955> | Toll Free - 877-699-FLUX<tel:877-699-FLUX>
Web - http://www.fluxlabs.net<http://www.fluxlabs.net/>


On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:

An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.

On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk<mailto:mark at workshopit.co.uk>> wrote:
Hi All,

If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2

Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).

For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)

It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.

Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?

Thanks!


--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/listinfo/mailscanner




--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/listinfo/mailscanner





--
Mark Adams
Workshop IT:

5 Cowcross Street
London EC1M 6DW
020 7183 0498
www.workshopit.co.uk<http://www.workshopit.co.uk/>
Registered in England and Wales: 8366747


--
MailScanner mailing list
mailscanner at lists.mailscanner.info<mailto:mailscanner at lists.mailscanner.info>
http://lists.mailscanner.info/listinfo/mailscanner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/74b35fc7/attachment.html>

From mark at workshopit.co.uk  Wed Jul 29 14:36:03 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Wed, 29 Jul 2015 15:36:03 +0100
Subject: Duplicated messages
In-Reply-To: <0721F5F8-9A09-4D55-9683-ECC9A384B5C7@fluxlabs.net>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
 <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
 <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>
 <CABF=4+w8EEbJM1d2=rEXCwEoVn8hhdn+MJW7ypdT-PnpE8Gt6g@mail.gmail.com>
 <11C72614-623D-4E8C-931F-484CDC8A2BE7@fluxlabs.net>
 <CABF=4+ySoMqGpu6CfhfHmVUhKEcsyZx3jsyuFmib5D6i5h9wWg@mail.gmail.com>
 <4C8BEE26-3CA7-450A-BD4E-226C0A461202@fluxlabs.net>
 <CABF=4+z8LopwFLOzBmE=Md8r+L0SVs+m+cRmKxaO7V0_OmiWwQ@mail.gmail.com>
 <288C4909-64BE-428B-BC76-B095BF2E63A7@fluxlabs.net>
 <CABF=4+ze+FC7JCpofK5sGMQ2QoBE6pSbEtWtNNiOkoE6mYSu2A@mail.gmail.com>
 <0721F5F8-9A09-4D55-9683-ECC9A384B5C7@fluxlabs.net>
Message-ID: <CABF=4+zz+AjP3B9j=tx70+EXALO1Bo6sso4Vt3LdHFGTBdEnpg@mail.gmail.com>

Good point, I'll just keep a dump of the DB elsewhere in case I need to
look for something on that day in the future. So just "delete from maillog
where date='2015-07-24';" is the best way to clean this out?

Regards,
Mark

On 29 July 2015 at 15:17, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:

>  Sure. But your looping through 9 million entries. Your better off to
> drop where date=
>
>  --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
> Web - http://www.fluxlabs.net
>
>
> On Jul 29, 2015, at 9:15 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
>   Very small, and I can't argue with you about that....
>
>  Yes it grew pretty much the whole 14G in one day. Ideally I'd like to
> keep the stuff from before the problem occurred on that day, is it not ok
> to delete from maillog where id="XX" ?
>
> On 29 July 2015 at 15:12, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>
>>  How small is that drive ? Pretty bad planning on someone's part .. Then
>> to be reliant on a single MX for mail flow.
>>
>>  Either way .. Dump all entries from that day. Did I read it grew 10gb
>> in a day ?
>>
>>  --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>> Web - http://www.fluxlabs.net
>>
>>
>> On Jul 29, 2015, at 9:08 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>>   It's not 1000 entries, its about that many (some id's with a huge
>> amount more) per message ID. The Mailscanner DB has over 9,000,000 entries
>> for that day. I was planning on looping through and deleting all the
>> offending entries for the 300+ emails that seemed to get stuck.
>>
>>  The mailscanner DB is currently 14G (maillog.MYD)
>>
>> On 29 July 2015 at 15:04, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>
>>>  1000 entries from a sql DB may be about 5mb of space.
>>> I would do a little more digging on the / partition to see what's eating
>>> space. Probably better off dropping some older archive dir
>>>
>>>  --
>>> Jeremy McSpadden | Flux Labs
>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>> Web - http://www.fluxlabs.net
>>>
>>>
>>> On Jul 29, 2015, at 9:00 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>
>>>   I agree with you completely, however that doesn't help my immediate
>>> situation. Can you provide advice on deleting from the mailscanner DB? is
>>> there any other tables I need to remove the offending ID's entries from?
>>>
>>> Regards,
>>> Mark
>>>
>>> On 29 July 2015 at 14:34, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>>
>>>>  It's 2015 .. That shouldn't be an excuse. It's like 10 cents per
>>>> 100gb of drive... Upgrade
>>>>
>>>>  --
>>>> Jeremy McSpadden | Flux Labs
>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>> Web - http://www.fluxlabs.net
>>>>
>>>>
>>>> On Jul 29, 2015, at 8:27 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>
>>>>   Hi Jeremy,
>>>>
>>>>  No I haven't yet - I'm short on space on my root partition still
>>>> because of the large mysql DB so I want to clean that up first. Can you
>>>> advise how best to do this? Is it safe enough to do delete from maillog
>>>> where id='XXX';  for all the ID's with the dupes? is there any other tables
>>>> that need to be cleared?
>>>>
>>>>  Regards,
>>>> Mark
>>>>
>>>> On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>>>
>>>>>  Log won't show taint issues. Setup log rotation.
>>>>>
>>>>>  Have you enabled debug in mailscanner config like I stated yesterday
>>>>> ?
>>>>>
>>>>>  --
>>>>> Jeremy McSpadden | Flux Labs
>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>> Web - http://www.fluxlabs.net
>>>>>
>>>>>
>>>>> On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>
>>>>>   Hi all,
>>>>>
>>>>>  So I have resolved getting the missing mails delivered from the
>>>>> quarantine. The main problem stopping this from being easy from the command
>>>>> line was the fact that "Quarantine Whole Messages As Queue Files = no" was
>>>>> set, whilst the MTA in use is exim. I've changed that setting to "yes"
>>>>> now...
>>>>>
>>>>>  I've read that if its postfix you can just send that "message" file
>>>>> back to the queue, I guess the headers are kept with the message in the
>>>>> quarantine with postfix. With exim they seem to be split between the
>>>>> database and the message file.
>>>>>
>>>>>  I first put the message ID's in to a file "idlist.txt" that had been
>>>>> put in to the quarantine with the "Other Bad Content Detected" error (every
>>>>> single email after a certain time on that day), then pulled the header from
>>>>> the db and combined them with the following simple loop;
>>>>>
>>>>>  -------
>>>>> #!/bin/bash
>>>>>  for msgid in `cat idlist.txt`;
>>>>> do
>>>>>     /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from
>>>>> maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *"
>>>>> >> with-headers/$msgid &&
>>>>>     /bin/cat 20150724/$msgid/message >> with-headers/$msgid
>>>>> done
>>>>>  -------
>>>>>
>>>>>  now I'm sending them out slowly (every 30 secs) with another simple
>>>>> loop...
>>>>>
>>>>>  -------
>>>>>  #!/bin/bash
>>>>> for msgs in with-headers/*;
>>>>> do
>>>>>     cat $msgs | exim -ti
>>>>>     mv $msgs with-headers-processed/
>>>>>     sleep 30
>>>>> done
>>>>>  -------
>>>>>
>>>>>  So at least the missing mail is now going to users.. but I'm no
>>>>> closer to knowing exactly why this happened in the first place. Jeremy
>>>>> mentioned a known "taint" issue? Can anyone elaborate on that?
>>>>>
>>>>>  I've also found now that Archive is enabled, and is set to "Archive
>>>>> Mail = " which I guess just defaults to the quarantine dir, as they seem to
>>>>> go to the "nonspam" folder in there (interestingly in a exim usable
>>>>> format!!) That couldn't have anything to do with the loop that appears to
>>>>> have killed my mailcleaner DB? I wouldn't think so as this has been running
>>>>> for years like this and not had this issue before but thought it worth
>>>>> mentioning.
>>>>>
>>>>>  Any other theories or places to check for clues? unfortunately the
>>>>> mail.log of the day got removed by the first person looking at the issue to
>>>>> try to free up space as it was over 4GB.
>>>>>
>>>>>  Regards,
>>>>> Mark
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>
>>>>>> Hi Jerry,
>>>>>>
>>>>>>  If you wanted to pull a bunch of items from the quarantine from the
>>>>>> command line and re-process them through Mailcleaner, how would you do that?
>>>>>>
>>>>>>  Regards,
>>>>>> Mark
>>>>>>
>>>>>> On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I am not sure on what parameters Mailwatch calls and logs “other bad
>>>>>>> content”.  The MailScanner setting is "Notify Senders of Other Blocked
>>>>>>> Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked
>>>>>>> content” for all we know. You are going to have to follow the below
>>>>>>> suggestion and enable debug or see if you can get an idea from
>>>>>>> /var/log/maillog.
>>>>>>>
>>>>>>> -
>>>>>>>  Jerry Benton
>>>>>>>  www.mailborder.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>   On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>> wrote:
>>>>>>>
>>>>>>>  Of course, apologies - I'm using Mailwatch. Any advice on how to
>>>>>>> most efficiently pull things out of quarantine via command-line? (note they
>>>>>>> are stored as "message" rather than queue items, that would be too easy..)
>>>>>>>
>>>>>>>  I don't have Archive enabled, everything has gone in to the
>>>>>>> quarantine because of this "Other Bad Content Detected"
>>>>>>>
>>>>>>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> By the way, there is no web interface in the MailScanner package.
>>>>>>>> There are 3rd party products of course (I created one myself) but those
>>>>>>>> questions would need to be directed to those support forums or mailing
>>>>>>>> lists.
>>>>>>>>
>>>>>>>> -
>>>>>>>>  Jerry Benton
>>>>>>>>  www.mailborder.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>   On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>  How do I try send them through again? At the moment they are just
>>>>>>>> "message" in the quarantine, and if I try open them through the web
>>>>>>>> interface it times out, I guess because its trying to open each one of the
>>>>>>>> dupes?
>>>>>>>>
>>>>>>>>  "Fatal error: Maximum execution time of 30 seconds exceeded in
>>>>>>>> /var/www/html/mailscanner/functions.php on line 1022"
>>>>>>>>
>>>>>>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>  Yup. Turn on debug and watch it pass through. Last time I saw
>>>>>>>>> these it was a taint issue .. Which I am assuming has been fixed by now.
>>>>>>>>>
>>>>>>>>>  --
>>>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile -
>>>>>>>>> 850-890-2543
>>>>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>>>>> Web - http://www.fluxlabs.net
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>   Hi Jeremy,
>>>>>>>>>
>>>>>>>>>  Are you saying that something in these messages is crashing
>>>>>>>>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>>>>>>>>> (all different types and from different servers) are now in the quarantine
>>>>>>>>> because of "Other Bad Content Detected" with the report "MailScanner:
>>>>>>>>> Message attempted to kill MailScanner". It seems it succeeded...
>>>>>>>>>
>>>>>>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>  It's probably looping/crashing mailscanner. Drop MS into debug
>>>>>>>>>> mode and watch logs.
>>>>>>>>>>
>>>>>>>>>>  --
>>>>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile -
>>>>>>>>>> 850-890-2543
>>>>>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>>>>>> Web - http://www.fluxlabs.net
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>   An update to this, the "2 or 4" duplicates showing in the exim
>>>>>>>>>> log look like they are actually just separate deliveries to other
>>>>>>>>>> addresses, so not duplicates. In 1 example there is a single email with 2
>>>>>>>>>> recipients (2 entries in exim log) that has over 1500+ entries in the
>>>>>>>>>> mailcleaner DB. It looks like this email hasn't been delivered to the
>>>>>>>>>> recipient at all either.
>>>>>>>>>>
>>>>>>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>>  If anyone could provide advice that would be great. Running
>>>>>>>>>>> Debian Wheezy Mailscanner 4.79.11-2.2
>>>>>>>>>>>
>>>>>>>>>>>  Our incoming dir filled up just before the weekend so we
>>>>>>>>>>> didn't see the issue for a couple of days. Normally we would just shut down
>>>>>>>>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>>>>>>>>> However on this occasion, the root partition also become full because of
>>>>>>>>>>> the mysql DB (it got to 14G in 2 days..).
>>>>>>>>>>>
>>>>>>>>>>>  For some reason everything started duplicating. I can see lots
>>>>>>>>>>> of incoming messages in the exim logs with duplication (2 or 4 of what
>>>>>>>>>>> looks like the same email) but in the mailscanner database there is
>>>>>>>>>>> hundreds of each email listed (apparently there was over 9 million messages
>>>>>>>>>>> delivered on 1 day compared with the server average of about 1500!)
>>>>>>>>>>>
>>>>>>>>>>>  It seems like some sort of loop, but afaik nothing specific
>>>>>>>>>>> was changed in the config apart from the fact incoming became full. Space
>>>>>>>>>>> has been cleared on the root partition and incoming, and everything appears
>>>>>>>>>>> to be running as normal right now.
>>>>>>>>>>>
>>>>>>>>>>>  Any advice on debugging this would be much appreciated, also,
>>>>>>>>>>> how best should I clear out the DB of all the dupes?
>>>>>>>>>>>
>>>>>>>>>>>  Thanks!
>>>>>>>>>>>
>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/7ec84142/attachment-0001.html>

From jerry.benton at mailborder.com  Wed Jul 29 18:05:42 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Wed, 29 Jul 2015 14:05:42 -0400
Subject: Duplicated messages
In-Reply-To: <CABF=4+zz+AjP3B9j=tx70+EXALO1Bo6sso4Vt3LdHFGTBdEnpg@mail.gmail.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
 <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
 <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>
 <CABF=4+w8EEbJM1d2=rEXCwEoVn8hhdn+MJW7ypdT-PnpE8Gt6g@mail.gmail.com>
 <11C72614-623D-4E8C-931F-484CDC8A2BE7@fluxlabs.net>
 <CABF=4+ySoMqGpu6CfhfHmVUhKEcsyZx3jsyuFmib5D6i5h9wWg@mail.gmail.com>
 <4C8BEE26-3CA7-450A-BD4E-226C0A461202@fluxlabs.net>
 <CABF=4+z8LopwFLOzBmE=Md8r+L0SVs+m+cRmKxaO7V0_OmiWwQ@mail.gmail.com>
 <288C4909-64BE-428B-BC76-B095BF2E63A7@fluxlabs.net>
 <CABF=4+ze+FC7JCpofK5sGMQ2QoBE6pSbEtWtNNiOkoE6mYSu2A@mail.gmail.com>
 <0721F5F8-9A09-4D55-9683-ECC9A384B5C7@fluxlabs.net>
 <CABF=4+zz+AjP3B9j=tx70+EXALO1Bo6sso4Vt3LdHFGTBdEnpg@mail.gmail.com>
Message-ID: <832142D3-F10F-47EF-826D-4499BEF8B680@mailborder.com>

Yes. You can also purge binary logs. 

-
Jerry Benton
www.mailborder.com



> On Jul 29, 2015, at 10:36 AM, Mark Adams <mark at workshopit.co.uk> wrote:
> 
> Good point, I'll just keep a dump of the DB elsewhere in case I need to look for something on that day in the future. So just "delete from maillog where date='2015-07-24';" is the best way to clean this out?
> 
> Regards,
> Mark
> 
> On 29 July 2015 at 15:17, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
> Sure. But your looping through 9 million entries. Your better off to drop where date=
> 
> --
> Jeremy McSpadden | Flux Labs
> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
> 
> 
> On Jul 29, 2015, at 9:15 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
> 
>> Very small, and I can't argue with you about that....
>> 
>> Yes it grew pretty much the whole 14G in one day. Ideally I'd like to keep the stuff from before the problem occurred on that day, is it not ok to delete from maillog where id="XX" ?
>> 
>> On 29 July 2015 at 15:12, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>> How small is that drive ? Pretty bad planning on someone's part .. Then to be reliant on a single MX for mail flow. 
>> 
>> Either way .. Dump all entries from that day. Did I read it grew 10gb in a day ? 
>> 
>> --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>> 
>> 
>> On Jul 29, 2015, at 9:08 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>> 
>>> It's not 1000 entries, its about that many (some id's with a huge amount more) per message ID. The Mailscanner DB has over 9,000,000 entries for that day. I was planning on looping through and deleting all the offending entries for the 300+ emails that seemed to get stuck.
>>> 
>>> The mailscanner DB is currently 14G (maillog.MYD)
>>> 
>>> On 29 July 2015 at 15:04, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>>> 1000 entries from a sql DB may be about 5mb of space. 
>>> I would do a little more digging on the / partition to see what's eating space. Probably better off dropping some older archive dir
>>> 
>>> --
>>> Jeremy McSpadden | Flux Labs
>>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>>> 
>>> 
>>> On Jul 29, 2015, at 9:00 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>> 
>>>> I agree with you completely, however that doesn't help my immediate situation. Can you provide advice on deleting from the mailscanner DB? is there any other tables I need to remove the offending ID's entries from?
>>>> 
>>>> Regards,
>>>> Mark
>>>> 
>>>> On 29 July 2015 at 14:34, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>>>> It's 2015 .. That shouldn't be an excuse. It's like 10 cents per 100gb of drive... Upgrade 
>>>> 
>>>> --
>>>> Jeremy McSpadden | Flux Labs
>>>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>>>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>>>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>>>> 
>>>> 
>>>> On Jul 29, 2015, at 8:27 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>> 
>>>>> Hi Jeremy,
>>>>> 
>>>>> No I haven't yet - I'm short on space on my root partition still because of the large mysql DB so I want to clean that up first. Can you advise how best to do this? Is it safe enough to do delete from maillog where id='XXX';  for all the ID's with the dupes? is there any other tables that need to be cleared?
>>>>> 
>>>>> Regards,
>>>>> Mark
>>>>> 
>>>>> On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>>>>> Log won't show taint issues. Setup log rotation. 
>>>>> 
>>>>> Have you enabled debug in mailscanner config like I stated yesterday ? 
>>>>> 
>>>>> --
>>>>> Jeremy McSpadden | Flux Labs
>>>>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>>>>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>>>>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>>>>> 
>>>>> 
>>>>> On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> So I have resolved getting the missing mails delivered from the quarantine. The main problem stopping this from being easy from the command line was the fact that "Quarantine Whole Messages As Queue Files = no" was set, whilst the MTA in use is exim. I've changed that setting to "yes" now...
>>>>>> 
>>>>>> I've read that if its postfix you can just send that "message" file back to the queue, I guess the headers are kept with the message in the quarantine with postfix. With exim they seem to be split between the database and the message file.
>>>>>> 
>>>>>> I first put the message ID's in to a file "idlist.txt" that had been put in to the quarantine with the "Other Bad Content Detected" error (every single email after a certain time on that day), then pulled the header from the db and combined them with the following simple loop;
>>>>>> 
>>>>>> -------
>>>>>> #!/bin/bash
>>>>>> for msgid in `cat idlist.txt`;
>>>>>> do
>>>>>>     /usr/bin/mysql -u root --password=XXXXX -N -e "select headers from maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row *" >> with-headers/$msgid &&
>>>>>>     /bin/cat 20150724/$msgid/message >> with-headers/$msgid
>>>>>> done
>>>>>> -------
>>>>>> 
>>>>>> now I'm sending them out slowly (every 30 secs) with another simple loop...
>>>>>> 
>>>>>> -------
>>>>>> #!/bin/bash
>>>>>> for msgs in with-headers/*;
>>>>>> do
>>>>>>     cat $msgs | exim -ti
>>>>>>     mv $msgs with-headers-processed/
>>>>>>     sleep 30
>>>>>> done
>>>>>> -------
>>>>>> 
>>>>>> So at least the missing mail is now going to users.. but I'm no closer to knowing exactly why this happened in the first place. Jeremy mentioned a known "taint" issue? Can anyone elaborate on that?
>>>>>> 
>>>>>> I've also found now that Archive is enabled, and is set to "Archive Mail = " which I guess just defaults to the quarantine dir, as they seem to go to the "nonspam" folder in there (interestingly in a exim usable format!!) That couldn't have anything to do with the loop that appears to have killed my mailcleaner DB? I wouldn't think so as this has been running for years like this and not had this issue before but thought it worth mentioning.
>>>>>> 
>>>>>> Any other theories or places to check for clues? unfortunately the mail.log of the day got removed by the first person looking at the issue to try to free up space as it was over 4GB.
>>>>>> 
>>>>>> Regards,
>>>>>> Mark
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>>>> Hi Jerry,
>>>>>> 
>>>>>> If you wanted to pull a bunch of items from the quarantine from the command line and re-process them through Mailcleaner, how would you do that?
>>>>>> 
>>>>>> Regards,
>>>>>> Mark
>>>>>> 
>>>>>> On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com <mailto:jerry.benton at mailborder.com>> wrote:
>>>>>> I am not sure on what parameters Mailwatch calls and logs “other bad content”.  The MailScanner setting is "Notify Senders of Other Blocked Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked content” for all we know. You are going to have to follow the below suggestion and enable debug or see if you can get an idea from /var/log/maillog.
>>>>>> 
>>>>>> -
>>>>>> Jerry Benton
>>>>>> www.mailborder.com <http://www.mailborder.com/>
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>>>>> 
>>>>>>> Of course, apologies - I'm using Mailwatch. Any advice on how to most efficiently pull things out of quarantine via command-line? (note they are stored as "message" rather than queue items, that would be too easy..) 
>>>>>>> 
>>>>>>> I don't have Archive enabled, everything has gone in to the quarantine because of this "Other Bad Content Detected"
>>>>>>> 
>>>>>>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com <mailto:jerry.benton at mailborder.com>> wrote:
>>>>>>> By the way, there is no web interface in the MailScanner package. There are 3rd party products of course (I created one myself) but those questions would need to be directed to those support forums or mailing lists.
>>>>>>> 
>>>>>>> -
>>>>>>> Jerry Benton
>>>>>>> www.mailborder.com <http://www.mailborder.com/>
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>>>>>> 
>>>>>>>> How do I try send them through again? At the moment they are just "message" in the quarantine, and if I try open them through the web interface it times out, I guess because its trying to open each one of the dupes?
>>>>>>>> 
>>>>>>>> "Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/mailscanner/functions.php on line 1022"
>>>>>>>> 
>>>>>>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>>>>>>>> Yup. Turn on debug and watch it pass through. Last time I saw these it was a taint issue .. Which I am assuming has been fixed by now. 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>>>>>>>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>>>>>>>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Jeremy,
>>>>>>>>> 
>>>>>>>>> Are you saying that something in these messages is crashing Mailscanner? Everything seems to be OK right now, but all 70 of the emails (all different types and from different servers) are now in the quarantine because of "Other Bad Content Detected" with the report "MailScanner: Message attempted to kill MailScanner". It seems it succeeded...
>>>>>>>>> 
>>>>>>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net <mailto:jeremy at fluxlabs.net>> wrote:
>>>>>>>>> It's probably looping/crashing mailscanner. Drop MS into debug mode and watch logs. 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>>>> Local - 850-250-5590x501 <tel:850-250-5590;501> | Mobile - 850-890-2543 <tel:850-890-2543> 
>>>>>>>>> Fax - 850-254-2955 <tel:850-254-2955> | Toll Free - 877-699-FLUX <tel:877-699-FLUX>
>>>>>>>>> Web - http://www.fluxlabs.net <http://www.fluxlabs.net/>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>>>>>>> 
>>>>>>>>>> An update to this, the "2 or 4" duplicates showing in the exim log look like they are actually just separate deliveries to other addresses, so not duplicates. In 1 example there is a single email with 2 recipients (2 entries in exim log) that has over 1500+ entries in the mailcleaner DB. It looks like this email hasn't been delivered to the recipient at all either.
>>>>>>>>>> 
>>>>>>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk <mailto:mark at workshopit.co.uk>> wrote:
>>>>>>>>>> Hi All,
>>>>>>>>>> 
>>>>>>>>>> If anyone could provide advice that would be great. Running Debian Wheezy Mailscanner 4.79.11-2.2
>>>>>>>>>> 
>>>>>>>>>> Our incoming dir filled up just before the weekend so we didn't see the issue for a couple of days. Normally we would just shut down mailcleaner and delete the dir then start it up again and all would be ok. However on this occasion, the root partition also become full because of the mysql DB (it got to 14G in 2 days..).
>>>>>>>>>> 
>>>>>>>>>> For some reason everything started duplicating. I can see lots of incoming messages in the exim logs with duplication (2 or 4 of what looks like the same email) but in the mailscanner database there is hundreds of each email listed (apparently there was over 9 million messages delivered on 1 day compared with the server average of about 1500!)
>>>>>>>>>> 
>>>>>>>>>> It seems like some sort of loop, but afaik nothing specific was changed in the config apart from the fact incoming became full. Space has been cleared on the root partition and incoming, and everything appears to be running as normal right now.
>>>>>>>>>> 
>>>>>>>>>> Any advice on debugging this would be much appreciated, also, how best should I clear out the DB of all the dupes?
>>>>>>>>>> 
>>>>>>>>>> Thanks!
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150729/32c72d41/attachment.html>

From mark at workshopit.co.uk  Thu Jul 30 01:50:27 2015
From: mark at workshopit.co.uk (Mark Adams)
Date: Thu, 30 Jul 2015 02:50:27 +0100
Subject: Duplicated messages
In-Reply-To: <832142D3-F10F-47EF-826D-4499BEF8B680@mailborder.com>
References: <CABF=4+wfMDFhyVHAshAfuz7E7PcuafgPZ-JkuExkOX+=CsMvnQ@mail.gmail.com>
 <CABF=4+xZPmHuhN8ha19P0M2uBqcGDRz=8Mj_2fisFFSGaqcLaQ@mail.gmail.com>
 <8C6C9141-001D-44D0-9ED9-FF0DE8F63DD9@fluxlabs.net>
 <CABF=4+zCTB4c1pyrPEA1bWL=ZLMP3VCh+N=ny6M1H=3DSDy6aw@mail.gmail.com>
 <B3D3B9C9-8D74-4F04-B8E2-DA70ADE33CE2@fluxlabs.net>
 <CABF=4+xE2ocf926eRAFbkTAQMeSZut00+=f5nXS5kGuKUcEZ_Q@mail.gmail.com>
 <A7C49679-C7D3-453C-94C4-64A15B925012@mailborder.com>
 <CABF=4+y3S5Zb=0ZyYud3AA_9NRwxyy9DNrPAwD5xtFgKoLN5_g@mail.gmail.com>
 <4A045EF4-7415-4201-8E63-90163706ECBC@mailborder.com>
 <CABF=4+zUz68oXmtzyKX8O33zQsFCjaKKTvUW1r9_GwPmuO8GqA@mail.gmail.com>
 <CABF=4+xqabCj-QzqtnoQ9acCeEupWtYF1DR8hxjTkf+_=3d4Bw@mail.gmail.com>
 <2C54DDA4-FE44-40C2-9860-38A9348157AB@fluxlabs.net>
 <CABF=4+w8EEbJM1d2=rEXCwEoVn8hhdn+MJW7ypdT-PnpE8Gt6g@mail.gmail.com>
 <11C72614-623D-4E8C-931F-484CDC8A2BE7@fluxlabs.net>
 <CABF=4+ySoMqGpu6CfhfHmVUhKEcsyZx3jsyuFmib5D6i5h9wWg@mail.gmail.com>
 <4C8BEE26-3CA7-450A-BD4E-226C0A461202@fluxlabs.net>
 <CABF=4+z8LopwFLOzBmE=Md8r+L0SVs+m+cRmKxaO7V0_OmiWwQ@mail.gmail.com>
 <288C4909-64BE-428B-BC76-B095BF2E63A7@fluxlabs.net>
 <CABF=4+ze+FC7JCpofK5sGMQ2QoBE6pSbEtWtNNiOkoE6mYSu2A@mail.gmail.com>
 <0721F5F8-9A09-4D55-9683-ECC9A384B5C7@fluxlabs.net>
 <CABF=4+zz+AjP3B9j=tx70+EXALO1Bo6sso4Vt3LdHFGTBdEnpg@mail.gmail.com>
 <832142D3-F10F-47EF-826D-4499BEF8B680@mailborder.com>
Message-ID: <CABF=4+z8X+=6t14PDN-2XN-E5-Ki1LksKd6PkDtC95M1=qRbWQ@mail.gmail.com>

I've deleted as per the previous mentioned sql command and then done an
optimize maillog

I have most of the disk space back now, but no answer for the trigger so
far...maybe that's a new thread :)

On 29 July 2015 at 19:05, Jerry Benton <jerry.benton at mailborder.com> wrote:

> Yes. You can also purge binary logs.
>
> -
> Jerry Benton
> www.mailborder.com
>
>
>
> On Jul 29, 2015, at 10:36 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>
> Good point, I'll just keep a dump of the DB elsewhere in case I need to
> look for something on that day in the future. So just "delete from maillog
> where date='2015-07-24';" is the best way to clean this out?
>
> Regards,
> Mark
>
> On 29 July 2015 at 15:17, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>
>>  Sure. But your looping through 9 million entries. Your better off to
>> drop where date=
>>
>>  --
>> Jeremy McSpadden | Flux Labs
>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>> Web - http://www.fluxlabs.net
>>
>>
>> On Jul 29, 2015, at 9:15 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>
>>   Very small, and I can't argue with you about that....
>>
>>  Yes it grew pretty much the whole 14G in one day. Ideally I'd like to
>> keep the stuff from before the problem occurred on that day, is it not ok
>> to delete from maillog where id="XX" ?
>>
>> On 29 July 2015 at 15:12, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>
>>>  How small is that drive ? Pretty bad planning on someone's part ..
>>> Then to be reliant on a single MX for mail flow.
>>>
>>>  Either way .. Dump all entries from that day. Did I read it grew 10gb
>>> in a day ?
>>>
>>>  --
>>> Jeremy McSpadden | Flux Labs
>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>> Web - http://www.fluxlabs.net
>>>
>>>
>>> On Jul 29, 2015, at 9:08 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>
>>>   It's not 1000 entries, its about that many (some id's with a huge
>>> amount more) per message ID. The Mailscanner DB has over 9,000,000 entries
>>> for that day. I was planning on looping through and deleting all the
>>> offending entries for the 300+ emails that seemed to get stuck.
>>>
>>>  The mailscanner DB is currently 14G (maillog.MYD)
>>>
>>> On 29 July 2015 at 15:04, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>>
>>>>  1000 entries from a sql DB may be about 5mb of space.
>>>> I would do a little more digging on the / partition to see what's
>>>> eating space. Probably better off dropping some older archive dir
>>>>
>>>>  --
>>>> Jeremy McSpadden | Flux Labs
>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>> Web - http://www.fluxlabs.net
>>>>
>>>>
>>>> On Jul 29, 2015, at 9:00 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>
>>>>   I agree with you completely, however that doesn't help my immediate
>>>> situation. Can you provide advice on deleting from the mailscanner DB? is
>>>> there any other tables I need to remove the offending ID's entries from?
>>>>
>>>> Regards,
>>>> Mark
>>>>
>>>> On 29 July 2015 at 14:34, Jeremy McSpadden <jeremy at fluxlabs.net> wrote:
>>>>
>>>>>  It's 2015 .. That shouldn't be an excuse. It's like 10 cents per
>>>>> 100gb of drive... Upgrade
>>>>>
>>>>>  --
>>>>> Jeremy McSpadden | Flux Labs
>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>> Web - http://www.fluxlabs.net
>>>>>
>>>>>
>>>>> On Jul 29, 2015, at 8:27 AM, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>
>>>>>   Hi Jeremy,
>>>>>
>>>>>  No I haven't yet - I'm short on space on my root partition still
>>>>> because of the large mysql DB so I want to clean that up first. Can you
>>>>> advise how best to do this? Is it safe enough to do delete from maillog
>>>>> where id='XXX';  for all the ID's with the dupes? is there any other tables
>>>>> that need to be cleared?
>>>>>
>>>>>  Regards,
>>>>> Mark
>>>>>
>>>>> On 29 July 2015 at 14:16, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>> wrote:
>>>>>
>>>>>>  Log won't show taint issues. Setup log rotation.
>>>>>>
>>>>>>  Have you enabled debug in mailscanner config like I stated
>>>>>> yesterday ?
>>>>>>
>>>>>>  --
>>>>>> Jeremy McSpadden | Flux Labs
>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile - 850-890-2543
>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>> Web - http://www.fluxlabs.net
>>>>>>
>>>>>>
>>>>>> On Jul 29, 2015, at 8:13 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>> wrote:
>>>>>>
>>>>>>   Hi all,
>>>>>>
>>>>>>  So I have resolved getting the missing mails delivered from the
>>>>>> quarantine. The main problem stopping this from being easy from the command
>>>>>> line was the fact that "Quarantine Whole Messages As Queue Files = no" was
>>>>>> set, whilst the MTA in use is exim. I've changed that setting to "yes"
>>>>>> now...
>>>>>>
>>>>>>  I've read that if its postfix you can just send that "message" file
>>>>>> back to the queue, I guess the headers are kept with the message in the
>>>>>> quarantine with postfix. With exim they seem to be split between the
>>>>>> database and the message file.
>>>>>>
>>>>>>  I first put the message ID's in to a file "idlist.txt" that had
>>>>>> been put in to the quarantine with the "Other Bad Content Detected" error
>>>>>> (every single email after a certain time on that day), then pulled the
>>>>>> header from the db and combined them with the following simple loop;
>>>>>>
>>>>>>  -------
>>>>>> #!/bin/bash
>>>>>>  for msgid in `cat idlist.txt`;
>>>>>> do
>>>>>>     /usr/bin/mysql -u root --password=XXXXX -N -e "select headers
>>>>>> from maillog where id='$msgid' limit 1 \G;" mailscanner | grep -v "* 1. row
>>>>>> *" >> with-headers/$msgid &&
>>>>>>     /bin/cat 20150724/$msgid/message >> with-headers/$msgid
>>>>>> done
>>>>>>  -------
>>>>>>
>>>>>>  now I'm sending them out slowly (every 30 secs) with another simple
>>>>>> loop...
>>>>>>
>>>>>>  -------
>>>>>>  #!/bin/bash
>>>>>> for msgs in with-headers/*;
>>>>>> do
>>>>>>     cat $msgs | exim -ti
>>>>>>     mv $msgs with-headers-processed/
>>>>>>     sleep 30
>>>>>> done
>>>>>>  -------
>>>>>>
>>>>>>  So at least the missing mail is now going to users.. but I'm no
>>>>>> closer to knowing exactly why this happened in the first place. Jeremy
>>>>>> mentioned a known "taint" issue? Can anyone elaborate on that?
>>>>>>
>>>>>>  I've also found now that Archive is enabled, and is set to "Archive
>>>>>> Mail = " which I guess just defaults to the quarantine dir, as they seem to
>>>>>> go to the "nonspam" folder in there (interestingly in a exim usable
>>>>>> format!!) That couldn't have anything to do with the loop that appears to
>>>>>> have killed my mailcleaner DB? I wouldn't think so as this has been running
>>>>>> for years like this and not had this issue before but thought it worth
>>>>>> mentioning.
>>>>>>
>>>>>>  Any other theories or places to check for clues? unfortunately the
>>>>>> mail.log of the day got removed by the first person looking at the issue to
>>>>>> try to free up space as it was over 4GB.
>>>>>>
>>>>>>  Regards,
>>>>>> Mark
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 28 July 2015 at 17:07, Mark Adams <mark at workshopit.co.uk> wrote:
>>>>>>
>>>>>>> Hi Jerry,
>>>>>>>
>>>>>>>  If you wanted to pull a bunch of items from the quarantine from
>>>>>>> the command line and re-process them through Mailcleaner, how would you do
>>>>>>> that?
>>>>>>>
>>>>>>>  Regards,
>>>>>>> Mark
>>>>>>>
>>>>>>> On 28 July 2015 at 17:00, Jerry Benton <jerry.benton at mailborder.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I am not sure on what parameters Mailwatch calls and logs “other
>>>>>>>> bad content”.  The MailScanner setting is "Notify Senders of Other Blocked
>>>>>>>> Content”. Mailwatch could be calling a trigger of a spam RBL “other blocked
>>>>>>>> content” for all we know. You are going to have to follow the below
>>>>>>>> suggestion and enable debug or see if you can get an idea from
>>>>>>>> /var/log/maillog.
>>>>>>>>
>>>>>>>>
>>>>>>>> -
>>>>>>>>  Jerry Benton
>>>>>>>>  www.mailborder.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>   On Jul 28, 2015, at 11:49 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>  Of course, apologies - I'm using Mailwatch. Any advice on how to
>>>>>>>> most efficiently pull things out of quarantine via command-line? (note they
>>>>>>>> are stored as "message" rather than queue items, that would be too easy..)
>>>>>>>>
>>>>>>>>  I don't have Archive enabled, everything has gone in to the
>>>>>>>> quarantine because of this "Other Bad Content Detected"
>>>>>>>>
>>>>>>>> On 28 July 2015 at 16:43, Jerry Benton <jerry.benton at mailborder.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> By the way, there is no web interface in the MailScanner package.
>>>>>>>>> There are 3rd party products of course (I created one myself) but those
>>>>>>>>> questions would need to be directed to those support forums or mailing
>>>>>>>>> lists.
>>>>>>>>>
>>>>>>>>> -
>>>>>>>>>  Jerry Benton
>>>>>>>>>  www.mailborder.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>   On Jul 28, 2015, at 11:34 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>  How do I try send them through again? At the moment they are
>>>>>>>>> just "message" in the quarantine, and if I try open them through the web
>>>>>>>>> interface it times out, I guess because its trying to open each one of the
>>>>>>>>> dupes?
>>>>>>>>>
>>>>>>>>>  "Fatal error: Maximum execution time of 30 seconds exceeded in
>>>>>>>>> /var/www/html/mailscanner/functions.php on line 1022"
>>>>>>>>>
>>>>>>>>> On 28 July 2015 at 16:31, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>  Yup. Turn on debug and watch it pass through. Last time I saw
>>>>>>>>>> these it was a taint issue .. Which I am assuming has been fixed by now.
>>>>>>>>>>
>>>>>>>>>>  --
>>>>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile -
>>>>>>>>>> 850-890-2543
>>>>>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>>>>>> Web - http://www.fluxlabs.net
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Jul 28, 2015, at 10:20 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>   Hi Jeremy,
>>>>>>>>>>
>>>>>>>>>>  Are you saying that something in these messages is crashing
>>>>>>>>>> Mailscanner? Everything seems to be OK right now, but all 70 of the emails
>>>>>>>>>> (all different types and from different servers) are now in the quarantine
>>>>>>>>>> because of "Other Bad Content Detected" with the report "MailScanner:
>>>>>>>>>> Message attempted to kill MailScanner". It seems it succeeded...
>>>>>>>>>>
>>>>>>>>>> On 28 July 2015 at 15:59, Jeremy McSpadden <jeremy at fluxlabs.net>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>  It's probably looping/crashing mailscanner. Drop MS into debug
>>>>>>>>>>> mode and watch logs.
>>>>>>>>>>>
>>>>>>>>>>>  --
>>>>>>>>>>> Jeremy McSpadden | Flux Labs
>>>>>>>>>>> Local - 850-250-5590x501 <850-250-5590;501> | Mobile -
>>>>>>>>>>> 850-890-2543
>>>>>>>>>>> Fax - 850-254-2955 | Toll Free - 877-699-FLUX
>>>>>>>>>>> Web - http://www.fluxlabs.net
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Jul 28, 2015, at 9:54 AM, Mark Adams <mark at workshopit.co.uk>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>   An update to this, the "2 or 4" duplicates showing in the
>>>>>>>>>>> exim log look like they are actually just separate deliveries to other
>>>>>>>>>>> addresses, so not duplicates. In 1 example there is a single email with 2
>>>>>>>>>>> recipients (2 entries in exim log) that has over 1500+ entries in the
>>>>>>>>>>> mailcleaner DB. It looks like this email hasn't been delivered to the
>>>>>>>>>>> recipient at all either.
>>>>>>>>>>>
>>>>>>>>>>> On 28 July 2015 at 15:14, Mark Adams <mark at workshopit.co.uk>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>>  If anyone could provide advice that would be great. Running
>>>>>>>>>>>> Debian Wheezy Mailscanner 4.79.11-2.2
>>>>>>>>>>>>
>>>>>>>>>>>>  Our incoming dir filled up just before the weekend so we
>>>>>>>>>>>> didn't see the issue for a couple of days. Normally we would just shut down
>>>>>>>>>>>> mailcleaner and delete the dir then start it up again and all would be ok.
>>>>>>>>>>>> However on this occasion, the root partition also become full because of
>>>>>>>>>>>> the mysql DB (it got to 14G in 2 days..).
>>>>>>>>>>>>
>>>>>>>>>>>>  For some reason everything started duplicating. I can see
>>>>>>>>>>>> lots of incoming messages in the exim logs with duplication (2 or 4 of what
>>>>>>>>>>>> looks like the same email) but in the mailscanner database there is
>>>>>>>>>>>> hundreds of each email listed (apparently there was over 9 million messages
>>>>>>>>>>>> delivered on 1 day compared with the server average of about 1500!)
>>>>>>>>>>>>
>>>>>>>>>>>>  It seems like some sort of loop, but afaik nothing specific
>>>>>>>>>>>> was changed in the config apart from the fact incoming became full. Space
>>>>>>>>>>>> has been cleared on the root partition and incoming, and everything appears
>>>>>>>>>>>> to be running as normal right now.
>>>>>>>>>>>>
>>>>>>>>>>>>  Any advice on debugging this would be much appreciated, also,
>>>>>>>>>>>> how best should I clear out the DB of all the dupes?
>>>>>>>>>>>>
>>>>>>>>>>>>  Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
>
>
>
>
>
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/listinfo/mailscanner
>
>
>


-- 
Mark Adams
*Workshop IT:*

5 Cowcross Street
London EC1M 6DW
020 7183 0498
www.workshopit.co.uk
Registered in England and Wales: 8366747
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150730/00c0e278/attachment.html>

From heino.backhaus at fink-computer.de  Thu Jul 30 12:12:40 2015
From: heino.backhaus at fink-computer.de (Heino Backhaus)
Date: Thu, 30 Jul 2015 14:12:40 +0200
Subject: MailScanner: allowing attachments identified as text/plain by
 file -i
In-Reply-To: <C7EB6B94-8471-4282-9B2F-FC68948401B6@kirchenweg.de>
References: <C7EB6B94-8471-4282-9B2F-FC68948401B6@kirchenweg.de>
Message-ID: <55BA14B8.9080508@fink-computer.de>

Hello Volker,


 > If the "mime type" *and* the filetype fields are both specified (and
 > are not "-")

I do not think that the dash is the source of the problem.
Eventualy this is a case of spaces instead of tabs.
Please double check that you've used Tabs. Otherwise
the line is ignored.

# NOTE: Fields are separated by TAB characters --- Important!

This can happen easily by editing the configfile in a Putty-Session
with vi and using copy and paste.
I had a similar thing today.

Mit freundlichen Gruessen

H. Backhaus

Fink-Computer Systeme
Heggrabenstr. 9, 35435 Wettenberg
Email: heino.backhaus at fink-computer.de
Web: www.fink-computer.de
Fax: +49-641-98444638
Fon: +49-641-98444640
UST-ID: DE151040770
HRB: 2143 Gießen
GF: Fredi Fink

"In retrospect it becomes clear that hindsight is definitely overrated!"

   -Alfred E. Neumann

Am 22.07.2015 um 14:21 schrieb Volker Dose:
> Hi list,
>
> I am struggling with the ”magic”  fifth field in filetype.rules.conf –
> as so many others in the past, as far as I understand old posting.
>
> Let me explain my settings:
>
> I have a list of attachments, I do allow in filetype.rules.conf (like
> text, pics, html, pdf and other stuff) and  the last line is a deny for
> every other attachment. I did this, because I do not want to get
> anything to my mailserver, where I am not 100% sure of the filetype – so
> executables are banned and also every unknown  filetype.
>
> This file looks like this:
>
>
> -------
> allow   ASCII text      ASCII text      ASCII text
> allow   PC bitmap       PC bitmap       PC bitmap
> allow   Emacs v18       Emacs v18       Emacs v18
> allow   C++ source      C++ source      C++ source
> allow   source          diverse source  diverse source
> […]
> deny            .*      Deny unidentified attachments
> Deny unidentified attachments
> ----------
>
>
> But  from time to time I get a false positive, often non-english
> text-parts are not very good identified, like Finnish or east-European
> languages.  Often the pdf attachment is identified fine and mailscanner
> processes it,  but txt and html-parts are too often blocked.
>
>
>
> But using the file –I command I have a much higher rate of messages
> identified as text or html mail-part.
>
> So I wanted to use this  feature Julian implemented 2008:
>
>
> ------------
> This 5th field is optional, and specifies a regular expression which is
> matched against the MIME type as determined by the "file -i" command.
>
> If it is never specified, then the "file -i" command will never be run
> on your message attachments so there is no appreciable overhead on the
> speed of MailScanner caused by this new feature.
>
> If the "mime type" *and* the filetype fields are both specified (and are
> not "-") then either matching will cause the rule to fire. In a "deny"
> rule like the example above, then *either* test firing will cause the
> attachment to be blocked. In an "allow" rule then *both* of the tests
> must pass to cause the attachment to be allowed and hence no more rules
> to be checked. This sounds a bit odd but actually ends up doing pretty
> much what you expect it to. I'm sure you'll let me know if I'm wrong
> there :-)
> ---------
>
> I added a line like this in my filetype.rules.conf:
>
> allow         -                            text/plain
>     -                       -
>
> But the message mentioned above still triggered my last line
>
> deny            .*      Deny unidentified attachments
> Deny unidentified attachments
>
>
> For example: Yesterday I realized, the text-message of an email
> (starting with the string “THX!”) war identified as “*AHX version*” from
> my file (version 5.14) command but as *text/plain* with „file -i"
>
> I understand the text from Julian, that both the “file” and the “file
> -i”-field has to match  and added a line like this:
>
> allow   AHX version     text/plain      -       -
>
> Which works – but only because I  have added the “file”-regex to that
> line, too.
>
> I am looking for a “match all” at that point – the dash “-“ did not work
> for me.
>
> I wonder if there is a  way to allow  any attachments, that give you a
> “text/plain” when using “file –i”.
>
>
> Any help appreciated!
>
> I am using MS-4.84.6-1 on a CentOS 6.6 32 bit.
>
> And by the way: I love MailScanner – thanks to all of you helping make
> the software work.
>
> Best regards
> Volker
>
>
>
>

From dana at troubleschuett.com  Thu Jul 30 14:52:31 2015
From: dana at troubleschuett.com (Dana Schuett)
Date: Thu, 30 Jul 2015 14:52:31 +0000
Subject: Clam AV / Perl Modules
Message-ID: <em9c9dc327-3550-40f5-a9fa-e1f53300885f@ares>

I noticed in the newest version of MailScanner (4.85.2-2) that they 
recommend that you let the install script install Clam AV. I have always 
installed clamd manually. I guess my question is, when letting the 
script install clam, why does it use calmscan instead of clamd? Clamscan 
causes a considerably high CPU load if you have any kind of mail volume. 
Wouldn't it make much more sense to use clamd instead? I know clamd has 
a significantly less CPU load. Any insight on this would be much 
appreciated.

Also, I noticed that the install script tries to install perl modules 
via Yum using the following format: Mail::ClamAV. In Yum, perl modules 
use this format perl-Mail-ClamAV, so of course they all fail. I can't 
believe this wasn't noticed by the devs.

- Dana
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150730/6476dde7/attachment.html>

From jerry.benton at mailborder.com  Thu Jul 30 19:40:49 2015
From: jerry.benton at mailborder.com (Jerry Benton)
Date: Thu, 30 Jul 2015 15:40:49 -0400
Subject: Clam AV / Perl Modules
In-Reply-To: <em9c9dc327-3550-40f5-a9fa-e1f53300885f@ares>
References: <em9c9dc327-3550-40f5-a9fa-e1f53300885f@ares>
Message-ID: <2292EDA6-1BFF-4EF8-9C31-466CE6683E31@mailborder.com>

When you install clamd, clamscan also gets installed as a dependency. For MailScanner to use clamd, you need to set it as your virus scanner in MailScanner.conf.

I wrote the install scripts. You can install module on RHEL like this: "yum install perl(Mail::ClamAV)” and it will install the yum package that contains that module if it is available. This is done so that perl modules are installed from the package management system if available and then installed via CPAN later down the line if the module is still missing. 

I can’t believe you don’t know this. 

-
Jerry Benton
www.mailborder.com



> On Jul 30, 2015, at 10:52 AM, Dana Schuett <dana at troubleschuett.com> wrote:
> 
> I noticed in the newest version of MailScanner (4.85.2-2) that they recommend that you let the install script install Clam AV. I have always installed clamd manually. I guess my question is, when letting the script install clam, why does it use calmscan instead of clamd? Clamscan causes a considerably high CPU load if you have any kind of mail volume. Wouldn't it make much more sense to use clamd instead? I know clamd has a significantly less CPU load. Any insight on this would be much appreciated.
>  
> Also, I noticed that the install script tries to install perl modules via Yum using the following format: Mail::ClamAV. In Yum, perl modules use this format perl-Mail-ClamAV, so of course they all fail. I can't believe this wasn't noticed by the devs.
>  
> - Dana
> 
> 
> -- 
> MailScanner mailing list
> mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info>
> http://lists.mailscanner.info/listinfo/mailscanner <http://lists.mailscanner.info/listinfo/mailscanner>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20150730/e7bbeeeb/attachment.html>

From danny at tweegy.nl  Fri Jul 31 22:20:59 2015
From: danny at tweegy.nl (Danny)
Date: Sat, 1 Aug 2015 00:20:59 +0200
Subject: 7z and other archive formats support
Message-ID: <mpgscg$59f$1@ger.gmane.org>

Hello,

I was wondering if MailScanner has 7z (binary) support or are the devs 
working on this or is it on the roadmap?
It would be nice to be able to look inside 7z, arj, cpio etc.

Regards,
Danny