Phishing Server Change

Jerry Benton jerry.benton at mailborder.com
Fri Aug 4 17:42:24 UTC 2017


Thom,

 

I can add a .master version of both files. That is not a problem. I will make a note to do that this weekend.

 

 

--

Jerry Benton

 <http://www.mailborder.com/> www.mailborder.com
+1   (843) 800-8605

+44 (020) 3883-8605



 

From: MailScanner [mailto:mailscanner-bounces+jerry.benton=mailborder.com at lists.mailscanner.info] On Behalf Of Thom van der Boon
Sent: Friday, August 4, 2017 1:22 PM
To: MailScanner Discussion <mailscanner at lists.mailscanner.info>
Subject: Re: Phishing Server Change

 

Jerry,

 

>> The logic of the script does not work as the current script. The current script downloads the file as .master and uses that file plus a .custom file to build the final list. It looks like this script was based on a very old version of the update script.

 

>> The script would have to keep a copy of the .master file instead of deleting it as it does in the current script in order to determine if the file on the update server has changed.

 

The problem is wget. To save bandwidth you want to run wget with the -N option (wget will only download if the remote file is newer than the local file), but the -O does not work together with the -N (curl does not have this problem)

 

To put it simple:

 

curl -z $CONFIGDIR/phishing.bad.sites.conf --compressed -o $CONFIGDIR/phishing.bad.sites.conf.master $THEURL

 

curl only downloads the remote phishing.bad.sites.conf to the local file to $CONFIGDIR/phishing.bad.sites.conf.master if the remote file is newer than the local phishing.bad.sites.conf

 

(curl is great) 

 

wget does not support checking against an other file

 

 

If you want the script to work like you want it to work, there are two options:

a) we have to drop wget support because you can't check wether a newer version of the file is available

b) on the server there need to be both a phishing.bad.sites.conf but also a phishing.bad.sites.conf.master (same file, but wget needs to check against something with the same filename wether it has changed)

 

 

>> The script should not restart mailscanner if mailscanner has been manually stopped. (mailscanner.off file is present.)

>> The existence of /etc/init.d/mailscanner needs to be validated before being used. This file will not exist on a fully implemented systemd server and thus will not work across all platforms.

 

Will add this to the script

 

 

Met vriendelijke groet, Best regards,

 


Thom van der Boon
E-Mail: thom at vdb.nl <mailto:thom at vdb.nl> 

 



=====

 

 

 

Thom.H. van der Boon b.v.
Transito 4

6909 DA  Babberich
Tel.: +31 (0)88 4272727
Fax: +31 (0)88 4272789
Home Page: http://www.vdb.nl/

 

  _____  

Van: "Jerry Benton" <jerry.benton at mailborder.com <mailto:jerry.benton at mailborder.com> >
Aan: "MailScanner Discussion" <mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> >
Verzonden: Vrijdag 4 augustus 2017 17:18:03
Onderwerp: RE: Phishing Server Change

 

I briefly reviewed this script. A couple of problems:

 

*	The logic of the script does not work as the current script. The current script downloads the file as .master and uses that file plus a .custom file to build the final list. It looks like this script was based on a very old version of the update script. 
*	The script should not restart mailscanner if mailscanner has been manually stopped. (mailscanner.off file is present.)
*	The existence of /etc/init.d/mailscanner needs to be validated before being used. This file will not exist on a fully implemented systemd server and thus will not work across all platforms. 

 

 

--

Jerry Benton

www.mailborder.com <http://www.mailborder.com/> 
+1   (843) 800-8605

+44 (020) 3883-8605

 

From: MailScanner [mailto:mailscanner-bounces+jerry.benton=mailborder.com at lists.mailscanner.info] On Behalf Of Thom van der Boon
Sent: Friday, August 4, 2017 1:52 AM
To: MailScanner Discussion <mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> >
Subject: Re: Phishing Server Change

 

Jerry,

 

Attached a almost totally rewritten update_bad_phishing_sites script.It uses curl and as a fallback wget. If the remote file is not updated, it will not download (with both curl and wget).

 

New function: If a new phishing.bad.sites.conf is downloaded, mailscanner is reloaded (if that fails, mailscanner will be restarted)

 

Testing of the script and feedback is highly appriciated.

 

I will do the same for the other script in the upcoming day.

 

Met vriendelijke groet, Best regards,

 


Thom van der Boon
E-Mail: thom at vdb.nl <mailto:thom at vdb.nl> 

 



=====

 

 

 

Thom.H. van der Boon b.v.
Transito 4

6909 DA  Babberich
Tel.: +31 (0)88 4272727
Fax: +31 (0)88 4272789
Home Page: http://www.vdb.nl/

 

  _____  

Van: "Jerry Benton" <jerry.benton at mailborder.com <mailto:jerry.benton at mailborder.com> >
Aan: "MailScanner Discussion" <mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> >
Verzonden: Donderdag 3 augustus 2017 16:42:51
Onderwerp: RE: Phishing Server Change

 

Thom,

 

Yes, that is correct. I can disable the rejections so you can do your testing. 

 

Ok … done. I disabled the rejects. 

 

 

--

Jerry Benton

www.mailborder.com <http://www.mailborder.com/> 
+1   (843) 800-8605

+44 (020) 3883-8605

 

From: MailScanner [mailto:mailscanner-bounces+jerry.benton=mailborder.com at lists.mailscanner.info] On Behalf Of Thom van der Boon
Sent: Thursday, August 3, 2017 9:46 AM
To: MailScanner Discussion <mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> >
Subject: Re: Phishing Server Change

 

Jerry,

 

I am currently debugging a update to the script 

 

I am currently trying to get the wget part working that i only downloads when the remote file is newer... but the server refuses wget... is that correct?

 

 

 

Met vriendelijke groet, Best regards,

 


Thom van der Boon
E-Mail: thom at vdb.nl <mailto:thom at vdb.nl> 

 



=====

 

 

 

Thom.H. van der Boon b.v.
Transito 4

6909 DA  Babberich
Tel.: +31 (0)88 4272727
Fax: +31 (0)88 4272789
Home Page: http://www.vdb.nl/

 

  _____  

Van: "Jerry Benton" <jerry.benton at mailborder.com <mailto:jerry.benton at mailborder.com> >
Aan: "MailScanner Discussion" <mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> >
Verzonden: Donderdag 3 augustus 2017 07:03:07
Onderwerp: Phishing Server Change

 

Please use the updated version of the phishing updates script from
http://phishing.mailscanner.info/

I am trying to reduce bandwidth and the new script uses curl as the primary
method, which support gzip by default. The fallback is wget, which uses more
bandwidth. Most servers are still using the older update script version that
uses wget as the primary method. (You cannot just tell wget to use gzip
because the downloaded file ends up staying compressed and thus useless.) 

The phishing server does about 550GB per month in transfers. Using wget the
bad phishing file transfer size is 280081 and with curl it is 119027. That
is almost half the transfer size. I would like to try to get that 550GB
number down. I don't want to have to pull out the big stick and start
blocking wget user agents, so please help me out here. 

As a side note, there is no point in updating the phishing files every hour.
They get updated six times per day. If you are running every hour, please
change your cron settings to something less taxing. Again, I don't want to
pull out the big stick and start throttling IPs that are updating too often.


--
Jerry Benton
www.mailborder.com <http://www.mailborder.com> 
+1   (843) 800-8605
+44 (020) 3883-8605





-- 
MailScanner mailing list
mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> 
http://lists.mailscanner.info/mailman/listinfo/mailscanner




-- 
MailScanner mailing list
mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> 
http://lists.mailscanner.info/mailman/listinfo/mailscanner




-- 
MailScanner mailing list
mailscanner at lists.mailscanner.info <mailto:mailscanner at lists.mailscanner.info> 
http://lists.mailscanner.info/mailman/listinfo/mailscanner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20170804/8eff2742/attachment.html>


More information about the MailScanner mailing list