Phishing Server Change

Sat Aug 5 02:43:41 UTC 2017

Hi 

Attached the two new scripts for testing 

Both curl and wget will only download when a new file is on the remote server. curl is the prefered download method, wget as a backup 

Met vriendelijke groet, Best regards, 

Thom van der Boon 
E-Mail: thom at vdb.nl 

===== 

Thom.H. van der Boon b.v. 
Transito 4 
6909 DA Babberich 
Tel.: +31 (0)88 4272727 
Fax: +31 (0)88 4272789 
Home Page: http://www.vdb.nl/ 

Van: "Jerry Benton" <jerry.benton at mailborder.com> 
Aan: "MailScanner Discussion" <mailscanner at lists.mailscanner.info> 
Verzonden: Vrijdag 4 augustus 2017 21:16:11 
Onderwerp: RE: Phishing Server Change 

.master copy files have been added to the update server. 

-- 

Jerry Benton 

www.mailborder.com 
+1 (843) 800-8605 

+44 (020) 3883-8605 

From: Jerry Benton [mailto:jerry.benton at mailborder.com] 
Sent: Friday, August 4, 2017 1:42 PM 
To: 'MailScanner Discussion' <mailscanner at lists.mailscanner.info> 
Subject: RE: Phishing Server Change 

Thom, 

I can add a .master version of both files. That is not a problem. I will make a note to do that this weekend. 

-- 

Jerry Benton 

www.mailborder.com 
+1 (843) 800-8605 

+44 (020) 3883-8605 

From: MailScanner [ mailto:mailscanner-bounces+jerry.benton=mailborder.com at lists.mailscanner.info ] On Behalf Of Thom van der Boon 
Sent: Friday, August 4, 2017 1:22 PM 
To: MailScanner Discussion < mailscanner at lists.mailscanner.info > 
Subject: Re: Phishing Server Change 

Jerry, 

>> The logic of the script does not work as the current script. The current script downloads the file as .master and uses that file plus a .custom file to build the final list. It looks like this script was based on a very old version of the update script. 

>> The script would have to keep a copy of the .master file instead of deleting it as it does in the current script in order to determine if the file on the update server has changed. 

The problem is wget. To save bandwidth you want to run wget with the -N option (wget will only download if the remote file is newer than the local file), but the -O does not work together with the -N (curl does not have this problem) 

To put it simple: 

curl -z $CONFIGDIR/phishing.bad.sites.conf --compressed -o $CONFIGDIR/phishing.bad.sites.conf.master $THEURL 

curl only downloads the remote phishing.bad.sites.conf to the local file to $CONFIGDIR/phishing.bad.sites.conf.master if the remote file is newer than the local phishing.bad.sites.conf 

(curl is great) 

wget does not support checking against an other file 

If you want the script to work like you want it to work, there are two options: 

a) we have to drop wget support because you can't check wether a newer version of the file is available 

b) on the server there need to be both a phishing.bad.sites.conf but also a phishing.bad.sites.conf.master (same file, but wget needs to check against something with the same filename wether it has changed) 

>> The script should not restart mailscanner if mailscanner has been manually stopped. (mailscanner.off file is present.) 

>> The existence of /etc/init.d/mailscanner needs to be validated before being used. This file will not exist on a fully implemented systemd server and thus will not work across all platforms. 

Will add this to the script 

Met vriendelijke groet, Best regards, 

Thom van der Boon 
E-Mail: thom at vdb.nl 

===== 

Thom.H. van der Boon b.v. 
Transito 4 

6909 DA Babberich 
Tel.: +31 (0)88 4272727 
Fax: +31 (0)88 4272789 
Home Page: http://www.vdb.nl/ 

Van: "Jerry Benton" < jerry.benton at mailborder.com > 
Aan: "MailScanner Discussion" < mailscanner at lists.mailscanner.info > 
Verzonden: Vrijdag 4 augustus 2017 17:18:03 
Onderwerp: RE: Phishing Server Change 

I briefly reviewed this script. A couple of problems: 

    * The logic of the script does not work as the current script. The current script downloads the file as .master and uses that file plus a .custom file to build the final list. It looks like this script was based on a very old version of the update script. 
    * The script should not restart mailscanner if mailscanner has been manually stopped. (mailscanner.off file is present.) 
    * The existence of /etc/init.d/mailscanner needs to be validated before being used. This file will not exist on a fully implemented systemd server and thus will not work across all platforms. 

-- 

Jerry Benton 

www.mailborder.com 
+1 (843) 800-8605 

+44 (020) 3883-8605 

From: MailScanner [ mailto:mailscanner-bounces+jerry.benton=mailborder.com at lists.mailscanner.info ] On Behalf Of Thom van der Boon 
Sent: Friday, August 4, 2017 1:52 AM 
To: MailScanner Discussion < mailscanner at lists.mailscanner.info > 
Subject: Re: Phishing Server Change 

Jerry, 

Attached a almost totally rewritten update_bad_phishing_sites script.It uses curl and as a fallback wget. If the remote file is not updated, it will not download (with both curl and wget). 

New function: If a new phishing.bad.sites.conf is downloaded, mailscanner is reloaded (if that fails, mailscanner will be restarted) 

Testing of the script and feedback is highly appriciated. 

I will do the same for the other script in the upcoming day. 

Met vriendelijke groet, Best regards, 

Thom van der Boon 
E-Mail: thom at vdb.nl 

===== 

Thom.H. van der Boon b.v. 
Transito 4 

6909 DA Babberich 
Tel.: +31 (0)88 4272727 
Fax: +31 (0)88 4272789 
Home Page: http://www.vdb.nl/ 

Van: "Jerry Benton" < jerry.benton at mailborder.com > 
Aan: "MailScanner Discussion" < mailscanner at lists.mailscanner.info > 
Verzonden: Donderdag 3 augustus 2017 16:42:51 
Onderwerp: RE: Phishing Server Change 

Thom, 

Yes, that is correct. I can disable the rejections so you can do your testing. 

Ok … done. I disabled the rejects. 

-- 

Jerry Benton 

www.mailborder.com 
+1 (843) 800-8605 

+44 (020) 3883-8605 

From: MailScanner [ mailto:mailscanner-bounces+jerry.benton=mailborder.com at lists.mailscanner.info ] On Behalf Of Thom van der Boon 
Sent: Thursday, August 3, 2017 9:46 AM 
To: MailScanner Discussion < mailscanner at lists.mailscanner.info > 
Subject: Re: Phishing Server Change 

Jerry, 

I am currently debugging a update to the script 

I am currently trying to get the wget part working that i only downloads when the remote file is newer... but the server refuses wget... is that correct? 

Met vriendelijke groet, Best regards, 

Thom van der Boon 
E-Mail: thom at vdb.nl 

===== 

Thom.H. van der Boon b.v. 
Transito 4 

6909 DA Babberich 
Tel.: +31 (0)88 4272727 
Fax: +31 (0)88 4272789 
Home Page: http://www.vdb.nl/ 

Van: "Jerry Benton" < jerry.benton at mailborder.com > 
Aan: "MailScanner Discussion" < mailscanner at lists.mailscanner.info > 
Verzonden: Donderdag 3 augustus 2017 07:03:07 
Onderwerp: Phishing Server Change 

Please use the updated version of the phishing updates script from 
http://phishing.mailscanner.info/ 

I am trying to reduce bandwidth and the new script uses curl as the primary 
method, which support gzip by default. The fallback is wget, which uses more 
bandwidth. Most servers are still using the older update script version that 
uses wget as the primary method. (You cannot just tell wget to use gzip 
because the downloaded file ends up staying compressed and thus useless.) 

The phishing server does about 550GB per month in transfers. Using wget the 
bad phishing file transfer size is 280081 and with curl it is 119027. That 
is almost half the transfer size. I would like to try to get that 550GB 
number down. I don't want to have to pull out the big stick and start 
blocking wget user agents, so please help me out here. 

As a side note, there is no point in updating the phishing files every hour. 
They get updated six times per day. If you are running every hour, please 
change your cron settings to something less taxing. Again, I don't want to 
pull out the big stick and start throttling IPs that are updating too often. 

-- 
Jerry Benton 
www.mailborder.com 
+1 (843) 800-8605 
+44 (020) 3883-8605 

-- 
MailScanner mailing list 
mailscanner at lists.mailscanner.info 
http://lists.mailscanner.info/mailman/listinfo/mailscanner 

-- 
MailScanner mailing list 
mailscanner at lists.mailscanner.info 
http://lists.mailscanner.info/mailman/listinfo/mailscanner 

-- 
MailScanner mailing list 
mailscanner at lists.mailscanner.info 
http://lists.mailscanner.info/mailman/listinfo/mailscanner 

-- 
MailScanner mailing list 
mailscanner at lists.mailscanner.info 
http://lists.mailscanner.info/mailman/listinfo/mailscanner 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20170805/b26d1282/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: update_bad_phishing_sites
Type: application/x-shellscript
Size: 7140 bytes
Desc: not available
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20170805/b26d1282/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: update_phishing_sites
Type: application/x-shellscript
Size: 7184 bytes
Desc: not available
URL: <http://lists.mailscanner.info/pipermail/mailscanner/attachments/20170805/b26d1282/attachment-0001.bin>