NO! Dont go! ms2cgp problem

Vasiliy Boulytchev vasiliy at linuxspecial.com
Wed Jan 17 18:10:46 CET 2007


OK, a bit more information...

cd /usr
grep -R QueueDirectory *

lib/MailScanner/MailScanner/CustomConfig.pm:#    sendmail -q30m 
-OQueueDirectory=/var/spool/mqueue.spam
lib/MailScanner/MailScanner/CustomConfig.pm:#    sendmail -q60m 
-OQueueDirectory=/var/spool/mqueue.highspam
lib/MailScanner/MailScanner/Sendmail.pm:      $args = " 
-OQueueDirectory=$outqdir " if $outqdir;

src/mailscanner/src/MailScanner-install-4.57.6/perl-tar/MailScanner-4.57.6/lib/MailScanner/CustomConfig.pm:#    
sendmail -q30m -OQueueDirectory=/var/spool/mqueue.spam
src/mailscanner/src/MailScanner-install-4.57.6/perl-tar/MailScanner-4.57.6/lib/MailScanner/CustomConfig.pm:#    
sendmail -q60m -OQueueDirectory=/var/spool/mqueue.highspam
src/mailscanner/src/MailScanner-install-4.57.6/perl-tar/MailScanner-4.57.6/lib/MailScanner/Sendmail.pm:      
$args = " -OQueueDirectory=$outqdir " if $outqdir;
src/mailscanner/src/MailScanner-install-4.57.6/perl-tar/MailScanner-4.57.6/INSTALL.OpenBSD:      
sendmail_flags="-bd -OPrivacyOptions=noetrn -ODeliveryMode=queueonly 
-OQueueDirectory=/var/spool/mqueue.in"

Any suggestions? :)

Again, I really dont need sendmail, I dont know why its everywhere.  
probably my misunderstanding....

Vasiliy Boulytchev
vasiliy at linuxspecial.com



Vasiliy Boulytchev wrote:
> John,
>    First of all, THANK YOU SO MUCH for helping us out!  I have been 
> using your cgp2ms and ms2cgp scripts for years flawlessly.
>    Now that you decide to leave us, I get a problem with ms2cgp on a 
> new install...
>
>    It seems the MS is not giving the correct $job value to the ms2cgp 
> script.... syslog:
>
> -- 
>    Jan 17 11:02:20 mailscanner1 ms2cgp[19781]: job = 
> -OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out
> Jan 17 11:02:20 mailscanner1 ms2cgp[19781]: qf = 
> /etc/MailScanner/shared/queues/ms1/out/qf-OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out 
>
> Jan 17 11:02:20 mailscanner1 ms2cgp[19781]: df = 
> /etc/MailScanner/shared/queues/ms1/out/df-OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out 
>
> Jan 17 11:02:20 mailscanner1 ms2cgp[19781]: msg = 
> /etc/MailScanner/mail.cluster/submx00/MS1.ms2cgp.-OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out.19781.tmp 
>
> Jan 17 11:02:20 mailscanner1 ms2cgp[19781]: Job 
> -OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out writing to 
> /etc/MailScanner/mail.cluster/submx00/MS1.ms2cgp.-OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out.19781.tmp 
>
> Jan 17 11:02:20 mailscanner1 ms2cgp[19781]: Open input 
> /etc/MailScanner/shared/queues/ms1/out/qf-OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out 
> failed, dying
> -- 
>
>    Now, if I call ms2cgp from commandline via this script:
>   #!/usr/bin/perl -w
>
> opendir(DIR, ".");
> @files = grep(/qf/,readdir(DIR));
> closedir(DIR);
>
> foreach $file (@files) {
>        substr($file, 0, 1) = "";
>        substr($file, 0, 1) = "";
>        system ("/usr/local/etc/ms2cgp $file");
> }
>
>    The batch processes just fine, syslog:
>
> -- 
> Jan 17 11:07:53 mailscanner1 ms2cgp[19859]: job = 19659
> Jan 17 11:07:53 mailscanner1 ms2cgp[19859]: qf = 
> /etc/MailScanner/shared/queues/ms1/out/qf19659
> Jan 17 11:07:53 mailscanner1 ms2cgp[19859]: df = 
> /etc/MailScanner/shared/queues/ms1/out/df19659
> Jan 17 11:07:53 mailscanner1 ms2cgp[19859]: msg = 
> /etc/MailScanner/mail.cluster/submx00/MS1.ms2cgp.19659.19859.tmp
> Jan 17 11:07:53 mailscanner1 ms2cgp[19859]: Job 19659 writing to 
> /etc/MailScanner/mail.cluster/submx00/MS1.ms2cgp.19659.19859.tmp
> -- 
>
>    So it seems to me, MS is not passing the correct $job value... 
> furthermore, why is it making it -OQueueDirectory!?!?!?
>
>    What do you guys think?
>
>    THANK YOU VERY MUCH!!!  My cgp2ms and ms2cgp scripts are attached.
>
> Vasiliy Boulytchev
> vasiliy at linuxspecial.com
>
>
>
> John Rudd wrote:
>>
>>
>> Just wanted to thank everyone at this list, starting with Julian, but 
>> also several of the subscribers who given a lot of information and 
>> input over the years I've been here.  At some points, this was an 
>> incredibly valuable and informative mailing list for me.
>>
>> And, what has changed is not the list.  The list, and the vast 
>> majority of its subscribers, are still great.  What changed were my 
>> anti-spam/anti-virus needs.  I'm moving to solutions that operate 
>> entirely during the SMTP session.  As a result, I decommissioned my 
>> last mailscanner system 2 weeks ago.
>>
>> After a short waiting period to be sure I wasn't going to need to 
>> roll back, or have any other mailscanner related questions, I'm going 
>> to be unsubscribing from the list later today.
>>
>> I wish all of you the best, and thank you for a great set of 
>> software, a great source of information resources, and a great source 
>> of conversation over the last 4 or 5 years.
> ------------------------------------------------------------------------
>
> #!/usr/bin/perl
>
> #   cgp2ms - part of a MailScanner to CommuniGate Pro gateway
> #   Copyright (C) 2003  The Regents of the University of California
> #
> #   This program is free software; you can redistribute it and/or modify
> #   it under the terms of the GNU General Public License as published by
> #   the Free Software Foundation; either version 2 of the License, or
> #   (at your option) any later version.
> #
> #   This program is distributed in the hope that it will be useful,
> #   but WITHOUT ANY WARRANTY; without even the implied warranty of
> #   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> #   GNU General Public License for more details.
> #
> #   You should have received a copy of the GNU General Public License
> #   along with this program; if not, write to the Free Software 
> #   Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
> #
> #   The Author, John Rudd, can be by email at
> #      jrudd at ucsc.edu
> #
> # cgp2ms -f file   Actually process the given file
> #  or
> # cgp2ms -c   Just check the queue sizes and print result 
> #
> # This program is used by CommuniGate Pro as an "execute" filter.  It takes
> # in arguments that define where the CommuniGate Pro (CGP) queue file lives
> # and uses the queue file to create a sendmail formated queue file (qf) and
> # data file (df) pair.  These two files are then inserted into the
> # MailScanner (MS) incoming queue directory.  When MS is done, it will use a
> # program which is a companion to this one, ms2cgp, # to give the message
> # back to CGP.
> #
> # Modified 10/2004 by Vasiliy Boulytchev and Randy Lindsey
> #   Added load balancing and sped up processing for larger volume
> # 12/19/2004 Randy Lindsey
> #   Rather than read queuedirs for qf* files across remote shares,
> #   this version reads a set of files created on each system containing the queue count
> #   
> # To take a host offline, just comment out its line in hostlist
> %hostlist = (
> #	IA => '/var/CommuniGate/MailScanner/Incoming',
> #	TX2 => '/usr/local/mailscanner.tx2',
> #	AZ => '/usr/local/mailscanner.az',
> #	TMP => '/usr/local/mailscanner.tmp',
> #	CA => '/usr/local/mailscanner.ak/mailscanner.in',
> 	MS1 => '/var/CommuniGate/MS.Status/queues/ms1/in',
> #	MS2 => '/usr/local/mailscanner.ak/mailscanner.ms2.in',
> #	MS3 => '/usr/local/mailscanner.ak/mailscanner.ms3.in',
> #	MS4 => '/usr/local/mailscanner.ak/mailscanner.ms4.in',
> #	MS5 => '/usr/local/mailscanner.ak/mailscanner.ms5.in',
> #	MS6 => '/usr/local/mailscanner.ak/mailscanner.ms6.in',
> #	MS7 => '/usr/local/mailscanner.ak/mailscanner.ms7.in',
> #	MS8 => '/usr/local/mailscanner.ak/mailscanner.ms8.in',
> #	MS9 => '/usr/local/mailscanner.ak/mailscanner.ms9.in',
> );
>
> # hostspeed - larger numbers are faster
> %hostspeed = (
> #	IA => 1,
> #	TX2 => 4,
> #	AZ => 0.6,
> #	CA => 9,
> #	TMP => 100,
> 	MS1 => 2,
> 	MS2 => 2,
> 	MS3 => 2,
> 	MS4 => 15,
> 	MS5 => 15,
> 	MS6 => 30,
> 	MS7 => 30,
> 	MS8 => 30,
> 	MS9 => 30,
> );
>
> sub check_one { #$queuedirname
> 	open DH, "</var/CommuniGate/MS.Status/queues/count/$_[0]" or return 999999;
> 	read(DH, $MsgsInQueue, 10);
> 	close DH;
> 	chomp $MsgsInQueue;
> 	return $MsgsInQueue;
> }
>
> sub check_queues {
>
> 	$totfiles = 0;
> 	$totpoints = 0;
> 	foreach my $hostid (keys %hostlist) {
> 		$hostfiles{$hostid} = check_one(lc($hostid));
> 		$totfiles += $hostfiles{$hostid};
> 		# Assign points based on relative system speed
> 		#   We scale it up by 737 to reduce rounding errors with small numbers
> 		if ($hostspeed{$hostid}) {
> 			$hostpoints{$hostid} = ($hostfiles{$hostid} * 737) / $hostspeed{$hostid};
> 		}
> 		else {
> 			$hostpoints{$hostid} = $hostfiles{$hostid};
> 		}
> 	}
>
> 	@hostsort = sort { $hostpoints{$a} <=> $hostpoints{$b} } keys %hostpoints;
> 	printf ("MailScanner to %s - Tot:%2u ",
> 		$hostsort[0], $totfiles);  # displays in CommuniGate logs
> 	foreach my $hostid (keys %hostlist) {
> 		printf ("%s:%2u/%2u ", $hostid, $hostfiles{$hostid}, $hostpoints{$hostid});
> 	}
> 	print ("\n");
> 	return $hostlist{$hostsort[0]};
> }
>
> sub copy_stdin {
> 	# Extra headers added by prior rules
> 	my $hdrline;
> 	while (defined($hdrline = <STDIN>)) {
> 		print QF "H$hdrline";
> 		print " $hdrline ";	# display in logs
> 	}
> }
>
> my ($i, $file, $from, $rcpt, @tempv, @rcpt, @argv, $inbody, $inhdr, $inmeta);
>
> $checkonly = 0;
> for ($i = 0; $i <= $#ARGV; $i++) {  # parse in the arguments
>    if (defined($ARGV[$i])) {
>       if ($ARGV[$i] eq "-f") {      # the location of the cgp queue file
>          $i++;
>          $file = $ARGV[$i];
>          }
>       elsif ($ARGV[$i] eq "-c") {   # just check queue sizes and print result
> 	 $checkonly = 1;
>          }
>       else {                        # left over from debugging, these
>          push (@argv, $ARGV[$i]);   # aren't actually used
>          }
>       }
>    }
> if (! $file) {
> 	$checkonly = 1;		# default if no parameters passed
> }
>
> $qdir = check_queues();		# load balance between servers
> if ($checkonly) {
> 	exit(0);
> }
>
> open (JOB, "<$file");
> open (QF, ">$qdir/msqf$$");    # minimize file moving by writing to final dir but
> open (DF, ">$qdir/msdf$$");    #  under a different name to prevent Mailscanner taking it
>
> print QF "V4\n";              # I don't think MS actually uses these
> print QF "T" . time . "\n";   # qf lines, but it helps the qf file
> print QF "K0\n";              # be a little more authentic
> print QF "N0\n";              #
> print QF "P150900\n";         #
>   
> $inmeta = 1;  # when reading the job file, are we still in the meta data?
> $inhdr = 0;   # when reading the job file, are we in the rfc822 headers?
> $inbody = 0;  # when reading the job file, are we in the rfc822 body/data?
> $rec = 0;     # have we read the most recent Received header yet?
>               # the most recent (first listed) Received header will
>               # tell us the information we'll use in the qf file's $_
>               # which is "which host relayed this message to us"
>
> while (defined ($line = <JOB>)) {
>    chomp $line;
>    if ($line eq "") {  # blank lines demark meta, headers, and body sections
>       if ($inmeta) {
>          $inmeta = 0; $inhdr = 1;
>          }
>       elsif ($inhdr) {
>          $inhdr = 0; $inbody = 1;
> 	 copy_stdin;
>          if ( !($rec) ) {    # no received header = from localhost
>             $rec = 1;
>             print QF "\$_localhost [127.0.0.1]\n";    # print relay host
>             print QF "S$from\n";      # print sender
>             foreach $to (@rcpt) {     # print the recipient list 
>                print QF "RPFD:$to\n";
>                }
>             }
>          }
>       }
>    elsif ($inmeta) {
>       if ($line =~ /^P/) {
>          @tempv = split(/ /, $line);
>          $from = $tempv[7];
>          }
>       elsif ($line =~ /^R/) {
>          @tempv = split(/ /, $line);
>          $rcpt = $tempv[7];
>          push (@rcpt, $rcpt);
>          }
>       }
>    elsif ($inhdr) {
>       if ( (!($rec)) && ($line =~ /^Received:/) ) { # get relay host
>          $rec = 1;
>          if ($line =~ /^Received: from (.*) \((.* )?(\[.*\]).*/) {
>             if (defined $2) {
>                $h = $2;
>                $h =~ s/\s*$//;
>                }
>             else {
>                $h = $1;
>                }
>             $a = $3;
>             print QF "\$_$h $a\n";    # print relay host
>             }
>          else {
>             $line =~ /^Received: from (\[.*\]).*/;
>             $a = $1;
>             print QF "\$_$a\n";    # print relay host
>             }
>          print QF "S$from\n";      # print sender
>          foreach $to (@rcpt) {     # print the recipient list 
>             print QF "RPFD:$to\n";
>             }
>          print QF "H$line\n";  # then print the Received header
>          }
>       elsif ($line  !~ /^\s/) {    # get a header
>          print QF "H$line\n";
>          }
>       else {                       # get the rest of a multi-line header
>          print QF "$line\n";
>          }
>       }
>    if ($inbody) {
>       print DF "$line\n";
>       }
>    }
>
> print QF ".\n"; # bat book 23.9.19, qf file should end in a "^\.$" line.
>
> close (DF);
> close (QF);
> close(JOB);
>
> # Rename the df file first, as Mailscanner looks for the qf and might interfere
> #   Note that this used to mv the files from /tmp to qdir, but a fraction of the time
> #   this failed due to race conditions with Mailscanner
> rename ("$qdir/msdf$$", "$qdir/df$$");
> rename ("$qdir/msqf$$", "$qdir/qf$$");
>
> exit(0);
>   
> ------------------------------------------------------------------------
>
> #!/usr/bin/perl
>
> #   ms2cgp - part of a MailScanner to CommuniGate Pro gateway
> #   Copyright (C) 2003  The Regents of the University of California
> #
> #   This program is free software; you can redistribute it and/or modify
> #   it under the terms of the GNU General Public License as published by
> #   the Free Software Foundation; either version 2 of the License, or
> #   (at your option) any later version.
> #
> #   This program is distributed in the hope that it will be useful,
> #   but WITHOUT ANY WARRANTY; without even the implied warranty of
> #   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> #   GNU General Public License for more details.
> #
> #   You should have received a copy of the GNU General Public License
> #   along with this program; if not, write to the Free Software
> #   Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
> #
> #   The Author, John Rudd, can be by email at
> #      jrudd at ucsc.edu
> #
> #
> # ms2cgp -qI(id) ...
> #
> # This program acts as Mailscanner's outgoing "Sendmail2" program, taking
> # an argument for each message Mailscanner (MS) has finished processing,
> # and using that argument to find the pair of sendmail mqueue files that
> # make up the message.  This program then re-combines them into an rfc822
> # file and submits them back to Communigate Pro (CGP) via its Submitted directory
> #
> # Modified 11/4/04 by Randy Lindsey to skip "sendmail" program and just
> #   write directly to Submitted in RFC822 format.
>
> my $ServerPrefix = "MS1";	# make this unique per server
> my $QDIR = "/etc/MailScanner/shared/queues/ms1/out";             # where MS sticks outgoing msgs.
> my $SubDir = "/etc/MailScanner/mail.cluster/submx00";	# CG's Submitted directory
> # my $Archive = "/extra/archivems2cgp/ia";
>
> my ($job, $from, $rcpt, $df, $qf, $msg, $line);
>
> use Sys::Syslog;
> openlog('ms2cgp', 'pid', 'user');
>
> foreach $job (@ARGV) { 
>    $job =~ s/^-qI//;
>    $qf = $QDIR . "/qf" . $job;     # the sendmail formatted queue file
>    $df = $QDIR . "/df" . $job;     # the sendmail formatted data file
>    $msg = $SubDir . "/" . $ServerPrefix . ".ms2cgp.$job.$$.tmp";    # the tempfile we'll give to CGP
>    $rcpt = "";
>
>    # syslog('info', "Job $job copying to archive $Archive");
>    # system("/bin/cp -f $qf $Archive/qf$job");
>    # system("/bin/cp -f $df $Archive/df$job");
>
>    	syslog ('info', "job = $job");
> 		closelog();
>    	syslog ('info', "qf = $qf");
> 		closelog();
> 	syslog ('info', "df = $df");
> 		closelog();
> 	syslog ('info', "msg = $msg");
> 		closelog();
>
>
> 	
>
>
>    syslog('info', "Job $job writing to $msg");
>    if (! open (QF, "<$qf")) {
>       syslog('info', "Open input $qf failed, dying");
>       closelog();
>       die "Open input $qf failed!";
>    }
>    if (-e $msg) {
>       syslog('info', "Output file exists $msg so failed, dying");
>       closelog();
>       die "Output file exists $msg";
>    }
>    if (! open (MSG, ">$msg")) {
>       syslog('info', "Open output $msg failed, dying");
>       closelog();
>       die "Open output $msg failed!";
>    }
>
>    while (defined ($line = <QF>) ) {
>       chomp $line;
>
>       last if ($line =~ /^\./);  # Bat book 23.9.19
>
>       if ($line =~ /^R/) {     # This is needed for Bcc per CGate Pipe specs (see help file)
>          $line =~ s/^R[A-Z]*:/Envelope-To: /;  # change sendmail RPFD to Envelope-To
>          print MSG "$line\n";
>          }
>       elsif ($line =~ /^H/) {     # get the headers and put them in the msg
>          $line =~ s/^H//;
>          print MSG "$line\n";
>          }
>       elsif ($line =~ /^\s/) {    # these should only be on line-wrapped
>          print MSG "$line\n";     # headers, so put them in the msg
>          }
>       # there's no "else" because we don't care about the other lines
>       }
>    
>    print MSG "\n"; # put in a blank line to make sure there's one between the
>                    # headers and the data
>
>    close (QF);
>    close (MSG);
>
>    # append the sendmail data file to the cgp message
>    if (system ("/bin/cat $df >> $msg") == 0) {
>       if (rename ($msg, "$msg.sub")) {
> 	 system ("/bin/rm $df $qf");
>       } else {
> 	 syslog('info', "rename $msg $msg.sub failed");
>       }
>    } else {
>       syslog('info', "cat $df to $msg failed");
>    }
> }
> closelog();
> exit(0);
>
>   


More information about the MailScanner mailing list