NO! Dont go! ms2cgp problem
Martin.Hepworth
martinh at solidstatelogic.com
Wed Jan 17 18:22:06 CET 2007
Vasiliy
Nothing to do with the recent change in MS's lock type is it??
--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
> -----Original Message-----
> From: mailscanner-bounces at lists.mailscanner.info [mailto:mailscanner-
> bounces at lists.mailscanner.info] On Behalf Of Vasiliy Boulytchev
> Sent: 17 January 2007 17:11
> To: MailScanner discussion
> Cc: jrudd at ucsc.edu
> Subject: Re: NO! Dont go! ms2cgp problem
>
> OK, a bit more information...
>
> cd /usr
> grep -R QueueDirectory *
>
> lib/MailScanner/MailScanner/CustomConfig.pm:# sendmail -q30m
> -OQueueDirectory=/var/spool/mqueue.spam
> lib/MailScanner/MailScanner/CustomConfig.pm:# sendmail -q60m
> -OQueueDirectory=/var/spool/mqueue.highspam
> lib/MailScanner/MailScanner/Sendmail.pm: $args = "
> -OQueueDirectory=$outqdir " if $outqdir;
>
> src/mailscanner/src/MailScanner-install-4.57.6/perl-tar/MailScanner-
> 4.57.6/lib/MailScanner/CustomConfig.pm:#
> sendmail -q30m -OQueueDirectory=/var/spool/mqueue.spam
> src/mailscanner/src/MailScanner-install-4.57.6/perl-tar/MailScanner-
> 4.57.6/lib/MailScanner/CustomConfig.pm:#
> sendmail -q60m -OQueueDirectory=/var/spool/mqueue.highspam
> src/mailscanner/src/MailScanner-install-4.57.6/perl-tar/MailScanner-
> 4.57.6/lib/MailScanner/Sendmail.pm:
> $args = " -OQueueDirectory=$outqdir " if $outqdir;
> src/mailscanner/src/MailScanner-install-4.57.6/perl-tar/MailScanner-
> 4.57.6/INSTALL.OpenBSD:
> sendmail_flags="-bd -OPrivacyOptions=noetrn -ODeliveryMode=queueonly
> -OQueueDirectory=/var/spool/mqueue.in"
>
> Any suggestions? :)
>
> Again, I really dont need sendmail, I dont know why its everywhere.
> probably my misunderstanding....
>
> Vasiliy Boulytchev
> vasiliy at linuxspecial.com
>
>
>
> Vasiliy Boulytchev wrote:
> > John,
> > First of all, THANK YOU SO MUCH for helping us out! I have been
> > using your cgp2ms and ms2cgp scripts for years flawlessly.
> > Now that you decide to leave us, I get a problem with ms2cgp on a
> > new install...
> >
> > It seems the MS is not giving the correct $job value to the
ms2cgp
> > script.... syslog:
> >
> > --
> > Jan 17 11:02:20 mailscanner1 ms2cgp[19781]: job =
> > -OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out
> > Jan 17 11:02:20 mailscanner1 ms2cgp[19781]: qf =
> > /etc/MailScanner/shared/queues/ms1/out/qf-
> OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out
> >
> > Jan 17 11:02:20 mailscanner1 ms2cgp[19781]: df =
> > /etc/MailScanner/shared/queues/ms1/out/df-
> OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out
> >
> > Jan 17 11:02:20 mailscanner1 ms2cgp[19781]: msg =
> > /etc/MailScanner/mail.cluster/submx00/MS1.ms2cgp.-
> OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out.19781.tmp
> >
> > Jan 17 11:02:20 mailscanner1 ms2cgp[19781]: Job
> > -OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out writing to
> > /etc/MailScanner/mail.cluster/submx00/MS1.ms2cgp.-
> OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out.19781.tmp
> >
> > Jan 17 11:02:20 mailscanner1 ms2cgp[19781]: Open input
> > /etc/MailScanner/shared/queues/ms1/out/qf-
> OQueueDirectory=/etc/MailScanner/shared/queues/ms1/out
> > failed, dying
> > --
> >
> > Now, if I call ms2cgp from commandline via this script:
> > #!/usr/bin/perl -w
> >
> > opendir(DIR, ".");
> > @files = grep(/qf/,readdir(DIR));
> > closedir(DIR);
> >
> > foreach $file (@files) {
> > substr($file, 0, 1) = "";
> > substr($file, 0, 1) = "";
> > system ("/usr/local/etc/ms2cgp $file");
> > }
> >
> > The batch processes just fine, syslog:
> >
> > --
> > Jan 17 11:07:53 mailscanner1 ms2cgp[19859]: job = 19659
> > Jan 17 11:07:53 mailscanner1 ms2cgp[19859]: qf =
> > /etc/MailScanner/shared/queues/ms1/out/qf19659
> > Jan 17 11:07:53 mailscanner1 ms2cgp[19859]: df =
> > /etc/MailScanner/shared/queues/ms1/out/df19659
> > Jan 17 11:07:53 mailscanner1 ms2cgp[19859]: msg =
> > /etc/MailScanner/mail.cluster/submx00/MS1.ms2cgp.19659.19859.tmp
> > Jan 17 11:07:53 mailscanner1 ms2cgp[19859]: Job 19659 writing to
> > /etc/MailScanner/mail.cluster/submx00/MS1.ms2cgp.19659.19859.tmp
> > --
> >
> > So it seems to me, MS is not passing the correct $job value...
> > furthermore, why is it making it -OQueueDirectory!?!?!?
> >
> > What do you guys think?
> >
> > THANK YOU VERY MUCH!!! My cgp2ms and ms2cgp scripts are
attached.
> >
> > Vasiliy Boulytchev
> > vasiliy at linuxspecial.com
> >
> >
> >
> > John Rudd wrote:
> >>
> >>
> >> Just wanted to thank everyone at this list, starting with Julian,
but
> >> also several of the subscribers who given a lot of information and
> >> input over the years I've been here. At some points, this was an
> >> incredibly valuable and informative mailing list for me.
> >>
> >> And, what has changed is not the list. The list, and the vast
> >> majority of its subscribers, are still great. What changed were my
> >> anti-spam/anti-virus needs. I'm moving to solutions that operate
> >> entirely during the SMTP session. As a result, I decommissioned my
> >> last mailscanner system 2 weeks ago.
> >>
> >> After a short waiting period to be sure I wasn't going to need to
> >> roll back, or have any other mailscanner related questions, I'm
going
> >> to be unsubscribing from the list later today.
> >>
> >> I wish all of you the best, and thank you for a great set of
> >> software, a great source of information resources, and a great
source
> >> of conversation over the last 4 or 5 years.
> >
------------------------------------------------------------------------
> >
> > #!/usr/bin/perl
> >
> > # cgp2ms - part of a MailScanner to CommuniGate Pro gateway
> > # Copyright (C) 2003 The Regents of the University of California
> > #
> > # This program is free software; you can redistribute it and/or
modify
> > # it under the terms of the GNU General Public License as
published by
> > # the Free Software Foundation; either version 2 of the License,
or
> > # (at your option) any later version.
> > #
> > # This program is distributed in the hope that it will be useful,
> > # but WITHOUT ANY WARRANTY; without even the implied warranty of
> > # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > # GNU General Public License for more details.
> > #
> > # You should have received a copy of the GNU General Public
License
> > # along with this program; if not, write to the Free Software
> > # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
02111-1307
> USA
> > #
> > # The Author, John Rudd, can be by email at
> > # jrudd at ucsc.edu
> > #
> > # cgp2ms -f file Actually process the given file
> > # or
> > # cgp2ms -c Just check the queue sizes and print result
> > #
> > # This program is used by CommuniGate Pro as an "execute" filter.
It
> takes
> > # in arguments that define where the CommuniGate Pro (CGP) queue
file
> lives
> > # and uses the queue file to create a sendmail formated queue file
(qf)
> and
> > # data file (df) pair. These two files are then inserted into the
> > # MailScanner (MS) incoming queue directory. When MS is done, it
will
> use a
> > # program which is a companion to this one, ms2cgp, # to give the
> message
> > # back to CGP.
> > #
> > # Modified 10/2004 by Vasiliy Boulytchev and Randy Lindsey
> > # Added load balancing and sped up processing for larger volume
> > # 12/19/2004 Randy Lindsey
> > # Rather than read queuedirs for qf* files across remote shares,
> > # this version reads a set of files created on each system
containing
> the queue count
> > #
> > # To take a host offline, just comment out its line in hostlist
> > %hostlist = (
> > # IA => '/var/CommuniGate/MailScanner/Incoming',
> > # TX2 => '/usr/local/mailscanner.tx2',
> > # AZ => '/usr/local/mailscanner.az',
> > # TMP => '/usr/local/mailscanner.tmp',
> > # CA => '/usr/local/mailscanner.ak/mailscanner.in',
> > MS1 => '/var/CommuniGate/MS.Status/queues/ms1/in',
> > # MS2 => '/usr/local/mailscanner.ak/mailscanner.ms2.in',
> > # MS3 => '/usr/local/mailscanner.ak/mailscanner.ms3.in',
> > # MS4 => '/usr/local/mailscanner.ak/mailscanner.ms4.in',
> > # MS5 => '/usr/local/mailscanner.ak/mailscanner.ms5.in',
> > # MS6 => '/usr/local/mailscanner.ak/mailscanner.ms6.in',
> > # MS7 => '/usr/local/mailscanner.ak/mailscanner.ms7.in',
> > # MS8 => '/usr/local/mailscanner.ak/mailscanner.ms8.in',
> > # MS9 => '/usr/local/mailscanner.ak/mailscanner.ms9.in',
> > );
> >
> > # hostspeed - larger numbers are faster
> > %hostspeed = (
> > # IA => 1,
> > # TX2 => 4,
> > # AZ => 0.6,
> > # CA => 9,
> > # TMP => 100,
> > MS1 => 2,
> > MS2 => 2,
> > MS3 => 2,
> > MS4 => 15,
> > MS5 => 15,
> > MS6 => 30,
> > MS7 => 30,
> > MS8 => 30,
> > MS9 => 30,
> > );
> >
> > sub check_one { #$queuedirname
> > open DH, "</var/CommuniGate/MS.Status/queues/count/$_[0]" or
return
> 999999;
> > read(DH, $MsgsInQueue, 10);
> > close DH;
> > chomp $MsgsInQueue;
> > return $MsgsInQueue;
> > }
> >
> > sub check_queues {
> >
> > $totfiles = 0;
> > $totpoints = 0;
> > foreach my $hostid (keys %hostlist) {
> > $hostfiles{$hostid} = check_one(lc($hostid));
> > $totfiles += $hostfiles{$hostid};
> > # Assign points based on relative system speed
> > # We scale it up by 737 to reduce rounding errors with
small
> numbers
> > if ($hostspeed{$hostid}) {
> > $hostpoints{$hostid} = ($hostfiles{$hostid} *
737) /
> $hostspeed{$hostid};
> > }
> > else {
> > $hostpoints{$hostid} = $hostfiles{$hostid};
> > }
> > }
> >
> > @hostsort = sort { $hostpoints{$a} <=> $hostpoints{$b} } keys
> %hostpoints;
> > printf ("MailScanner to %s - Tot:%2u ",
> > $hostsort[0], $totfiles); # displays in CommuniGate
logs
> > foreach my $hostid (keys %hostlist) {
> > printf ("%s:%2u/%2u ", $hostid, $hostfiles{$hostid},
> $hostpoints{$hostid});
> > }
> > print ("\n");
> > return $hostlist{$hostsort[0]};
> > }
> >
> > sub copy_stdin {
> > # Extra headers added by prior rules
> > my $hdrline;
> > while (defined($hdrline = <STDIN>)) {
> > print QF "H$hdrline";
> > print " $hdrline "; # display in logs
> > }
> > }
> >
> > my ($i, $file, $from, $rcpt, @tempv, @rcpt, @argv, $inbody, $inhdr,
> $inmeta);
> >
> > $checkonly = 0;
> > for ($i = 0; $i <= $#ARGV; $i++) { # parse in the arguments
> > if (defined($ARGV[$i])) {
> > if ($ARGV[$i] eq "-f") { # the location of the cgp queue
file
> > $i++;
> > $file = $ARGV[$i];
> > }
> > elsif ($ARGV[$i] eq "-c") { # just check queue sizes and
print
> result
> > $checkonly = 1;
> > }
> > else { # left over from debugging,
these
> > push (@argv, $ARGV[$i]); # aren't actually used
> > }
> > }
> > }
> > if (! $file) {
> > $checkonly = 1; # default if no parameters passed
> > }
> >
> > $qdir = check_queues(); # load balance between servers
> > if ($checkonly) {
> > exit(0);
> > }
> >
> > open (JOB, "<$file");
> > open (QF, ">$qdir/msqf$$"); # minimize file moving by writing to
> final dir but
> > open (DF, ">$qdir/msdf$$"); # under a different name to prevent
> Mailscanner taking it
> >
> > print QF "V4\n"; # I don't think MS actually uses these
> > print QF "T" . time . "\n"; # qf lines, but it helps the qf file
> > print QF "K0\n"; # be a little more authentic
> > print QF "N0\n"; #
> > print QF "P150900\n"; #
> >
> > $inmeta = 1; # when reading the job file, are we still in the meta
> data?
> > $inhdr = 0; # when reading the job file, are we in the rfc822
headers?
> > $inbody = 0; # when reading the job file, are we in the rfc822
> body/data?
> > $rec = 0; # have we read the most recent Received header yet?
> > # the most recent (first listed) Received header will
> > # tell us the information we'll use in the qf file's
$_
> > # which is "which host relayed this message to us"
> >
> > while (defined ($line = <JOB>)) {
> > chomp $line;
> > if ($line eq "") { # blank lines demark meta, headers, and body
> sections
> > if ($inmeta) {
> > $inmeta = 0; $inhdr = 1;
> > }
> > elsif ($inhdr) {
> > $inhdr = 0; $inbody = 1;
> > copy_stdin;
> > if ( !($rec) ) { # no received header = from localhost
> > $rec = 1;
> > print QF "\$_localhost [127.0.0.1]\n"; # print relay
host
> > print QF "S$from\n"; # print sender
> > foreach $to (@rcpt) { # print the recipient list
> > print QF "RPFD:$to\n";
> > }
> > }
> > }
> > }
> > elsif ($inmeta) {
> > if ($line =~ /^P/) {
> > @tempv = split(/ /, $line);
> > $from = $tempv[7];
> > }
> > elsif ($line =~ /^R/) {
> > @tempv = split(/ /, $line);
> > $rcpt = $tempv[7];
> > push (@rcpt, $rcpt);
> > }
> > }
> > elsif ($inhdr) {
> > if ( (!($rec)) && ($line =~ /^Received:/) ) { # get relay host
> > $rec = 1;
> > if ($line =~ /^Received: from (.*) \((.* )?(\[.*\]).*/) {
> > if (defined $2) {
> > $h = $2;
> > $h =~ s/\s*$//;
> > }
> > else {
> > $h = $1;
> > }
> > $a = $3;
> > print QF "\$_$h $a\n"; # print relay host
> > }
> > else {
> > $line =~ /^Received: from (\[.*\]).*/;
> > $a = $1;
> > print QF "\$_$a\n"; # print relay host
> > }
> > print QF "S$from\n"; # print sender
> > foreach $to (@rcpt) { # print the recipient list
> > print QF "RPFD:$to\n";
> > }
> > print QF "H$line\n"; # then print the Received header
> > }
> > elsif ($line !~ /^\s/) { # get a header
> > print QF "H$line\n";
> > }
> > else { # get the rest of a multi-line
header
> > print QF "$line\n";
> > }
> > }
> > if ($inbody) {
> > print DF "$line\n";
> > }
> > }
> >
> > print QF ".\n"; # bat book 23.9.19, qf file should end in a "^\.$"
line.
> >
> > close (DF);
> > close (QF);
> > close(JOB);
> >
> > # Rename the df file first, as Mailscanner looks for the qf and
might
> interfere
> > # Note that this used to mv the files from /tmp to qdir, but a
> fraction of the time
> > # this failed due to race conditions with Mailscanner
> > rename ("$qdir/msdf$$", "$qdir/df$$");
> > rename ("$qdir/msqf$$", "$qdir/qf$$");
> >
> > exit(0);
> >
> >
------------------------------------------------------------------------
> >
> > #!/usr/bin/perl
> >
> > # ms2cgp - part of a MailScanner to CommuniGate Pro gateway
> > # Copyright (C) 2003 The Regents of the University of California
> > #
> > # This program is free software; you can redistribute it and/or
modify
> > # it under the terms of the GNU General Public License as
published by
> > # the Free Software Foundation; either version 2 of the License,
or
> > # (at your option) any later version.
> > #
> > # This program is distributed in the hope that it will be useful,
> > # but WITHOUT ANY WARRANTY; without even the implied warranty of
> > # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > # GNU General Public License for more details.
> > #
> > # You should have received a copy of the GNU General Public
License
> > # along with this program; if not, write to the Free Software
> > # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
02111-1307
> USA
> > #
> > # The Author, John Rudd, can be by email at
> > # jrudd at ucsc.edu
> > #
> > #
> > # ms2cgp -qI(id) ...
> > #
> > # This program acts as Mailscanner's outgoing "Sendmail2" program,
> taking
> > # an argument for each message Mailscanner (MS) has finished
processing,
> > # and using that argument to find the pair of sendmail mqueue files
that
> > # make up the message. This program then re-combines them into an
> rfc822
> > # file and submits them back to Communigate Pro (CGP) via its
Submitted
> directory
> > #
> > # Modified 11/4/04 by Randy Lindsey to skip "sendmail" program and
just
> > # write directly to Submitted in RFC822 format.
> >
> > my $ServerPrefix = "MS1"; # make this unique per server
> > my $QDIR = "/etc/MailScanner/shared/queues/ms1/out"; #
where
> MS sticks outgoing msgs.
> > my $SubDir = "/etc/MailScanner/mail.cluster/submx00"; # CG's
> Submitted directory
> > # my $Archive = "/extra/archivems2cgp/ia";
> >
> > my ($job, $from, $rcpt, $df, $qf, $msg, $line);
> >
> > use Sys::Syslog;
> > openlog('ms2cgp', 'pid', 'user');
> >
> > foreach $job (@ARGV) {
> > $job =~ s/^-qI//;
> > $qf = $QDIR . "/qf" . $job; # the sendmail formatted queue
file
> > $df = $QDIR . "/df" . $job; # the sendmail formatted data
file
> > $msg = $SubDir . "/" . $ServerPrefix . ".ms2cgp.$job.$$.tmp";
#
> the tempfile we'll give to CGP
> > $rcpt = "";
> >
> > # syslog('info', "Job $job copying to archive $Archive");
> > # system("/bin/cp -f $qf $Archive/qf$job");
> > # system("/bin/cp -f $df $Archive/df$job");
> >
> > syslog ('info', "job = $job");
> > closelog();
> > syslog ('info', "qf = $qf");
> > closelog();
> > syslog ('info', "df = $df");
> > closelog();
> > syslog ('info', "msg = $msg");
> > closelog();
> >
> >
> >
> >
> >
> > syslog('info', "Job $job writing to $msg");
> > if (! open (QF, "<$qf")) {
> > syslog('info', "Open input $qf failed, dying");
> > closelog();
> > die "Open input $qf failed!";
> > }
> > if (-e $msg) {
> > syslog('info', "Output file exists $msg so failed, dying");
> > closelog();
> > die "Output file exists $msg";
> > }
> > if (! open (MSG, ">$msg")) {
> > syslog('info', "Open output $msg failed, dying");
> > closelog();
> > die "Open output $msg failed!";
> > }
> >
> > while (defined ($line = <QF>) ) {
> > chomp $line;
> >
> > last if ($line =~ /^\./); # Bat book 23.9.19
> >
> > if ($line =~ /^R/) { # This is needed for Bcc per CGate
Pipe
> specs (see help file)
> > $line =~ s/^R[A-Z]*:/Envelope-To: /; # change sendmail
RPFD to
> Envelope-To
> > print MSG "$line\n";
> > }
> > elsif ($line =~ /^H/) { # get the headers and put them in
the
> msg
> > $line =~ s/^H//;
> > print MSG "$line\n";
> > }
> > elsif ($line =~ /^\s/) { # these should only be on
line-wrapped
> > print MSG "$line\n"; # headers, so put them in the msg
> > }
> > # there's no "else" because we don't care about the other
lines
> > }
> >
> > print MSG "\n"; # put in a blank line to make sure there's one
> between the
> > # headers and the data
> >
> > close (QF);
> > close (MSG);
> >
> > # append the sendmail data file to the cgp message
> > if (system ("/bin/cat $df >> $msg") == 0) {
> > if (rename ($msg, "$msg.sub")) {
> > system ("/bin/rm $df $qf");
> > } else {
> > syslog('info', "rename $msg $msg.sub failed");
> > }
> > } else {
> > syslog('info', "cat $df to $msg failed");
> > }
> > }
> > closelog();
> > exit(0);
> >
> >
> --
> MailScanner mailing list
> mailscanner at lists.mailscanner.info
> http://lists.mailscanner.info/mailman/listinfo/mailscanner
>
> Before posting, read http://wiki.mailscanner.info/posting
>
> Support MailScanner development - buy the book off the website!
**********************************************************************
Confidentiality : This e-mail and any attachments are intended for the
addressee only and may be confidential. If they come to you in error
you must take no action based on them, nor must you copy or show them
to anyone. Please advise the sender by replying to this e-mail
immediately and then delete the original from your computer.
Opinion : Any opinions expressed in this e-mail are entirely those of
the author and unless specifically stated to the contrary, are not
necessarily those of the author's employer.
Security Warning : Internet e-mail is not necessarily a secure
communications medium and can be subject to data corruption. We advise
that you consider this fact when e-mailing us.
Viruses : We have taken steps to ensure that this e-mail and any
attachments are free from known viruses but in keeping with good
computing practice, you should ensure that they are virus free.
Red Lion 49 Ltd T/A Solid State Logic
Registered as a limited company in England and Wales
(Company No:5362730)
Registered Office: 25 Spring Hill Road, Begbroke, Oxford OX5 1RU,
United Kingdom
**********************************************************************
More information about the MailScanner
mailing list