From f3b0d8e33bfc8f0fba67d7a85b7ba9c6a72f9597 Mon Sep 17 00:00:00 2001 From: Sven Hoexter Date: Wed, 20 Aug 2025 16:32:59 +0200 Subject: [PATCH] New upstream version 1.1.12 --- ChangeLog | 69 ++- ToDo | 2 +- pffrombyto.1 | 2 +- pflogsumm | 1128 +++++++++++++++++++++++++++++++++++++------------- pflogsumm.1 | 220 +++++++--- pftobyfrom.1 | 2 +- 6 files changed, 1070 insertions(+), 353 deletions(-) diff --git a/ChangeLog b/ChangeLog index 87b2476..040ddc9 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,10 +1,75 @@ ChangeLog for pflogsumm - [Note: Let me know if you would like to be notified as new versions + [Notes: Let me know if you would like to be notified as new versions are released. The latest released version can always be found at - http://jimsun.LinxNet.com/postfix_contrib.html.] + http://jimsun.LinxNet.com/postfix_contrib.html. + As of 2025-07-30 the pflogsumm project page, above, supports an Atom + feed for update notifications.] + + +rel-1.1.12 20250819 + + *** Breaking Changes *** + + Date::Calc now Required due to date(-range) enhancements. + + UI Changes + + Options using underscores have been restored (reversing + removal in v1.1.8). Their use now results in a prominent + "deprecated" message in the report. Support for underscores + will be removed after a suitable deprecation period. (N.B: + This was accomplished without re-introducing Bugzilla bug + 1931403.) + + Renamed option --unprocd to --unprocd-file for better clarity. + (--unprocd was introduced in v1.1.11.) + + Changed --pscrn-detail option to behave like all other + report-limiting options: Full detail unless this option + is specified to limit or suppress it. (--pscrn-detail was + introduced in v1.1.11.) + + Added support for a config file via --config. Command-line arguments + override config file for non-boolean options. + + Note: This option requires the Config::Simple Perl module. + (Only required if --config is used.) + + Added long-form options --date-range, --extended-detail, + --host-cnt, --quiet, and --user-cnt to facilitate more user- + friendly configuration files. Short-form options are retained for + command-line use. + + Added --dump-config as a configuration creation/debugging aid. + + Expanded -d/--date-range options to include today, yesterday, this/last + week/month, and specific date and date ranges in ISO 8601/RFC 3339 format. + + N.B.: Sadly, the extended date range options come with an + unavoidable processing performance penalty. This is because the + only reasonable way to do it was to convert everything into + decimal values. + + Added --dow0mon (day of week 0 is Monday) for use in conjunction with + this/last week date ranges. (Default is Sunday.) + + Added support for log entries with RFC 5424/3164-style fields and + optional syslog version (e.g., "<123>1 ..."). + + Reports now display a date range at the top when processing multi-day + or multi-date logs. + + Set default output column width to 80 columns, consistent with + documentation. + + Squashed another, hopefully the last, bug that would sometimes lead + to inaccurate tallying of messages received. + + Belated note: --rej-add-from/--rej-add-to always include "<" and + ">", respectively, since v1.1.7. rel-1.1.11 20250530 diff --git a/ToDo b/ToDo index 36db0b1..3f17097 100644 --- a/ToDo +++ b/ToDo @@ -3,7 +3,7 @@ To Be Done (Maybe) Fix parsing for "451 4.3.5 Server configuration error;" - date ranges, "lastweek", etc.? + date ranges, "lastweek", etc. (Done) (options for?) break-down by local vs. non-local?, further "drill-downs" to sender/recipient domains? diff --git a/pffrombyto.1 b/pffrombyto.1 index a5836e0..d1bef7b 100644 --- a/pffrombyto.1 +++ b/pffrombyto.1 @@ -55,7 +55,7 @@ .\" ======================================================================== .\" .IX Title "PFFROMBYTO 1" -.TH PFFROMBYTO 1 2025-05-22 1.1.11 "User Contributed Perl Documentation" +.TH PFFROMBYTO 1 2025-05-22 1.1.12 "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l diff --git a/pflogsumm b/pflogsumm index 202e1f5..437ecb9 100755 --- a/pflogsumm +++ b/pflogsumm @@ -6,22 +6,25 @@ eval 'exec perl -S $0 "$@"' pflogsumm - Produce Postfix MTA logfile summary -Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 +Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.12 =head1 SYNOPSIS - pflogsumm -[eq] [-d ] [--detail ] - [--bounce-detail ] [--colwidth ] [--deferral-detail ] - [-h ] [-i|--ignore-case] [--iso-date-time] [--mailq] - [-m|--uucp-mung] [--no-no-msg-size] [--problems-first] - [--pscrn-detail [cnt] [--pscrn-stats] [--rej-add-from] [--rej-add-to] - [--reject-detail ] [--smtp-detail ] [--smtpd-stats] - [--smtpd-warning-detail ] [--srs-mung] [--syslog-name=string] - [-u ] [--unprocd ] [--use-orig-to] - [--verbose-msg-detail] [--verp-mung[=] [-x] [--zero-fill] - [file1 [filen]] + pflogsumm [--config ] [--bounce-detail ] + [--colwidth ] [--deferral-detail ] [--detail ] + [-d ] [--dow0mon] [-e] [-h ] [-i] + [--iso-date-time] [--mailq] [-m] [--no-no-msg-size] + [--problems-first] [--pscrn-detail ] [--pscrn-stats] + [-q] [--rej-add-from] [--rej-add-to] [--reject-detail ] + [--smtp-detail ] [--smtpd-stats] [--smtpd-warning-detail ] + [--srs-mung] [--syslog-name=string] [-u ] + [--unprocd-file ] [--use-orig-to] [--verbose-msg-detail] + [--verp-mung[=]] [-x] [--zero-fill] [file1 [filen]] - pflogsumm -[help|version] + pflogsumm --[dump-config|help|version] + + Note: Where both long- and short-form options exist only the + latter are shown above. See man page for long-form equivalents. If no file(s) specified, reads from stdin. Output is to stdout. Errors and debug to stderr. @@ -44,15 +47,80 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 Limit detailed bounce reports to the top . 0 to suppress entirely. - --colwidth + --config + + Path to a configuration file containing pflogsumm + options. + + Supports all standard command-line options (without the + leading "-" or "--"). Options like "config", "dump-config", + "help", and "version" technically work here, too, though + they're not particularly useful in this context. + Command-line arguments override config file values except + for boolean options. + + --colwidth Maximum report output width. Default is 80 columns. 0 = unlimited. N.B.: --verbose-msg-detail overrides - -d today generate report for just today - -d yesterday generate report for just "yesterday" + -d + --date-range + + Limits the report to the specified date or range. + + Accepted values: + + today + yesterday + "this week" / "last week" + "this month" / "last month" + YYYY-MM[-DD] + "YYYY-MM[-DD] YYYY-MM[-DD]" + + These options do what they suggest, with one + important caveat: + + ISO 8601 / RFC 3339-style dates and ranges may + not yield accurate results when used with + traditional log formats lacking year information + ("month day-of-month"). + + In such cases, pflogsumm assumes log entries + are from the current year. For example, if the + current month is April and a log contains "Apr + NN" entries from the previous year, they will + be interpreted as from the *current* April. + + As such, date-based filtering is only reliable + for entries less than ~365 days old for + old-/traditional-style logfiles. + + Arguments containing spaces must be quoted! + + This/last week/month arguments can take underscores, + rather than spaces, to avoid quoting: E.g.: + + --date-range last_week + + ISO 8601/RFC 3339 date ranges may optionally use a + hyphen or the word "to" for readability. E.g.: + + "2025-08-01 to 2025-08-08" + + If an optional day (DD) is omitted, the range becomes + the full month. E.g.: + + 2025-08 == 2025-08-01 through 2025-08-31 + + "2025-07 - 2025-08" == 2025-07-01 - 2025-08-31 + + --dow0mon + First day of the week is Monday, rather than Sunday. + + (Used only for this/last week calculations.) --deferral-detail @@ -60,13 +128,26 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 to suppress entirely. --detail - Sets all --*-detail, -h and -u to . Is over-ridden by individual settings. --detail 0 suppresses *all* detail. - -e extended (extreme? excessive?) detail + --dump-config + Dump the config to STDOUT and exit. + + This can be used as both a debugging aid and as a way + to develop your first config file. For the latter: + Simply run your usual pflogsumm command line, adding + --dump-config to it, and redirect STDOUT to a file. + + To make it cleaner: Remove unset configs: + pflogsumm --dump-config |grep -v ' = $' + + -e + --extended-detail + + Extended (extreme? excessive?) detail Emit detailed reports. At present, this includes only a per-message report, sorted by sender domain, then user-in-domain, then by queue i.d. @@ -75,8 +156,10 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 quickly consume very large amounts of memory if a lot of log entries are processed! - -h top to display in host/domain reports. - + -h + --host-cnt + + top to display in host/domain reports. 0 = none. See also: "-u" and "--*-detail" options for further @@ -89,7 +172,8 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 know: lame.) -i - --ignore-case Handle complete email address in a case-insensitive + --ignore-case + Handle complete email address in a case-insensitive manner. Normally pflogsumm lower-cases only the host and @@ -104,7 +188,6 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 rather than "Mon DD CCYY" and "HHMM". -m modify (mung?) UUCP-style bang-paths - --uucp-mung This is for use when you have a mix of Internet-style domain addresses and UUCP-style bang-paths in the log. @@ -118,6 +201,8 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 extended detail report (-e), to help ensure that by- domain-by-name sorting is more accurate. + See also: --uucp-mung + --mailq Run "mailq" command at end of report. Merely a convenience feature. (Assumes that "mailq" @@ -142,11 +227,13 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 Emit "problems" reports (bounces, defers, warnings, etc.) before "normal" stats. - --pscrn-detail [cnt] - Emit postscreen detail. + --pscrn-detail - If the optional cnt is included: Limits postscreen detail - reports to the top cnt. + Limit postscreen detail reporting to top lines of + each event. 0 to suppress entirely. + + Note: Postscreen rejects are collected and reported + in any event. --pscrn-stats Collect and emit postscreen summary stats. @@ -160,13 +247,14 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 each listing. (Does not apply to "Improper use of SMTP command pipelining" report.) - -q quiet - don't print headings for empty reports + -q + --quiet + quiet - don't print headings for empty reports note: headings for warning, fatal, and "master" messages will always be printed. --rej-add-to - For sender reject reports: Add the intended recipient address. @@ -181,7 +269,6 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 0 to suppress entirely. --smtpd-stats - Generate smtpd connection statistics. The "per-day" report is not generated for single-day @@ -194,7 +281,6 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 0 to suppress entirely. --srs-mung - Undo SRS address munging. If your postfix install has an SRS plugin running, many @@ -223,12 +309,15 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 See the discussion about the use of this option under "NOTES," below. - -u top to display in user reports. 0 == none. + -u + --user-cnt - See also: "-h" and "--*-detail" options for further - report-limiting options. + top to display in user reports. 0 == none. - --unprocd + See also: "-h" and "--*-detail" options for further + report-limiting options. + + --unprocd-file Emit unprocessed logfile lines to file @@ -237,6 +326,11 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 Where "orig_to" fields are found, report that in place of the "to" address. + --uucp-mung + modify (mung?) UUCP-style bang-paths + + See also: -m + --verbose-msg-detail For the message deferral, bounce and reject summaries: @@ -287,30 +381,48 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 pflogsumm -d yesterday /var/log/maillog - A report of prior week's activities (after logs rotated): + A report of prior week's activities: - pflogsumm /var/log/maillog.0 + pflogsumm -d last_week /var/log/maillog.0 What's happened so far today: pflogsumm -d today /var/log/maillog Crontab entry to generate a report of the previous day's activity - at 10 minutes after midnight. + at 10 minutes after midnight: 10 0 * * * /usr/local/sbin/pflogsumm -d yesterday /var/log/maillog - 2>&1 |/usr/bin/mailx -s "`uname -n` daily mail stats" postmaster + 2>&1 |/usr/bin/mailx -s "`uname -n` daily mail stats" postmaster Crontab entry to generate a report for the prior week's activity. - (This example assumes one rotates ones mail logs weekly, some time - before 4:10 a.m. on Sunday.) - 10 4 * * 0 /usr/local/sbin/pflogsumm /var/log/maillog.0 - 2>&1 |/usr/bin/mailx -s "`uname -n` weekly mail stats" postmaster + 10 4 * * 0 /usr/local/sbin/pflogsumm -d "last week" /var/log/maillog.0 + 2>&1 |/usr/bin/mailx -s "`uname -n` weekly mail stats" postmaster + + (The two crontab examples, above, must actually be a single line + each. They're broken-up into two-or-more lines due to page + formatting issues.) + + Using a config file: + + pflogsumm --config /usr/local/etc/pflogusmm/daily.conf - The two crontab examples, above, must actually be a single line - each. They're broken-up into two-or-more lines due to page - formatting issues. + Using a config file, overriding a config file options on the command + line: + + pflogsumm --config /usr/local/etc/pflogsumm/daily.conf + --detail 30 + + This would override *all* detail settings in the config + file, setting them all to 30. + + pflogsumm --config /usr/local/etc/pflogsumm/daily.conf + --detail 30 --host-cnt 10 + + This would override all detail settings in the config + file, setting them all to 30, with the global detail + setting in turn being overridden to 10 for host count. =head1 SEE ALSO @@ -320,6 +432,11 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 =head1 NOTES + Some options, such as date range, have both short-form and + long-form names. In the interest of brevity, only the + short-form options are shown in the SYNOPSIS and in + pflogsumm's "help" output. + Pflogsumm makes no attempt to catch/parse non-Postfix log entries. Unless it has "postfix/" in the log entry, it will be ignored. @@ -397,11 +514,12 @@ Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.11 =head1 REQUIREMENTS - Requires Perl 5.10, minimum + Requires Perl 5.10, minimum, and Date::Calc + + For --config, Pflogsumm requires the Config::Simple module. - For certain options (e.g.: --smtpd-stats), Pflogsumm requires the - Date::Calc module, which can be obtained from CPAN at - http://www.perl.com. + Both of the above can be obtained from CPAN at http://www.perl.com + or from your distro's repository. Pflogsumm is currently written and tested under Perl 5.38. As of version 19990413-02, pflogsumm worked with Perl 5.003, but @@ -434,19 +552,22 @@ use strict; use locale; use Getopt::Long; use List::Util qw(reduce); -eval { require Date::Calc }; -my $haveDateCalc = $@ ? 0 : 1; +use Time::Local; +use Date::Calc qw(Add_Delta_Days Week_of_Year Delta_DHMS Day_of_Week + Monday_of_Week Days_in_Month); +use POSIX qw(strftime); +eval { require Config::Simple }; +my $haveConfigSimple = $@ ? 0 : 1; my $mailqCmd = "mailq"; -my $release = "1.1.11"; +my $release = "1.1.12"; # Variables and constants used throughout pflogsumm -use vars qw( - $progName - $usageMsg - %opts - @monthNames %monthNums $thisYr $thisMon @dowNames - $isoDateTime +our ( + $progName, + $usageMsg, + @monthNames, %monthNums, $thisYr, $thisMon, @dowNames, + %fromDate, %thruDate, %qidTracker ); # Some constants used by display routines. I arbitrarily chose to @@ -489,7 +610,7 @@ use constant { my ( $cmd, $qid, $addr, $orig_to, $size, $relay, $status, $delay, - $dateStr, $dateStrRFC3339, $dow, + $strtDate, $endDate, %panics, %fatals, %warnings, %masterMsgs, %deferred, %bounced, %noMsgSize, %msgDetail, @@ -505,7 +626,7 @@ my ( $msgsDfrdCnt, $msgsDfrd, %msgDfrdFlgs, %connTime, %smtpdPerDay, %smtpdPerDom, $smtpdConnCnt, $smtpdTotTime, %pscrnConnTime, %pscrnPerDay, %pscrnPerIP, $pscrnConnCnt, $pscrnTotTime, - %smtpMsgs + %smtpMsgs, $sizeDataExists, @deprecated ); $dayCnt = $smtpdConnCnt = $smtpdTotTime = 0; @@ -573,87 +694,257 @@ my %pscrnHits; ($progName = $0) =~ s/^.*\///; $usageMsg = - "usage: $progName -[eq] [-d ] [--detail ] - [--bounce-detail ] [--colwidth ] [--deferral-detail ] - [-h ] [-i|--ignore-case] [--iso-date-time] [--mailq] - [-m|--uucp-mung] [--no-no-msg-size] [--problems-first] - [--pscrn-detail [cnt] [--pscrn-stats] [--rej-add-from] [--rej-add-to] - [--reject-detail ] [--smtp-detail ] [--smtpd-stats] - [--smtpd-warning-detail ] [--srs-mung] [--syslog-name=string] - [-u ] [--unprocd ] [--use-orig-to] - [--verbose-msg-detail] [--verp-mung[=]] [-x] [--zero-fill] - [file1 [filen]] - - $progName --[version|help]"; - -# Some pre-inits for convenience -$isoDateTime = 0; # Don't use ISO date/time formats -GetOptions( - "bounce-detail=i" => \$opts{'bounceDetail'}, - "colwidth=i" => \$opts{'colWidth'}, - "d=s" => \$opts{'d'}, - "deferral-detail=i" => \$opts{'deferralDetail'}, - "detail=i" => \$opts{'detail'}, - "e" => \$opts{'e'}, - "help" => \$opts{'help'}, - "h=i" => \$opts{'h'}, - "ignore-case" => \$opts{'i'}, - "i" => \$opts{'i'}, - "iso-date-time" => \$isoDateTime, - "mailq" => \$opts{'mailq'}, - "m" => \$opts{'m'}, - "no-no-msg-size" => \$opts{'noNoMsgSize'}, - "problems-first" => \$opts{'pf'}, - "pscrn-detail:i" => \$opts{'pscrnDetail'}, - "pscrn-stats" => \$opts{'pscrnStats'}, - "q" => \$opts{'q'}, - "rej-add-from" => \$opts{'rejAddFrom'}, - "rej-add-to" => \$opts{'rejAddTo'}, - "reject-detail=i" => \$opts{'rejectDetail'}, - "smtp-detail=i" => \$opts{'smtpDetail'}, - "smtpd-stats" => \$opts{'smtpdStats'}, - "smtpd-warning-detail=i" => \$opts{'smtpdWarnDetail'}, - "srs-mung" => \$opts{'srsMung'}, - "syslog-name=s" => \$opts{'syslogName'}, - "u=i" => \$opts{'u'}, - "unprocd=s" => \$opts{'unProcdFN'}, - "use-orig-to" => \$opts{'useOrigTo'}, - "uucp-mung" => \$opts{'m'}, - "verbose-msg-detail" => \$opts{'verbMsgDetail'}, - "verp-mung:i" => \$opts{'verpMung'}, - "version" => \$opts{'version'}, - "x" => \$opts{'debug'}, - "zero-fill" => \$opts{'zeroFill'} -) || die "$usageMsg\n"; + "usage: $progName [--config ] [--bounce-detail ] + [--colwidth ] [--deferral-detail ] [--detail ] + [-d ] [--dow0mon] [-e] [-h ] [-i] + [--iso-date-time] [--mailq] [-m] [--no-no-msg-size] + [--problems-first] [--pscrn-detail ] [--pscrn-stats] + [-q] [--rej-add-from] [--rej-add-to] [--reject-detail ] + [--smtp-detail ] [--smtpd-stats] [--smtpd-warning-detail ] + [--srs-mung] [--syslog-name=string] [-u ] + [--unprocd-file ] [--use-orig-to] [--verbose-msg-detail] + [--verp-mung[=]] [-x] [--zero-fill] [file1 [filen]] + + $progName --[dump-config|help|version] + + Note: Where both long- and short-form options exist only the + latter are shown above. See man page for long-form equivalents."; + +# +# Central options specifications. This allows us to create a unified set +# of arguments to GetOpts, for processing Config::Simple, and for dumping +# the configuration. +# +# type: s = string, i = integer, b = boolean, f = float (validated manually) +# Notes: "i" and "s" are used in the GetOpts hash. "f" is translated to "s". +# Short options are ignored by Config::Simple processing. +my %optionSpec = ( + 'bounce-detail' => { type => 'i' }, + 'colwidth' => { type => 'i' }, + 'config' => { type => 's' }, # not exposed as CLI short option + 'date-range' => { type => 's', short => 'd' }, + 'debug' => { type => 'b', short => 'x' }, + 'deferral-detail' => { type => 'i' }, + 'detail' => { type => 'i' }, + 'dow0mon' => { type => 'b' }, + 'dump-config' => { type => 'b' }, + 'extended-detail' => { type => 'b', short => 'e' }, + 'help' => { type => 'b' }, + 'host-cnt' => { type => 'i', short => 'h' }, + 'ignore-case' => { type => 'b', short => 'i' }, + 'iso-date-time' => { type => 'b' }, + 'mailq' => { type => 'b' }, + 'no-no-msg-size' => { type => 'b' }, + 'problems-first' => { type => 'b' }, + 'pscrn-detail' => { type => 'i' }, # optional arg + 'pscrn-stats' => { type => 'b' }, + 'quiet' => { type => 'b', short => 'q' }, + 'rej-add-from' => { type => 'b' }, + 'rej-add-to' => { type => 'b' }, + 'reject-detail' => { type => 'i' }, + 'smtp-detail' => { type => 'i' }, + 'smtpd-stats' => { type => 'b' }, + 'smtpd-warning-detail' => { type => 'i' }, + 'srs-mung' => { type => 'b' }, + 'syslog-name' => { type => 's' }, + 'unprocd-file' => { type => 's' }, + 'use-orig-to' => { type => 'b' }, + 'user-cnt' => { type => 'i', short => 'u' }, + 'uucp-mung' => { type => 'b', short => 'm' }, + 'verbose-msg-detail' => { type => 'b' }, + 'verp-mung' => { type => 'i' }, # optional arg + 'version' => { type => 'b' }, +); + +# Storage for actual values +our %opts; + +# Dynamically build GetOptions argument list +my @getopt_args; +for my $long (sort keys %optionSpec) { + my $type = $optionSpec{$long}->{type}; + my $short = $optionSpec{$long}->{short}; + + my $opt_string = $long; + if ($type eq 'f') { + $opt_string .= "=s"; + } elsif ($type ne 'b') { + $opt_string .= "=$type"; + } + push @getopt_args, $opt_string => \$opts{$long}; + + if (defined $short) { + my $short_string = $short; + if ($type eq 'f') { + $short_string .= "=s"; + } elsif ($type ne 'b') { + $short_string .= "=$type"; + } + push @getopt_args, $short_string => \$opts{$long}; + } +} + +# Ok, this is kind of ugly, but it solves a problem: We don't want to +# *require* Config::Simple, but we also don't want to warn about it if it's +# not needed, so... +my $configFile; +for (my $i = 0; $i < @ARGV; $i++) { + if($ARGV[$i] eq '--config' && defined $ARGV[$i + 1]) { + $configFile = $ARGV[$i + 1]; + splice @ARGV, $i, 2; # Remove from ARGV + last; + } +} + +if($haveConfigSimple) { + # manually import the Config::Simple routines we want + no warnings 'once'; + *ConfigSimpleNew = sub { Config::Simple->new(@_) }; + *ConfigSimpleVars = *Config::Simple::vars; + *ConfigSimpleError = *Config::Simple::error; + +} elsif(defined($configFile)) { + # If user specified --config but doesn't have Config::Simple + # installed, die with friendly help message. + die <" used only once" if -w is on. (No, - # $^W = 0 doesn't work in this context.) - *Delta_DHMS = *Date::Calc::Delta_DHMS; - *Delta_DHMS = *Date::Calc::Delta_DHMS; - *Day_of_Week = *Date::Calc::Day_of_Week; - *Day_of_Week = *Date::Calc::Day_of_Week; - -} elsif(defined($opts{'smtpdStats'}) || defined($opts{'pscrnStats'})) { - # If user specified --smtpd-stats or --pscrn-stats but doesn't - # have Date::Calc installed, die with friendly help message. - die < length($b) ? $a : $b } keys %opts; + # indent a little... + my $fmtStr = sprintf "%%%ds =", length($longestKey) + 2; + + foreach my $key (sort keys(%opts)) { + next if $key eq 'dump-config'; + if($optionSpec{$key}{'type'} eq 'b') { + printf "${fmtStr} %s\n", $key, defined($opts{$key})? "true" : ""; + } elsif(looks_like_number($opts{$key})) { + # internally: 0 == none, undefined == -1 == all + my $val = $opts{$key} == 0? "none" : ($opts{$key} == -1? "all" : $opts{$key}); + printf "${fmtStr} $val\n", $key; + } else { + printf "${fmtStr} %s\n", $key, defined($opts{$key})? $opts{$key} : ""; + } + } + exit 0; } -($dateStr, $dateStrRFC3339, $dow) = get_datestrs($opts{'d'}, $haveDateCalc) if(defined($opts{'d'})); - # debugging my $unProcd; -if($opts{'unProcdFN'}) { - open($unProcd, "> $opts{'unProcdFN'}") || - die "couldn't open \"$opts{'unProcdFN'}\": $!\n"; +if($opts{'unprocd-file'}) { + open($unProcd, "> $opts{'unprocd-file'}") || + die "couldn't open \"$opts{'unprocd-file'}\": $!\n"; } while(<>) { - next if(defined($dateStr) && ! (/^${dateStr} / || /^${dateStrRFC3339}T/)); s/: \[ID \d+ [^\]]+\] /: /; # lose "[ID nnnnnn some.thing]" stuff my $logRmdr; - # "Traditional" timestamp format? - if((($msgMonStr, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) = - /^(...) {1,2}(\d{1,2}) (\d{2}):(\d{2}):(\d{2}) \S+ (.+)$/) == 6) - { - # Convert string to numeric value for later "month rollover" check - $msgMon = $monthNums{$msgMonStr}; - } else { - # RFC 3339 timestamp format? - next unless((($msgYr, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) = - /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.\d+)?(?:[\+\-](?:\d{2}):(?:\d{2})|Z) \S+ (.+)$/) == 7); - # RFC 3339 months start at "1", we index from 0 - --$msgMon; - } + next unless((($msgYr, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) = line_matches_dates($_, $strtDate, $endDate)) == 7); + + # Snag first date seen + ($fromDate{'yr'}, $fromDate{'mon'}, $fromDate{'day'}) = ($msgYr, $msgMon, $msgDay) unless($fromDate{'mon'}); + # Snag last date seen + ($thruDate{'yr'}, $thruDate{'mon'}, $thruDate{'day'}) = ($msgYr, $msgMon, $msgDay); unless((($cmd, $qid) = $logRmdr =~ m#^(?:postfix|$syslogName)(?:/(?:smtps|submission))?/([^\[:]*).*?: ([^:\s]+)#o) == 2 || (($cmd, $qid) = $logRmdr =~ m#^((?:postfix)(?:-script)?)(?:\[\d+\])?: ([^:\s]+)#o) == 2) @@ -729,11 +1009,6 @@ while(<>) { } chomp; - # If the log line's month is greater than our current month, - # we've probably had a year rollover - # FIXME: For processing old logfiles: This is a broken test! - $msgYr = ($msgMon > $thisMon? $thisYr - 1 : $thisYr); - # the following test depends on one getting more than one message a # month--or at least that successive messages don't arrive on the # same month-day in successive months :-) @@ -741,7 +1016,7 @@ while(<>) { $lastMsgDay = $msgDay; $revMsgDateStr = sprintf "%d%02d%02d", $msgYr, $msgMon, $msgDay; ++$dayCnt; - if(defined($opts{'zeroFill'})) { + if(defined($opts{'zero-fill'})) { ${$msgsPerDay{$revMsgDateStr}}[4] = 0; } } @@ -750,21 +1025,26 @@ while(<>) { if($cmd eq "cleanup" && (my($rejSubTyp, $rejReas, $rejRmdr) = $logRmdr =~ /\/cleanup\[\d+\]: .*?\b((?:milter-)?reject|warning|hold|discard): (header|body|END-OF-MESSAGE) (.*)$/) == 3) { - $rejRmdr =~ s/( from \S+?)?; from=<.*$// unless($opts{'verbMsgDetail'}); + $rejRmdr =~ s/( from \S+?)?; from=<.*$// unless($opts{'verbose-msg-detail'}); # FIXME: In retrospect: I've no idea where I came up with the magic numbers I pass to this function. $rejRmdr = string_trimmer($rejRmdr, 64); - if($rejSubTyp eq "reject" or $rejSubTyp eq "milter-reject") { - ++$rejects{$cmd}{$rejReas}{$rejRmdr} unless($opts{'rejectDetail'} == 0); + if($rejSubTyp eq "reject" or $rejSubTyp eq "milter-reject") { + ++$rejects{$cmd}{$rejReas}{$rejRmdr} unless($opts{'reject-detail'} == 0); ++$msgsRjctd; - --$msgsRcvd; # It will have already been counted as "Received," even though it ultimately is not + if($opts{'debug'}) { + push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$rejSubTyp: $rejSubTyp, --\$msgsRcvd"); + ++$qidTracker{$qid}{'lateRejects'}; + } + --$msgsRcvd; # Late Reject: It will have already been counted as "Received," even though it ultimately is not } elsif($rejSubTyp eq "warning") { - ++$warns{$cmd}{$rejReas}{$rejRmdr} unless($opts{'rejectDetail'} == 0); + ++$warns{$cmd}{$rejReas}{$rejRmdr} unless($opts{'reject-detail'} == 0); ++$msgsWrnd; } elsif($rejSubTyp eq "hold") { - ++$holds{$cmd}{$rejReas}{$rejRmdr} unless($opts{'rejectDetail'} == 0); + ++$holds{$cmd}{$rejReas}{$rejRmdr} unless($opts{'reject-detail'} == 0); ++$msgsHld; } elsif($rejSubTyp eq "discard") { - ++$discards{$cmd}{$rejReas}{$rejRmdr} unless($opts{'rejectDetail'} == 0); + push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$rejSubTyp: $rejSubTyp") if $opts{'debug'}; + ++$discards{$cmd}{$rejReas}{$rejRmdr} unless($opts{'reject-detail'} == 0); ++$msgsDscrdd; } delete($rcvdMsg{$qid}); # We're done with this @@ -772,7 +1052,7 @@ while(<>) { ++${$msgsPerDay{$revMsgDateStr}}[4]; } elsif($qid eq 'warning') { (my $warnReas = $logRmdr) =~ s/^.*warning: //; - unless($opts{'verbMsgDetail'}) { + unless($opts{'verbose-msg-detail'}) { # Condense smtpd and other warnings $warnReas =~ s/^(Unable to look up (?:MX|NS) host) for .+(: Host not found(?:,try again)?)/$1$2/ || $warnReas =~ s/^(hostname ).+ (does not resolve to address) [0-9A-F:\.]+$/$1$2/ || @@ -794,7 +1074,7 @@ while(<>) { $warnReas =~ s/(process .+) pid \d+ (exit status \d+)/$1 $2/; } $warnReas = string_trimmer($warnReas, 66); - unless($cmd eq "smtpd" && $opts{'smtpdWarnDetail'} == 0) { + unless($cmd eq "smtpd" && $opts{'smtpd-warning-detail'} == 0) { ++$warnings{$cmd}{$warnReas}; } } elsif($qid eq 'fatal') { @@ -820,29 +1100,43 @@ while(<>) { } elsif($cmd eq 'master') { ++$masterMsgs{(split(/^.*master.*: /, $logRmdr))[1]}; } elsif($cmd eq 'smtpd' || $cmd eq 'postscreen') { - if($logRmdr =~ /\[\d+\]: \w+: client=(.+?)(,|$)/) { + if((my $clientInfo = $logRmdr) =~ /\[\d+\]: \w+: client=(.+?)(?:,|$)/) { # # Warning: this code in two places! # ++$rcvPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[0]; + if($opts{'debug'}) { + push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, ++\$msgsRcvd"); + ++$qidTracker{$qid}{'rcvdCnt'}; + } ++$msgsRcvd; - $rcvdMsg{$qid}{'whence'} = gimme_domain($1); # Whence it came + $rcvdMsg{$qid}{'whence'} = gimme_domain($clientInfo); # Whence it came } elsif(my($rejSubTyp) = $logRmdr =~ /\[\d+\]: \w+: (reject(?:_warning)?|hold|discard): /) { if($rejSubTyp eq 'reject') { proc_smtpd_reject($logRmdr, \%rejects, \$msgsRjctd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); - delete($rcvdMsg{$qid}) if($rcvdMsg{$qid}); # If it's rejected later in the game + # Experimental + unless($qid eq 'NOQUEUE') { + if($opts{'debug'}) { + push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$rejSubTyp: $rejSubTyp, --\$msgsRcvd"); + ++$qidTracker{$qid}{'lateRejects'}; + } + --$msgsRcvd # Late reject: It's been counted as received already + } + delete($rcvdMsg{$qid}) if($rcvdMsg{$qid}); # Late Reject: If it's rejected later in the game } elsif($rejSubTyp eq 'reject_warning') { proc_smtpd_reject($logRmdr, \%warns, \$msgsWrnd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($rejSubTyp eq 'hold') { + push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$rejSubTyp: $rejSubTyp") if $opts{'debug'}; proc_smtpd_reject($logRmdr, \%holds, \$msgsHld, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($rejSubTyp eq 'discard') { + push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$rejSubTyp: $rejSubTyp") if $opts{'debug'}; proc_smtpd_reject($logRmdr, \%discards, \$msgsDscrdd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); @@ -850,7 +1144,7 @@ while(<>) { } else { if($cmd eq 'smtpd') { - next unless(defined($opts{'smtpdStats'})); + next unless(defined($opts{'smtpd-stats'})); if($logRmdr =~ /: connect from /) { $logRmdr =~ /\/smtpd\[(\d+)\]: /; @{$connTime{$1}} = @@ -890,11 +1184,11 @@ while(<>) { $smtpdTotTime += $tSecs; } } - } elsif($cmd eq 'postscreen' && (defined $opts{'pscrnStats'} || $opts{'pscrnDetail'})) { + } elsif($cmd eq 'postscreen' && (defined $opts{'pscrn-stats'} || $opts{'pscrn-detail'})) { my ($pscrnAct, $clientIP, $clientPort, $pscrnAddl, $capCnt); print STDERR "\n" if($opts{'debug'}); - print STDERR "\$opts{'pscrnStats'}: " . ($opts{'pscrnStats'} // 0) .", \$opts{'pscrnDetail'}: $opts{'pscrnDetail'}\n" if($opts{'debug'}); + print STDERR "\$opts{'pscrn-stats'}: " . ($opts{'pscrn-stats'} // 0) .", \$opts{'pscrn-detail'}: $opts{'pscrn-detail'}\n" if($opts{'debug'}); foreach my $regEx (@pscrnRegexs) { print STDERR "\$regEx->{'expr'}: \"$regEx->{'expr'}\"\n" if($opts{'debug'}); if(($capCnt = (($pscrnAct, $clientIP, $clientPort, $pscrnAddl) = $logRmdr =~ /$regEx->{'expr'}/)) >= 3) { @@ -913,11 +1207,11 @@ while(<>) { my $bump_capt_cnt = sub { if($capCnt == 4) { print STDERR "Bumping \$pscrnHits{\"$pscrnAct $pscrnAddl\"}{\"$clientIP\"} on \$logRmdr: \"$logRmdr\"\n" if($opts{'debug'}); - ++$pscrnHits{"$pscrnAct $pscrnAddl"}{$clientIP} if($opts{'pscrnDetail'}); + ++$pscrnHits{"$pscrnAct $pscrnAddl"}{$clientIP} if($opts{'pscrn-detail'}); print STDERR "\$cmd: \"$cmd\", \$logRmdr: \"$logRmdr\"\n" if($opts{'debug'}); } else { print STDERR "Bumping \$pscrnHits{\"$pscrnAct\"}{\"$clientIP\"} on \$logRmdr: \"$logRmdr\"\n" if($opts{'debug'}); - ++$pscrnHits{$pscrnAct}{$clientIP} if($opts{'pscrnDetail'}); + ++$pscrnHits{$pscrnAct}{$clientIP} if($opts{'pscrn-detail'}); print STDERR "\$cmd: \"$cmd\", \$logRmdr: \"$logRmdr\"\n" if($opts{'debug'}); } }; @@ -963,16 +1257,16 @@ while(<>) { $pscrnTotTime += $tSecs; # Want the per-postscreen-action stats? - $bump_capt_cnt->() if($opts{'pscrnDetail'} != 0 && $pscrnAct =~ /^PASS (NEW|OLD)$/); + $bump_capt_cnt->() if($opts{'pscrn-detail'} && $pscrnAct =~ /^PASS (NEW|OLD)$/); } } else { - $bump_capt_cnt->() if($opts{'pscrnDetail'}); # Want the per-postscreen-action stats? + $bump_capt_cnt->() if($opts{'pscrn-detail'}); # Want the per-postscreen-action stats? } } elsif($capCnt == 4) { - $bump_capt_cnt->() if($opts{'pscrnDetail'}); # Want the per-postscreen-action stats? + $bump_capt_cnt->() if($opts{'pscrn-detail'}); # Want the per-postscreen-action stats? } else { - print $unProcd "[02]: $_\n" if($unProcd && (defined $opts{'pscrnStats'} || $opts{'pscrnDetail'} != 0)); + print $unProcd "[02]: $_\n" if($unProcd && (defined $opts{'pscrn-stats'} || $opts{'pscrn-detail'})); } } } @@ -980,20 +1274,21 @@ while(<>) { my $toRmdr; if((($addr, $size) = $logRmdr =~ /from=<([^>]*)>, size=(\d+)/) == 2) { + ++$sizeDataExists; # Flag for orphan rcvdMsg cleanup: Older logs won't have size data next if($rcvdMsg{$qid}{'size'}); # avoid double-counting! if($addr) { - if($opts{'m'} && $addr =~ /^(.*!)*([^!]+)!([^!@]+)@([^\.]+)$/) { + if($opts{'uucp-mung'} && $addr =~ /^(.*!)*([^!]+)!([^!@]+)@([^\.]+)$/) { $addr = "$4!" . ($1? "$1" : "") . $3 . "\@$2"; } - $addr =~ s/(@.+)/\L$1/ unless($opts{'i'}); - $addr = lc($addr) if($opts{'i'}); + $addr =~ s/(@.+)/\L$1/ unless($opts{'ignore-case'}); + $addr = lc($addr) if($opts{'ignore-case'}); $addr = verp_mung($addr); $addr = srs_mung($addr); } else { $addr = "from=<>" } $rcvdMsg{$qid}{'size'} = $size; - push(@{$msgDetail{$qid}}, $addr) if($opts{'e'}); + push(@{$msgDetail{$qid}}, $addr) if($opts{'extended-detail'}); # Avoid counting forwards if($rcvdMsg{$qid}{'whence'}) { # Get the domain out of the sender's address. If there is @@ -1015,19 +1310,20 @@ while(<>) { elsif((($addr, $orig_to, $relay, $delay, $status, $toRmdr) = $logRmdr =~ /to=<([^>]*)>, (?:orig_to=<([^>]*)>, )?relay=([^,]+), (?:conn_use=[^,]+, )?delay=([^,]+), (?:delays=[^,]+, )?(?:dsn=[^,]+, )?status=(\S+)(.*)$/) >= 4) { - $addr = $orig_to if($opts{'useOrigTo'} && $orig_to); + $addr = $orig_to if($opts{'use-orig-to'} && $orig_to); - if($opts{'m'} && $addr =~ /^(.*!)*([^!]+)!([^!@]+)@([^\.]+)$/) { + if($opts{'uucp-mung'} && $addr =~ /^(.*!)*([^!]+)!([^!@]+)@([^\.]+)$/) { $addr = "$4!" . ($1? "$1" : "") . $3 . "\@$2"; } - $addr =~ s/(@.+)/\L$1/ unless($opts{'i'}); - $addr = lc($addr) if($opts{'i'}); - $relay = lc($relay) if($opts{'i'}); + $addr =~ s/(@.+)/\L$1/ unless($opts{'ignore-case'}); + $addr = lc($addr) if($opts{'ignore-case'}); + $relay = lc($relay) if($opts{'ignore-case'}); (my $domAddr = $addr) =~ s/^[^@]+\@//; # get domain only if($status eq 'sent') { # was it actually forwarded, rather than delivered? if((my $newQid) = ($toRmdr =~ /\(forwarded as ([^\)]+)\)/)) { + push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$status: $status, forwarded as new qid $1, ++\$msgsFwdd") if $opts{'debug'}; ++$msgsFwdd; delete($rcvdMsg{$qid}); # We're done with this next; @@ -1044,6 +1340,10 @@ while(<>) { ++${$recipUser{$addr}}[MSG_CNT_I]; ++$dlvPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[1]; + if($opts{'debug'}) { + push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$status: $status, ++\$msgsDlvrd"); + ++$qidTracker{$qid}{'dlvrdCnt'}; + } ++$msgsDlvrd; if($rcvdMsg{$qid}{'size'}) { ${$recipDom{$domAddr}}[MSG_SIZE_I] += $rcvdMsg{$qid}{'size'}; @@ -1052,16 +1352,16 @@ while(<>) { } else { ${$recipDom{$domAddr}}[MSG_SIZE_I] += 0; ${$recipUser{$addr}}[MSG_SIZE_I] += 0; - $noMsgSize{$qid} = $addr unless($opts{'noNoMsgSize'}); - push(@{$msgDetail{$qid}}, "(sender not in log)") if($opts{'e'}); + $noMsgSize{$qid} = $addr unless($opts{'no-no-msg-size'}); + push(@{$msgDetail{$qid}}, "(sender not in log)") if($opts{'extended-detail'}); # put this back later? mebbe with -v? # msg_warn("no message size for qid: $qid"); } - push(@{$msgDetail{$qid}}, $addr) if($opts{'e'}); + push(@{$msgDetail{$qid}}, $addr) if($opts{'extended-detail'}); } elsif($status eq 'deferred') { - unless($opts{'deferralDetail'} == 0) { + unless($opts{'deferral-detail'} == 0) { my ($deferredReas) = $logRmdr =~ /, status=deferred \(([^\)]+)/; - if(!defined($opts{'verbMsgDetail'})) { + if(!defined($opts{'verbose-msg-detail'})) { my ($host, $reason, $moreReason); # More ugliness :/ unless((($host, $reason) = ($deferredReas =~ /^host (\S+) (?:said|refused to talk to me): ([^(]+)/)) || (($host, $reason) = ($deferredReas =~ /^(?:delivery temporarily suspended: )?connect to (.+?(?::\d+)?): ([^)]+)$/)) || @@ -1085,6 +1385,7 @@ while(<>) { } ++$dfrPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[2]; + push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$status: $status, ++\$msgsDfrd") if $opts{'debug'}; ++$msgsDfrdCnt; ++$msgsDfrd unless($msgDfrdFlgs{$qid}++); ++${$recipDom{$domAddr}}[MSG_DFRS_I]; @@ -1094,15 +1395,16 @@ while(<>) { ${$recipDom{$domAddr}}[MSG_DLY_MAX_I] = $delay } } elsif($status eq 'bounced') { - unless($opts{'bounceDetail'} == 0) { + unless($opts{'bounce-detail'} == 0) { my ($bounceReas) = $logRmdr =~ /, status=bounced \((.+)\)/; - unless(defined($opts{'verbMsgDetail'})) { + unless(defined($opts{'verbose-msg-detail'})) { $bounceReas = said_string_trimmer($bounceReas, 66); } ++$bounced{$relay}{$bounceReas}; } ++$bncPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[3]; + push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$status: $status, ++\$msgsBncd") if $opts{'debug'}; ++$msgsBncd; } else { print $unProcd "[03]: $_\n" if $unProcd; @@ -1114,10 +1416,14 @@ while(<>) { # ++$rcvPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[0]; + if($opts{'debug'}) { + push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, ++\$msgsRcvd"); + ++$qidTracker{$qid}{'rcvdCnt'}; + } ++$msgsRcvd; $rcvdMsg{$qid}{'whence'} = "pickup"; # Whence it came } - elsif($cmd eq 'smtp' && $opts{'smtpDetail'} != 0) { + elsif($cmd eq 'smtp' && $opts{'smtp-detail'} != 0) { # Was an IPv6 problem here if($logRmdr =~ /.* connect to (\S+?): ([^;]+); address \S+ port.*$/) { ++$smtpMsgs{lc($2)}{$1}; @@ -1137,20 +1443,76 @@ while(<>) { } } -# Experimental: +# Experimental heuristic: # # If messages were "received" but undelivered, unforwarded, and not # rejected in cleanup, odds are nothing was ever really received—not # even a 0-length message. # +# N.B.: This may result in wonky outcomes for older Postfix logs +# where some of the data in newer logs isn't availble. +# if(my $noSizeCnt = scalar grep { !exists $rcvdMsg{$_}{'size'} } keys %rcvdMsg) { - $msgsRcvd -= $noSizeCnt; + foreach my $qid (keys %rcvdMsg) { + push(@{$qidTracker{$qid}{'status'}}, "No \$rcvdMsg{$qid}{'size'} at end of processing: --\$msgsRcvd") if $opts{'debug'}; + } + $msgsRcvd -= $noSizeCnt if $sizeDataExists; +} + +# Extensive queue I.D. lifetime tracking +if($opts{'debug'} && scalar keys %qidTracker) { + my ($qidCnt, $rcvdDlvrd, $dlvrdCnt, $addlDlvr, $multiDlvrCnt, $noSizeCnt, + $addlRcvd, $multiRcvdCnt, $noRcvdCnt, $lateRejects) = ((0) x 10); + + foreach my $qid (sort keys %qidTracker) { + ++$qidCnt; + print STDERR "qid: $qid\n"; + if(exists $qidTracker{$qid}{'dlvrdCnt'}) { + ++$rcvdDlvrd; + $dlvrdCnt += $qidTracker{$qid}{'dlvrdCnt'}; + if($qidTracker{$qid}{'dlvrdCnt'} > 1) { + $addlDlvr += $qidTracker{$qid}{'dlvrdCnt'} - 1; + ++$multiDlvrCnt; + } + print STDERR " delivered cnt: $qidTracker{$qid}{'dlvrdCnt'}\n" + } else { + print STDERR " delivered cnt: 0\n"; + } + if(! $qidTracker{$qid}{'rcvdCnt'}) { + print STDERR " received cnt: 0\n"; + ++$noRcvdCnt; + } elsif($qidTracker{$qid}{'rcvdCnt'} > 1) { + $addlRcvd += $qidTracker{$qid}{'rcvdCnt'} - 1; + ++$multiRcvdCnt; + print STDERR " received cnt: $qidTracker{$qid}{'rcvdCnt'}\n"; + } + $lateRejects += $qidTracker{$qid}{'lateRejects'} if $qidTracker{$qid}{'lateRejects'}; + foreach my $event (@{$qidTracker{$qid}{'status'}}) { + print STDERR " $event\n"; + } + if(exists $rcvdMsg{$qid} && ! exists $rcvdMsg{$qid}{'size'}) { + print STDERR " no size data\n"; + ++$noSizeCnt; + } + } + printf STDERR "\n %6d%s qids\n", adj_int_units($qidCnt); + printf STDERR " %6d%s qids delivered\n", adj_int_units($rcvdDlvrd); + printf STDERR " %6d%s qids w/multi-deliveries\n", adj_int_units($multiDlvrCnt); + printf STDERR " %6d%s total add'l deliveries\n", adj_int_units($addlDlvr); + printf STDERR " %6d%s qids w/multi-received\n", adj_int_units($multiRcvdCnt); + printf STDERR " %6d%s total add'l received\n", adj_int_units($addlRcvd); + printf STDERR " %6d%s qids w/no received count\n", adj_int_units($noRcvdCnt); + printf STDERR " %6d%s forwarded\n", adj_int_units($msgsFwdd); + printf STDERR " %6d%s delivered by cnt\n", adj_int_units($dlvrdCnt); + printf STDERR " %6d%s discarded\n", adj_int_units($msgsDscrdd); + printf STDERR " %6d%s qids w/no size data\n", adj_int_units($noSizeCnt); + printf STDERR " %6d%s late rejects (rec'd but not dlvrd)\n", adj_int_units($lateRejects); } # debugging if($unProcd) { close($unProcd) || - warn "problem closing \"$opts{'unProcdFN'}\": $!\n"; + warn "problem closing \"$opts{'unprocd-file'}\": $!\n"; } # Calculate percentage of messages rejected and discarded @@ -1162,13 +1524,46 @@ if(my $msgsTotal = $msgsDlvrd + $msgsRjctd + $msgsDscrdd) { } print "Postfix Log Summaries"; -if(defined($dateStr)) { - (my $dispDate = $dateStr) =~ s/\[ 0\]// if($dateStr); - $dow .= ", " if(length($dow)); - print " for ${dow}${dispDate}"; +if (defined($thruDate{'mon'}) && defined($thruDate{'day'})) { + # We can safely assume that if we've a thruDate we've a fromDate + my $monName = $monthNames[ $fromDate{'mon'}]; + my $day = $fromDate{'day'}; + my $yr = $fromDate{'yr'} // $thisYr; + + # st00pid Day_of_Week requires months indexed from 1, not 0 + my $dowIdx = Day_of_Week($yr, $fromDate{'mon'} + 1, $day); + my $dowStr = $dowNames[$dowIdx]; + $day =~ s/^0//; + + print " for $dowStr, $monName $day $yr"; + + # One or both of these could be undefined, so... + my $fromYr = $fromDate{'yr'} // $thisYr; + my $thruYr = $thruDate{'yr'} // $thisYr; + + unless($fromDate{'mon'} == $thruDate{'mon'} && + $fromDate{'day'} == $thruDate{'day'} && + $fromYr == $thruYr) + { + my $monName = $monthNames[ $thruDate{'mon'}]; + my $day = $thruDate{'day'}; + my $yr = $thruDate{'yr'} // $thisYr; + + my $dowIdx = Day_of_Week($yr, $thruDate{'mon'} + 1, $day); + my $dowStr = $dowNames[$dowIdx]; + $day =~ s/^0//; + + print " through $dowStr, $monName $day $yr"; + } } print "\n"; +# Did they use any deprecated "_" options? +if(scalar @deprecated) { + print "\n"; + print "$_\n" foreach (@deprecated); +} + print_subsect_title("Grand Totals"); print "messages\n\n"; printf " %6d%s received\n", adj_int_units($msgsRcvd); @@ -1190,7 +1585,7 @@ printf " %6d%s sending hosts/domains\n", adj_int_units($sendgDomCnt); printf " %6d%s recipients\n", adj_int_units($recipUserCnt); printf " %6d%s recipient hosts/domains\n", adj_int_units($recipDomCnt); -if(defined($opts{'smtpdStats'})) { +if(defined($opts{'smtpd-stats'})) { print "\nsmtpd\n\n"; printf " %6d%s connections\n", adj_int_units($smtpdConnCnt); printf " %6d%s hosts/domains\n", adj_int_units(int(keys %smtpdPerDom)); @@ -1203,7 +1598,7 @@ if(defined($opts{'smtpdStats'})) { } } -if(defined($opts{'pscrnStats'})) { +if(defined($opts{'pscrn-stats'})) { print "\npostscreen\n\n"; printf " %6d%s connections\n", adj_int_units($pscrnConnCnt); printf " %6d%s IP addresses\n", adj_int_units(int(keys %pscrnPerIP)); @@ -1219,58 +1614,58 @@ if(defined($opts{'pscrnStats'})) { print "\n"; -print_problems_reports() if(defined($opts{'pf'})); +print_problems_reports() if(defined($opts{'problems-first'})); print_per_day_summary(\%msgsPerDay) if($dayCnt > 1); print_per_hour_summary(\@rcvPerHr, \@dlvPerHr, \@dfrPerHr, \@bncPerHr, \@rejPerHr, $dayCnt); -print_recip_domain_summary(\%recipDom, $opts{'h'}); -print_sending_domain_summary(\%sendgDom, $opts{'h'}); +print_recip_domain_summary(\%recipDom, $opts{'host-cnt'}); +print_sending_domain_summary(\%sendgDom, $opts{'host-cnt'}); -if(defined($opts{'smtpdStats'})) { +if(defined($opts{'smtpd-stats'})) { print_per_day_smtpd(\%smtpdPerDay, $dayCnt) if($dayCnt > 1); print_per_hour_smtpd(\@smtpdPerHr, $dayCnt); - print_domain_smtpd_summary(\%smtpdPerDom, $opts{'h'}); + print_domain_smtpd_summary(\%smtpdPerDom, $opts{'host-cnt'}); } -print_user_data(\%sendgUser, "Senders by message count", MSG_CNT_I, $opts{'u'}, $opts{'q'}); -print_user_data(\%recipUser, "Recipients by message count", MSG_CNT_I, $opts{'u'}, $opts{'q'}); -print_user_data(\%sendgUser, "Senders by message size", MSG_SIZE_I, $opts{'u'}, $opts{'q'}); -print_user_data(\%recipUser, "Recipients by message size", MSG_SIZE_I, $opts{'u'}, $opts{'q'}); +print_user_data(\%sendgUser, "Senders by message count", MSG_CNT_I, $opts{'user-cnt'}, $opts{'quiet'}); +print_user_data(\%recipUser, "Recipients by message count", MSG_CNT_I, $opts{'user-cnt'}, $opts{'quiet'}); +print_user_data(\%sendgUser, "Senders by message size", MSG_SIZE_I, $opts{'user-cnt'}, $opts{'quiet'}); +print_user_data(\%recipUser, "Recipients by message size", MSG_SIZE_I, $opts{'user-cnt'}, $opts{'quiet'}); print_hash_by_key(\%noMsgSize, "Messages with no size data", 0, 1); -print_problems_reports() unless(defined($opts{'pf'})); +print_problems_reports() unless(defined($opts{'problems-first'})); -print_detailed_msg_data(\%msgDetail, "Message detail", $opts{'q'}) if($opts{'e'}); +print_detailed_msg_data(\%msgDetail, "Message detail", $opts{'quiet'}) if($opts{'extended-detail'}); # Print "problems" reports sub print_problems_reports { - unless($opts{'deferralDetail'} == 0) { - print_nested_hash(\%deferred, "message deferral detail", $opts{'deferralDetail'}, $opts{'q'}); + unless($opts{'deferral-detail'} == 0) { + print_nested_hash(\%deferred, "message deferral detail", $opts{'deferral-detail'}, $opts{'quiet'}); } - unless($opts{'bounceDetail'} == 0) { - print_nested_hash(\%bounced, "message bounce detail (by relay)", $opts{'bounceDetail'}, $opts{'q'}); + unless($opts{'bounce-detail'} == 0) { + print_nested_hash(\%bounced, "message bounce detail (by relay)", $opts{'bounce-detail'}, $opts{'quiet'}); } - unless($opts{'rejectDetail'} == 0) { - print_nested_hash(\%rejects, "message reject detail", $opts{'rejectDetail'}, $opts{'q'}); - print_nested_hash(\%warns, "message reject warning detail", $opts{'rejectDetail'}, $opts{'q'}); - print_nested_hash(\%holds, "message hold detail", $opts{'rejectDetail'}, $opts{'q'}); - print_nested_hash(\%discards, "message discard detail", $opts{'rejectDetail'}, $opts{'q'}); + unless($opts{'reject-detail'} == 0) { + print_nested_hash(\%rejects, "message reject detail", $opts{'reject-detail'}, $opts{'quiet'}); + print_nested_hash(\%warns, "message reject warning detail", $opts{'reject-detail'}, $opts{'quiet'}); + print_nested_hash(\%holds, "message hold detail", $opts{'reject-detail'}, $opts{'quiet'}); + print_nested_hash(\%discards, "message discard detail", $opts{'reject-detail'}, $opts{'quiet'}); } - unless($opts{'smtpDetail'} == 0) { - print_nested_hash(\%smtpMsgs, "smtp delivery failures", $opts{'smtpDetail'}, $opts{'q'}); + unless($opts{'smtp-detail'} == 0) { + print_nested_hash(\%smtpMsgs, "smtp delivery failures", $opts{'smtp-detail'}, $opts{'quiet'}); } - unless($opts{'smtpdWarnDetail'} == 0) { - print_nested_hash(\%warnings, "Warnings", $opts{'smtpdWarnDetail'}, $opts{'q'}); + unless($opts{'smtpd-warning-detail'} == 0) { + print_nested_hash(\%warnings, "Warnings", $opts{'smtpd-warning-detail'}, $opts{'quiet'}); } - print_nested_hash(\%pscrnHits, "postscreen actions", $opts{'pscrnDetail'}, $opts{'q'}) if($opts{'pscrnDetail'}); + print_nested_hash(\%pscrnHits, "postscreen actions", $opts{'pscrn-detail'}, $opts{'quiet'}) if($opts{'pscrn-detail'}); - print_nested_hash(\%fatals, "Fatal Errors", 0, $opts{'q'}); - print_nested_hash(\%panics, "Panics", 0, $opts{'q'}); - print_hash_by_cnt_vals(\%masterMsgs,"Master daemon messages", 0, $opts{'q'}); + print_nested_hash(\%fatals, "Fatal Errors", 0, $opts{'quiet'}); + print_nested_hash(\%panics, "Panics", 0, $opts{'quiet'}); + print_hash_by_cnt_vals(\%masterMsgs,"Master daemon messages", 0, $opts{'quiet'}); } if($opts{'mailq'}) { @@ -1295,7 +1690,7 @@ End_Of_Per_Day_Heading foreach (sort { $a <=> $b } keys(%$msgsPerDay)) { my ($msgYr, $msgMon, $msgDay) = unpack("A4 A2 A2", $_); - if($isoDateTime) { + if($opts{'iso-date-time'}) { printf " %04d-%02d-%02d ", $msgYr, $msgMon + 1, $msgDay } else { my $msgMonStr = $monthNames[$msgMon]; @@ -1323,7 +1718,7 @@ sub print_per_hour_summary { End_Of_Per_Hour_Heading for($hour = 0; $hour < 24; ++$hour) { - if($isoDateTime) { + if($opts{'iso-date-time'}) { printf " %02d:00-%02d:00", $hour, $hour + 1; } else { printf " %02d00-%02d00 ", $hour, $hour + 1; @@ -1464,7 +1859,7 @@ End_Of_Per_Hour_Smtp } my($sec, $min, $hr) = get_smh($smtpdPerHr[$hour]->[1]); - if($isoDateTime) { + if($opts{'iso-date-time'}) { printf " %02d:00-%02d:00", $hour, $hour + 1; } else { printf " %02d00-%02d00 ", $hour, $hour + 1; @@ -1495,7 +1890,7 @@ End_Of_Per_Day_Smtp foreach (sort { $a <=> $b } keys(%$smtpdPerDay)) { my ($msgYr, $msgMon, $msgDay) = unpack("A4 A2 A2", $_); - if($isoDateTime) { + if($opts{'iso-date-time'}) { printf " %04d-%02d-%02d ", $msgYr, $msgMon + 1, $msgDay } else { my $msgMonStr = $monthNames[$msgMon]; @@ -1802,26 +2197,144 @@ sub by_count_then_size { } } -# return traditional and RFC3339 date strings to match in log -sub get_datestrs { - my ($dateOpt, $haveDateCalc) = @_; +# Get range of dates to parse +sub get_dates { + my ($range, $day0mon, $currTime) = @_; + my ($startYr, $startMon, $startDay, $endYr, $endMon, $endDay); + + $currTime //= time(); + my ($sec, $min, $hour, $day, $mon, $yr) = localtime($currTime); + $yr += 1900; + $mon += 1; + + # Normalize + $range =~ s/_/ /g; + + if ($range eq 'today') { + ($startYr, $startMon, $startDay) = ($yr, $mon, $day); + ($endYr, $endMon, $endDay) = ($yr, $mon, $day); + } + elsif ($range eq 'yesterday') { + ($startYr, $startMon, $startDay) = Add_Delta_Days($yr, $mon, $day, -1); + ($endYr, $endMon, $endDay) = ($startYr, $startMon, $startDay); + } + elsif ($range eq 'this week' or $range eq 'last week') { + # 1) Get local calendar date for "now" + my ($sec,$min,$hour,$d,$mo,$y) = localtime($currTime); + my $midnight_now = timelocal(0,0,0, $d, $mo, $y); # local midnight of "now" + + # 2) Day-of-week at local midnight (0=Sun..6=Sat) + my $dow = (localtime($midnight_now))[6]; + + # 3) Days since start-of-week (Sun-start vs Mon-start) + my $since_start = $day0mon ? ($dow == 0 ? 6 : $dow - 1) : $dow; + + # 4) Convert to Y-M-D, then use calendar math only + my ($ny,$nmon,$nday) = ($y + 1900, $mo + 1, $d); + + my $offset = -$since_start + ($range eq 'last week' ? -7 : 0); + my ($sy,$sm,$sd) = Add_Delta_Days($ny, $nmon, $nday, $offset); # start (Y-M-D) + my ($ey,$em,$ed) = Add_Delta_Days($sy, $sm, $sd, 6); # end (Y-M-D) + + # 5) Back to epochs at local midnight (isdst auto-handled) + my $start_epoch = timelocal(0,0,0, $sd, $sm-1, $sy-1900); + my $end_epoch = timelocal(0,0,0, $ed, $em-1, $ey-1900); + + ($startYr,$startMon,$startDay) = ($sy,$sm,$sd); + ($endYr, $endMon, $endDay) = ($ey,$em,$ed); + } + elsif ($range eq 'this month') { + ($startYr, $startMon, $startDay) = ($yr, $mon, 1); + ($endYr, $endMon, $endDay) = Add_Delta_Days($yr, $mon, 1, Days_in_Month($yr, $mon) - 1); + } + elsif ($range eq 'last month') { + my ($lastYr, $lastMon) = ($mon == 1) ? ($yr - 1, 12) : ($yr, $mon - 1); + ($startYr, $startMon, $startDay) = ($lastYr, $lastMon, 1); + ($endYr, $endMon, $endDay) = Add_Delta_Days($lastYr, $lastMon, 1, Days_in_Month($lastYr, $lastMon) - 1); + } + elsif ($range =~ /^(\d{4})-(\d{2})(?:-(\d{2}))?$/) { + ($startYr, $startMon, $startDay) = ($1, $2, $3); + unless(defined($startDay)) { + $startDay = 1; + ($endYr, $endMon, $endDay) = ($1, $2, Days_in_Month($startYr, $startMon)); + } else { + ($endYr, $endMon, $endDay) = ($1, $2, $3); + } + } + elsif ($range =~ /^(\d{4}-\d{2}(?:-\d{2})?)\s+(?:(?:to|-)\s+)?(\d{4}-\d{2}(?:-\d{2})?)$/) { + my ($s, $e) = ($1, $2); + ($startYr, $startMon, $startDay) = split(/-/, $s); + $startDay = 1 unless($startDay); + ($endYr, $endMon, $endDay) = split(/-/, $e); + $endDay = Days_in_Month($endYr, $endMon) unless($endDay); + } else { + die "Invalid date range format: '$range'\n"; + } + + my $start_time = timelocal(0, 0, 0, $startDay, $startMon - 1, $startYr - 1900); + my $end_time = timelocal(0, 0, 0, $endDay, $endMon - 1, $endYr - 1900); + + die "End date precedes start date: '$range'" if $end_time < $start_time; + + return ($start_time, $end_time); +} + +# +# If a line matches the desired date (range): Return the year, month, day, hour, minutes, seconds, and log remainder +# +# N.B.: Year is returned adj. to +1900 +# Month is returned as 0-11 +# +sub line_matches_dates { + my ($line, $startEpoch, $endEpoch) = @_; + my $now = time(); + + my ($epoch, $msgYr, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr); + + # Try RFC 3339 / ISO 8601 first + if (scalar (($msgYr, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) = + ($line =~ /^(?:<\d{1,3}>(?:[1-9]\d*\s+|\s*))?(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.\d+)?(?:[\+\-](?:\d{2}):(?:\d{2})|Z)? \S+ (.+)$/)) == 7) + { + # RFC 3339 months start at "1", we index from 0 + --$msgMon; + + return ($msgYr, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) unless(defined($startEpoch) && defined($endEpoch)); - my $time = time(); + $epoch = eval { timelocal(0, 0, 0, $msgDay, $msgMon, $msgYr - 1900) }; + return ($epoch >= $startEpoch && $epoch <= $endEpoch) + ? ($msgYr, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) + : (undef); + } - if($dateOpt eq "yesterday") { - # Back up to yesterday - $time -= ((localtime($time))[2] + 2) * 3600; - } elsif($dateOpt ne "today") { - die "$usageMsg\n"; + # Try traditional syslog format + my $monStr; + if(scalar (($monStr, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) = + ($line =~ /^(?:<\d{1,3}>(?:[1-9]\d*\s+|\s*))?(\w{3}) {1,2}(\d{1,2}) (\d{2}):(\d{2}):(\d{2}) \S+ (.+)$/)) == 6) + { + return (undef) unless defined($msgMon = $monthNums{$monStr}); + #$msgMon = $monthNums{$monStr}; + #unless(defined($msgMon)) { + # print "dbg: \$msgMon undefined from \$monStr: \"$monStr\"\n"; + # return (undef); + #} + my ($currMon, $currYr) = (localtime($now))[4,5]; + # If month in logfile line is > current month the logfile line must be from last year + --$currYr if($msgMon > $currMon); + + return ($currYr + 1900, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) unless (defined($startEpoch) && defined($endEpoch)); + + $epoch = eval { timelocal(0, 0, 0, $msgDay, $msgMon, $currYr) }; + return (defined $epoch && $epoch >= $startEpoch && $epoch <= $endEpoch) + ? ($currYr + 1900, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) + : (undef); } - my ($t_mday, $t_mon, $t_year) = (localtime($time))[3,4,5]; - my $dow = ($dateOpt && $haveDateCalc)? $dowNames[Day_of_Week($t_year + 1900, $t_mon + 1, $t_mday)] : ""; - return @{[map {s/ (\d)$/[ 0]$1/; $_} sprintf("%s %2d", $monthNames[$t_mon], $t_mday)]}, - sprintf("%04d-%02d-%02d", $t_year+1900, $t_mon+1, $t_mday), $dow; + return (undef); # Not a parsable line } -# if there's a real domain: uses that. Otherwise uses the IP addr. + +# if there's a real hostname/domain: uses that. Otherwise uses +# the IP addr. # # N.B.: in-addr.arpa and ip6.arpa FQDNs return IP addrs # @@ -1848,14 +2361,31 @@ sub gimme_domain { $fqdn = "unknown" unless($fqdn); $ipaddr = "unknown" unless($ipaddr); - my $domain; - if($fqdn eq "unknown" || $fqdn =~ /\.(in-addr|ip6)\.arpa$/) { - $domain = $ipaddr; - } else { - ($domain = $fqdn) =~ s/^(.*)\.([^\.]+)\.([^\.]{3,15}|[^\.]{2,3}\.[^\.]{2})$/\L$2.$3/; + return $ipaddr if($fqdn eq "unknown" || $fqdn =~ /\.(in-addr|ip6)\.arpa$/); + + my $domain = lc $fqdn; + + # Skip if no dot (single-label or malformed) + return $domain unless $domain =~ /\./; + + my @parts = split /\./, $domain; + my $tld = $parts[-1]; + my $sld = $parts[-2]; + my %original_tlds = map { $_ => 1 } qw(com net org gov mil edu); + + if ($original_tlds{$tld}) { + # Collapse to second-level domain: example.com + return "$sld.$tld"; } - return $domain; + # Otherwise elide leftmost: host.example.co.uk → example.co.uk + # if more than 3 elements + if (@parts > 3) { + shift @parts; + return join('.', @parts); + } else { + return $domain; + } } # Return (value, units) for integer @@ -1913,8 +2443,8 @@ sub said_string_trimmer { sub string_trimmer { my($trimmedString, $maxLen) = @_; - unless($opts{'colWidth'} == 0) { - $maxLen += $opts{'colWidth'} - 80 if($opts{'colWidth'} > 0); + unless($opts{'colwidth'} == 0) { + $maxLen += $opts{'colwidth'} - 80 if($opts{'colwidth'} > 0); if(length($trimmedString) > $maxLen) { $trimmedString = substr($trimmedString, 0, $maxLen - 3) . "..."; } @@ -1950,7 +2480,7 @@ sub proc_smtpd_reject { # Hate the sub-calling overhead if we're not doing reject details # anyway, but this is the only place we can do this. - return if($opts{'rejectDetail'} == 0); + return if($opts{'reject-detail'} == 0); # This could get real ugly! @@ -1962,7 +2492,7 @@ sub proc_smtpd_reject { # Next: get the reject "reason" $rejReas = $rejRmdr; - unless(defined($opts{'verbMsgDetail'})) { + unless(defined($opts{'verbose-msg-detail'})) { if($rejTyp eq "RCPT" || $rejTyp eq "DATA" || $rejTyp eq "CONNECT" || $rejTyp eq "BDAT") { # special treatment :-( # If there are "<>"s immediately following the reject code, that's # an email address or HELO string. There can be *anything* in @@ -1993,7 +2523,7 @@ sub proc_smtpd_reject { (($to) = $rejRmdr =~ /\d{3} <([^>]+)>: User unknown /) || (($to) = $rejRmdr =~ /to=<(.*?)(?:[, ]|$)/) || ($to = "<>"); - $to = lc($to) if($opts{'i'}); + $to = lc($to) if($opts{'ignore-case'}); # Snag sender address (($from) = $rejRmdr =~ /from=<([^>]+)>/) || ($from = "<>"); @@ -2001,7 +2531,7 @@ sub proc_smtpd_reject { if(defined($from)) { $from = verp_mung($from); $from = srs_mung($from); - $from = lc($from) if($opts{'i'}); + $from = lc($from) if($opts{'ignore-case'}); } # stash in "triple-subscripted-array" @@ -2009,7 +2539,7 @@ sub proc_smtpd_reject { # Sender address rejected: Domain not found # Sender address rejected: need fully-qualified address my $rejData = $from; - $rejData .= " ($to)" if($opts{'rejAddTo'} && $to); + $rejData .= " ($to)" if($opts{'rej-add-to'} && $to); ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } elsif($rejReas =~ m/^(Recipient address rejected:|User unknown( |$))/) { # Recipient address rejected: Domain not found @@ -2017,7 +2547,7 @@ sub proc_smtpd_reject { # User unknown (in local/relay recipient table) #++$rejects->{$rejTyp}{$rejReas}{$to}; my $rejData = $to; - if($opts{'rejAddFrom'}) { + if($opts{'rej-add-from'}) { $rejData .= " (" . ($from? $from : gimme_domain($rejFrom)) . ")"; } ++$rejects->{$rejTyp}{$rejReas}{$rejData}; @@ -2027,20 +2557,24 @@ sub proc_smtpd_reject { ++$rejects->{$rejTyp}{$rejReas}{$src}; } elsif($rejReas =~ s/^.*?\d{3} (Message size exceeds fixed limit);.*$/$1/) { my $rejData = gimme_domain($rejFrom); - $rejData .= " ($from)" if($opts{'rejAddFrom'}); + $rejData .= " ($from)" if($opts{'rej-add-from'}); ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } elsif($rejReas =~ s/^.*?\d{3} (Server configuration (?:error|problem));.*$/(Local) $1/) { my $rejData = gimme_domain($rejFrom); - $rejData .= " ($from)" if($opts{'rejAddFrom'}); + $rejData .= " ($from)" if($opts{'rej-add-from'}); + ++$rejects->{$rejTyp}{$rejReas}{$rejData}; + } elsif($rejReas =~ m/^Helo command rejected: Invalid name$/) { + my $rejData = gimme_domain($rejFrom); + $rejData .= " ($from)" if($opts{'rej-add-from'}); ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } else { print STDERR "dbg: unknown/un-enumerated reject reason: \$rejReas: \"$rejReas\", \$rejTyp: \"$rejTyp\", \$rejFrom: \"$rejFrom\"!\n" if($opts{'debug'}); my $rejData = gimme_domain($rejFrom); - if($opts{'rejAddFrom'} && $opts{'rejAddTo'} && $to) { + if($opts{'rej-add-from'} && $opts{'rej-add-to'} && $to) { $rejData .= " ($from -> $to)"; - } elsif($opts{'rejAddFrom'}) { + } elsif($opts{'rej-add-from'}) { $rejData .= " (< $from)"; - } elsif($opts{'rejAddTo'} && $to) { + } elsif($opts{'rej-add-to'} && $to) { $rejData .= " (> $to)"; } ++$rejects->{$rejTyp}{$rejReas}{$rejData}; @@ -2057,9 +2591,9 @@ sub proc_smtpd_reject { sub verp_mung { my $addr = $_[0]; - if(defined($opts{'verpMung'})) { + if(defined($opts{'verp-mung'})) { $addr =~ s/((?:bounce[ds]?|no(?:list|reply|response)|return|sentto|\d+).*?)(?:[\+_\.\*-]\d+\b)+/$1-ID/i; - if($opts{'verpMung'} > 1) { + if($opts{'verp-mung'} > 1) { $addr =~ s/[\*-](\d+[\*-])?[^=\*-]+[=\*][^\@]+\@/\@/; } } @@ -2070,7 +2604,7 @@ sub verp_mung { sub srs_mung { my $addr = $_[0]; - if(defined($opts{'srsMung'})) { + if(defined($opts{'srs-mung'})) { $addr =~ s/^SRS(?:[01])(?:[=+-])(?:[^=]+=[\w\.]+==)*(?:[^=]+=[^=]+=)([\w\.]+)=(.+)@[\w\.]+$/$2\@$1/i; } diff --git a/pflogsumm.1 b/pflogsumm.1 index cb0ca17..5c709bf 100644 --- a/pflogsumm.1 +++ b/pflogsumm.1 @@ -55,7 +55,7 @@ .\" ======================================================================== .\" .IX Title "PFLOGSUMM 1" -.TH PFLOGSUMM 1 2025-06-07 1.1.11 "User Contributed Perl Documentation" +.TH PFLOGSUMM 1 2025-08-19 1.1.12 "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l @@ -63,22 +63,25 @@ .SH NAME pflogsumm \- Produce Postfix MTA logfile summary .PP -Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 +Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.12 .SH SYNOPSIS .IX Header "SYNOPSIS" .Vb 10 -\& pflogsumm \-[eq] [\-d ] [\-\-detail ] -\& [\-\-bounce\-detail ] [\-\-colwidth ] [\-\-deferral\-detail ] -\& [\-h ] [\-i|\-\-ignore\-case] [\-\-iso\-date\-time] [\-\-mailq] -\& [\-m|\-\-uucp\-mung] [\-\-no\-no\-msg\-size] [\-\-problems\-first] -\& [\-\-pscrn\-detail [cnt] [\-\-pscrn\-stats] [\-\-rej\-add\-from] [\-\-rej\-add\-to] -\& [\-\-reject\-detail ] [\-\-smtp\-detail ] [\-\-smtpd\-stats] -\& [\-\-smtpd\-warning\-detail ] [\-\-srs\-mung] [\-\-syslog\-name=string] -\& [\-u ] [\-\-unprocd ] [\-\-use\-orig\-to] -\& [\-\-verbose\-msg\-detail] [\-\-verp\-mung[=] [\-x] [\-\-zero\-fill] -\& [file1 [filen]] -\& -\& pflogsumm \-[help|version] +\& pflogsumm [\-\-config ] [\-\-bounce\-detail ] +\& [\-\-colwidth ] [\-\-deferral\-detail ] [\-\-detail ] +\& [\-d ] [\-\-dow0mon] [\-e] [\-h ] [\-i] +\& [\-\-iso\-date\-time] [\-\-mailq] [\-m] [\-\-no\-no\-msg\-size] +\& [\-\-problems\-first] [\-\-pscrn\-detail ] [\-\-pscrn\-stats] +\& [\-q] [\-\-rej\-add\-from] [\-\-rej\-add\-to] [\-\-reject\-detail ] +\& [\-\-smtp\-detail ] [\-\-smtpd\-stats] [\-\-smtpd\-warning\-detail ] +\& [\-\-srs\-mung] [\-\-syslog\-name=string] [\-u ] +\& [\-\-unprocd\-file ] [\-\-use\-orig\-to] [\-\-verbose\-msg\-detail] +\& [\-\-verp\-mung[=]] [\-x] [\-\-zero\-fill] [file1 [filen]] +\& +\& pflogsumm \-\-[dump\-config|help|version] +\& +\& Note: Where both long\- and short\-form options exist only the +\& latter are shown above. See man page for long\-form equivalents. \& \& If no file(s) specified, reads from stdin. Output is to stdout. Errors \& and debug to stderr. @@ -103,15 +106,80 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& Limit detailed bounce reports to the top . 0 \& to suppress entirely. \& -\& \-\-colwidth +\& \-\-config +\& +\& Path to a configuration file containing pflogsumm +\& options. +\& +\& Supports all standard command\-line options (without the +\& leading "\-" or "\-\-"). Options like "config", "dump\-config", +\& "help", and "version" technically work here, too, though +\& they\*(Aqre not particularly useful in this context. +\& +\& Command\-line arguments override config file values except +\& for boolean options. \& +\& \-\-colwidth \& Maximum report output width. Default is 80 columns. \& 0 = unlimited. \& \& N.B.: \-\-verbose\-msg\-detail overrides \& -\& \-d today generate report for just today -\& \-d yesterday generate report for just "yesterday" +\& \-d +\& \-\-date\-range +\& +\& Limits the report to the specified date or range. +\& +\& Accepted values: +\& +\& today +\& yesterday +\& "this week" / "last week" +\& "this month" / "last month" +\& YYYY\-MM[\-DD] +\& "YYYY\-MM[\-DD] YYYY\-MM[\-DD]" +\& +\& These options do what they suggest, with one +\& important caveat: +\& +\& ISO 8601 / RFC 3339\-style dates and ranges may +\& not yield accurate results when used with +\& traditional log formats lacking year information +\& ("month day\-of\-month"). +\& +\& In such cases, pflogsumm assumes log entries +\& are from the current year. For example, if the +\& current month is April and a log contains "Apr +\& NN" entries from the previous year, they will +\& be interpreted as from the *current* April. +\& +\& As such, date\-based filtering is only reliable +\& for entries less than ~365 days old for +\& old\-/traditional\-style logfiles. +\& +\& Arguments containing spaces must be quoted! +\& +\& This/last week/month arguments can take underscores, +\& rather than spaces, to avoid quoting: E.g.: +\& +\& \-\-date\-range last_week +\& +\& ISO 8601/RFC 3339 date ranges may optionally use a +\& hyphen or the word "to" for readability. E.g.: +\& +\& "2025\-08\-01 to 2025\-08\-08" +\& +\& If an optional day (DD) is omitted, the range becomes +\& the full month. E.g.: +\& +\& 2025\-08 == 2025\-08\-01 through 2025\-08\-31 +\& +\& "2025\-07 \- 2025\-08" == 2025\-07\-01 \- 2025\-08\-31 +\& +\& \-\-dow0mon +\& First day of the week is Monday, rather than Sunday. +\& +\& (Used only for this/last week calculations.) \& \& \-\-deferral\-detail \& @@ -119,13 +187,26 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& to suppress entirely. \& \& \-\-detail -\& \& Sets all \-\-*\-detail, \-h and \-u to . Is \& over\-ridden by individual settings. \-\-detail 0 \& suppresses *all* detail. \& -\& \-e extended (extreme? excessive?) detail +\& \-\-dump\-config +\& Dump the config to STDOUT and exit. +\& +\& This can be used as both a debugging aid and as a way +\& to develop your first config file. For the latter: +\& Simply run your usual pflogsumm command line, adding +\& \-\-dump\-config to it, and redirect STDOUT to a file. +\& +\& To make it cleaner: Remove unset configs: +\& +\& pflogsumm \-\-dump\-config |grep \-v \*(Aq = $\*(Aq \& +\& \-e +\& \-\-extended\-detail +\& +\& Extended (extreme? excessive?) detail \& Emit detailed reports. At present, this includes \& only a per\-message report, sorted by sender domain, \& then user\-in\-domain, then by queue i.d. @@ -134,8 +215,10 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& quickly consume very large amounts of memory if a \& lot of log entries are processed! \& -\& \-h top to display in host/domain reports. -\& +\& \-h +\& \-\-host\-cnt +\& +\& top to display in host/domain reports. \& 0 = none. \& \& See also: "\-u" and "\-\-*\-detail" options for further @@ -148,7 +231,8 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& know: lame.) \& \& \-i -\& \-\-ignore\-case Handle complete email address in a case\-insensitive +\& \-\-ignore\-case +\& Handle complete email address in a case\-insensitive \& manner. \& \& Normally pflogsumm lower\-cases only the host and @@ -163,7 +247,6 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& rather than "Mon DD CCYY" and "HHMM". \& \& \-m modify (mung?) UUCP\-style bang\-paths -\& \-\-uucp\-mung \& \& This is for use when you have a mix of Internet\-style \& domain addresses and UUCP\-style bang\-paths in the log. @@ -177,6 +260,8 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& extended detail report (\-e), to help ensure that by\- \& domain\-by\-name sorting is more accurate. \& +\& See also: \-\-uucp\-mung +\& \& \-\-mailq Run "mailq" command at end of report. \& \& Merely a convenience feature. (Assumes that "mailq" @@ -201,11 +286,13 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& Emit "problems" reports (bounces, defers, warnings, \& etc.) before "normal" stats. \& -\& \-\-pscrn\-detail [cnt] -\& Emit postscreen detail. +\& \-\-pscrn\-detail \& -\& If the optional cnt is included: Limits postscreen detail -\& reports to the top cnt. +\& Limit postscreen detail reporting to top lines of +\& each event. 0 to suppress entirely. +\& +\& Note: Postscreen rejects are collected and reported +\& in any event. \& \& \-\-pscrn\-stats \& Collect and emit postscreen summary stats. @@ -219,13 +306,14 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& each listing. (Does not apply to "Improper use of \& SMTP command pipelining" report.) \& -\& \-q quiet \- don\*(Aqt print headings for empty reports +\& \-q +\& \-\-quiet +\& quiet \- don\*(Aqt print headings for empty reports \& \& note: headings for warning, fatal, and "master" \& messages will always be printed. \& \& \-\-rej\-add\-to -\& \& For sender reject reports: Add the intended recipient \& address. \& @@ -240,7 +328,6 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& 0 to suppress entirely. \& \& \-\-smtpd\-stats -\& \& Generate smtpd connection statistics. \& \& The "per\-day" report is not generated for single\-day @@ -253,7 +340,6 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& 0 to suppress entirely. \& \& \-\-srs\-mung -\& \& Undo SRS address munging. \& \& If your postfix install has an SRS plugin running, many @@ -282,12 +368,15 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& See the discussion about the use of this option under \& "NOTES," below. \& -\& \-u top to display in user reports. 0 == none. +\& \-u +\& \-\-user\-cnt \& -\& See also: "\-h" and "\-\-*\-detail" options for further -\& report\-limiting options. +\& top to display in user reports. 0 == none. +\& +\& See also: "\-h" and "\-\-*\-detail" options for further +\& report\-limiting options. \& -\& \-\-unprocd +\& \-\-unprocd\-file \& \& Emit unprocessed logfile lines to file \& @@ -296,6 +385,11 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& Where "orig_to" fields are found, report that in place \& of the "to" address. \& +\& \-\-uucp\-mung +\& modify (mung?) UUCP\-style bang\-paths +\& +\& See also: \-m +\& \& \-\-verbose\-msg\-detail \& \& For the message deferral, bounce and reject summaries: @@ -349,30 +443,48 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 \& \& pflogsumm \-d yesterday /var/log/maillog \& -\& A report of prior week\*(Aqs activities (after logs rotated): +\& A report of prior week\*(Aqs activities: \& -\& pflogsumm /var/log/maillog.0 +\& pflogsumm \-d last_week /var/log/maillog.0 \& \& What\*(Aqs happened so far today: \& \& pflogsumm \-d today /var/log/maillog \& \& Crontab entry to generate a report of the previous day\*(Aqs activity -\& at 10 minutes after midnight. +\& at 10 minutes after midnight: \& \& 10 0 * * * /usr/local/sbin/pflogsumm \-d yesterday /var/log/maillog -\& 2>&1 |/usr/bin/mailx \-s "\`uname \-n\` daily mail stats" postmaster +\& 2>&1 |/usr/bin/mailx \-s "\`uname \-n\` daily mail stats" postmaster \& \& Crontab entry to generate a report for the prior week\*(Aqs activity. -\& (This example assumes one rotates ones mail logs weekly, some time -\& before 4:10 a.m. on Sunday.) \& -\& 10 4 * * 0 /usr/local/sbin/pflogsumm /var/log/maillog.0 -\& 2>&1 |/usr/bin/mailx \-s "\`uname \-n\` weekly mail stats" postmaster +\& 10 4 * * 0 /usr/local/sbin/pflogsumm \-d "last week" /var/log/maillog.0 +\& 2>&1 |/usr/bin/mailx \-s "\`uname \-n\` weekly mail stats" postmaster +\& +\& (The two crontab examples, above, must actually be a single line +\& each. They\*(Aqre broken\-up into two\-or\-more lines due to page +\& formatting issues.) +\& +\& Using a config file: +\& +\& pflogsumm \-\-config /usr/local/etc/pflogusmm/daily.conf +\& +\& Using a config file, overriding a config file options on the command +\& line: \& -\& The two crontab examples, above, must actually be a single line -\& each. They\*(Aqre broken\-up into two\-or\-more lines due to page -\& formatting issues. +\& pflogsumm \-\-config /usr/local/etc/pflogsumm/daily.conf +\& \-\-detail 30 +\& +\& This would override *all* detail settings in the config +\& file, setting them all to 30. +\& +\& pflogsumm \-\-config /usr/local/etc/pflogsumm/daily.conf +\& \-\-detail 30 \-\-host\-cnt 10 +\& +\& This would override all detail settings in the config +\& file, setting them all to 30, with the global detail +\& setting in turn being overridden to 10 for host count. .Ve .SH "SEE ALSO" .IX Header "SEE ALSO" @@ -383,7 +495,12 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 .Ve .SH NOTES .IX Header "NOTES" -.Vb 3 +.Vb 4 +\& Some options, such as date range, have both short\-form and +\& long\-form names. In the interest of brevity, only the +\& short\-form options are shown in the SYNOPSIS and in +\& pflogsumm\*(Aqs "help" output. +\& \& Pflogsumm makes no attempt to catch/parse non\-Postfix log \& entries. Unless it has "postfix/" in the log entry, it will be \& ignored. @@ -462,11 +579,12 @@ Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.11 .SH REQUIREMENTS .IX Header "REQUIREMENTS" .Vb 1 -\& Requires Perl 5.10, minimum +\& Requires Perl 5.10, minimum, and Date::Calc +\& +\& For \-\-config, Pflogsumm requires the Config::Simple module. \& -\& For certain options (e.g.: \-\-smtpd\-stats), Pflogsumm requires the -\& Date::Calc module, which can be obtained from CPAN at -\& http://www.perl.com. +\& Both of the above can be obtained from CPAN at http://www.perl.com +\& or from your distro\*(Aqs repository. \& \& Pflogsumm is currently written and tested under Perl 5.38. \& As of version 19990413\-02, pflogsumm worked with Perl 5.003, but diff --git a/pftobyfrom.1 b/pftobyfrom.1 index 64659ed..2d991fa 100644 --- a/pftobyfrom.1 +++ b/pftobyfrom.1 @@ -55,7 +55,7 @@ .\" ======================================================================== .\" .IX Title "PFTOBYFROM 1" -.TH PFTOBYFROM 1 2025-05-22 1.1.11 "User Contributed Perl Documentation" +.TH PFTOBYFROM 1 2025-05-22 1.1.12 "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l -- 2.39.5