1 ChangeLog for pflogsumm.pl
4 [Note: Let me know if you would like to be notified as new versions
5 are released. The latest released version can always be found at
6 http://jimsun.LinxNet.com/postfix_contrib.html.]
11 Fixed RFC 3339 support. Releases 1.1.3 and 1.1.4 were badly broken
12 in this respect. Thanks and a tip o' the hat to Sven Hoexter
13 (sven-at-timegate-dot-de) for the help.
17 Modified for compatibility with -o syslog_name=blurfl/submission
18 and -o syslog_name=blurfl/smtps set in master.cf. (These are the
19 defaults in Postfix 2.9 and beyond.)
21 N.B.: This doesn't mean you'll get submission and
22 smtps broken-out separately from plain old smtp,
23 it simply means the presence of the new sub-strings
24 won't break Pflogsumm.
26 Changed "_"s (underscores) in option switches to "-"s (dashes).
27 (Underscores are still accepted.) Thanks and a tip o' the hat
28 to David Landgren (david-at-landgren-dot-net) for the suggestion.
30 Removed switches deprecated in 1.1.3 from the docs.
32 Improved ISO timestamp parsing to account for optional fractional
33 seconds part. (This is thrown-away by Pflogsumm.)
35 Minor updates to the FAQ.
37 Replaced "depreciated" with "deprecated" throughout. Thanks and
38 a tip o' the hat to Rob Arista for the heads-up.
40 Fixed bug in host normalization function that was broken for
45 Added long-awaited switches to optionally reduce detail reporting:
46 --bounce_detail=N, --deferral_detail=N, --reject_detail=N,
47 --smtp_detail=N, smtpd_warning_detail=N, and --detail=N. Setting
48 any of them to 0 suppresses that detail entirely. --detail=N sets
49 the default for all of them, as well as for -u=N and -h=N.
51 With the above enhancements, the following switches are deprecated,
52 and will eventually be removed: --no_bounce_detail,
53 --no_deferral_detail, --no_reject_detail and --no_smtpd_warnings.
54 They are replaced by setting the desired --*_detail=0. They still
55 work, but using them generates a warning.
57 Added support for parsing logs with RFC 3339 timestamps. Thanks
58 and a tip o' the hat to sftf-at-yandrex-dot-ru for the heads-up
59 and the code contribution. (N.B.: My code does not require a
60 command-line switch. The format is detected automatically.)
62 Fixed some --ignore-case inconsistincies. Thanks and a tip o'
63 the hat to Richard Blanchet (richard-dot-blanchet-at-free-dot-fr)
64 for the heads-up and the diff.
66 Fixed parsing bug that resulted in attempts to treat
67 kind-of-IPv4-looking strings as IPv4 addresses. (I really need to
68 improve reject/defer/etc. "reason" parsing to fix this properly.)
69 Thanks to Joseph Vit (jvit-at-certicon-dot-cz) for the bug
74 Fixed bug with calculating yesterday's date in vicinity of DST
75 changes. (Thanks and a tip o' the hat to Wieland Chmielewski
76 for bringing the problem to my attention.)
78 Added missing "underlining" to some (sub-)section titles for
84 Fixed to parse Postfix-2.3 (and beyond) logfiles. Thanks to
85 whomever contributed to
87 http://bugs.gentoo.org/show_bug.cgi?id=144236
89 Removed support for vmailer.
91 Removed "SMTPD_STATS_SUPPORT" "fences" in code in favour of code
92 to automatically detect the availability of Date::Calc. If
93 --smtpd_stats is specified and Date::Calc is not installed, now
94 bails-out with friendly message. (Adapted from suggestion and
95 examples provided by David Landgren <david-at-landgren-dot-net>.
98 Removed rem_smtpd_stats_supp.pl utility from distribution. (No
101 Memory footprint improvement: Pflogsumm no longer stores data for
102 reports that are supressed via --no_<mumble> switches.
104 Removed extraneous arguments in two calls to print_nested_hash
105 that would result in the "quiet" flag being ignored. Thanks to
106 Pavel Urban (pupu-at-pupu-dot-cz) for bringing that to my
109 Added notes to FAQ about translations and i18n, about mismatching
110 "received"/"delivered" counts, about bug in calculating "yesterday,"
111 and about John Fawcett's "prepflog."
116 Promoted 1.0.18 (Beta) to "production/stable" version release.
121 Fixed reject parsing for "DATA" smtpd rejects.
126 Fixed reject parsing to properly recognize bare "User unknown".
127 (Thanks to J.D. Bronson" <jeff_bronson-at-wixb-dot-com> for the
128 bug-report and sample logfile lines.)
133 Re-worked "to" and "from" field parsing in reject report handling to
134 make it more robust in pathological cases. (Thanks to Paul Brooks
135 <paul-dot-brooks-at-metro1-dot-com> and Lars Hecking
136 <lhecking-at-nmrc-dot-ie> for the bug-reports and sample logfile
139 Fixed warnings resulting from non-standard, extraneous syslog input.
140 (Thanks to Mathias Behrle <m123-at-arcor-dot-de> for the report.)
142 Fixed reject parsing to account for really atrocious garbage in
143 HELO strings, sender addresses and recipient addresses. (Thanks to
144 Lars Hecking <lhecking-at-nmrc-dot-ie> for the bug-report and sample
147 Fixed reject parsing to properly recognize "CONNECT" smtpd rejects.
148 (Thanks to Mike Vanecek <postfix_list-at-mm-vanecek-dot-cc> for the
151 Fixed reject parsing to properly recognize "User unknown in relay
152 recipient table." (Thanks to Lars Hecking <lhecking-at-nmrc-dot-ie>
153 for the bug-report and sample logfile lines.)
155 Some code optimization resulting in 3-5% performance improvement.
160 Pflogsumm *should* now properly parse and handle log entries with
161 IPv6 addresses in them. (Adapted from idea and code submitted by
162 Stefan `Sec` Zehl <sec-at-42-dot-org>.)
164 Fixed "User unknown in local recipient table" reject reports to
165 show target recipient address, rather then sending domain, to be
166 consistent with other "recipient" reports. (Thanks to WC Jones
167 <sx-at-insecurity-dot-org> for the suggestion.)
169 Fixed parsing of "Recipient address rejected" for recipient
170 address verification (RAV) rejects. (Thanks to Len Conrad
171 <LConrad-at-Go2France-dot-com> for the suggestion.)
173 FAQ additions regarding recommendations on how to format custom
174 reject reports for "best" results in Pflogsumm's output and note
175 regarding "non-standard" syslogd's.
180 Fixed bug in parsing for "Host/Domain Summary: Messages Received"
181 report improvement (rel-1.0.13) that resulted from (unexpected, to
184 ... postfix/smtpd[31430]: E02DDB04E: client=blurfl[1.2.3.4],
185 sasl_method=LOGIN, sasl_username=phred
189 The "Host/Domain Summary: Messages Received" report would show simply
190 "from=<>", for the host/domain, for postmaster bounces. Pflogsumm now
191 substitutes the client hostname or IP address for these, unless it's
192 from the pickup daemon, in which case "from=<>" is retained. (Note
193 that "Senders by message count/size" reports are unaffected by this
196 "Senders by message count" and "Recipients by message count" reports
197 are now secondarily sorted by domain, host and user parts. (As a
198 side-effect: So are "Senders by message size" and "Recipients by
199 message size" but, being as the odds are against numerous senders and
200 recipients having the same total message sizes, this change hasn't much
205 Rejects, warns, etc. now print sub-category totals. E.g.:
207 message reject detail
208 ---------------------
210 Relay access denied (total: 6)
212 (Adapted from idea and code submitted by blake7-at-blake7-dot-org.)
214 Reject, warning, etc. reports are now sorted by 2nd column (e.g.: IP
215 address, domain, etc.) within count. (Adapted from idea and code
216 submitted by David Landgren <david-at-landgren-dot-net>.)
218 Added --no_smtpd_warnings (report) option.
220 Added --no_no_msg_size (report) option.
222 A couple of minor improvements to reject parsing/reporting.
226 This is a bug-fix release.
228 There was a problem in the way pflogsumm-1.0.8 through 1.0.10
229 handled the --syslog_name option: When --syslog_name was
230 specified, some log entries with the default "postfix" name would
231 be missed. This revision may introduce incompatibilities if
232 you're logging two or more instances of Postfix to the same log.
233 See the docs included in the tarball for details.
237 Re-worked "% rejected" calculation to include messages discarded
238 and added "% discarded" calculation/display.
242 Bugfix: If Perl's -w is specified at run-time and there were no
243 messages delivered or rejected, uninitialized variable warnings
244 would be issued in the percent rejected calculation code. Thanks
245 for Larry Hansford (and many others since!) for the bug report.
249 Bugfix: Fixed problem with "orig_to=<blurfl>" being parsed as
250 "to=<blurfl>". This resulted in *very* wrong output. Thanks to
251 Bjorn Swift for the report.
253 Added "% rejected" to Grand Totals "rejected" figure. This is
254 calculated as: rejected / (delivered + rejected). (I did this
255 purely because it amuses me.)
257 Bugfix: Fix, in reject processing, for truncated overly-long "to"
258 fields. Thanks to Rick Troxel for reporting the problem.
260 Added --syslog_name option. Thanks to Ben Rosengart for the
265 Corrected and improved message reject/reject warn/hold/discard
266 parsing. Again. (Thanks to Peter Santiago for reporting the
267 problem that initiated these improvements.)
271 Added support for reporting message reject warnings, holds and
274 Note: Message rejects, reject warnings, holds and discards
275 are all reported under the "rejects" column for the Per-Hour
276 and Per-Day traffic summaries.
278 More aggressive verp munging (again). (Prompted, in part, by a
279 suggestion from Casey Peel. Thanks!)
281 Verp munging now applied to sender addresses in smtpd reject
284 WARNING: Please note that verp munging is highly experimental!
286 Pflogsumm distribution changed to gzip'd tarball format.
288 Tightened-up parsing. Thanks for Ralf Hildebrandt for noting and
289 reporting the problem.
291 Docs at the top of pflogsumm.pl changed to POD format for automated
296 Automatically-generated manpage added.
298 "To Do" moved out of ChangeLog into separate file.
300 Package now includes convenience Perl script for removing smtpd
301 stats support for those who don't have Date::Calc, don't want to
302 install it and don't care about smtpd stats reporting.
304 Belated thanks to Len Conrad in regards to the Sender Address
305 Verification work in 1.0.5.
309 Fixed to parse smtpd rejects for Postfix versions as of 20021026.
310 (Retained compatibility with older versions of Postfix.)
312 Note: smtpd and header-/body-checks warn, hold and discard
313 messages are *not* currently parsed/reported. I'll need to
314 get some logfile entries.
316 Fixed parsing to handle the new "sender address verification"
319 Added "--zero_fill" option to put zeros in columns that might
320 otherwise be blank in some reports. (Suggestion by Matthias
323 Fixed "Message size exceeds fixed limit" parsing for reject
328 Added "--no_*_detail" options. (Suppresses some of the "detail"
331 Added "--version" option. (Thanks to "Guillaume")
333 Improved handling of "[ID nnnnnn some.thing]" stuff (Thanks to
336 Repaired and optimized some of the "reject" parsing.
338 Added processing and report of smtp delivery failures.
340 Added --rej_add_from option: For those reject reports that list
341 IP addresses or host/domain names: append the email from address
342 to each listing. (Note: does not apply to "Improper use of SMTP
343 command pipelining" report.)
348 Minor re-work of "reject: RCPT" parsing to account for Yet Another
349 Change in the logfile format. (Semi-colon changed to a comma in
350 "blocked using rbl.maps.vix.com,".)
355 Took another whack at "verp" munging. *sigh*
357 Added code to summarize "Improper use of SMTP command pipelining"
363 Modified to catch "reject: header" log entries changed as of
364 postfix release-20010228 (?). Prior versions of postfix had the
365 string "warning: " (where the qid normally is). Thanks to Glen
366 Eustace <root@godzone.net.nz>, Len Conrad
367 <lconrad@go2france.com>, Daniel Roesen
368 <droesen@entire-systems.com>, Milivoj Ivkovic <mi@alma.ch> and
369 j_zuilkowski@hotmail.com (Jon Zuilkowski) for reports and/or
372 Fixed a couple of "uninitialized variable" problems.
374 Committed (actually starting with 20000925-01beta) to CVS.
379 Added a line to compensate for (new?) "[ID nnnnnn some.thing]"
380 sub-strings that appear in logfile entries under Sun Solaris 8.
383 Note: Upon being committed to CVS, this became rel-0.9.0.
388 Forgot to add "--problems_first" to the "usage" output and in the
389 synopsis at the top of the comments.
394 Re-did what 20000907-02beta was *supposed* to be! To wit:
395 replaced missing "--ignore_case" bugfix, "panic" entry processing,
396 improvements to "fatal" and "warning" message reporting and
397 missing "--mailq" option. (Obviously: 20000907-02beta was
398 derived from the wrong code base.)
403 Fixed bug in ISO date formatting that caused the month to be off
404 by one. Thanks to Kurt Andersen <kurta@sitefs1a.spk.agilent.com>
405 for the report and the patch.
407 Fixed overflow of connect time reporting into days. (Can happen
408 during weekly summaries for sites with large volumes of email.)
409 Thanks again to Kurt Andersen <kurta@sitefs1a.spk.agilent.com>
410 for the report and the fix.
412 Improved "rejects" reporting *again*. Thanks to Thomas Parmelan
413 <tom@proxad.net> for the patch.
415 Added "--problems_first" option to print "problem" reports such as
416 bounces, defers, warnings, fatal errors, etc. before "normal"
422 Fixed bug in code that prevented "--ignore_case" from actually
423 doing anything. Thanks to Nadeem Hasan <nhasan@usa.net> for
424 reporting this and supplying the fix.
429 Added the following caveat to the "Notes" section of Pflogsumm:
431 -------------------------------------------------------------
432 IMPORTANT: Pflogsumm makes no attempt to catch/parse non-
433 postfix/vmailer daemon log entries. (I.e.: Unless
434 it has "postfix/" or "vmailer/" in the log entry,
436 -------------------------------------------------------------
438 Added reporting of "panic" log messages. This was missed until
441 Increased reporting detail of "fatal" and "warning" entries.
442 (Actually, "warning" detail was increased in 19991120-01beta.
443 Neglected to note it then.)
446 19991123-01 (unreleased)
448 Added "--mailq" option. (Convenience factor.) Runs Postfix's
449 "mailq" command at the end of the other reports.
451 -------------------------------------------------------
452 NOTE: If Postfix's "mailq" command isn't in your $PATH,
453 you'll have to edit the "$mailqCmd" variable located
454 near the top of pflogsumm to path it explicitly.
455 -------------------------------------------------------
460 Tried once again to improve parsing of reject log entries.
461 Specifically: those associated with "RCPT" rejects.
464 19991016-01 (not generally released)
466 Added --smtpd_stats. Generates smtpd connection statistics.
468 ---------------------------------------------------------------
469 NOTE: Support for --smtpd_stats requires the Date::Calc module
470 (available from CPAN). If you don't want to go to the trouble
471 of fetching & installing that module, and you don't want smtpd
472 stats anyway, *carefully* identify all of the code sections
473 delimited by "# ---Begin: SMTPD_STATS_SUPPORT---" and
474 "# ---End: SMTPD_STATS_SUPPORT---" and remove them.
475 ---------------------------------------------------------------
478 19990909-01 (not generally released)
480 Added -i and --ignore_case options. Causes entire email address
481 to be lower-cased instead of just the host/domain part.
483 Added "use locale". (This means that the sorting order within
484 reports may be different from before--depending on how you have
485 your machine's locale set.)
490 Improved "reason" parsing and reporting for bounced and deferred
493 Added parsing of "cleanup" reject lines to catch PCRE/regexp
496 Added "reject" stats to per-hour and (on multi-day reports) per-
499 Improved "warnings" report to show details.
501 A single message deferred multiple times showed up as multiple
502 deferrals--implying that multiple messages were deferred. Now
503 shows "how many messages were deferred" and "how many deferrals"
506 Changed display of "Grand Totals" to make it a bit more readable
509 Added "automatic perl finder" line for those systems that don't
510 support the "#!" notation.
512 By popular demand: added note to comments as to where pflogsumm
513 home page could be found :-).
518 Fixed problem with last octet of IP address getting truncated in
519 reports when IP address used in place of unknown hosts.
521 Changed the way a few internal variables were handled to be
522 compatible with Perl 5.003. Don't run it under Perl 5.003 with the
523 "-w" perl switch, tho! It will issue lots of warnings. All tests
524 I performed indicated that it produces the correct output, however.
526 ------------------------------------------------------------
527 NOTE: While this version was tested to work with Perl 5.003,
528 I recommend that you upgrade to 5.004 or later. I will not
529 guarantee that I'll remember to do the full regression-
530 testing that I usually do with 5.003 as well.
531 ------------------------------------------------------------
536 NOTICE: As of this version of pflogsumm.pl, the "-c" switch is
537 GONE! (As per the previous notice.)
539 Added "--help" option to emit short usage message and bail out.
541 Added "--iso_date_time" switch to change displays of dates and times
542 to ISO 8601 standard formats (CCYY-MM-DD and HH:MM), rather than
543 "Month-name Day-number CCYY" and "HHMM" formats (the default).
545 Added "--verbose_msg_detail" switch. This causes the full "reason"
546 to be displayed for the message deferral, bounce and reject summaries.
547 (Note: this can result in quite long lines in the report. Also note
548 that there have been a couple of subtle changes in the "reason"
549 parsing/reporting in the default mode (no "--verbose_msg_detail".)
551 Added "--verp_mung" option. The problem this addresses is "VERP"
552 generated (??? so far as I can tell!) addresses (?) of the form:
554 "list-return-NN-someuser=some.dom@host.sender.dom"
556 These result in mail from the same "user" and site to look like it
557 originated from different users, when in fact it originates from the
558 same "user." There are presently two "levels" of address munging
559 available. With no numeric argument (or any value less than 2), the
560 above address will be converted to:
562 "list-return-ID-someuser=some.dom@host.sender.dom"
564 In other words: the numeric value will be replaced with "ID".
566 By specifying "--verp_mung=2", the munging is more "aggressive",
567 converting the above address to something like:
569 "list@host.sender.dom"
571 Which looks more "normal."
573 (Actually: specifying anything less than 2 does the "simple" munging
574 and anything greater than 1 results in the more "aggressive" hack
577 Added "--uucp_mung" switch for consistence with "--verp_mung".
582 NOTICE: As of this version of pflogsumm.pl, versions of VMailer
583 prior to 19981023 are no longer supported. Sorry.
584 Pflogsumm-19990121-01.pl will be made permanently
585 available from now on for those with out-of-date versions
586 of VMailer prior to 19981023.
588 NOTICE: As of this version of pflogsumm.pl, the "-c" switch is
589 DEPRECATED. This version is transitional and retains it.
590 The next version will not have it. Subsequent versions
591 may re-use it for another purpose. Use the "-h" and "-u"
594 Added "-h" and "-u" switches to provide finer-grained control over
595 report output. Deprecated "-c".
597 Added "deferred" and "bounced" to "Grand Totals", "by-day" and "by-
600 Added "by-host/domain" reports. For sent (delivered) and received
601 messages: lists message count, total size of messages and
602 host/domain. For delivered messages: also lists number of deferred
603 messages and average and maximum delivery time. Both reports sorted
604 by message count in descending order.
606 Grand totals also now list number of recipient and sender
609 Re-wrote "by-user" data collection storage to reduce memory consumption
612 Moved "credits" from pflogsumm.pl to this file.
617 Now accounts for forwarded messages.
619 Side-effects of the above:
621 . Total messages size now broken-out into total bytes received
622 and total bytes delivered.
623 . Count of forwarded messages now reported.
624 . Postfix-internally-generated messages (e.g.: Postmaster
625 notifications of bounces) are no longer counted as "received".
626 (They do, however, show up as "delivered".)
627 . Forwarded addresses no longer show up as "recipients" (just
628 as with aliases and mailing lists).
630 Note that "delivered" will exceed "received" when messages
631 are forwarded because of additional header lines.
636 Added processing for "reject" log entries.
638 Expanded detail of "deferred" and "bounced" log entries to include
644 Added "messages received/delivered by hour" and "messages
645 received/delivered by day" reports. See the "Notes" section in the
646 documentation for details on how these behave.
648 Broke-out total message count to "messages received" and "messages
651 (For the above two enhancements: "postfix/pickup" and "postfix/smtpd"
652 lines are now processed. They used to be discarded.)
654 Renamed "summary" report to "Grand Totals".
656 Added code to parse date & time stamps from log entries. This was
657 needed, in part, for the "messages per-hour/day" reports. It would
658 have been necessary for future enhancements in the way of date- &
659 time-based processing anyway.
661 Added "Notes" section to docs at top of code.
666 Improved display of large integer values.
671 Bugfix only. Data for "extended detail" listing was being built
672 even if "-e" not specified. This resulted in unexpected excessive
673 memory consumption when crunching large amounts of data.
675 Added warning about memory consumption when "-e" option specified.
680 Further improvement to "accuracy" of by-domain-then-logname sort.
681 (Presently used only by "extended detail" listing). For comparison
682 purposes: mungs "machine(s).host.dom" into "host.dom.machine(s)" so
683 sort is keyed on "base" domain name before machines within the
684 domain. Does *not* attempt to reverse the order of the "machine(s)" -
685 so within a particular "base" domain, may not come out in quite the
686 right order. ("foo.bar.some.dom" will come out before
687 "sales.acme.some.dom", for example.)
688 Also works for 2x2-style domain names. (I.e.: "some.do.co")
691 19990102-01 (never released)
693 Added "mung UUCP-style bang-paths" switch (-m).
695 Improved performance and "accuracy" of by-domain-then-logname sort
696 used by (only at present) "extended detail" listing.
701 Added "extended detail" option (-e). At present this includes only a
702 per-message detail listing, which lists per-message detail sorted by
703 sender domain, then sender username, then by queue i.d.
710 Replaced warning message when message size unavailable in favor of
711 producing a report of these, sorted by queue i.d. Unlike the other
712 reports, this report header is not emitted at all if there are none of
713 these. (Always acts as if the -q switch had been specified).
718 Added experimental code to lower-case all domain names so that
719 "user@foo.dom" and "user@FOO.DOM" will come out the same.
721 Added test for existence of message size value when "to=" records are
722 being processed. This was necessary for cases in which the logfile
723 entry containing the "status=sent" record is not processed at the same
724 time as the logfile containing the "size=nnnn" record. Note that this
725 will produce a summary that will show recipient counts without
726 matching recipient sizes. The only way to cure this would be to
727 create a separate disk file to "memorize" message sizes. (Which would
728 introduce a whole new raft of problems.)
730 Added warning message (emitted to stderr) when the situation above is
733 Fixed "usage" message to indicate you can specify files on command
736 Wrapped a couple of long lines in the comments and code.
738 Added (temporary) version numbering scheme.
742 Other changes/enhancements since previous un-version-numbered
743 versions: deals with log entries for VMailer as well as Postfix, more
744 robust parsing of "to=" and "from=" fields (now handles spaces in
745 these), eliminated double-counting of message sizes (happened when
746 delivery was deferred), re-structured parsing to be more robust (not-
747 to-mention correct!), added "grand summary" report at top (total
748 messages, total size, number of senders and recipients).
753 [Note: The credits reflect suggestions and code contributions that
754 have actually been added. If your contribution doesn't appear
755 here, it may simply mean that it hasn't been added yet. (In which
756 case it should be on the list above.) On the other hand: if I
757 failed to credit you for something that *has* been added, please
760 Paul D. Robertson <proberts@clark.net>
762 For much testing and patience and many good suggestions on
763 how pflogsumm could be improved.
765 Simon J Mudd <simon.mudd@alltrading.com>
767 For the following code contributions:
769 Add "deferred" and "bounced" to "by hour" reports.
770 (I also added these to "by day" reports and "Grand
773 "VERP" (?) address munger (less-agressive version)
775 Suggestion for "by domain" delivery delay report.
777 For the --smtpd_stats suggestion.
779 Anders Arnholm <anders@arnholm.nu>
781 For pointing out the problem with forwarded messages.
783 Walcir Fontanini <walcir@densis.fee.unicamp.br>
785 For pointers to changes to make for Perl 5.003 compatibility.
786 (Added to 19990413-02beta.) (Which I will *try* to keep in
789 Eric Cholet <cholet@logilune.com>
791 For the --ignore_case patch.
793 Kurt Andersen <kurta@sitefs1a.spk.agilent.com>
795 For the ISO date formatting month-off-by-one patch and the
796 connect time overflow fix.
798 Thomas Parmelan <tom@proxad.net>
800 For improved "rejects" reporting patch.
802 Glen Eustace <root@godzone.net.nz>
804 Patch to fix "reject: header" matching after Wietse changed