1 ChangeLog for pflogsumm.pl
4 [Note: Let me know if you would like to be notified as new versions
5 are released. The latest released version can always be found at
6 http://jimsun.LinxNet.com/postfix_contrib.html.]
11 Modified for compatibility with -o syslog_name=blurfl/submission
12 and -o syslog_name=blurfl/smtps set in master.cf. (These are the
13 defaults in Postfix 2.9 and beyond.)
15 N.B.: This doesn't mean you'll get submission and
16 smtps broken-out separately from plain old smtp,
17 it simply means the presence of the new sub-strings
18 won't break Pflogsumm.
20 Changed "_"s (underscores) in option switches to "-"s (dashes).
21 (Underscores are still accepted.) Thanks and a tip o' the hat
22 to David Landgren (david-at-landgren-dot-net) for the suggestion.
24 Removed switches deprecated in 1.1.3 from the docs.
26 Improved ISO timestamp parsing to account for optional fractional
27 seconds part. (This is thrown-away by Pflogsumm.)
29 Minor updates to the FAQ.
31 Replaced "depreciated" with "deprecated" throughout. Thanks and
32 a tip o' the hat to Rob Arista for the heads-up.
34 Fixed bug in host normalization function that was broken for
39 Added long-awaited switches to optionally reduce detail reporting:
40 --bounce_detail=N, --deferral_detail=N, --reject_detail=N,
41 --smtp_detail=N, smtpd_warning_detail=N, and --detail=N. Setting
42 any of them to 0 suppresses that detail entirely. --detail=N sets
43 the default for all of them, as well as for -u=N and -h=N.
45 With the above enhancements, the following switches are deprecated,
46 and will eventually be removed: --no_bounce_detail,
47 --no_deferral_detail, --no_reject_detail and --no_smtpd_warnings.
48 They are replaced by setting the desired --*_detail=0. They still
49 work, but using them generates a warning.
51 Added support for parsing logs with RFC 3339 timestamps. Thanks
52 and a tip o' the hat to sftf-at-yandrex-dot-ru for the heads-up
53 and the code contribution. (N.B.: My code does not require a
54 command-line switch. The format is detected automatically.)
56 Fixed some --ignore-case inconsistincies. Thanks and a tip o'
57 the hat to Richard Blanchet (richard-dot-blanchet-at-free-dot-fr)
58 for the heads-up and the diff.
60 Fixed parsing bug that resulted in attempts to treat
61 kind-of-IPv4-looking strings as IPv4 addresses. (I really need to
62 improve reject/defer/etc. "reason" parsing to fix this properly.)
63 Thanks to Joseph Vit (jvit-at-certicon-dot-cz) for the bug
68 Fixed bug with calculating yesterday's date in vicinity of DST
69 changes. (Thanks and a tip o' the hat to Wieland Chmielewski
70 for bringing the problem to my attention.)
72 Added missing "underlining" to some (sub-)section titles for
78 Fixed to parse Postfix-2.3 (and beyond) logfiles. Thanks to
79 whomever contributed to
81 http://bugs.gentoo.org/show_bug.cgi?id=144236
83 Removed support for vmailer.
85 Removed "SMTPD_STATS_SUPPORT" "fences" in code in favour of code
86 to automatically detect the availability of Date::Calc. If
87 --smtpd_stats is specified and Date::Calc is not installed, now
88 bails-out with friendly message. (Adapted from suggestion and
89 examples provided by David Landgren <david-at-landgren-dot-net>.
92 Removed rem_smtpd_stats_supp.pl utility from distribution. (No
95 Memory footprint improvement: Pflogsumm no longer stores data for
96 reports that are supressed via --no_<mumble> switches.
98 Removed extraneous arguments in two calls to print_nested_hash
99 that would result in the "quiet" flag being ignored. Thanks to
100 Pavel Urban (pupu-at-pupu-dot-cz) for bringing that to my
103 Added notes to FAQ about translations and i18n, about mismatching
104 "received"/"delivered" counts, about bug in calculating "yesterday,"
105 and about John Fawcett's "prepflog."
110 Promoted 1.0.18 (Beta) to "production/stable" version release.
115 Fixed reject parsing for "DATA" smtpd rejects.
120 Fixed reject parsing to properly recognize bare "User unknown".
121 (Thanks to J.D. Bronson" <jeff_bronson-at-wixb-dot-com> for the
122 bug-report and sample logfile lines.)
127 Re-worked "to" and "from" field parsing in reject report handling to
128 make it more robust in pathological cases. (Thanks to Paul Brooks
129 <paul-dot-brooks-at-metro1-dot-com> and Lars Hecking
130 <lhecking-at-nmrc-dot-ie> for the bug-reports and sample logfile
133 Fixed warnings resulting from non-standard, extraneous syslog input.
134 (Thanks to Mathias Behrle <m123-at-arcor-dot-de> for the report.)
136 Fixed reject parsing to account for really atrocious garbage in
137 HELO strings, sender addresses and recipient addresses. (Thanks to
138 Lars Hecking <lhecking-at-nmrc-dot-ie> for the bug-report and sample
141 Fixed reject parsing to properly recognize "CONNECT" smtpd rejects.
142 (Thanks to Mike Vanecek <postfix_list-at-mm-vanecek-dot-cc> for the
145 Fixed reject parsing to properly recognize "User unknown in relay
146 recipient table." (Thanks to Lars Hecking <lhecking-at-nmrc-dot-ie>
147 for the bug-report and sample logfile lines.)
149 Some code optimization resulting in 3-5% performance improvement.
154 Pflogsumm *should* now properly parse and handle log entries with
155 IPv6 addresses in them. (Adapted from idea and code submitted by
156 Stefan `Sec` Zehl <sec-at-42-dot-org>.)
158 Fixed "User unknown in local recipient table" reject reports to
159 show target recipient address, rather then sending domain, to be
160 consistent with other "recipient" reports. (Thanks to WC Jones
161 <sx-at-insecurity-dot-org> for the suggestion.)
163 Fixed parsing of "Recipient address rejected" for recipient
164 address verification (RAV) rejects. (Thanks to Len Conrad
165 <LConrad-at-Go2France-dot-com> for the suggestion.)
167 FAQ additions regarding recommendations on how to format custom
168 reject reports for "best" results in Pflogsumm's output and note
169 regarding "non-standard" syslogd's.
174 Fixed bug in parsing for "Host/Domain Summary: Messages Received"
175 report improvement (rel-1.0.13) that resulted from (unexpected, to
178 ... postfix/smtpd[31430]: E02DDB04E: client=blurfl[1.2.3.4],
179 sasl_method=LOGIN, sasl_username=phred
183 The "Host/Domain Summary: Messages Received" report would show simply
184 "from=<>", for the host/domain, for postmaster bounces. Pflogsumm now
185 substitutes the client hostname or IP address for these, unless it's
186 from the pickup daemon, in which case "from=<>" is retained. (Note
187 that "Senders by message count/size" reports are unaffected by this
190 "Senders by message count" and "Recipients by message count" reports
191 are now secondarily sorted by domain, host and user parts. (As a
192 side-effect: So are "Senders by message size" and "Recipients by
193 message size" but, being as the odds are against numerous senders and
194 recipients having the same total message sizes, this change hasn't much
199 Rejects, warns, etc. now print sub-category totals. E.g.:
201 message reject detail
202 ---------------------
204 Relay access denied (total: 6)
206 (Adapted from idea and code submitted by blake7-at-blake7-dot-org.)
208 Reject, warning, etc. reports are now sorted by 2nd column (e.g.: IP
209 address, domain, etc.) within count. (Adapted from idea and code
210 submitted by David Landgren <david-at-landgren-dot-net>.)
212 Added --no_smtpd_warnings (report) option.
214 Added --no_no_msg_size (report) option.
216 A couple of minor improvements to reject parsing/reporting.
220 This is a bug-fix release.
222 There was a problem in the way pflogsumm-1.0.8 through 1.0.10
223 handled the --syslog_name option: When --syslog_name was
224 specified, some log entries with the default "postfix" name would
225 be missed. This revision may introduce incompatibilities if
226 you're logging two or more instances of Postfix to the same log.
227 See the docs included in the tarball for details.
231 Re-worked "% rejected" calculation to include messages discarded
232 and added "% discarded" calculation/display.
236 Bugfix: If Perl's -w is specified at run-time and there were no
237 messages delivered or rejected, uninitialized variable warnings
238 would be issued in the percent rejected calculation code. Thanks
239 for Larry Hansford (and many others since!) for the bug report.
243 Bugfix: Fixed problem with "orig_to=<blurfl>" being parsed as
244 "to=<blurfl>". This resulted in *very* wrong output. Thanks to
245 Bjorn Swift for the report.
247 Added "% rejected" to Grand Totals "rejected" figure. This is
248 calculated as: rejected / (delivered + rejected). (I did this
249 purely because it amuses me.)
251 Bugfix: Fix, in reject processing, for truncated overly-long "to"
252 fields. Thanks to Rick Troxel for reporting the problem.
254 Added --syslog_name option. Thanks to Ben Rosengart for the
259 Corrected and improved message reject/reject warn/hold/discard
260 parsing. Again. (Thanks to Peter Santiago for reporting the
261 problem that initiated these improvements.)
265 Added support for reporting message reject warnings, holds and
268 Note: Message rejects, reject warnings, holds and discards
269 are all reported under the "rejects" column for the Per-Hour
270 and Per-Day traffic summaries.
272 More aggressive verp munging (again). (Prompted, in part, by a
273 suggestion from Casey Peel. Thanks!)
275 Verp munging now applied to sender addresses in smtpd reject
278 WARNING: Please note that verp munging is highly experimental!
280 Pflogsumm distribution changed to gzip'd tarball format.
282 Tightened-up parsing. Thanks for Ralf Hildebrandt for noting and
283 reporting the problem.
285 Docs at the top of pflogsumm.pl changed to POD format for automated
290 Automatically-generated manpage added.
292 "To Do" moved out of ChangeLog into separate file.
294 Package now includes convenience Perl script for removing smtpd
295 stats support for those who don't have Date::Calc, don't want to
296 install it and don't care about smtpd stats reporting.
298 Belated thanks to Len Conrad in regards to the Sender Address
299 Verification work in 1.0.5.
303 Fixed to parse smtpd rejects for Postfix versions as of 20021026.
304 (Retained compatibility with older versions of Postfix.)
306 Note: smtpd and header-/body-checks warn, hold and discard
307 messages are *not* currently parsed/reported. I'll need to
308 get some logfile entries.
310 Fixed parsing to handle the new "sender address verification"
313 Added "--zero_fill" option to put zeros in columns that might
314 otherwise be blank in some reports. (Suggestion by Matthias
317 Fixed "Message size exceeds fixed limit" parsing for reject
322 Added "--no_*_detail" options. (Suppresses some of the "detail"
325 Added "--version" option. (Thanks to "Guillaume")
327 Improved handling of "[ID nnnnnn some.thing]" stuff (Thanks to
330 Repaired and optimized some of the "reject" parsing.
332 Added processing and report of smtp delivery failures.
334 Added --rej_add_from option: For those reject reports that list
335 IP addresses or host/domain names: append the email from address
336 to each listing. (Note: does not apply to "Improper use of SMTP
337 command pipelining" report.)
342 Minor re-work of "reject: RCPT" parsing to account for Yet Another
343 Change in the logfile format. (Semi-colon changed to a comma in
344 "blocked using rbl.maps.vix.com,".)
349 Took another whack at "verp" munging. *sigh*
351 Added code to summarize "Improper use of SMTP command pipelining"
357 Modified to catch "reject: header" log entries changed as of
358 postfix release-20010228 (?). Prior versions of postfix had the
359 string "warning: " (where the qid normally is). Thanks to Glen
360 Eustace <root@godzone.net.nz>, Len Conrad
361 <lconrad@go2france.com>, Daniel Roesen
362 <droesen@entire-systems.com>, Milivoj Ivkovic <mi@alma.ch> and
363 j_zuilkowski@hotmail.com (Jon Zuilkowski) for reports and/or
366 Fixed a couple of "uninitialized variable" problems.
368 Committed (actually starting with 20000925-01beta) to CVS.
373 Added a line to compensate for (new?) "[ID nnnnnn some.thing]"
374 sub-strings that appear in logfile entries under Sun Solaris 8.
377 Note: Upon being committed to CVS, this became rel-0.9.0.
382 Forgot to add "--problems_first" to the "usage" output and in the
383 synopsis at the top of the comments.
388 Re-did what 20000907-02beta was *supposed* to be! To wit:
389 replaced missing "--ignore_case" bugfix, "panic" entry processing,
390 improvements to "fatal" and "warning" message reporting and
391 missing "--mailq" option. (Obviously: 20000907-02beta was
392 derived from the wrong code base.)
397 Fixed bug in ISO date formatting that caused the month to be off
398 by one. Thanks to Kurt Andersen <kurta@sitefs1a.spk.agilent.com>
399 for the report and the patch.
401 Fixed overflow of connect time reporting into days. (Can happen
402 during weekly summaries for sites with large volumes of email.)
403 Thanks again to Kurt Andersen <kurta@sitefs1a.spk.agilent.com>
404 for the report and the fix.
406 Improved "rejects" reporting *again*. Thanks to Thomas Parmelan
407 <tom@proxad.net> for the patch.
409 Added "--problems_first" option to print "problem" reports such as
410 bounces, defers, warnings, fatal errors, etc. before "normal"
416 Fixed bug in code that prevented "--ignore_case" from actually
417 doing anything. Thanks to Nadeem Hasan <nhasan@usa.net> for
418 reporting this and supplying the fix.
423 Added the following caveat to the "Notes" section of Pflogsumm:
425 -------------------------------------------------------------
426 IMPORTANT: Pflogsumm makes no attempt to catch/parse non-
427 postfix/vmailer daemon log entries. (I.e.: Unless
428 it has "postfix/" or "vmailer/" in the log entry,
430 -------------------------------------------------------------
432 Added reporting of "panic" log messages. This was missed until
435 Increased reporting detail of "fatal" and "warning" entries.
436 (Actually, "warning" detail was increased in 19991120-01beta.
437 Neglected to note it then.)
440 19991123-01 (unreleased)
442 Added "--mailq" option. (Convenience factor.) Runs Postfix's
443 "mailq" command at the end of the other reports.
445 -------------------------------------------------------
446 NOTE: If Postfix's "mailq" command isn't in your $PATH,
447 you'll have to edit the "$mailqCmd" variable located
448 near the top of pflogsumm to path it explicitly.
449 -------------------------------------------------------
454 Tried once again to improve parsing of reject log entries.
455 Specifically: those associated with "RCPT" rejects.
458 19991016-01 (not generally released)
460 Added --smtpd_stats. Generates smtpd connection statistics.
462 ---------------------------------------------------------------
463 NOTE: Support for --smtpd_stats requires the Date::Calc module
464 (available from CPAN). If you don't want to go to the trouble
465 of fetching & installing that module, and you don't want smtpd
466 stats anyway, *carefully* identify all of the code sections
467 delimited by "# ---Begin: SMTPD_STATS_SUPPORT---" and
468 "# ---End: SMTPD_STATS_SUPPORT---" and remove them.
469 ---------------------------------------------------------------
472 19990909-01 (not generally released)
474 Added -i and --ignore_case options. Causes entire email address
475 to be lower-cased instead of just the host/domain part.
477 Added "use locale". (This means that the sorting order within
478 reports may be different from before--depending on how you have
479 your machine's locale set.)
484 Improved "reason" parsing and reporting for bounced and deferred
487 Added parsing of "cleanup" reject lines to catch PCRE/regexp
490 Added "reject" stats to per-hour and (on multi-day reports) per-
493 Improved "warnings" report to show details.
495 A single message deferred multiple times showed up as multiple
496 deferrals--implying that multiple messages were deferred. Now
497 shows "how many messages were deferred" and "how many deferrals"
500 Changed display of "Grand Totals" to make it a bit more readable
503 Added "automatic perl finder" line for those systems that don't
504 support the "#!" notation.
506 By popular demand: added note to comments as to where pflogsumm
507 home page could be found :-).
512 Fixed problem with last octet of IP address getting truncated in
513 reports when IP address used in place of unknown hosts.
515 Changed the way a few internal variables were handled to be
516 compatible with Perl 5.003. Don't run it under Perl 5.003 with the
517 "-w" perl switch, tho! It will issue lots of warnings. All tests
518 I performed indicated that it produces the correct output, however.
520 ------------------------------------------------------------
521 NOTE: While this version was tested to work with Perl 5.003,
522 I recommend that you upgrade to 5.004 or later. I will not
523 guarantee that I'll remember to do the full regression-
524 testing that I usually do with 5.003 as well.
525 ------------------------------------------------------------
530 NOTICE: As of this version of pflogsumm.pl, the "-c" switch is
531 GONE! (As per the previous notice.)
533 Added "--help" option to emit short usage message and bail out.
535 Added "--iso_date_time" switch to change displays of dates and times
536 to ISO 8601 standard formats (CCYY-MM-DD and HH:MM), rather than
537 "Month-name Day-number CCYY" and "HHMM" formats (the default).
539 Added "--verbose_msg_detail" switch. This causes the full "reason"
540 to be displayed for the message deferral, bounce and reject summaries.
541 (Note: this can result in quite long lines in the report. Also note
542 that there have been a couple of subtle changes in the "reason"
543 parsing/reporting in the default mode (no "--verbose_msg_detail".)
545 Added "--verp_mung" option. The problem this addresses is "VERP"
546 generated (??? so far as I can tell!) addresses (?) of the form:
548 "list-return-NN-someuser=some.dom@host.sender.dom"
550 These result in mail from the same "user" and site to look like it
551 originated from different users, when in fact it originates from the
552 same "user." There are presently two "levels" of address munging
553 available. With no numeric argument (or any value less than 2), the
554 above address will be converted to:
556 "list-return-ID-someuser=some.dom@host.sender.dom"
558 In other words: the numeric value will be replaced with "ID".
560 By specifying "--verp_mung=2", the munging is more "aggressive",
561 converting the above address to something like:
563 "list@host.sender.dom"
565 Which looks more "normal."
567 (Actually: specifying anything less than 2 does the "simple" munging
568 and anything greater than 1 results in the more "aggressive" hack
571 Added "--uucp_mung" switch for consistence with "--verp_mung".
576 NOTICE: As of this version of pflogsumm.pl, versions of VMailer
577 prior to 19981023 are no longer supported. Sorry.
578 Pflogsumm-19990121-01.pl will be made permanently
579 available from now on for those with out-of-date versions
580 of VMailer prior to 19981023.
582 NOTICE: As of this version of pflogsumm.pl, the "-c" switch is
583 DEPRECATED. This version is transitional and retains it.
584 The next version will not have it. Subsequent versions
585 may re-use it for another purpose. Use the "-h" and "-u"
588 Added "-h" and "-u" switches to provide finer-grained control over
589 report output. Deprecated "-c".
591 Added "deferred" and "bounced" to "Grand Totals", "by-day" and "by-
594 Added "by-host/domain" reports. For sent (delivered) and received
595 messages: lists message count, total size of messages and
596 host/domain. For delivered messages: also lists number of deferred
597 messages and average and maximum delivery time. Both reports sorted
598 by message count in descending order.
600 Grand totals also now list number of recipient and sender
603 Re-wrote "by-user" data collection storage to reduce memory consumption
606 Moved "credits" from pflogsumm.pl to this file.
611 Now accounts for forwarded messages.
613 Side-effects of the above:
615 . Total messages size now broken-out into total bytes received
616 and total bytes delivered.
617 . Count of forwarded messages now reported.
618 . Postfix-internally-generated messages (e.g.: Postmaster
619 notifications of bounces) are no longer counted as "received".
620 (They do, however, show up as "delivered".)
621 . Forwarded addresses no longer show up as "recipients" (just
622 as with aliases and mailing lists).
624 Note that "delivered" will exceed "received" when messages
625 are forwarded because of additional header lines.
630 Added processing for "reject" log entries.
632 Expanded detail of "deferred" and "bounced" log entries to include
638 Added "messages received/delivered by hour" and "messages
639 received/delivered by day" reports. See the "Notes" section in the
640 documentation for details on how these behave.
642 Broke-out total message count to "messages received" and "messages
645 (For the above two enhancements: "postfix/pickup" and "postfix/smtpd"
646 lines are now processed. They used to be discarded.)
648 Renamed "summary" report to "Grand Totals".
650 Added code to parse date & time stamps from log entries. This was
651 needed, in part, for the "messages per-hour/day" reports. It would
652 have been necessary for future enhancements in the way of date- &
653 time-based processing anyway.
655 Added "Notes" section to docs at top of code.
660 Improved display of large integer values.
665 Bugfix only. Data for "extended detail" listing was being built
666 even if "-e" not specified. This resulted in unexpected excessive
667 memory consumption when crunching large amounts of data.
669 Added warning about memory consumption when "-e" option specified.
674 Further improvement to "accuracy" of by-domain-then-logname sort.
675 (Presently used only by "extended detail" listing). For comparison
676 purposes: mungs "machine(s).host.dom" into "host.dom.machine(s)" so
677 sort is keyed on "base" domain name before machines within the
678 domain. Does *not* attempt to reverse the order of the "machine(s)" -
679 so within a particular "base" domain, may not come out in quite the
680 right order. ("foo.bar.some.dom" will come out before
681 "sales.acme.some.dom", for example.)
682 Also works for 2x2-style domain names. (I.e.: "some.do.co")
685 19990102-01 (never released)
687 Added "mung UUCP-style bang-paths" switch (-m).
689 Improved performance and "accuracy" of by-domain-then-logname sort
690 used by (only at present) "extended detail" listing.
695 Added "extended detail" option (-e). At present this includes only a
696 per-message detail listing, which lists per-message detail sorted by
697 sender domain, then sender username, then by queue i.d.
704 Replaced warning message when message size unavailable in favor of
705 producing a report of these, sorted by queue i.d. Unlike the other
706 reports, this report header is not emitted at all if there are none of
707 these. (Always acts as if the -q switch had been specified).
712 Added experimental code to lower-case all domain names so that
713 "user@foo.dom" and "user@FOO.DOM" will come out the same.
715 Added test for existence of message size value when "to=" records are
716 being processed. This was necessary for cases in which the logfile
717 entry containing the "status=sent" record is not processed at the same
718 time as the logfile containing the "size=nnnn" record. Note that this
719 will produce a summary that will show recipient counts without
720 matching recipient sizes. The only way to cure this would be to
721 create a separate disk file to "memorize" message sizes. (Which would
722 introduce a whole new raft of problems.)
724 Added warning message (emitted to stderr) when the situation above is
727 Fixed "usage" message to indicate you can specify files on command
730 Wrapped a couple of long lines in the comments and code.
732 Added (temporary) version numbering scheme.
736 Other changes/enhancements since previous un-version-numbered
737 versions: deals with log entries for VMailer as well as Postfix, more
738 robust parsing of "to=" and "from=" fields (now handles spaces in
739 these), eliminated double-counting of message sizes (happened when
740 delivery was deferred), re-structured parsing to be more robust (not-
741 to-mention correct!), added "grand summary" report at top (total
742 messages, total size, number of senders and recipients).
747 [Note: The credits reflect suggestions and code contributions that
748 have actually been added. If your contribution doesn't appear
749 here, it may simply mean that it hasn't been added yet. (In which
750 case it should be on the list above.) On the other hand: if I
751 failed to credit you for something that *has* been added, please
754 Paul D. Robertson <proberts@clark.net>
756 For much testing and patience and many good suggestions on
757 how pflogsumm could be improved.
759 Simon J Mudd <simon.mudd@alltrading.com>
761 For the following code contributions:
763 Add "deferred" and "bounced" to "by hour" reports.
764 (I also added these to "by day" reports and "Grand
767 "VERP" (?) address munger (less-agressive version)
769 Suggestion for "by domain" delivery delay report.
771 For the --smtpd_stats suggestion.
773 Anders Arnholm <anders@arnholm.nu>
775 For pointing out the problem with forwarded messages.
777 Walcir Fontanini <walcir@densis.fee.unicamp.br>
779 For pointers to changes to make for Perl 5.003 compatibility.
780 (Added to 19990413-02beta.) (Which I will *try* to keep in
783 Eric Cholet <cholet@logilune.com>
785 For the --ignore_case patch.
787 Kurt Andersen <kurta@sitefs1a.spk.agilent.com>
789 For the ISO date formatting month-off-by-one patch and the
790 connect time overflow fix.
792 Thomas Parmelan <tom@proxad.net>
794 For improved "rejects" reporting patch.
796 Glen Eustace <root@godzone.net.nz>
798 Patch to fix "reject: header" matching after Wietse changed
projects / sven / pflogsumm.git / blob