1 ChangeLog for pflogsumm.pl
4 [Note: Let me know if you would like to be notified as new versions
5 are released. The latest released version can always be found at
6 http://jimsun.LinxNet.com/postfix_contrib.html.]
11 Added long-awaited switches to optionally reduce detail reporting:
12 --bounce_detail=N, --deferral_detail=N, --reject_detail=N,
13 --smtp_detail=N, smtpd_warning_detail=N, and --detail=N. Setting
14 any of them to 0 suppresses that detail entirely. --detail=N sets
15 the default for all of them, as well as for -u=N and -h=N.
17 With the above enhancements, the following switches are depreciated,
18 and will eventually be removed: --no_bounce_detail,
19 --no_deferral_detail, --no_reject_detail and --no_smtpd_warnings.
20 They are replaced by setting the desired --*_detail=0. They still
21 work, but using them generates a warning.
23 Added support for parsing logs with RFC 3339 timestamps. Thanks
24 and a tip o' the hat to sftf-at-yandrex-dot-ru for the heads-up
25 and the code contribution. (N.B.: My code does not require a
26 command-line switch. The format is detected automatically.)
28 Fixed some --ignore-case inconsistincies. Thanks and a tip o'
29 the hat to Richard Blanchet (richard-dot-blanchet-at-free-dot-fr)
30 for the heads-up and the diff.
32 Fixed parsing bug that resulted in attempts to treat
33 kind-of-IPv4-looking strings as IPv4 addresses. (I really need to
34 improve reject/defer/etc. "reason" parsing to fix this properly.)
35 Thanks to Joseph Vit (jvit-at-certicon-dot-cz) for the bug
40 Fixed bug with calculating yesterday's date in vicinity of DST
41 changes. (Thanks and a tip o' the hat to Wieland Chmielewski
42 for bringing the problem to my attention.)
44 Added missing "underlining" to some (sub-)section titles for
50 Fixed to parse Postfix-2.3 (and beyond) logfiles. Thanks to
51 whomever contributed to
53 http://bugs.gentoo.org/show_bug.cgi?id=144236
55 Removed support for vmailer.
57 Removed "SMTPD_STATS_SUPPORT" "fences" in code in favour of code
58 to automatically detect the availability of Date::Calc. If
59 --smtpd_stats is specified and Date::Calc is not installed, now
60 bails-out with friendly message. (Adapted from suggestion and
61 examples provided by David Landgren <david-at-landgren-dot-net>.
64 Removed rem_smtpd_stats_supp.pl utility from distribution. (No
67 Memory footprint improvement: Pflogsumm no longer stores data for
68 reports that are supressed via --no_<mumble> switches.
70 Removed extraneous arguments in two calls to print_nested_hash
71 that would result in the "quiet" flag being ignored. Thanks to
72 Pavel Urban (pupu-at-pupu-dot-cz) for bringing that to my
75 Added notes to FAQ about translations and i18n, about mismatching
76 "received"/"delivered" counts, about bug in calculating "yesterday,"
77 and about John Fawcett's "prepflog."
82 Promoted 1.0.18 (Beta) to "production/stable" version release.
87 Fixed reject parsing for "DATA" smtpd rejects.
92 Fixed reject parsing to properly recognize bare "User unknown".
93 (Thanks to J.D. Bronson" <jeff_bronson-at-wixb-dot-com> for the
94 bug-report and sample logfile lines.)
99 Re-worked "to" and "from" field parsing in reject report handling to
100 make it more robust in pathological cases. (Thanks to Paul Brooks
101 <paul-dot-brooks-at-metro1-dot-com> and Lars Hecking
102 <lhecking-at-nmrc-dot-ie> for the bug-reports and sample logfile
105 Fixed warnings resulting from non-standard, extraneous syslog input.
106 (Thanks to Mathias Behrle <m123-at-arcor-dot-de> for the report.)
108 Fixed reject parsing to account for really atrocious garbage in
109 HELO strings, sender addresses and recipient addresses. (Thanks to
110 Lars Hecking <lhecking-at-nmrc-dot-ie> for the bug-report and sample
113 Fixed reject parsing to properly recognize "CONNECT" smtpd rejects.
114 (Thanks to Mike Vanecek <postfix_list-at-mm-vanecek-dot-cc> for the
117 Fixed reject parsing to properly recognize "User unknown in relay
118 recipient table." (Thanks to Lars Hecking <lhecking-at-nmrc-dot-ie>
119 for the bug-report and sample logfile lines.)
121 Some code optimization resulting in 3-5% performance improvement.
126 Pflogsumm *should* now properly parse and handle log entries with
127 IPv6 addresses in them. (Adapted from idea and code submitted by
128 Stefan `Sec` Zehl <sec-at-42-dot-org>.)
130 Fixed "User unknown in local recipient table" reject reports to
131 show target recipient address, rather then sending domain, to be
132 consistent with other "recipient" reports. (Thanks to WC Jones
133 <sx-at-insecurity-dot-org> for the suggestion.)
135 Fixed parsing of "Recipient address rejected" for recipient
136 address verification (RAV) rejects. (Thanks to Len Conrad
137 <LConrad-at-Go2France-dot-com> for the suggestion.)
139 FAQ additions regarding recommendations on how to format custom
140 reject reports for "best" results in Pflogsumm's output and note
141 regarding "non-standard" syslogd's.
146 Fixed bug in parsing for "Host/Domain Summary: Messages Received"
147 report improvement (rel-1.0.13) that resulted from (unexpected, to
150 ... postfix/smtpd[31430]: E02DDB04E: client=blurfl[1.2.3.4],
151 sasl_method=LOGIN, sasl_username=phred
155 The "Host/Domain Summary: Messages Received" report would show simply
156 "from=<>", for the host/domain, for postmaster bounces. Pflogsumm now
157 substitutes the client hostname or IP address for these, unless it's
158 from the pickup daemon, in which case "from=<>" is retained. (Note
159 that "Senders by message count/size" reports are unaffected by this
162 "Senders by message count" and "Recipients by message count" reports
163 are now secondarily sorted by domain, host and user parts. (As a
164 side-effect: So are "Senders by message size" and "Recipients by
165 message size" but, being as the odds are against numerous senders and
166 recipients having the same total message sizes, this change hasn't much
171 Rejects, warns, etc. now print sub-category totals. E.g.:
173 message reject detail
174 ---------------------
176 Relay access denied (total: 6)
178 (Adapted from idea and code submitted by blake7-at-blake7-dot-org.)
180 Reject, warning, etc. reports are now sorted by 2nd column (e.g.: IP
181 address, domain, etc.) within count. (Adapted from idea and code
182 submitted by David Landgren <david-at-landgren-dot-net>.)
184 Added --no_smtpd_warnings (report) option.
186 Added --no_no_msg_size (report) option.
188 A couple of minor improvements to reject parsing/reporting.
192 This is a bug-fix release.
194 There was a problem in the way pflogsumm-1.0.8 through 1.0.10
195 handled the --syslog_name option: When --syslog_name was
196 specified, some log entries with the default "postfix" name would
197 be missed. This revision may introduce incompatibilities if
198 you're logging two or more instances of Postfix to the same log.
199 See the docs included in the tarball for details.
203 Re-worked "% rejected" calculation to include messages discarded
204 and added "% discarded" calculation/display.
208 Bugfix: If Perl's -w is specified at run-time and there were no
209 messages delivered or rejected, uninitialized variable warnings
210 would be issued in the percent rejected calculation code. Thanks
211 for Larry Hansford (and many others since!) for the bug report.
215 Bugfix: Fixed problem with "orig_to=<blurfl>" being parsed as
216 "to=<blurfl>". This resulted in *very* wrong output. Thanks to
217 Bjorn Swift for the report.
219 Added "% rejected" to Grand Totals "rejected" figure. This is
220 calculated as: rejected / (delivered + rejected). (I did this
221 purely because it amuses me.)
223 Bugfix: Fix, in reject processing, for truncated overly-long "to"
224 fields. Thanks to Rick Troxel for reporting the problem.
226 Added --syslog_name option. Thanks to Ben Rosengart for the
231 Corrected and improved message reject/reject warn/hold/discard
232 parsing. Again. (Thanks to Peter Santiago for reporting the
233 problem that initiated these improvements.)
237 Added support for reporting message reject warnings, holds and
240 Note: Message rejects, reject warnings, holds and discards
241 are all reported under the "rejects" column for the Per-Hour
242 and Per-Day traffic summaries.
244 More aggressive verp munging (again). (Prompted, in part, by a
245 suggestion from Casey Peel. Thanks!)
247 Verp munging now applied to sender addresses in smtpd reject
250 WARNING: Please note that verp munging is highly experimental!
252 Pflogsumm distribution changed to gzip'd tarball format.
254 Tightened-up parsing. Thanks for Ralf Hildebrandt for noting and
255 reporting the problem.
257 Docs at the top of pflogsumm.pl changed to POD format for automated
262 Automatically-generated manpage added.
264 "To Do" moved out of ChangeLog into separate file.
266 Package now includes convenience Perl script for removing smtpd
267 stats support for those who don't have Date::Calc, don't want to
268 install it and don't care about smtpd stats reporting.
270 Belated thanks to Len Conrad in regards to the Sender Address
271 Verification work in 1.0.5.
275 Fixed to parse smtpd rejects for Postfix versions as of 20021026.
276 (Retained compatibility with older versions of Postfix.)
278 Note: smtpd and header-/body-checks warn, hold and discard
279 messages are *not* currently parsed/reported. I'll need to
280 get some logfile entries.
282 Fixed parsing to handle the new "sender address verification"
285 Added "--zero_fill" option to put zeros in columns that might
286 otherwise be blank in some reports. (Suggestion by Matthias
289 Fixed "Message size exceeds fixed limit" parsing for reject
294 Added "--no_*_detail" options. (Suppresses some of the "detail"
297 Added "--version" option. (Thanks to "Guillaume")
299 Improved handling of "[ID nnnnnn some.thing]" stuff (Thanks to
302 Repaired and optimized some of the "reject" parsing.
304 Added processing and report of smtp delivery failures.
306 Added --rej_add_from option: For those reject reports that list
307 IP addresses or host/domain names: append the email from address
308 to each listing. (Note: does not apply to "Improper use of SMTP
309 command pipelining" report.)
314 Minor re-work of "reject: RCPT" parsing to account for Yet Another
315 Change in the logfile format. (Semi-colon changed to a comma in
316 "blocked using rbl.maps.vix.com,".)
321 Took another whack at "verp" munging. *sigh*
323 Added code to summarize "Improper use of SMTP command pipelining"
329 Modified to catch "reject: header" log entries changed as of
330 postfix release-20010228 (?). Prior versions of postfix had the
331 string "warning: " (where the qid normally is). Thanks to Glen
332 Eustace <root@godzone.net.nz>, Len Conrad
333 <lconrad@go2france.com>, Daniel Roesen
334 <droesen@entire-systems.com>, Milivoj Ivkovic <mi@alma.ch> and
335 j_zuilkowski@hotmail.com (Jon Zuilkowski) for reports and/or
338 Fixed a couple of "uninitialized variable" problems.
340 Committed (actually starting with 20000925-01beta) to CVS.
345 Added a line to compensate for (new?) "[ID nnnnnn some.thing]"
346 sub-strings that appear in logfile entries under Sun Solaris 8.
349 Note: Upon being committed to CVS, this became rel-0.9.0.
354 Forgot to add "--problems_first" to the "usage" output and in the
355 synopsis at the top of the comments.
360 Re-did what 20000907-02beta was *supposed* to be! To wit:
361 replaced missing "--ignore_case" bugfix, "panic" entry processing,
362 improvements to "fatal" and "warning" message reporting and
363 missing "--mailq" option. (Obviously: 20000907-02beta was
364 derived from the wrong code base.)
369 Fixed bug in ISO date formatting that caused the month to be off
370 by one. Thanks to Kurt Andersen <kurta@sitefs1a.spk.agilent.com>
371 for the report and the patch.
373 Fixed overflow of connect time reporting into days. (Can happen
374 during weekly summaries for sites with large volumes of email.)
375 Thanks again to Kurt Andersen <kurta@sitefs1a.spk.agilent.com>
376 for the report and the fix.
378 Improved "rejects" reporting *again*. Thanks to Thomas Parmelan
379 <tom@proxad.net> for the patch.
381 Added "--problems_first" option to print "problem" reports such as
382 bounces, defers, warnings, fatal errors, etc. before "normal"
388 Fixed bug in code that prevented "--ignore_case" from actually
389 doing anything. Thanks to Nadeem Hasan <nhasan@usa.net> for
390 reporting this and supplying the fix.
395 Added the following caveat to the "Notes" section of Pflogsumm:
397 -------------------------------------------------------------
398 IMPORTANT: Pflogsumm makes no attempt to catch/parse non-
399 postfix/vmailer daemon log entries. (I.e.: Unless
400 it has "postfix/" or "vmailer/" in the log entry,
402 -------------------------------------------------------------
404 Added reporting of "panic" log messages. This was missed until
407 Increased reporting detail of "fatal" and "warning" entries.
408 (Actually, "warning" detail was increased in 19991120-01beta.
409 Neglected to note it then.)
412 19991123-01 (unreleased)
414 Added "--mailq" option. (Convenience factor.) Runs Postfix's
415 "mailq" command at the end of the other reports.
417 -------------------------------------------------------
418 NOTE: If Postfix's "mailq" command isn't in your $PATH,
419 you'll have to edit the "$mailqCmd" variable located
420 near the top of pflogsumm to path it explicitly.
421 -------------------------------------------------------
426 Tried once again to improve parsing of reject log entries.
427 Specifically: those associated with "RCPT" rejects.
430 19991016-01 (not generally released)
432 Added --smtpd_stats. Generates smtpd connection statistics.
434 ---------------------------------------------------------------
435 NOTE: Support for --smtpd_stats requires the Date::Calc module
436 (available from CPAN). If you don't want to go to the trouble
437 of fetching & installing that module, and you don't want smtpd
438 stats anyway, *carefully* identify all of the code sections
439 delimited by "# ---Begin: SMTPD_STATS_SUPPORT---" and
440 "# ---End: SMTPD_STATS_SUPPORT---" and remove them.
441 ---------------------------------------------------------------
444 19990909-01 (not generally released)
446 Added -i and --ignore_case options. Causes entire email address
447 to be lower-cased instead of just the host/domain part.
449 Added "use locale". (This means that the sorting order within
450 reports may be different from before--depending on how you have
451 your machine's locale set.)
456 Improved "reason" parsing and reporting for bounced and deferred
459 Added parsing of "cleanup" reject lines to catch PCRE/regexp
462 Added "reject" stats to per-hour and (on multi-day reports) per-
465 Improved "warnings" report to show details.
467 A single message deferred multiple times showed up as multiple
468 deferrals--implying that multiple messages were deferred. Now
469 shows "how many messages were deferred" and "how many deferrals"
472 Changed display of "Grand Totals" to make it a bit more readable
475 Added "automatic perl finder" line for those systems that don't
476 support the "#!" notation.
478 By popular demand: added note to comments as to where pflogsumm
479 home page could be found :-).
484 Fixed problem with last octet of IP address getting truncated in
485 reports when IP address used in place of unknown hosts.
487 Changed the way a few internal variables were handled to be
488 compatible with Perl 5.003. Don't run it under Perl 5.003 with the
489 "-w" perl switch, tho! It will issue lots of warnings. All tests
490 I performed indicated that it produces the correct output, however.
492 ------------------------------------------------------------
493 NOTE: While this version was tested to work with Perl 5.003,
494 I recommend that you upgrade to 5.004 or later. I will not
495 guarantee that I'll remember to do the full regression-
496 testing that I usually do with 5.003 as well.
497 ------------------------------------------------------------
502 NOTICE: As of this version of pflogsumm.pl, the "-c" switch is
503 GONE! (As per the previous notice.)
505 Added "--help" option to emit short usage message and bail out.
507 Added "--iso_date_time" switch to change displays of dates and times
508 to ISO 8601 standard formats (CCYY-MM-DD and HH:MM), rather than
509 "Month-name Day-number CCYY" and "HHMM" formats (the default).
511 Added "--verbose_msg_detail" switch. This causes the full "reason"
512 to be displayed for the message deferral, bounce and reject summaries.
513 (Note: this can result in quite long lines in the report. Also note
514 that there have been a couple of subtle changes in the "reason"
515 parsing/reporting in the default mode (no "--verbose_msg_detail".)
517 Added "--verp_mung" option. The problem this addresses is "VERP"
518 generated (??? so far as I can tell!) addresses (?) of the form:
520 "list-return-NN-someuser=some.dom@host.sender.dom"
522 These result in mail from the same "user" and site to look like it
523 originated from different users, when in fact it originates from the
524 same "user." There are presently two "levels" of address munging
525 available. With no numeric argument (or any value less than 2), the
526 above address will be converted to:
528 "list-return-ID-someuser=some.dom@host.sender.dom"
530 In other words: the numeric value will be replaced with "ID".
532 By specifying "--verp_mung=2", the munging is more "aggressive",
533 converting the above address to something like:
535 "list@host.sender.dom"
537 Which looks more "normal."
539 (Actually: specifying anything less than 2 does the "simple" munging
540 and anything greater than 1 results in the more "aggressive" hack
543 Added "--uucp_mung" switch for consistence with "--verp_mung".
548 NOTICE: As of this version of pflogsumm.pl, versions of VMailer
549 prior to 19981023 are no longer supported. Sorry.
550 Pflogsumm-19990121-01.pl will be made permanently
551 available from now on for those with out-of-date versions
552 of VMailer prior to 19981023.
554 NOTICE: As of this version of pflogsumm.pl, the "-c" switch is
555 DEPRECIATED. This version is transitional and retains it.
556 The next version will not have it. Subsequent versions
557 may re-use it for another purpose. Use the "-h" and "-u"
560 Added "-h" and "-u" switches to provide finer-grained control over
561 report output. Depreciated "-c".
563 Added "deferred" and "bounced" to "Grand Totals", "by-day" and "by-
566 Added "by-host/domain" reports. For sent (delivered) and received
567 messages: lists message count, total size of messages and
568 host/domain. For delivered messages: also lists number of deferred
569 messages and average and maximum delivery time. Both reports sorted
570 by message count in descending order.
572 Grand totals also now list number of recipient and sender
575 Re-wrote "by-user" data collection storage to reduce memory consumption
578 Moved "credits" from pflogsumm.pl to this file.
583 Now accounts for forwarded messages.
585 Side-effects of the above:
587 . Total messages size now broken-out into total bytes received
588 and total bytes delivered.
589 . Count of forwarded messages now reported.
590 . Postfix-internally-generated messages (e.g.: Postmaster
591 notifications of bounces) are no longer counted as "received".
592 (They do, however, show up as "delivered".)
593 . Forwarded addresses no longer show up as "recipients" (just
594 as with aliases and mailing lists).
596 Note that "delivered" will exceed "received" when messages
597 are forwarded because of additional header lines.
602 Added processing for "reject" log entries.
604 Expanded detail of "deferred" and "bounced" log entries to include
610 Added "messages received/delivered by hour" and "messages
611 received/delivered by day" reports. See the "Notes" section in the
612 documentation for details on how these behave.
614 Broke-out total message count to "messages received" and "messages
617 (For the above two enhancements: "postfix/pickup" and "postfix/smtpd"
618 lines are now processed. They used to be discarded.)
620 Renamed "summary" report to "Grand Totals".
622 Added code to parse date & time stamps from log entries. This was
623 needed, in part, for the "messages per-hour/day" reports. It would
624 have been necessary for future enhancements in the way of date- &
625 time-based processing anyway.
627 Added "Notes" section to docs at top of code.
632 Improved display of large integer values.
637 Bugfix only. Data for "extended detail" listing was being built
638 even if "-e" not specified. This resulted in unexpected excessive
639 memory consumption when crunching large amounts of data.
641 Added warning about memory consumption when "-e" option specified.
646 Further improvement to "accuracy" of by-domain-then-logname sort.
647 (Presently used only by "extended detail" listing). For comparison
648 purposes: mungs "machine(s).host.dom" into "host.dom.machine(s)" so
649 sort is keyed on "base" domain name before machines within the
650 domain. Does *not* attempt to reverse the order of the "machine(s)" -
651 so within a particular "base" domain, may not come out in quite the
652 right order. ("foo.bar.some.dom" will come out before
653 "sales.acme.some.dom", for example.)
654 Also works for 2x2-style domain names. (I.e.: "some.do.co")
657 19990102-01 (never released)
659 Added "mung UUCP-style bang-paths" switch (-m).
661 Improved performance and "accuracy" of by-domain-then-logname sort
662 used by (only at present) "extended detail" listing.
667 Added "extended detail" option (-e). At present this includes only a
668 per-message detail listing, which lists per-message detail sorted by
669 sender domain, then sender username, then by queue i.d.
676 Replaced warning message when message size unavailable in favor of
677 producing a report of these, sorted by queue i.d. Unlike the other
678 reports, this report header is not emitted at all if there are none of
679 these. (Always acts as if the -q switch had been specified).
684 Added experimental code to lower-case all domain names so that
685 "user@foo.dom" and "user@FOO.DOM" will come out the same.
687 Added test for existence of message size value when "to=" records are
688 being processed. This was necessary for cases in which the logfile
689 entry containing the "status=sent" record is not processed at the same
690 time as the logfile containing the "size=nnnn" record. Note that this
691 will produce a summary that will show recipient counts without
692 matching recipient sizes. The only way to cure this would be to
693 create a separate disk file to "memorize" message sizes. (Which would
694 introduce a whole new raft of problems.)
696 Added warning message (emitted to stderr) when the situation above is
699 Fixed "usage" message to indicate you can specify files on command
702 Wrapped a couple of long lines in the comments and code.
704 Added (temporary) version numbering scheme.
708 Other changes/enhancements since previous un-version-numbered
709 versions: deals with log entries for VMailer as well as Postfix, more
710 robust parsing of "to=" and "from=" fields (now handles spaces in
711 these), eliminated double-counting of message sizes (happened when
712 delivery was deferred), re-structured parsing to be more robust (not-
713 to-mention correct!), added "grand summary" report at top (total
714 messages, total size, number of senders and recipients).
719 [Note: The credits reflect suggestions and code contributions that
720 have actually been added. If your contribution doesn't appear
721 here, it may simply mean that it hasn't been added yet. (In which
722 case it should be on the list above.) On the other hand: if I
723 failed to credit you for something that *has* been added, please
726 Paul D. Robertson <proberts@clark.net>
728 For much testing and patience and many good suggestions on
729 how pflogsumm could be improved.
731 Simon J Mudd <simon.mudd@alltrading.com>
733 For the following code contributions:
735 Add "deferred" and "bounced" to "by hour" reports.
736 (I also added these to "by day" reports and "Grand
739 "VERP" (?) address munger (less-agressive version)
741 Suggestion for "by domain" delivery delay report.
743 For the --smtpd_stats suggestion.
745 Anders Arnholm <anders@arnholm.nu>
747 For pointing out the problem with forwarded messages.
749 Walcir Fontanini <walcir@densis.fee.unicamp.br>
751 For pointers to changes to make for Perl 5.003 compatibility.
752 (Added to 19990413-02beta.) (Which I will *try* to keep in
755 Eric Cholet <cholet@logilune.com>
757 For the --ignore_case patch.
759 Kurt Andersen <kurta@sitefs1a.spk.agilent.com>
761 For the ISO date formatting month-off-by-one patch and the
762 connect time overflow fix.
764 Thomas Parmelan <tom@proxad.net>
766 For improved "rejects" reporting patch.
768 Glen Eustace <root@godzone.net.nz>
770 Patch to fix "reject: header" matching after Wietse changed
projects / sven / pflogsumm.git / blob