Pflogsumm - The Postfix Log Entry Summarizer



Home


Pflogsumm is a log analyzer for Postfix. It can mail out a summery on a daily, weekly or monthly basis and is fully configurable. The generated report can show you patterns in email traffic so you can better determine bandwidth limitations, mail server abusers or if your mail server needs to be upgraded. Statistics can be very helpful in showing, for example, when most of your mail is sent or received or if that one mail user is sending out more than their fair share of mail. Pflogsumm will give you the ability to quickly determine what machines on your network are the top (ab)users and the proof to follow up company policies with numbers.

Pflogsumm is a single perl file and its only dependancies are perl, the perl module Date::Calc (which is probably already installed on your system) and the proper location of a Postfix log file.



What the email report looks like

Before we get started setting up pflogsumm lets take a look at the output of an example email report. We think it is important to see what you are going to get out of a tool before you spend the time and energy setting it up.

Below is a scrollable window and you will see all of the information gathered from a half day on the calomel.org mail server (hosts are obfuscated in the example). Scroll though the email and notice all the patterns shown. Also notice the email is text only. There are no pictures or "manager-friendly" pie graphs available in pflogsumm. Only the data in an easy to read format.


From: root@your_host.com (Root User)
To: root@your_host.com
Date: Mon,  1 Jan 2010 11:00:00 -0400 (EDT)
Subject: Postal Statistics Mon Jan  1 11:00:00 EDT 2007

Postfix log summaries for Jan  1

Grand Totals
------------
messages

    845   received
   1208   delivered
      0   forwarded
      2   deferred  (3  deferrals)
     12   bounced
    822   rejected (40%)
      2   reject warnings
      0   held
      0   discarded (0%)

  86983k  bytes received
 168531k  bytes delivered
    214   senders
    157   sending hosts/domains
    176   recipients
     77   recipient hosts/domains


message deferral detail
-----------------------
  smtp (total: 3)
         3   mailbox unavailable (in reply to RCPT TO command

message bounce detail (by relay)
--------------------------------
  c.mx.maal.yahoo.com[256.39.55.3]:25 (total: 3)
         3   dd This user doesn't have a yahoo.com account (dothingoctam_267...
  none (total: 1)
         1   spam body
  smtp.iccm.rcenaria.es[153.145.82.2]:25 (total: 2)
         2   User unknown in local recipient table (in reply to RCPT TO comm...

message reject detail
---------------------
  RCPT
    Client host rejected: DHCP1 check (total: 271)
          98   arcor-ip.net
          78   paltel.net
          14   t-dialin.net
          10   charter.com
           9   ctbcnetsuper.com.br
           9   i59F51603.versanet.de
           6   catv-5062d7f9.catv.broadband.hu
           6   tkb.net.pl
           5   bezeqint.net
           4   brasiltelecom.net.br
           4   sbb.co.yu
           3   cia.com
           3   adsl-81-7-96-45.zebra.lt
           3   verizon.net
           3   195-240-166-103-static.dsl.ip.tiscali.nl
           3   p2d215.traco.pl
           3   ppp85-140-244-82.pppoe.mtu-net.ru
           2   gibconnect.com
           1   fibertel.com.ar
           1   bigpond.net.au
           1   dslcom3-125.express.oricom.ca
           1   203-233-222-201.adsl.terra.cl
           1   fdn.com
           1   mindspring.com
    Client host rejected: DHCP2 check (total: 17)
          12   veloxzone.com.br
           3   chello089079077128.chello.pl
           1   dial050238.pool.invitel.hu
           1   cc1206100-a.mp1.dr.home.nl
    Helo command rejected: need fully-qualified hostname (total: 2)
           1   tm.net.my
           1   dyn-85.204.185.47.tm.upcnet.ro
    Recipient address rejected: User unknown (total: 13)
           2   time@your_host.com
           2   toogp@your_host.com
           2   restel@your_host.com
           1   tewaslio@your_host.com
           1   e@your_host.com
           1   gfu@your_host.com
           1   twased@your_host.com
           1   fellow@your_host.com
           1   odsnmeifg@your_host.com
           1   msdhad@your_host.com
    Sender address rejected: Domain not found (total: 10)
           9   aw-confirm@email.ebay.com
           1   jun-liprashant@amefi.org
    cannot find your hostname (total: 463)
         145   194.24.251.235
         130   64.32.178.103
          24   77.73.21.114
          24   189.12.229.24
          18   41.233.124.21
          12   62.182.2.101
          11   209.120.212.32
           9   59.22.242.31
           9   82.223.40.74
           9   29.21.131.58
           9   121.15.248.102
           9   223.13.161.146
           9   228.209.159.149
           9   222.127.127.249
           8   82.149.82.219
           6   52.187.55.161
           6   72.52.24.23
           6   82.214.224.190
           6   22.162.254.162
           6   82.105.61.171
           6   82.245.236.2
           6   211.226.145.149
           5   64.222.14.254
           5   203.62.52.61
           5   218.227.20.61
           5   222.122.49.9
           4   72.9.222.102
           4   212.129.197.148
  cleanup
    body (total: 35)
          27   And you will At last your new life! Like a real man with a re...
           1   60% of long-term relationship breakups report that sexual pro...
           1   And you will Finally  your new life! Like a real man with a r...
           1   Hello! I am bored this evening. I am nice girl that would lik...
           1   Hello! I am tired today. I am nice girl that would like to ch...
           1   Hello! I am tired this evening. I am nice girl that would lik...
           1   Hello! I am bored this afternoon. I am nice girl that would l...
    header (total: 11)
           4   Content-Type: application/x-msdownload; name="Attachments001....
           3   Content-Type: application/x-msdownload; name="WinZip.BHX"
           1   Content-Type: application/x-msdownload; name="SeX.mim"
           1   Received: from lyris.networkworld.info (Lyris.networkworld.in...
           1   Subject: Emails
           1   Subject: ?o??o?o? ????? ? ?????o??o? ???????????? ???o???????...

message reject warning detail
-----------------------------
  RCPT
    Helo command rejected: Host not found (total: 2)
           1   telecam.net.ar
           1   ber246.neaplus.adsl.tpnet.pl

message hold detail: none

message discard detail: none

smtp delivery failures
----------------------
  connection refused (total: 3)
         1   mail.example.com
         1   hotmail.ten.org
         1   felix.com
  operation timed out (total: 3)
         2   timetested.com
         1   whataboutme.net

Warnings
--------
  smtpd (total: 187)
        30   64.55.178.153: hostname virginia39.seemeplayme.com verification...
        15   189.55.209.54: hostname 18912209024.user.veloxzone.com.br verif...
        12   52.181.2.151: address not listed for hostname return.wdc.pl
        11   259.190.252.32: address not listed for hostname mail.affinity-n...
         9   45.233.154.21: hostname host-41.233.134.21.tedata.net verificat...
         9   85.21.151.58: hostname host58-131-21-89.tz.ru verification fail...
         8   85.149.52.219: address not listed for hostname hosted.by.mostwo...
         6   55.187.55.161: hostname adsl-dynamic-pool-xxx.fpt.vn verificati...
         6   85.152.554.162: hostname 143-254-162.dsl.primorye.ru verificati...
         6   55.155.61.171: hostname dsl.static8510561171.ttnet.net.tr verif...
         6   85.215.224.190: hostname dsl.dynamic81214224190.ttnet.net.tr ve...
         5   65.255.14.254: hostname 254-14-251-64.serverpronto.com verifica...
         5   255.65.52.61: address not listed for hostname mail.fenying.com.tw
         3   85.97.55.163: hostname dsl.dynamic859745163.ttnet.net.tr verifi...
         3   253.155.103.185: hostname 203-150-103-185.inter.net.th verifica...
         3   256.245.30.24: hostname unknown.hostforweb.com verification fai...
         3   85.215.204.3: hostname dsl.static812152043.ttnet.net.tr verific...
         3   85.242.76.53: hostname dsl88.242-19509.ttnet.net.tr verificatio...
         3   159.165.114.63: hostname dsl-189-165-114-63.prod-infinitum.com....
         3   85.251.254.66: address not listed for hostname smtp.dgcsystems.net

Fatal Errors: none

Panics: none

Master daemon messages: none

Per-Hour Traffic Summary
    time          received  delivered   deferred    bounced     rejected
    --------------------------------------------------------------------
    0000-0100          33         40          0          0         50
    0100-0200          51         64          1          1         41
    0200-0300          44         63          0          0         43
    0300-0400          84        162          0          9         36
    0400-0500          89        147          0          0         26
    0500-0600          74         95          0          1        112
    0600-0700          72         91          0          1         79
    0700-0800          41         52          0          0        178
    0800-0900          50         87          2          0        105
    0900-1000         186        256          0          0        126
    1000-1100         120        150          0          0         27
    1100-1200           1          1          0          0          1
    1200-1300           0          0          0          0          0
    1300-1400           0          0          0          0          0
    1400-1500           0          0          0          0          0
    1500-1600           0          0          0          0          0
    1600-1700           0          0          0          0          0
    1700-1800           0          0          0          0          0
    1800-1900           0          0          0          0          0
    1900-2000           0          0          0          0          0
    2000-2100           0          0          0          0          0
    2100-2200           0          0          0          0          0
    2200-2300           0          0          0          0          0
    2300-2400           0          0          0          0          0

Host/Domain Summary: Message Delivery (top 5)
 sent cnt  bytes   defers   avg dly max dly host/domain
 -------- -------  -------  ------- ------- -----------
    871   159992k       0     2.8 s    1.6 m  your_host.com
     62     5632k       0     3.7 s   36.0 s  another_host.com
     48    96048        0    25.8 s    5.2 m  yahoo.com
     33    82959        0    17.8 s    2.0 m  teldes.ney
     17    62542        0     3.5 s    6.6 s  gmail.com

Host/Domain Summary: Messages Received (top 5)
 msg cnt   bytes   host/domain
 -------- -------  -----------
    455    73589k  your_host.com
     44   120031   another_host.com
     34     3052k  google.com
     24      548k  netmail.net
     19      589k  yahoo.com

top 5 Senders by message count
------------------------------
     99   user1@your_host.com
     76   dat@your_host.com
     66   host@your_host.com
     44   root@your_host.com
     44   telme@your_host.com

top 5 Recipients by message count
---------------------------------
    106   myuser@your_host.com
     76   main@your_host.com
     72   hello@your_host.com
     59   felix@your_host.com
     48   foul@your_host.com

top 5 Senders by message size
-----------------------------
  45738k  twat@your_host.com
  23968k  halloart@your_host.com
   5881k  yuer@your_host.com
   2673k  twotone@your_host.com
   2534k  whatup@your_host.com

top 5 Recipients by message size
--------------------------------
  24666k  me@your_host.com
  24449k  geter@your_host.com
  24335k  what@your_host.com
  24316k  whodat@your_host.com
  23960k  finallydone@your_host.com



If this looks like a tool you could use then lets take a look at the quick three-step setup.



Get pflogsumm setup and running

Step 1: To get started you first need to download the pflogsumm.pl perl script.

Download pflogsumm.pl here.

Step 2: Extract the files from the tar ball and put _only_ the perl script pflogsumm.pl into /usr/local/bin/ . Make sure the permissions are 700 for security.

Step 3: Setup a cron job to mail out the report every day at 11:59pm (23:59).

#minute (0-59)
#|   hour (0-23)
#|   |    day of the month (1-31)
#|   |    |   month of the year (1-12 or Jan-Dec)
#|   |    |   |   day of the week (0-6 with 0=Sun or Sun-Sat)
#|   |    |   |   |   commands
#|   |    |   |   |   |
#### pflogsumm mail report
59  23    *   *  *    /usr/local/bin/pflogsumm -u 5 -h 5 --problems_first \
                      -d today /var/log/maillog | mail -s "pflogsumm report `date`" root

This is the same line used to generate the example email shown in the scrollable table above. The cron job is going to go through the postfix log in /var/log/maillog and report todays stats from 12:00am to 11:59pm. We are going the see the top five(5) senders and receivers of email by volume and size. The report is going to be mailed to root with the subject like "pflogsumm report Mon Jan 1 11:00:00 EDT 2007".

I would suggest running the cron job line at least once to make sure everything works. If you do not get any errors and the email comes through then you are done. If you experience problems, then take a look at the question and answer section at the bottom of this page. Finally, if you want more information on mail tools or postfix itself including and "how to" setup of the Postfix config then checkout the Calomel mainpage.



HELPFUL HINT: For an added layer of protection again spam you can use a bayesian spam filter. Check out our Bogofilter "how to" Anti-Spam Guide. With a little time and understanding you could easily filter up to 99% of any remaining spam.



Questions?

Postfix is blocking my pflogsumm emails!!

If you are seeing that the postfix body_checks you put in place, which are now being reported by pflogsumm.pl are in turn being blocked by postfix then you need to put an exception into the body_checks file. This is easily done by adding this line to the _top_ of your postfix body_checks file:
### allow pflogsumm reports through postfix (body_checks file) ###
/^ {6,11}[[:digit:]]{1,6}[ km] /    OK

I get an error about Date::Calc no found! Whats the problem?

This is perl module pflogsumm uses to calculate the date. It needs to be installed on your system. You will find many distributions have rpm's for perl modules. You can also install Date::Calc using CPAN which is just as easy.

Can you show me more information about pflogsumm?

I highly suggest checking out the Pflogsumm FAQ for more information. It is a light hearted question and answer session that helps put to rest a lot of the emails the authors has recieved.





Questions, comments, or suggestions? Contact Calomel.org