Spamd tarpit/greylisting anti-spam "how to" (spamdb)

Greytrapping is the seeding of an email address so that spammers can find it, but normal users can not. If the email address is used then the sender must be a spammer and they are black listed.

Lets use the email address "greytrap@your_domain.org" as an example. If we took this email and put it in the source HTML of our web site normal humans would not see it. Spammers on the other hand use web page scrapers and bots to harvest email address; so they would find this address.

When the spammer sends mail with the destination address of "greytrap@your_domain.org" spamd knows this is a spammer and SPAMTRAP's them. When a host that is currently greylisted attempts to send mail to a spamtrap address, it is spamtrapped for 24 hours by adding the host to the spamd spamlist <spamd-greytrap>.

You can add as many spamtrapped email address as you want. We even suggest looking through the logs and seeing what invalid addresses spammers are sending mail to. You can then add these addresses too.

To enter a greytrap email into the spamd database use the following format. You can add multiple spamtrap email addresses.

spamdb -T -a 'greytrap@your_domain.org'

To seed the SPAMTRAP email address in HTML use the following format. It looks like a real email address to the spammer's bots, but it will not be visible to any humans looking at the web page.

<a href="mailto:greytrap@your_domain.org"></a>

Optional: Greytrapping all but allowed domains and email addresses

The file /etc/mail/spamd.alloweddomains can contain a list of allowed domains and addresses which are allowed to go through the grey listing process. They are _not_ whitelisted, just allowed to prove they are good hosts by going through the grey list process. Think of it as a white list of acceptable domains or emails this machine will be willing to receive. If a remote machine is not sending mail to a valid address then they should not be contacting us.

spamd.alloweddomains can be used to specify a list of domain name suffixes or full email addresses which must match each destination email address in the grey list. Any destination address which does not match one of the suffixes listed in spamd.alloweddomains will be trapped, exactly as if it were sent to a spamtrap address. When a host that is currently greylisted attempts to send mail to a spamtrap address, it is blacklisted for 24 hours by adding the host to the spamd blacklist <spamd-greytrap>.

In our pf example about we only have two rules, spamd-white and not (!) spamd-white. Since a host in the spamd-greytrap table is not in spamd-white then it will be stuttered at for as long as the remote machine is connected.

This is one of the most powerful options in spamd. If you can keep a list of valid email addresses current in the spamd.alloweddomains file then all other attempts will be grey trapped. At Calomel.org, this grey trapping option traps over 99% of the spammers and leaves our real mail server to handle real mail.

For example, if spamd.alloweddomains contained the following:

@your_domain.org
critical.com
mary@your_domain.org

The following addresses will _NOT_ be spam trapped:

mary@your_domain.org
frank@your_domain.org
bob@critical.com
bobby@nobby.critical.com
robert@whoop.critical.com

The following address _WILL_ be spam trapped:

baker@test.your_domain.org     - invalid sub domain
marge@machine.your_domain.org  - invalid sub domain
bob@wrong_nam.com              - not your hostname

Starting Spamd in grey list mode

To get spamd working you can reboot the box and make sure that the spamd processes have started. You should see the user "_spamd" running on ports 8025 and 8026. If you run "ps -aux | grep spamd" you should see the following processes listed:

root@machine: ps -aux | grep spamd
_spamd   22041  0.0  0.1  9804  1160 ??  Is    Mon11AM    0:01.59 spamd: (pf  update) (spamd)
_spamd   22052  0.0  1.0  9748 10276 ??  S     Mon11AM    0:01.99 spamd: [priv] (greylist) (spamd)
_spamd   22063  0.0  0.1  9848  1236 ??  S     Mon11AM    0:00.14 spamd: (/var/db/spamd update) (spamd)
_spamd   22074  0.0  0.1   560  1124 ??  Ss    Mon11AM    0:02.29 /usr/libexec/spamlogd

Grey list mode activated

Now, all you have to do is wait for remote mail servers to connect and get grey listed. Use the command "spamd" to list out the database and watch as the hosts get trapped. When a valid mail server connects after the grey list time (25 minutes) and before the greylist timeout (4 hours) they will be whitelisted. Those host will then connect directly to the real mail server. All other hosts who do not connect again will have their entries delete from the database and they will have to start the process all over again.

IMPORTANT NOTE: web mail sites that use multiple mail servers to deliver one email

Web mail services like Google, Hotmail and some other sites use multiple email servers. This would not normally be a problem for grey listing except those services randomly attempt to deliver a single email from any of the mail servers in the pool. The first attempts comes from server_1 then the second from server_3 and the third might be attempted from server_1 again. This causes a problem with grey listing because it is possible that none of the mail servers will be used more than a few times to try to deliver the email. Thus, none of the servers will get white listed.

To fix this problem we have a few options:

Normal Volume Site: If your site gets enough emails from services like Google, Hotmail, etc then they will eventually get whitelisted due to the amount of mail being sent. You would really only need 5-10 emails to be sent at once to your mail server for all of the Gmail servers to eventually be whitelisted within an hour or so. With enough email grey listing is not a problem.

Low Volume Site: If your mail server is a personal server and you do not get a lot of emails then you may have a problem with Google, Hotmail, etc. You also have a few options available.

Option 1: You could manually whitelist the servers according to the MX records registered to the services. You would only need to white list less than 50% of them as the odds are that a single email will hit at least one of the whitelisted servers during attempted delivery. The rest of the servers will be whitelisted automatically over time.

Option 2: You could watch or have a script monitor the logs and send yourself an alert if one of the servers in question connect. This is more hands on, but it should only take a week or so to find all of the ip in question.

Option 3: Once they are white listed you can extend the whitelist time of 864 hours (-G25:4:864) with a script. If you grep for all the white listed entries and just add the ips again using "spamdb -a "ip" it will extend the whitexp time another 36 days (864 hours). Here is a shell script called "spamd_extend_whitelist.sh" that will list out the WHITE ips from spamdb and refresh their white expire time to 36 days. Run it in a cron job on the first of the month to, in effect, have a permanent white list. FYI: every time you use "spamdb -a" on an ip it looks like another new email was delivered according to spamd. In effect, the last column in the spamdb database updates by one.

#!/bin/sh
#
## Calomel.org .:. spamd_extend_whitelist.sh
#
for i in `spamdb | grep WHITE | awk -F "|" '{print $2}'`;
 do
  /usr/sbin/spamdb -a $i
done

Option 4: If the white listed ips are always going to be allowed then setup another table in Pf, put the white listed ips in there and always forward them to the real mail server.

Option 5: You could setup your own account on the free services and send yourself 10 emails at once. Looking at the spamd database you will see all of the ips that connect and they should get white listed normally. If not, just manually white list them.

How about a script? White listing using "spamd_whitelist.sh"

Whitelist script: spamd_whitelist.sh :If you wanted to create a whitelist of the most common grey list friendly mail servers that would also be a good option. Here is a script that will dig the host names of the listed domains and make a text file with the results.

#!/bin/sh
#
## Calomel.org  spamd_whitelist.sh
#

FILE=spamd-spf.txt

rm -f $FILE
touch $FILE

### This first list are for domains who have SPF records.
for domain in \
        aol.com \
        apple.com \
        amazon.com \
        s._spf.ebay.com \
        m._spf.ebay.com \
        p._spf.ebay.com \
        p2._spf.ebay.com \
        c._spf.ebay.com \
        gmx.net \
        _spf.google.com \
        spf-a.hotmail.com \
        spf-b.hotmail.com \
        spf-c.hotmail.com \
        spf-d.hotmail.com \
        _spf-a.microsoft.com \
        _spf-b.microsoft.com \
        _spf-c.microsoft.com \
        southwest.com
do
echo \#$domain >> $FILE;
dig $domain TXT +short | tr "\ " "\n" | grep ^ip4: | cut -d: -f2 >> $FILE;
done

### This second list is for domains you specifically want to allow
### that may not be publicly available or do not have SPF records.
echo \#privatelist >> $FILE;
for privatelist in \
         10.0.0.0/8 \
         127.0.0.0/8 \
         172.16.0.0/12 \
         192.168.0.0/16
do
echo $privatelist >> $FILE;
done

Output of spamd_whitelist.sh would look similar to this:

#aol.com
152.163.225.0/24
205.188.139.0/24
205.188.144.0/24
205.188.156.0/23
205.188.159.0/24
64.12.136.0/23
64.12.138.0/24
64.12.143.99/32
64.12.143.100/32
216.34.51.0/24
#_spf-a.microsoft.com
216.99.5.67
216.99.5.68
207.46.50.82
#_spf-b.microsoft.com
131.107.65.22
217.77.141.52
217.77.141.59
#_spf-c.microsoft.com
203.32.4.25
131.107.70.12
131.107.70.16
86.61.88.25

...many more lines...

Now that you have a list of hosts you can add them all as white list entries to a PF table we will call "company-white". We have to make a PF table because we can not add CIDR-formatted network blocks (i.e. 192.168/16 or 10/8) to spamdb. Using the following three(3) lines in pf will populate the table, redirect traffic and pass the packets to the mail server.

################ Tables ####################################
table <company-white> persist file "/tools/pf_company_whitelist"

################ Translation ###############################
rdr on $ExtIf inet proto tcp from <company-white> to ($ExtIf) port smtp tag SMTPD -> lo0 port smtp

################ Filtering #################################
pass in log on $ExtIf inet proto tcp from <company-white> to lo0 port smtp flags S/SA synproxy state tagged SMTPD

Example 2: Run Spamd with the sole purpose of annoying spammers

If you do not run a real mail server, but wish to annoy spammers then using spamd in blacklist mode is perfect. It will trap any connection sent to port 25 of your ip/host and stutter the connection. This will waste the time and queue memory of the remote system. The theory being, if you do not have a real mail server then why would a host connect to you unless they are a spammer.

Add the following lines. The line "spamd_black=YES" turns black list only mode on.

## Add the following to /etc/rc.conf.local
#
spamd_flags="-G25:4:864 -h your_hostname.org -l127.0.0.1 -n \"Sendmail 8.11.4/8.11.1\" -S10 -s1 -v"
spamd_black=YES
spamlogd_flags=""

These will start spamd with the following options:

• -4 - return error code 450 "Requested mail action not taken: mailbox unavailable" to the spammer. This is really not needed. The default error code 451 "Requested action aborted: local error in processing" is more successful in getting spammers to reconnect.
• -G25:4:864 - adjust the three time parameters for grey listing to pass time of 25 minutes, greyexp to 4 hours, and white expire time to 864 hours, approximately 36 days
• -h your_hostname.org - tell yet to be whitelisted hosts your hostname is "your_hostname.org"
• -l127.0.0.1 - listen on local host only
• -n \"Sendmail 8.11.4/8.11.1\" - tell yet to be whitelisted hosts your email server is this string. Since spammers are malicious it is best to annonomize your mail server. You can put anything here like a false mail server name or even a somthing making fun of spammers.
• -S10 - stutter at greylisted/blacklist hosts for the first 10 seconds of the connection (allowed 1-90)
• -s1 - stutter at a speed of 1 character per second (allowed 1-10)
• -v - verbose logging to the "daemon" log file. This will tell spamd to report the From: and To: fields in the logs.

Setting up pf

Now, add the following to your pf.conf. All these rules do is direct all connections from any remote host to port 25 on your machine directly to spamd. There they will be stuttered at and their time wasted.

## Add the following to /etc/pf.conf
#
################ Translation ###############################
#
# Spamd ( external mail attempts to spamd server )
 rdr on $ExtIf inet proto tcp from any to ($ExtIf) port smtp tag SPAMD -> lo0 port spamd
#
################ Filtering #################################
#
# $ExtIf inbound
 pass in log on $ExtIf inet proto tcp from any to lo0 port spamd flags S/SA synproxy state tagged SPAMD

Checking /etc/mail/spamd.conf

The spamd.conf file is used to list out black list files. We do _not_ suggest using blacklists because of the unknowns involved in their collection. We do not know what affiliations the people collecting the ips may have or their agenda. To be safe we suggest avoiding blacklists. Spamd in blacklist mode stutters all connections so this is really a moot point.

The following is all you need in your /etc/mail/spamd.conf as to void out all black lists.

all:\
        ::

Starting Spamd in black list mode

To get spamd working you can reboot the box and make sure that the spamd processes have started. You should see he user "_spamd" running on port 8025. If you run "ps -aux | grep spamd" you should see the following processes listed:

root@machine: ps -aux | grep spamd
_spamd   22063  0.0  0.1  9848  1236 ??  S     Mon11AM    0:00.14 spamd: (/var/db/spamd update) (spamd)
_spamd   22074  0.0  0.1   560  1124 ??  Ss    Mon11AM    0:02.29 /usr/libexec/spamlogd

Black list mode activated

Now, all you have to do is wait for remote mail servers to connect and they will be blacklisted. If you use the program "pftop" or use the command "systat states" you can watch the connections.

Questions?

Clients connecting to Spamd are overloading my network connection! What can I do?

If spamd is using to much bandwitdh by sending too many packets try increasing the stutter time "-S90" and the stutter speed "-s5". For example, 600 connections total is 600 packets per 5 seconds. The network would need to handle 120 packets average per second each way; around 180 kilobytes in each direction. This might still be too much bandwidth for your connection, but you could increase the values as needed or decrease the amount of connections spamd will accept with maxcon "-c". Maxcon may not exceed kern.maxfiles - 200, and defaults to 800. Reduce maxcon to 200 or 100. Figure out how much bandwidth you can afford to use on spamd. Each packet sent to and from a client connection to spamd is 1.5 kilobytes. If you want to spam trap 100 clients at 1 packet per second each then you will need (1.5*100)/1 = 150 kilobytes per second. If you increased the stutter time to 5 seconds then you would need (1.5*100)/5 = 30 kilobytes per second.

Do you have a script I can run to tally the results of Spamd?

Yes. We have a script called Spamd Statistics. It will go through the logs and tally all of the time spammers wasted, the amount of bandwidth your host used to respond to spammers and more.

How do I list out the Spamd database?

Use the command "spamdb" or sort the list with "spamdb | sort".

How can I change the log file for spamd (/var/log/daemon to /var/log/spamd)?

Spamd sends log messages to syslogd using facility daemon and, with increasing verbosity, level err, warn, info, and debug. The following syslog.conf section can be used to log connection details to a dedicated file:

!spamd
daemon.err;daemon.warn;daemon.info  /var/log/spamd

How about using the Greyscanning script by Bob Beck?

The grey scanning perl script will watch the spamd logs and make decisions on hosts that connect. We highly suggest at least taking a look at the script as we find it an incredibly powerful tool. It will "process spamdb output and look for patterns. Mainly we look at the greylist entries, and make some decisions about them. if they look excessively spammish, we take some action against them, by running spamdb -t -a to add them as a TRAPPED entry to spamd, meaning spamd in greylist mode will tarpit them for the next 24 hours." Bob Beck's greyscanner.pl

How can I extract just the ip addresses from spamdb?

spamdb | grep WHITE | awk -F "|" '{print $2}'

How about a script to count each type of spamdb entree and show the ips and host names of white listed entries?

Sure. This script will show the a numerically sorted list of white listed ip address and their related host names. Also, there is a count of GREY, WHITE, SPAMTRAP and TRAPPED entries. We use this to quickly scan who has been whitelisted and if their hostname looks valid.

#!/bin/sh
#
## Calomel.org .:. spamdb.sh
#
# ./spamdb.sh white = list out whitelisted ips and matching hostnames
# ./spamdb.sh stats = spamd statistics
#
if [ $1 = "white" ]
   then
    echo " "
     for i in `spamdb | grep WHITE | awk -F "|" '{print $2}' | sort `;
      do
       echo -n $i; echo -n -e "\t"; host $i | awk '{print $5}'
      done
    echo " "
fi

if [ $1 = "stats" ]
   then
     echo " "
     echo "Spamdb classes"
     echo -n "GREY     = ";spamdb | grep -c ^GREY
     echo -n "SPAMTRAP = ";spamdb | grep -c ^SPAMTRAP
     echo -n "TRAPPED  = ";spamdb | grep -c ^TRAPPED
     echo -n "WHITE    = ";spamdb | grep -c ^WHITE

     echo " "
     echo "Top 25 To: addresses"
     cat /var/log/daemon | grep " spamd" | grep ": (" | awk '{print $10}' | sort | uniq -c | sort -r | head -25
fi

What is the maximum white expire (whiteexp) time I can set?

864 hours is the upper limit. You can not set the white expire time (-G25:4:864) above this. Take a look for the script "spamd_extend_whitelist.sh" in the examples above. It will 'touch' all of the white listed spamdb entries and re-expire them in another 36 days. Run this script in a cron job on the first of the month and you will have a permanent white list.

How can I manually WHITE list ips addresses in the database?

spamdb -a 127.0.0.1

How can I manually remove ips addresses in the database?

spamdb -d 127.0.0.1

How can I add a SPAMTRAP email address?

spamdb -T -a 'spamtrap@domain.org' 

How can I manually TRAP (i.e. spamd will ignore) an ip for 24 hours?

spamdb -t -a 127.0.0.1

How can I convert Unix time to standard date format ?

date -r 1200337384

Where can I find more information about Spamd?

man spamd  ...and... man spamdb

Questions, comments, or suggestions? Contact Calomel.org