Home
On most LAN networks the
single point of failure accessing the Internet is the firewall. When the
firewall goes down then access in or out of the network is lost. If this is a
local system then it is time to drop everything and walk to the server room. If
this is a data center location then it is time for a road trip and hope we did
not forget to bring anything we might need in the rush to leave. Either way you
are going to flooded with emails saying, "is the Internet down?". Wouldn't it
be nice if you had two or more firewalls that could fail over to each other if
one of them goes down? We could then take our time getting the secondary
firewall back up.
Thanks to the OpenBSD group, you now have that option. When a firewall fails over all state information will be inherited by the backup firewall. This means that none of the current TCP connections will ever drop and users will not notice an interruption in service. In an operational environment this is an absolute necessity.
CARP is the Common Address Redundancy Protocol. It's a free, secure alternative to the "Virtual Router Redundancy Protocol" and the "Hot Standby Router Protocol." CARP was created and is maintained by the OpenBSD project. Properly setup, Carp can be a secure insurance policy in case of software malfunction or hardware failure.
This document is here to help you setup your own firewall using OpenBSD and CARP. Make sure to use the latest version of the OS as there are always improvements being made. Also, check the errata to make sure you have applied any patches to the kernel. If you need assistance with patching check out our Patching OpenBSD kernel and packages Guide.
Lets take a look at the ideology of CARP, the procedure used to setup the firewall, what files need to edited and what to expect when it goes live.
If you have ever lost a network card or hard drive on your firewall, then you know it would have been nice to have a hot spare. Nothing is more frustrating than having to fix a downed machine while users are waiting and, if this is a business, loosing money. Using OpenBSD and CARP we can setup two firewalls. Not only to serve as a primary and secondary firewall, but also make the fail over so seamless almost no one will notice. Lets get started.
We are going to setup two(2) firewalls called "fw0" and "fw1". Firewall fw0 will be the "MASTER" and it is used exclusively as long as it is up, running properly and we have not "demoted" it. Firewall fw1 is the "BACKUP" firewall and used _only_ if the MASTER fw0 firewall goes down. They both exchange and update state information from each other over the CARP interface (em2) and are ready to activate if the other firewall becomes unresponsive. The following ASCII picture show our network topology:
WEB/Internet
|
+--------------------/ \------------------+
| carp1(em0) = 50.50.50.11 |
| |
50.50.50.10 em0 em0 50.50.50.20
| |
+-------+ 10.20.30.20 +-------+
| fw0 |- em2 ---- CARP Pfsync ---- em2 -| fw1 |
+-------+ 10.20.30.10 +-------+
| |
192.168.10.10 em1 em1 192.169.10.20
| |
| carp2(em1) = 192.196.10.11 |
+--------------------\ /------------------+
|
Internal LAN
(192.168.10/24)
Both firewalls have three(3) physical interfaces em0, em1 and em2 as well as two(2) virtual CARP interfaces; carp1 bound to em0 and carp2 bound to em1. Lets look at each in detail:
The following two text boxes are a list of all of the files on both firewalls that need to be edited according to the ASCII picture above. Files include the network interfaces, virtual carp interfaces and system config options to accept CARP communication.
Master CARP Firewall (fw0)
## External interface nic (em0) root@fw0: cat /etc/hostname.em0 inet 50.50.50.10 255.255.255.0 NONE ## Internal interface nic (em1) root@fw0: cat /etc/hostname.em1 inet 192.168.10.10 255.255.255.0 NONE ## Pfsync interface nic (em2) root@fw0: cat /etc/hostname.em2 inet 10.20.30.10 255.255.255.0 NONE ## Pfsync sync-device bound to the pfsyn nic (em2) root@fw0: cat /etc/hostname.pfsync0 up syncdev em2 ## Carp1 Virtual interface bound to the external nic (em0) root@fw0: cat /etc/hostname.carp1 inet 50.50.50.11 255.255.255.0 50.50.50.255 vhid 1 advbase 20 advskew 0 carpdev em0 pass 6f650f0bb2471295614139fd9d212b45 ## Carp2 Virtual interface bound to the internal nic (em1) root@fw0: cat /etc/hostname.carp2 inet 192.168.10.11 255.255.255.0 192.168.10.255 vhid 2 advbase 20 advskew 0 carpdev em1 pass d0ecb9f6ed1c899bdc484cb0d26542ba ## Packet forwarding and CARP preempt (add the following) root@fw0: cat /etc/sysctl.conf net.inet.carp.allow=1 # Allow the firewall to accept CARP packets net.inet.carp.preempt=1 # Allow firewalls to failover when one goes down net.inet.ip.forwarding=1 # Allow packet forwarding through the firewalls ## Enable Pf (add the following) root@fw0: cat /etc/rc.conf.local pf=YES pf_rules=/etc/pf.conf
Backup CARP Firewall (fw1)
## External interface nic (em0) root@fw1: cat /etc/hostname.em0 inet 50.50.50.20 255.255.255.0 NONE ## Internal interface nic (em1) root@fw1: cat /etc/hostname.em1 inet 192.168.10.20 255.255.255.0 NONE ## Pfsync interface nic (em2) root@fw1: cat /etc/hostname.em2 inet 10.20.30.20 255.255.255.0 NONE ## Pfsync sync-device bound to the pfsyn nic (em2) root@fw1: cat /etc/hostname.pfsync0 up syncdev em2 ## Carp1 Virtual interface bound to the external nic (em0) root@fw1: cat /etc/hostname.carp1 inet 50.50.50.11 255.255.255.0 50.50.50.255 vhid 1 advbase 20 advskew 10 carpdev em0 pass 6f650f0bb2471295614139fd9d212b45 ## Carp2 Virtual interface bound to the internal nic (em1) root@fw1: cat /etc/hostname.carp2 inet 192.168.10.11 255.255.255.0 192.168.10.255 vhid 2 advbase 20 advskew 10 carpdev em1 pass d0ecb9f6ed1c899bdc484cb0d26542ba ## Packet forwarding and CARP preempt (add the following) root@fw1: cat /etc/sysctl.conf net.inet.carp.allow=1 # Allow the firewall to accept CARP packets net.inet.carp.preempt=1 # Allow firewalls to failover when one goes down net.inet.ip.forwarding=1 # Allow packet forwarding through the firewalls ## Enable Pf (add the following) root@fw1: cat /etc/rc.conf.local pf=YES pf_rules=/etc/pf.conf
Explanation of the Carp files
The carp1 and carp2 devices are assigned to both firewalls, but only the box negotiated as the MASTER firewall activate the virtual ips. The config files are setup in the following manner. This explanation is for fw0 carp1:
vhid 1 is the Virtual Host ID. This is a unique number which is used to identify the redundancy group to other nodes on the network. Acceptable values are from 1 to 255. The "vhid" must match the same carp interface on both firewalls. Notice carp1 on fw0 matches the vhid of carp1 on fw1. To reduce confusion it is a good practice to keep your vhid equal to the carp number. For example, carp1 is vhid1 and carp2 is vhid2.
So that the option "advskew" and the command "carpdemote" work properly it is highly advisable to keep the advbase the same value on all of the carp devices on all of the firewalls.
An advskew of 0 means this box should always be the master and will be so immediately upon reboot. With an advskew of 10 the backup firewall fw1 will wait upon reboot and see if any other servers have a lower advskew. If the master is not up then the backup will become the master in about 20x3=60 seconds. To calculate the effect of advskew on the advbase use the following method, advbase+(advskew/256). In our example the master firewall (fw0) is 20+(0/256)=20 and the backup firewall (fw1) is 20+(10/256)=20.039. If we had a third firewall and its advskew was set to 200 then its value would be 20+(200/256)=20.781. This means the firewall that is active with the lowest value will be the master firewall. fw0 is the master unless down, then fw1 is the master.
carpdev em0 is the physical network interface that this carp virtual ip will bind to.
pass 6f650f... is the authentication password used when talking to other CARP-enabled hosts in this redundancy group. This must be the same on all members of the group. When carp1 on the first firewall intends to talk to carp1 on the second firewall they need to have matching passwords. CARP uses a cryptographically strong SHA-1 HMAC to protect each advertisement. We suggest a password of 30 characters which is the maximum length allowed.
To help calculate a random password for carp you can use the following line. It queries /dev/urandom and outputs a 30 character string which you can cut and paste in place.
cat /dev/urandom | hexdump -n 30| cut -d \ -f 2-| head -n 1 | tr -d " "
Protocols
Virtual MAC Address: The Carp virtual MAC is in the format 00-00-5e-00-01-xx where the last octet is filled in by the CARP vhid. For example if you look at the MASTER firewall you will see carp1 has the MAC address of 00-00-5e-00-01-01 and carp2 is 00-00-5e-00-01-02.
IP Protocol: CARP uses IP protocol number 112 (0x70).
Multicast Advertisements: CARP advertisements are multicast to the 224.0.0.18 for IPv4 or FF02::12 for IPv6 multicast groups.
TTL/Hop Limit: CARP packets are always sent with a TTL/HLIM of 255. This is so CARP packets which have crossed a subnet boundary, for example being passed on by a router, can be recognized and dropped.
Timers
The host that advertises the most frequently will become the master for the CARP group if adskew is the same. The timer values configured on each host are sent as part of the CARP advertisements so that all other hosts can make a determinate decision as to which host will become the master.
Advertisement Interval: This is the base interval at which CARP hellos will be sent. The default is 1 second and is configured with the advbase keyword.
Advertisement Skew: This value is used to skew the advertisement interval in order to make the host more or less preferred in becoming master. The valid range is 0 to 254, with lower values making the host more preferred to be master. The default is 0 and is configured with the advskew keyword.
The advertisement window is calculated by taking the skew, dividing it by 256 and adding it to the adbase interval. The CARP host will send a hello every "window" seconds.
Failover Timer: If a backup CARP host doesn't see an advertisement from the master for 3 consecutive advertisement windows then it assumes the master is down. It will take over the CARP group and start advertising itself as the master. The number of advertisement windows (3) to delay before assuming the master is down is hard-coded into CARP and is not tunable.
In the event that two or more hosts have the same timer values configured, the following behavior results:
- If preempt is disabled: whichever host starts advertising first (i.e., is configured first) will become the master.
- If preempt is enabled: whichever host starts advertising last (i.e., is configured last) will become the master.
We need to make sure pf knows how and where to send the state information between firewalls. The following lines in /etc/pf.conf are the absolute minimum necessary to get carp working. This will setup em2 as the CarpIf variable sending states to each firewall every second. The two pass lines will allow pf to accept Carpv2 broadcast packets from any Carp host on the network. Note: Only the master firewall at the time sends out broadcast packets. The backup firewalls just listen until they become the master.
Add the following lines or a similar setup to your existing pf.conf on both firewalls. Remember that the pf.conf on the master should be copied to all other firewalls. If the master goes down then the new master must act the same. If you need further assistance with Pf then check our OpenBSD Pf Firewall "how to" :
# cat /etc/pf.conf
#
################ Macros ###################################
### Interfaces ###
ExtIf="em0"
IntIf="em1"
CarpIf="em2"
################ Filtering #################################
# CARP firewall fail over
pass on $CarpIf inet proto pfsync keep state
pass on { $ExtIf, $IntIf } inet proto carp keep state
The following is a fully working pf.conf to use as an example. There is a lot in the file so take your time when reading through it. To show an example of the format we setup:
For more information about OpenBSD's Pf firewall and HFSC quality of service options check out our PF Config (pf.conf)
## Calomel.org Carp Example /etc/pf.conf
#
################ OpenBSD pf.conf ##########################
# Required order: options, normalization, queueing, translation, filtering.
# Note: translation rules are first match while filter rules are last match.
################ Macros ###################################
### Interfaces ###
ExtIf="em0" # 50.50.50.10 / 50.50.50.20
IntIf="em1" # 192.168.10.10 / 192.168.10.20
PFSync="em2" # 10.20.30.10 / 10.20.30.20
### Carp Virtual Interfaces ###
# carp1 -- 50.50.50.11
# carp2 -- 192.168.10.11
### Hosts ###
CarpExt="{50.50.50.10, 50.50.50.20}"
CarpInt="{192.168.10.10, 192.168.10.20}"
IntNet="192.168.10/24"
### Ports ###
LANPort="{80, 443}
SshPort="8022"
Webserver="192.168.10.100"
### Stateful Tracking Options ###
SshSTO ="(max 5, source-track rule, max-src-states 5, max-src-nodes 5, max-src-conn-rate 5/60)"
WebSTO ="(max 800, source-track rule, max-src-conn 200, max-src-nodes 500, max-src-conn-rate 300/120, overload flush global)"
################ Tables ####################################
table <BLOCKTEMP> persist
################ Options ##################################
# Misc Options
set debug urgent
set require-order yes
set block-policy drop
set loginterface $ExtIf
set state-policy if-bound
set fingerprints "/etc/pf.os"
set ruleset-optimization none
# Timeout Options
set optimization aggressive
set timeout { frag 5, tcp.established 3600 }
set timeout { tcp.first 20, tcp.closing 20, tcp.closed 20, tcp.finwait 20 }
set timeout { udp.first 20, udp.single 20, udp.multiple 20 }
set timeout { other.first 20, other.single 20, other.multiple 20 }
set timeout { adaptive.start 5000, adaptive.end 10000 }
################ Normalization #############################
scrub log on $ExtIf all random-id min-ttl 254 max-mss 1452 set-tos lowdelay reassemble tcp fragment reassemble
################ Translation ###############################
# Network Address Translation (NAT)
no rdr on lo0 from any to any
nat on egress from $IntNet to any tag EGRESS -> carp1 port 1024:65535
# Openssh
rdr on $IntIf inet proto tcp from $IntNet to $IntIf port ssh tag OPENSSH -> lo0 port $SshPort
# External Web traffic to an internal server
rdr on $ExtIf proto tcp from any to carp1 port www tag WEB -> $Webserver port www
# DENY rouge redirections
no rdr
################ Filtering #################################
# Deny spoofed packets
antispoof log quick for { lo0 $ExtIf $IntIf }
# Block blacklisted
block in quick on $ExtIf from <BLOCKTEMP> to any
# Block to/from illegal sources/destinations
block in log quick inet from no-route to any
block in log quick inet from any to 255.255.255.255
block quick inet6
# CARP firewall failover
pass quick log on $PFSync proto pfsync keep state (no-sync)
pass in quick log on $ExtIf proto carp from $CarpExt to 224.0.0.18 keep state
pass in quick log on $IntIf proto carp from $CarpInt to 224.0.0.18 keep state
# BLOCK all in/out on all interfaces and log
block in log on $ExtIf
block return in log on $IntIf
# IntIf Inbound
pass in log on $IntIf inet proto tcp from $IntNet to any port $LANPort $TcpState
pass in log on $IntIf inet proto tcp from $IntNet to lo0 port $SshPort $TcpState $SshIntSTO tagged OPENSSH
pass in log on $IntIf inet proto icmp from $IntNet to $IntIf icmp-type 8 code 0 $UdpState
# IntIf Outbound
pass out log on $IntIf inet proto icmp from $IntIf to $IntNet icmp-type 8 code 0 $UdpState
# ExtIf Inbound
pass in log on $ExtIf inet proto tcp from any to $Webserver port www $SynState $WebSTO tagged WEB
# $ExtIf Outbound
pass out log on $ExtIf inet proto tcp from $ExtIf to any $TcpState tagged EGRESS
pass out log on $ExtIf inet proto tcp from $ExtIf to any $TcpState
pass out log on $ExtIf inet proto udp from $ExtIf to any $UdpState tagged EGRESS
pass out log on $ExtIf inet proto udp from $ExtIf to any $UdpState
pass out log on $ExtIf inet proto icmp from $ExtIf to any icmp-type 8 code 0 $UdpState
################ END #######################################
If you have not already you are welcome to reboot the firewalls now. It does not matter which one reboots first or what order they come up in. According to the CARP settings fw0 will always be the master when both firewalls are up and running. Once the firewalls are up make sure all the interfaces are active and pf loads
Look at the interfaces with "ifconfig". You should see three physical interfaces (like em0, em1 and em2) and two carp interfaces (like carp1 and carp2). Make sure you also see pflog0 and pfsync0 in the list. If you do not see them then pf was not properly started with the lines in /etc/rc.conf.local (pf=YES). This is an example from our test box of what you should see on the MASTER firewall. Notice both carp1 and carp2 say "carp: MASTER" as this is an ifconfig of fw0.