CARP Firewall Failover for OpenBSD



Home


On most LAN networks the single point of failure accessing the Internet is the firewall. When the firewall goes down then access in or out of the network is lost. If this is a local system then it is time to drop everything and walk to the server room. If this is a data center location then it is time for a road trip and hope we did not forget to bring anything we might need in the rush to leave. Either way you are going to flooded with emails saying, "is the Internet down?". Wouldn't it be nice if you had two or more firewalls that could fail over to each other if one of them goes down? We could then take our time getting the secondary firewall back up.

Thanks to the OpenBSD group, you now have that option. When a firewall fails over all state information will be inherited by the backup firewall. This means that none of the current TCP connections will ever drop and users will not notice an interruption in service. In an operational environment this is an absolute necessity.

CARP is the Common Address Redundancy Protocol. It's a free, secure alternative to the "Virtual Router Redundancy Protocol" and the "Hot Standby Router Protocol." CARP was created and is maintained by the OpenBSD project. Properly setup, Carp can be a secure insurance policy in case of software malfunction or hardware failure.

This document is here to help you setup your own firewall using OpenBSD and CARP. Make sure to use the latest version of the OS as there are always improvements being made. Also, check the errata to make sure you have applied any patches to the kernel. If you need assistance with patching check out our Patching OpenBSD kernel and packages Guide.

Lets take a look at the ideology of CARP, the procedure used to setup the firewall, what files need to edited and what to expect when it goes live.





The ideology behind "CARP firewall fail over" and setup

If you have ever lost a network card or hard drive on your firewall, then you know it would have been nice to have a hot spare. Nothing is more frustrating than having to fix a downed machine while users are waiting and, if this is a business, loosing money. Using OpenBSD and CARP we can setup two firewalls. Not only to serve as a primary and secondary firewall, but also make the fail over so seamless almost no one will notice. Lets get started.

We are going to setup two(2) firewalls called "fw0" and "fw1". Firewall fw0 will be the "MASTER" and it is used exclusively as long as it is up, running properly and we have not "demoted" it. Firewall fw1 is the "BACKUP" firewall and used _only_ if the MASTER fw0 firewall goes down. They both exchange and update state information from each other over the CARP interface (em2) and are ready to activate if the other firewall becomes unresponsive. The following ASCII picture show our network topology:

                                   WEB/Internet
                                         |
                   +--------------------/ \------------------+
                   |         carp1(em0) = 50.50.50.11        |
                   |                                         |
      50.50.50.10 em0                                       em0 50.50.50.20
                   |                                         |
               +-------+                    10.20.30.20  +-------+
               |  fw0  |- em2 ---- CARP Pfsync ---- em2 -|  fw1  |
               +-------+  10.20.30.10                    +-------+
                   |                                         |
    192.168.10.10 em1                                       em1 192.169.10.20
                   |                                         |
                   |         carp2(em1) = 192.196.10.11      |
                   +--------------------\ /------------------+
                                         |
                                   Internal LAN 
                                  (192.168.10/24)



Both firewalls have three(3) physical interfaces em0, em1 and em2 as well as two(2) virtual CARP interfaces; carp1 bound to em0 and carp2 bound to em1. Lets look at each in detail:





Configuring the two firewalls

The following two text boxes are a list of all of the files on both firewalls that need to be edited according to the ASCII picture above. Files include the network interfaces, virtual carp interfaces and system config options to accept CARP communication.



Master CARP Firewall (fw0)

## External interface nic (em0)
root@fw0: cat /etc/hostname.em0 
inet 50.50.50.10 255.255.255.0 NONE 

## Internal interface nic (em1)
root@fw0: cat /etc/hostname.em1 
inet 192.168.10.10 255.255.255.0 NONE 

## Pfsync interface nic (em2)
root@fw0: cat /etc/hostname.em2
inet 10.20.30.10 255.255.255.0 NONE 

## Pfsync sync-device bound to the pfsyn nic (em2)
root@fw0: cat /etc/hostname.pfsync0 
up syncdev em2

## Carp1 Virtual interface bound to the external nic (em0)
root@fw0: cat /etc/hostname.carp1
inet 50.50.50.11 255.255.255.0 50.50.50.255 vhid 1 advbase 20 advskew 0 carpdev em0 pass 6f650f0bb2471295614139fd9d212b45

## Carp2 Virtual interface bound to the internal nic (em1)
root@fw0: cat /etc/hostname.carp2
inet 192.168.10.11 255.255.255.0 192.168.10.255 vhid 2 advbase 20 advskew 0 carpdev em1 pass d0ecb9f6ed1c899bdc484cb0d26542ba

## Packet forwarding and CARP preempt (add the following)
root@fw0: cat /etc/sysctl.conf
net.inet.carp.allow=1    # Allow the firewall to accept CARP packets
net.inet.carp.preempt=1  # Allow firewalls to failover when one goes down
net.inet.ip.forwarding=1 # Allow packet forwarding through the firewalls

## Enable Pf (add the following)
root@fw0: cat /etc/rc.conf.local
pf=YES
pf_rules=/etc/pf.conf



Backup CARP Firewall (fw1)

## External interface nic (em0)
root@fw1: cat /etc/hostname.em0
inet 50.50.50.20 255.255.255.0 NONE

## Internal interface nic (em1)
root@fw1: cat /etc/hostname.em1
inet 192.168.10.20 255.255.255.0 NONE

## Pfsync interface nic (em2)
root@fw1: cat /etc/hostname.em2
inet 10.20.30.20 255.255.255.0 NONE

## Pfsync sync-device bound to the pfsyn nic (em2)
root@fw1: cat /etc/hostname.pfsync0
up syncdev em2

## Carp1 Virtual interface bound to the external nic (em0)
root@fw1: cat /etc/hostname.carp1
inet 50.50.50.11 255.255.255.0 50.50.50.255 vhid 1 advbase 20 advskew 10 carpdev em0 pass 6f650f0bb2471295614139fd9d212b45

## Carp2 Virtual interface bound to the internal nic (em1)
root@fw1: cat /etc/hostname.carp2
inet 192.168.10.11 255.255.255.0 192.168.10.255 vhid 2 advbase 20 advskew 10 carpdev em1 pass d0ecb9f6ed1c899bdc484cb0d26542ba

## Packet forwarding and CARP preempt (add the following)
root@fw1: cat /etc/sysctl.conf
net.inet.carp.allow=1    # Allow the firewall to accept CARP packets
net.inet.carp.preempt=1  # Allow firewalls to failover when one goes down
net.inet.ip.forwarding=1 # Allow packet forwarding through the firewalls

## Enable Pf (add the following)
root@fw1: cat /etc/rc.conf.local
pf=YES
pf_rules=/etc/pf.conf





Explanation of the Carp files

The carp1 and carp2 devices are assigned to both firewalls, but only the box negotiated as the MASTER firewall activate the virtual ips. The config files are setup in the following manner. This explanation is for fw0 carp1:



CARP: general information and notes

Protocols

Virtual MAC Address: The Carp virtual MAC is in the format 00-00-5e-00-01-xx where the last octet is filled in by the CARP vhid. For example if you look at the MASTER firewall you will see carp1 has the MAC address of 00-00-5e-00-01-01 and carp2 is 00-00-5e-00-01-02.

IP Protocol: CARP uses IP protocol number 112 (0x70).

Multicast Advertisements: CARP advertisements are multicast to the 224.0.0.18 for IPv4 or FF02::12 for IPv6 multicast groups.

TTL/Hop Limit: CARP packets are always sent with a TTL/HLIM of 255. This is so CARP packets which have crossed a subnet boundary, for example being passed on by a router, can be recognized and dropped.

Timers

The host that advertises the most frequently will become the master for the CARP group if adskew is the same. The timer values configured on each host are sent as part of the CARP advertisements so that all other hosts can make a determinate decision as to which host will become the master.

Advertisement Interval: This is the base interval at which CARP hellos will be sent. The default is 1 second and is configured with the advbase keyword.

Advertisement Skew: This value is used to skew the advertisement interval in order to make the host more or less preferred in becoming master. The valid range is 0 to 254, with lower values making the host more preferred to be master. The default is 0 and is configured with the advskew keyword.

The advertisement window is calculated by taking the skew, dividing it by 256 and adding it to the adbase interval. The CARP host will send a hello every "window" seconds.

Failover Timer: If a backup CARP host doesn't see an advertisement from the master for 3 consecutive advertisement windows then it assumes the master is down. It will take over the CARP group and start advertising itself as the master. The number of advertisement windows (3) to delay before assuming the master is down is hard-coded into CARP and is not tunable.

In the event that two or more hosts have the same timer values configured, the following behavior results:



You can reduce the power consumption of your firewall and keep track of system temperatures by using Power Management with apmd and Sensorsd hardware monitor (sensorsd.conf).





Option 1 of 2: The minimum necessary to setup Pf with Carp

We need to make sure pf knows how and where to send the state information between firewalls. The following lines in /etc/pf.conf are the absolute minimum necessary to get carp working. This will setup em2 as the CarpIf variable sending states to each firewall every second. The two pass lines will allow pf to accept Carpv2 broadcast packets from any Carp host on the network. Note: Only the master firewall at the time sends out broadcast packets. The backup firewalls just listen until they become the master.

Add the following lines or a similar setup to your existing pf.conf on both firewalls. Remember that the pf.conf on the master should be copied to all other firewalls. If the master goes down then the new master must act the same. If you need further assistance with Pf then check our OpenBSD Pf Firewall "how to" :

# cat /etc/pf.conf
#
################ Macros ###################################
### Interfaces ###
 ExtIf="em0"
 IntIf="em1"
 CarpIf="em2"

################ Filtering #################################
# CARP firewall fail over
pass on $CarpIf inet proto pfsync keep state
pass on { $ExtIf, $IntIf } inet proto carp keep state





Option 2 of 2: A full working pf.conf to setup Pf with Carp

The following is a fully working pf.conf to use as an example. There is a lot in the file so take your time when reading through it. To show an example of the format we setup:

  • Internet access from LAN clients to ports 80 and 443. LAN clients will also be NAT'ed to the carp1 ip address.
  • www port 80 access through the firewall on the carp1 interface to an internal webserver at Webserver="192.168.10.100"
  • Carp broadcast traffic is limited by the ip addresses of the firewalls we own even though broadcast packets are easily spoof-able. This is just to keep out the junk traffic.
  • The table BLOCKTEMP was made to house the clients who abuse our webserver by connecting more than 300 times in 120 seconds. Not CARP specific, but it is good practice to set upper limits on your services.
  • For more information about OpenBSD's Pf firewall and HFSC quality of service options check out our PF Config (pf.conf)

    ## Calomel.org Carp Example /etc/pf.conf
    #
    ################ OpenBSD pf.conf ##########################
    # Required order: options, normalization, queueing, translation, filtering.
    # Note: translation rules are first match while filter rules are last match.
    ################ Macros ###################################
    
    ### Interfaces ###
     ExtIf="em0"  # 50.50.50.10   / 50.50.50.20
     IntIf="em1"  # 192.168.10.10 / 192.168.10.20
     PFSync="em2" # 10.20.30.10   / 10.20.30.20
    
    ### Carp Virtual Interfaces ###
    # carp1 -- 50.50.50.11
    # carp2 -- 192.168.10.11
    
    ### Hosts ###
     CarpExt="{50.50.50.10, 50.50.50.20}"
     CarpInt="{192.168.10.10, 192.168.10.20}"
     IntNet="192.168.10/24"
    
    ### Ports ###
     LANPort="{80, 443}
     SshPort="8022"
     Webserver="192.168.10.100"
    
    ### Stateful Tracking Options ###
     SshSTO ="(max 5,   source-track rule, max-src-states 5, max-src-nodes 5,   max-src-conn-rate 5/60)"
     WebSTO ="(max 800, source-track rule, max-src-conn 200, max-src-nodes 500, max-src-conn-rate 300/120, overload  flush global)"
    
    ################ Tables ####################################
     table <BLOCKTEMP> persist
    
    ################ Options ##################################
    # Misc Options
     set debug urgent
     set require-order yes
     set block-policy drop
     set loginterface $ExtIf
     set state-policy if-bound
     set fingerprints "/etc/pf.os"
     set ruleset-optimization none
    
    # Timeout Options
     set optimization aggressive
     set timeout { frag 5, tcp.established 3600 }
     set timeout { tcp.first 20, tcp.closing 20, tcp.closed 20, tcp.finwait 20 }
     set timeout { udp.first 20, udp.single 20, udp.multiple 20 }
     set timeout { other.first 20, other.single 20, other.multiple 20 }
     set timeout { adaptive.start 5000, adaptive.end 10000 }
    
    ################ Normalization #############################
     scrub log on $ExtIf all random-id min-ttl 254 max-mss 1452 set-tos lowdelay reassemble tcp fragment reassemble
    
    ################ Translation ###############################
    # Network Address Translation (NAT)
     no rdr on lo0 from any to any
     nat on egress from $IntNet to any tag EGRESS -> carp1 port 1024:65535
    
    # Openssh
     rdr on $IntIf inet proto tcp from $IntNet to $IntIf port ssh tag OPENSSH -> lo0 port $SshPort
    
    # External Web traffic to an internal server
     rdr on $ExtIf proto tcp from any to carp1 port www tag WEB -> $Webserver port www
    
    # DENY rouge redirections
     no rdr
    
    ################ Filtering #################################
    # Deny spoofed packets
     antispoof log quick for { lo0 $ExtIf $IntIf }
    
    # Block blacklisted
     block in quick on $ExtIf from <BLOCKTEMP> to any
    
    # Block to/from illegal sources/destinations
     block in log quick inet  from no-route to any
     block in log quick inet  from any to 255.255.255.255
     block        quick inet6
    
    # CARP firewall failover
     pass quick log on $PFSync proto pfsync keep state (no-sync)
     pass in quick log on $ExtIf proto carp from $CarpExt to 224.0.0.18 keep state
     pass in quick log on $IntIf proto carp from $CarpInt to 224.0.0.18 keep state
    
    # BLOCK all in/out on all interfaces and log
     block        in log on $ExtIf
     block return in log on $IntIf
    
    # IntIf Inbound
     pass in log on $IntIf inet proto tcp  from $IntNet to any port $LANPort $TcpState 
     pass in log on $IntIf inet proto tcp  from $IntNet to lo0 port $SshPort $TcpState $SshIntSTO tagged OPENSSH
     pass in log on $IntIf inet proto icmp from $IntNet to $IntIf icmp-type 8 code 0 $UdpState
    
    # IntIf Outbound
     pass out log on $IntIf inet proto icmp from $IntIf to $IntNet icmp-type 8 code 0 $UdpState
    
    # ExtIf Inbound
     pass in log on $ExtIf inet proto tcp from any to $Webserver port www $SynState $WebSTO tagged WEB
    
    # $ExtIf Outbound
     pass out log on $ExtIf inet proto tcp  from $ExtIf to any $TcpState tagged EGRESS
     pass out log on $ExtIf inet proto tcp  from $ExtIf to any $TcpState
     pass out log on $ExtIf inet proto udp  from $ExtIf to any $UdpState tagged EGRESS
     pass out log on $ExtIf inet proto udp  from $ExtIf to any $UdpState
     pass out log on $ExtIf inet proto icmp from $ExtIf to any icmp-type 8 code 0 $UdpState
    
    ################ END #######################################
    





    For more information about OpenBSD's Pf firewall and HFSC quality of service options check out our PF Config (pf.conf) and PF quality of service HFSC "how to's".




    Testing the firewalls

    If you have not already you are welcome to reboot the firewalls now. It does not matter which one reboots first or what order they come up in. According to the CARP settings fw0 will always be the master when both firewalls are up and running. Once the firewalls are up make sure all the interfaces are active and pf loads

    Look at the interfaces with "ifconfig". You should see three physical interfaces (like em0, em1 and em2) and two carp interfaces (like carp1 and carp2). Make sure you also see pflog0 and pfsync0 in the list. If you do not see them then pf was not properly started with the lines in /etc/rc.conf.local (pf=YES). This is an example from our test box of what you should see on the MASTER firewall. Notice both carp1 and carp2 say "carp: MASTER" as this is an ifconfig of fw0.

    root@fw0: ifconfig
    lo0: flags=8049 mtu 33208
            groups: lo
            inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    em0: flags=8943 mtu 1500
            lladdr 00:1c:33:30:36:d5
            groups: egress
            media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
            status: active
            inet 50.50.50.10 netmask 0xffffff00 broadcast 50.50.50.255
            inet6 fe80::31b:31ff:3e00:46d5%em0 prefixlen 64 scopeid 0x1
    em1: flags=8943 mtu 1500
            lladdr 00:33:33:28:33:5a
            media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
            status: active
            inet 192.169.10.10 netmask 0xffffff00 broadcast 192.168.10.255
            inet6 fe30::330:48ff:fe33:ca3a%em1 prefixlen 64 scopeid 0x2
    em2: flags=8843 mtu 1500
            lladdr 00:33:33:33:c3:5b
            media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
            status: active
            inet 10.20.30.10 netmask 0xffffff00 broadcast 10.20.30.255
            inet6 fe33::333:483f:3338:ca3b%em2 prefixlen 64 scopeid 0x3
    enc0: flags=0<> mtu 1536
    pfsync0: flags=41 mtu 1460
            pfsync: syncdev: em2 syncpeer: 224.0.0.240 maxupd: 128
            groups: carp pfsync
    pflog0: flags=141 mtu 33208
            groups: pflog
    carp1: flags=8843 mtu 1500
            lladdr 00:00:5e:00:01:01
            carp: MASTER carpdev em0 vhid 1 advbase 1 advskew 1
            groups: carp
            inet6 fe80::333:3eff:fe33:101%carp1 prefixlen 64 scopeid 0x7
            inet 50.50.50.11 netmask 0xffffff00 broadcast 33.33.33.255
    carp2: flags=8843 mtu 1500
            lladdr 00:00:5e:00:01:02
            carp: MASTER carpdev em1 vhid 2 advbase 1 advskew 1
            groups: carp
            inet6 fe83::330:333f:3333:302%carp2 prefixlen 64 scopeid 0x8
            inet 192.168.10.11 netmask 0xffffff00 broadcast 10.10.10.255
    





    Watching the 'state' of things...

    systat states (OpenBSD 4.4 and after) In later versions of OpenBSD after v4.4 pftop has been removed since the author did not maintain it. Instead you can use the new version of systat with the argument "states". Like so, "systat states" and it will similar information to what pftop did. You can also execute "systat" and hit number "8" to show the same states page.

    Pftop (OpenBSD 4.3 and before) is an excellent program to make sure CARP pfsyncing is working. If you have an older version of OpenBSD install it by using "pkg_add -i pftop". After installing pftop you can run it on both firewalls and watch the states in real time. You will see that when the state is created on the MASTER firewall, CARP will sync the state to the BACKUP firewall within a second through the pfsync interface (em2). This is how both firewalls are kept in sync and how the BACKUP firewall can take over the duties of the MASTER.





    Simulating a firewall failure

    To simulate a firewall going down we have a few choices. Lets first look at the current state of the firewalls. To check the status of the firewall use "ifconfig carp". Look to see if the carp1 and carp2 interfaces are in one of these three states:

    Interface States

    INIT : All CARP interfaces start in this state. Also, when a CARP interface is admin down, i.e. "ifconfig em0 down", it is put into this state. When a CARP interface is admin up, it immediately transitions to BACKUP. Note that in OpenBSD 3.8 and earlier, a bug exists which will cause the host to transition to MASTER right away if preempt is enabled.

    BACKUP : The host is listening for advertisements from the master. If no advertisements are seen after 3 advertisement windows (adbase=20 for example) then assume the master is down, transition to MASTER state and start sending advertisements. If an advertisement is seen with a worse (higher) advertisement window than ours, and if preempt is enabled, transition to MASTER and start sending advertisements.

    MASTER : The host is forwarding traffic directed to the CARP virtual/group IP address(es). The host is also sending advertisements once per advertisement window (adbase=20) to announce its presence to other CARP hosts within the CARP group. The host still listens for advertisements from other CARP hosts. If an advertisement is seen with a better (lower or equal to ours) advertisement window, transition to BACKUP and allow the other host to become MASTER.

    IMPORTANT NOTE: changing any values associated with a CARP interface (timers, password, etc) will automatically result in the interface being put into the INIT state.

    Under normal circumstances, there can be multiple hosts within a CARP group in the BACKUP state, but only one host should ever be in MASTER state.

    To simulate the MASTER firewall (fw0) going down you can:

    Explanation of the Demotion Counter

    Instead of halting the box and rebooting you can "demote" the firewall, perfect when you need to work on the box and you need the second firewall to take over. The demotion counter is a value advertised by CARP hosts which announces how "ready" a host is to take on the role of master.

    Check the demote count of the box by executing:

    root@fw0: ifconfig -g carp
    carp: carp demote count 0
    

    The carpdemote (default=0) argument works exactly like the advskew directive in the carp file. Lets say we have two firewalls like in the example above. We need to calculate the possibility that each machine will become the master firewall in the group. To do this use the following equation, advbase+(advskew/256)+(carpdemote/256) . This means the master fw0 has a value of 20+(0/256)+(0/256)=20 and the backup has a value of 20+(10/256)+(0/256)=20.039 .

    In order for the master firewall to be "demoted" past the backup firewall we need to add a carpdemote value high enough to beat 20.39 of fw1. So, we try carpdemote=200 on fw0 and get 20+(0/256)+(200/256)=20.78 . 20.78 is higher than fw1's 20.39 and fw1 now becomes the master firewall.

    With a carpdemote=200 the MASTER firewall has a demote count of 20.78 the BACKUP will now be the MASTER. In effect they will switch places. The "demote" function is very useful when you need to take a firewall down for maintenance or when you want the box to be rebooted and not take over as the MASTER when it comes back up. To demote a CARP firewall execute the following on the MASTER firewall.

    root@fw0: ifconfig -g carp carpdemote 200
    root@fw0: ifconfig -g carp
    carp: carp demote count 200
    

    To reverse a carpdemote use "-carpdemote" and the value you want to subtract. Since we added 200 to the demote count in the previous step, lets set the demote count back to zero(0). Once the demote count is set back to zero(0) fw0 will be the MASTER again.

    root@fw0: ifconfig -g carp -carpdemote 200
    root@fw0: ifconfig -g carp
    carp: carp demote count 0
    





    Monitoring the CARP interface

    In order to better keep track of which machine is the master or slave you may want to take a look at our page, Ifstated interface monitor "how to". With Ifstated you have the ability to have emails sent to you when the firewalls change state or any other operation you want to add.



    Want more speed? Make sure to also check out the Network Speed and Performance Guide. With a little time and understanding you could easily double your firewall's throughput.





    Questions, comments, or suggestions? Contact Calomel.org