High availability

Active/Standby clustering

On this page

Using HAProxy Fusion?

If you’re using HAProxy Fusion, then see the HAProxy Fusion - High availability topic instead.

In an active-standby cluster, one HAProxy Enterprise node receives traffic while a second node is put on standby. In case the active node fails, the standby takes over. With HAProxy Enterprise, you configure an active-standby cluster by installing the Virtual Router Redundancy Protocol (VRRP) module.

The HAProxy Enterprise VRRP module assigns a virtual, static IP address to your load balancer node. If the node fails, the IP address will instantly transfer to a backup node, avoiding any break in service. Your router can continue to send traffic to the same IP address for the load balancer, never knowing that there was a failover. The module utilizes a stable version of Keepalived, which implements VRRP.

In this guide, we will set up two load balancers: one active and the other on standby.

Active/Standby with VRRP

Prerequisites Jump to heading

If you have a firewall running on the server, such as firewalld (the default on Red Hat Enterprise Linux) or iptables, then be sure to add an exception for VRRP traffic.

Add an exception to firewalld

To add an exception for VRRP to firewalld:

  1. Check if firewalld is running:

    nix
    sudo firewall-cmd --state
    nix
    sudo firewall-cmd --state
    output
    running
    output
    running

  2. Check which firewalld zones are active for your network interfaces. Below, our active zone is named public:

    nix
    sudo firewall-cmd --get-active-zones
    nix
    sudo firewall-cmd --get-active-zones
    output
    public
  interfaces: eth0 eth1
    output
    public
  interfaces: eth0 eth1

  3. To ensure that existing runtime rules (those that are in effect) are persisted, save them as permanent before going further:

    nix
    sudo firewall-cmd --runtime-to-permanent 
    nix
    sudo firewall-cmd --runtime-to-permanent 
    output
    success
    output
    success

  4. Add the VRRP protocol to the list of rules. Below, we use the zone named public:

    nix
    sudo firewall-cmd --zone=public --add-protocol=vrrp --permanent
    sudo firewall-cmd --reload
    nix
    sudo firewall-cmd --zone=public --add-protocol=vrrp --permanent
    sudo firewall-cmd --reload
    output
    success
    output
    success

  5. Verify that the protocol has been added. Below, we use the zone named public:

    nix
    sudo firewall-cmd --zone=public --list-all | grep vrrp
    nix
    sudo firewall-cmd --zone=public --list-all | grep vrrp
    protocols: vrrp
    protocols: vrrp

  6. If you need to revert this change, run these commands:

    nix
    sudo firewall-cmd --zone=public --remove-protocol=vrrp --permanent
    sudo firewall-cmd --reload
    nix
    sudo firewall-cmd --zone=public --remove-protocol=vrrp --permanent
    sudo firewall-cmd --reload
Add an exception to iptables

To add an exception for VRRP to iptables:

  1. Save a backup of your current iptables configuration:

    nix
    sudo iptables-save > iptables.backup.rules
    nix
    sudo iptables-save > iptables.backup.rules
    output
    # Generated by iptables-save v1.8.5 on Wed Jun 26 20:40:43 2024
    output
    # Generated by iptables-save v1.8.5 on Wed Jun 26 20:40:43 2024

  2. Add the VRRP protocol to the list of rules. Below, we specify the interface eth0:

    nix
    sudo iptables -A INPUT -p vrrp -i eth0 -j ACCEPT && sudo iptables -A OUTPUT -p vrrp -j ACCEPT
    nix
    sudo iptables -A INPUT -p vrrp -i eth0 -j ACCEPT && sudo iptables -A OUTPUT -p vrrp -j ACCEPT

  3. Verify that the protocol number 112 (VRRP) has been added:

    nix
    sudo iptables -nvL | grep 112
    nix
    sudo iptables -nvL | grep 112
    output
    0     0 ACCEPT     112  --  eth0   *       0.0.0.0/0            0.0.0.0/0
0     0 ACCEPT     112  --  *      *       0.0.0.0/0            0.0.0.0/0
    output
    0     0 ACCEPT     112  --  eth0   *       0.0.0.0/0            0.0.0.0/0
0     0 ACCEPT     112  --  *      *       0.0.0.0/0            0.0.0.0/0

  4. Save the configuration:

    nix
    sudo iptables-save > /iptables.vrrp.rules
    nix
    sudo iptables-save > /iptables.vrrp.rules

  5. If you need to revert this change, run this command:

    nix
    sudo iptables-restore < iptables.backup.rules
    nix
    sudo iptables-restore < iptables.backup.rules

Installation Jump to heading

Follow the steps on your active and backup load balancers.

Active load balancer Jump to heading

On the active load balancer, follow these steps:

  1. Install the VRRP module using your system’s package manager:

    nix
    sudo apt-get install hapee-extras-vrrp
    nix
    sudo apt-get install hapee-extras-vrrp
    nix
    sudo yum install hapee-extras-vrrp
    nix
    sudo yum install hapee-extras-vrrp
    nix
    sudo zypper install hapee-extras-vrrp
    nix
    sudo zypper install hapee-extras-vrrp
    nix
    sudo pkg install hapee-extras-vrrp
    nix
    sudo pkg install hapee-extras-vrrp

  2. Edit the file /etc/hapee-extras/hapee-vrrp.cfg.

    • In the vrrp_instance section, change the interface value from eth0 to the name of a network interface that receives traffic on the server. This is the same interface that you the bind HAProxy Enterprise’s frontend to. For example, if traffic reaches the load balancer using interface enp0s8. Set this line to interface enp0s8.

    • Use the same interface name in the track_interface block.

    • Change the IP address listed in the virtual_ipaddress_excluded block to the virtual IP address you’d like to assign to this interface. HAProxy Enterprise will bind to this address to receive traffic. If needed, add more IP addresses here, each on its own line. The new address(es) should fall within the interface’s IP subnet but shouldn’t already be assigned to any server.

    hapee-vrrp.cfg
    text
    vrrp_instance vrrp_1 {
      interface enp0s8       # Change network interface name
      state MASTER
      virtual_router_id 51
      priority 101
      virtual_ipaddress_excluded {
        192.168.50.10        # New IP address
      }
      track_interface {
        enp0s8 weight -2     # Change network interface name
      }
      track_script {
        chk_sshd
        chk_lb
      }
    }
    hapee-vrrp.cfg
    text
    vrrp_instance vrrp_1 {
      interface enp0s8       # Change network interface name
      state MASTER
      virtual_router_id 51
      priority 101
      virtual_ipaddress_excluded {
        192.168.50.10        # New IP address
      }
      track_interface {
        enp0s8 weight -2     # Change network interface name
      }
      track_script {
        chk_sshd
        chk_lb
      }
    }

  3. Enable and start the hapee-extras-vrrp service:

    nix
    sudo systemctl enable hapee-extras-vrrp
    sudo systemctl unmask hapee-extras-vrrp
    sudo systemctl start hapee-extras-vrrp
    nix
    sudo systemctl enable hapee-extras-vrrp
    sudo systemctl unmask hapee-extras-vrrp
    sudo systemctl start hapee-extras-vrrp

  4. Allow the server to bind to the virtual IP address.

    Edit the file /etc/sysctl.conf and add the net.ipv4.ip_nonlocal_bind=1 directive. This directive allows the server to accept connections for IP addresses that aren’t bound to any of its interfaces, enabling the use of a floating, virtual IP.

    sysctl.conf
    ini
    net.ipv4.ip_nonlocal_bind=1
    sysctl.conf
    ini
    net.ipv4.ip_nonlocal_bind=1

  5. Configure your bind line in the HAProxy Enterprise configuration to use the virtual IP address.

    haproxy
    frontend myfrontend
      mode http
      bind 192.168.50.10:80
      default_backend web_servers
    haproxy
    frontend myfrontend
      mode http
      bind 192.168.50.10:80
      default_backend web_servers

  6. Reboot the server.

Standby load balancer Jump to heading

On the standby load balancer, follow these steps:

  1. Install the VRRP module using your system’s package manager:

    nix
    sudo apt-get install hapee-extras-vrrp
    nix
    sudo apt-get install hapee-extras-vrrp
    nix
    sudo yum install hapee-extras-vrrp
    nix
    sudo yum install hapee-extras-vrrp
    nix
    sudo zypper install hapee-extras-vrrp
    nix
    sudo zypper install hapee-extras-vrrp
    nix
    sudo pkg install hapee-extras-vrrp
    nix
    sudo pkg install hapee-extras-vrrp

  2. Edit the VRRP configuration /etc/hapee-extras/hapee-vrrp.cfg. It must be almost exactly the same as the file on the active load balancer, except:

    • change the priority field in the vrrp_instance block to have a lower number than the active node; the load balancer with the highest priority is promoted to be the active node. For example, if the priority on the active node is 101, then set the backup node’s priority to 100.

    • change state to BACKUP.

    hapee-vrrp.cfg
    text
    vrrp_instance vrrp_1 {
      interface enp0s8
      state BACKUP
      virtual_router_id 51
      priority 100              # A lower priority value than the primary
      virtual_ipaddress_excluded {
        192.168.50.10           # Same IP address as on primary
      }
      track_interface {
        enp0s8 weight -2
      }
      track_script {
        chk_sshd
        chk_lb
      }
    }
    hapee-vrrp.cfg
    text
    vrrp_instance vrrp_1 {
      interface enp0s8
      state BACKUP
      virtual_router_id 51
      priority 100              # A lower priority value than the primary
      virtual_ipaddress_excluded {
        192.168.50.10           # Same IP address as on primary
      }
      track_interface {
        enp0s8 weight -2
      }
      track_script {
        chk_sshd
        chk_lb
      }
    }

  3. Enable and start the hapee-extras-vrrp service:

    nix
    sudo systemctl enable hapee-extras-vrrp
    sudo systemctl unmask hapee-extras-vrrp
    sudo systemctl start hapee-extras-vrrp
    nix
    sudo systemctl enable hapee-extras-vrrp
    sudo systemctl unmask hapee-extras-vrrp
    sudo systemctl start hapee-extras-vrrp

  4. Allow the server to bind to the virtual IP address.

    Edit the file /etc/sysctl.conf and add the net.ipv4.ip_nonlocal_bind=1 directive. This directive allows the server to accept connections for IP addresses that aren’t bound to any of its interfaces, enabling the use of a floating, virtual IP.

    sysctl.conf
    ini
    net.ipv4.ip_nonlocal_bind=1
    sysctl.conf
    ini
    net.ipv4.ip_nonlocal_bind=1

  5. Configure your bind line in the HAProxy Enterprise configuration to use the virtual IP address.

    haproxy
    frontend myfrontend
      mode http
      bind 192.168.50.10:80
      default_backend web_servers
    haproxy
    frontend myfrontend
      mode http
      bind 192.168.50.10:80
      default_backend web_servers

  6. Reboot the server.

Verify the setup Jump to heading

To check that the VRRP service is running, use systemctl status:

nix
sudo systemctl status hapee-extras-vrrp --no-pager
nix
sudo systemctl status hapee-extras-vrrp --no-pager
output
text
● hapee-extras-vrrp.service - HAPEE VRRP : VRRP daemon (Keepalived)
     Loaded: loaded (/lib/systemd/system/hapee-extras-vrrp.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-10-18 18:01:59 UTC; 10min ago
   Main PID: 1191 (hapee-vrrp)
      Tasks: 2 (limit: 4555)
     Memory: 1.5M
        CPU: 4.632s
     CGroup: /system.slice/hapee-extras-vrrp.service
             ├─1191 /opt/hapee-extras/sbin/hapee-vrrp -D -f /etc/hapee-extras/hapee-vrrp.cfg -p /var/run/hapee-extras/hapee-vrrp.pid
             └─1192 /opt/hapee-extras/sbin/hapee-vrrp -D -f /etc/hapee-extras/hapee-vrrp.cfg -p /var/run/hapee-extras/hapee-vrrp.pid
Oct 18 18:02:01 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) forcing a new MASTER election
Oct 18 18:02:02 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Transition to MASTER STATE
Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Entering MASTER STATE
Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) setting protocol E-VIPs.
Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Sending gratuitous ARPs on eth0 for 172.1.1.4
output
text
● hapee-extras-vrrp.service - HAPEE VRRP : VRRP daemon (Keepalived)
     Loaded: loaded (/lib/systemd/system/hapee-extras-vrrp.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-10-18 18:01:59 UTC; 10min ago
   Main PID: 1191 (hapee-vrrp)
      Tasks: 2 (limit: 4555)
     Memory: 1.5M
        CPU: 4.632s
     CGroup: /system.slice/hapee-extras-vrrp.service
             ├─1191 /opt/hapee-extras/sbin/hapee-vrrp -D -f /etc/hapee-extras/hapee-vrrp.cfg -p /var/run/hapee-extras/hapee-vrrp.pid
             └─1192 /opt/hapee-extras/sbin/hapee-vrrp -D -f /etc/hapee-extras/hapee-vrrp.cfg -p /var/run/hapee-extras/hapee-vrrp.pid
Oct 18 18:02:01 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) forcing a new MASTER election
Oct 18 18:02:02 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Transition to MASTER STATE
Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Entering MASTER STATE
Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) setting protocol E-VIPs.
Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Sending gratuitous ARPs on eth0 for 172.1.1.4

Manual failover Jump to heading

To activate the standby load balancer while simultaneously putting the active load balancer into standby mode:

  1. Edit the VRRP configuration /etc/hapee-extras/hapee-vrrp.cfg on the active load balancer.

  2. Lower the priority value to less than the standby load balancer’s priority. This will put the active load balancer into the standby state and simultaneously activate the standby load balancer. For example, change it from 101 to 90.

  3. Restart the hapee-extras-vrrp service.

    Reverse these actions to restore the original state.

Failover triggers Jump to heading

The following events can trigger a failover:

  • The active node lowers its weight below one of the backup nodes due to a failed health check.
  • A backup node is reconfigured with a weight larger than the current active node.
  • The active node stops emitting its heartbeat packet to the cluster.

VRRP health check scripts Jump to heading

In the VRRP configuration, the vrrp_script blocks define health checks that can trigger a failover. By default, our configuration checks whether the SSH daemon is running and whether HAProxy Enterprise is running. You may want to add more checks.

Consider this block, which checks the status of the SSH daemon every five seconds. If the service is not available, the VRRP instance’s weight (its priority) is reduced by 4 points:

hapee-vrrp.cfg
text
vrrp_script chk_sshd {
  script "pkill -0 sshd"  # pkill -0 is cheaper than pidof
  interval 5              # check every 5 seconds
  weight -4               # remove 4 points of prio if missing
  fall 2                  # check twice before setting down
  rise 1                  # check once before setting up
}
hapee-vrrp.cfg
text
vrrp_script chk_sshd {
  script "pkill -0 sshd"  # pkill -0 is cheaper than pidof
  interval 5              # check every 5 seconds
  weight -4               # remove 4 points of prio if missing
  fall 2                  # check twice before setting down
  rise 1                  # check once before setting up
}

The HAProxy Enterprise health check is similar, except if the hapee-lb process is running, it adds 6 points. The reason is, in case one load balancer has SSH working but not hapee-lb, while the other has hapee-lb but not SSH, the one with hapee-lb will become the active node, even though it is not manageable remotely.

hapee-vrrp.cfg
text
vrrp_script chk_lb {
  script "pkill -0 hapee-lb"  # pkill -0 is cheaper than pidof
  interval 1                  # check every second
  weight 6                    # add 6 points of prio if present
  fall 2                      # check twice before setting down
  rise 1                      # check once before setting up
}
hapee-vrrp.cfg
text
vrrp_script chk_lb {
  script "pkill -0 hapee-lb"  # pkill -0 is cheaper than pidof
  interval 1                  # check every second
  weight 6                    # add 6 points of prio if present
  fall 2                      # check twice before setting down
  rise 1                      # check once before setting up
}

The following fields define a vrrp_script block:

Name Argument type Description
script string Command to run. Running pkill -0 checks if any process with the given name is running. If it returns 0, the check passes; if it returns 1, the check fails.
interval integer Interval (in seconds) between two checks.
weight integer Points to add or remove to instance weight.
fall integer Number of consecutive negative checks before considered as down.
rise integer Number of consecutive positive checks before considered as up.

VRRP instance reference Jump to heading

The following fields define a vrrp_instance block:

Name Argument type Description
vrrp_instance string Describes a new VRRP instance.
interface string Interface managed by the VRRP module.
state string State at startup until VRRP priority negotiation completes. Can be either MASTER or BACKUP.
virtual_router_id integer VRRP instance identifier, which must be common to all nodes of the same cluster. Its value ranges from 0 to 255.
priority integer Weight of local node in this VRRP instance. NOTE This won’t work on cloud provider networks that disable IP multicast.
virtual_ipaddress string List of virtual IP addresses to add when the state changes to MASTER or remove when it changes to BACKUP. All VRRP nodes in a cluster must own the same IP. NOTE This won’t work on cloud provider networks that disable IP multicast.
virtual_ipaddress_excluded string The same as virtual_ipaddress, but IPs aren’t announced in the VRRP heartbeat packet, which reduces bandwidth usage when sending packets.
track_interface string Tracks interface status and updates priority accordingly.
track_script string Runs the health check scripts and updates priority accordingly.

Troubleshooting Jump to heading

Does a packet capture reveal that packets are reaching the server, but the VRRP service doesn’t receive them?

  • Check if there is a firewall running that may be blocking the packets. See Prerequisites.

Do you have any suggestions on how we can improve the content of this page?

Previous page Active/Active clustering Next page License keys
© 2024 HAProxy Technologies, LLC. All Rights Reserved
Manage Cookie Preferences