Active/Standby clustering

Using HAProxy Fusion?

If you’re using HAProxy Fusion, then see the HAProxy Fusion - High availability topic instead.

In an active-standby cluster, one HAProxy Enterprise node receives traffic while a second node is put on standby. In case the active node fails, the standby takes over. With HAProxy Enterprise, you configure an active-standby cluster by installing the Virtual Router Redundancy Protocol (VRRP) module.

The HAProxy Enterprise VRRP module assigns a virtual, static IP address to your load balancer node. If the node fails, the IP address will instantly transfer to a backup node, avoiding any break in service. Your router can continue to send traffic to the same IP address for the load balancer, never knowing that there was a failover. The module utilizes a stable version of Keepalived, which implements VRRP.

In this guide, we will set up two load balancers: one active and the other on standby.

Active/Standby with VRRP

Prerequisites Jump to heading

If you have a firewall running on the server, such as firewalld (the default on Red Hat Enterprise Linux) or iptables, then be sure to add an exception for VRRP traffic.

Add an exception to firewalld

To add an exception for VRRP to firewalld:

Check if firewalld is running:

nix
sudo firewall-cmd --state

nix
sudo firewall-cmd --state

output
running

output
running

Check which firewalld zones are active for your network interfaces. Below, our active zone is named public:

nix
sudo firewall-cmd --get-active-zones

nix
sudo firewall-cmd --get-active-zones

output
public
  interfaces: eth0 eth1

output
public
  interfaces: eth0 eth1

To ensure that existing runtime rules (those that are in effect) are persisted, save them as permanent before going further:
```
nix
sudo firewall-cmd --runtime-to-permanent 
```
```
nix
sudo firewall-cmd --runtime-to-permanent 
```
```
output
success
```
```
output
success
```

Add the VRRP protocol to the list of rules. Below, we use the zone named public:

nix
sudo firewall-cmd --zone=public --add-protocol=vrrp --permanent
sudo firewall-cmd --reload

nix
sudo firewall-cmd --zone=public --add-protocol=vrrp --permanent
sudo firewall-cmd --reload

output
success

output
success

Verify that the protocol has been added. Below, we use the zone named public:

nix
sudo firewall-cmd --zone=public --list-all | grep vrrp

nix
sudo firewall-cmd --zone=public --list-all | grep vrrp

protocols: vrrp

protocols: vrrp

Make these changes persist:

nix
sudo firewall-cmd --runtime-to-permanent

nix
sudo firewall-cmd --runtime-to-permanent

output
success

output
success

If you need to revert this change, run these commands:

nix
sudo firewall-cmd --zone=public --remove-protocol=vrrp --permanent
sudo firewall-cmd --reload

nix
sudo firewall-cmd --zone=public --remove-protocol=vrrp --permanent
sudo firewall-cmd --reload

Add an exception to iptables

To add an exception for VRRP to iptables:

Save a backup of your current iptables configuration:

nix
sudo apt install iptables-persistent
sudo su -c 'iptables-save > /etc/iptables.backup.rules'

nix
sudo apt install iptables-persistent
sudo su -c 'iptables-save > /etc/iptables.backup.rules'

Add the VRRP protocol to the list of rules. Below, we specify the interface eth0:

nix
sudo iptables -A INPUT -p vrrp -i eth0 -j ACCEPT && sudo iptables -A OUTPUT -p vrrp -j ACCEPT

nix
sudo iptables -A INPUT -p vrrp -i eth0 -j ACCEPT && sudo iptables -A OUTPUT -p vrrp -j ACCEPT

Verify that the protocol number 112 (VRRP) has been added:

nix
sudo iptables -nvL | grep 112

nix
sudo iptables -nvL | grep 112

output
0     0 ACCEPT     112  --  eth0   *       0.0.0.0/0            0.0.0.0/0
0     0 ACCEPT     112  --  *      *       0.0.0.0/0            0.0.0.0/0

output
0     0 ACCEPT     112  --  eth0   *       0.0.0.0/0            0.0.0.0/0
0     0 ACCEPT     112  --  *      *       0.0.0.0/0            0.0.0.0/0

Save the configuration:

nix
sudo su -c 'iptables-save > /etc/iptables.vrrp.rules'

nix
sudo su -c 'iptables-save > /etc/iptables.vrrp.rules'

If you need to revert this change, run this command:

nix
sudo iptables-restore < /etc/iptables.backup.rules

nix
sudo iptables-restore < /etc/iptables.backup.rules

Installation Jump to heading

Follow the steps on your active and backup load balancers.

Active load balancer Jump to heading

On the active load balancer, follow these steps:

Install the VRRP module using your system’s package manager:

nix
sudo apt-get install hapee-extras-vrrp

nix
sudo apt-get install hapee-extras-vrrp

nix
sudo yum install hapee-extras-vrrp

nix
sudo yum install hapee-extras-vrrp

nix
sudo zypper install hapee-extras-vrrp

nix
sudo zypper install hapee-extras-vrrp

nix
sudo pkg install hapee-extras-vrrp

nix
sudo pkg install hapee-extras-vrrp

Decide on a unique Virtual Router Identifier (VRID) for the cluster. A VRID can be any number between 1 and 255. It allows the instances in a cluster to share a virtual router and virtual IP address. The VRID must be the same on all instances in a cluster. If there are multiple VRRP clusters in the environment, the VRID must be unique for each cluster.

Do not use a VRID already in use for another cluster. If the vrrp service has been started and enabled, you can list VRIDs already in use by executing the following command:
```
nix
sudo tcpdump -vvvenns0 -c 5 -i eth0 vrrp | grep -o "vrid [0-9]*"
```
```
nix
sudo tcpdump -vvvenns0 -c 5 -i eth0 vrrp | grep -o "vrid [0-9]*"
```
```
output
text
[...]
5 packets captured
6 packets received by filter
0 packets dropped by kernel
vrid 161
vrid 155
```
```
output
text
[...]
5 packets captured
6 packets received by filter
0 packets dropped by kernel
vrid 161
vrid 155
```
In our example configuration, we will use VRID 51.
Decide on a password for the cluster. The password is a clear-text string for use by all instances in this VRRP cluster. It does not provide security, but it ensures that VRRP instances from other clusters cannot join this cluster due to errors in configuration, such as a duplicate VRID.

Edit the file /etc/hapee-extras/hapee-vrrp.cfg.

In the vrrp_instance section, change the interface value from eth0 to the name of a network interface that receives traffic on the server. This is the same interface that you the bind HAProxy Enterprise’s frontend to. For example, if traffic reaches the load balancer using interface enp0s8. Set this line to interface enp0s8.
For the virtual_router_id value, specify the VRID determined previously.
Change the authentication password in auth_pass to the password determined previously.
Change the IP address listed in the virtual_ipaddress_excluded block to the virtual IP address you’d like to assign to this interface. HAProxy Enterprise will bind to this address to receive traffic. If needed, add more IP addresses here, each on its own line. The new address(es) should fall within the interface’s IP subnet but shouldn’t already be assigned to any server.
In the track_interface block, specify the same interface name that is specified in the interface field.

hapee-vrrp.cfg
text
vrrp_instance vrrp_1 {
  interface enp0s8       # Change network interface name
  state MASTER
  virtual_router_id 51   # VRID unique to this cluster
  authentication {
    auth_type PASS
    auth_pass 1234       # Text string unique to this cluster
  }
  priority 101
  virtual_ipaddress_excluded {
    192.168.50.10        # New IP address
  }
  track_interface {
    enp0s8 weight -2     # Change network interface name
  }
  track_script {
    chk_sshd
    chk_lb
  }
}

hapee-vrrp.cfg
text
vrrp_instance vrrp_1 {
  interface enp0s8       # Change network interface name
  state MASTER
  virtual_router_id 51   # VRID unique to this cluster
  authentication {
    auth_type PASS
    auth_pass 1234       # Text string unique to this cluster
  }
  priority 101
  virtual_ipaddress_excluded {
    192.168.50.10        # New IP address
  }
  track_interface {
    enp0s8 weight -2     # Change network interface name
  }
  track_script {
    chk_sshd
    chk_lb
  }
}

Enable and start the hapee-extras-vrrp service:

nix
sudo systemctl enable hapee-extras-vrrp
sudo systemctl unmask hapee-extras-vrrp
sudo systemctl start hapee-extras-vrrp

nix
sudo systemctl enable hapee-extras-vrrp
sudo systemctl unmask hapee-extras-vrrp
sudo systemctl start hapee-extras-vrrp

Allow the server to bind to the virtual IP address.

Edit the file /etc/sysctl.conf and add the net.ipv4.ip_nonlocal_bind=1 directive. This directive allows the server to accept connections for IP addresses that aren’t bound to any of its interfaces, enabling the use of a floating, virtual IP.
```
sysctl.conf
ini
net.ipv4.ip_nonlocal_bind=1
```
```
sysctl.conf
ini
net.ipv4.ip_nonlocal_bind=1
```

Configure your bind line in the HAProxy Enterprise configuration to use the virtual IP address.

haproxy
frontend myfrontend
  mode http
  bind 192.168.50.10:80
  default_backend web_servers

haproxy
frontend myfrontend
  mode http
  bind 192.168.50.10:80
  default_backend web_servers

Reboot the server.

Standby load balancer Jump to heading

On the standby load balancer, follow these steps:

Install the VRRP module using your system’s package manager:

nix
sudo apt-get install hapee-extras-vrrp

nix
sudo apt-get install hapee-extras-vrrp

nix
sudo yum install hapee-extras-vrrp

nix
sudo yum install hapee-extras-vrrp

nix
sudo zypper install hapee-extras-vrrp

nix
sudo zypper install hapee-extras-vrrp

nix
sudo pkg install hapee-extras-vrrp

nix
sudo pkg install hapee-extras-vrrp

Edit the VRRP configuration /etc/hapee-extras/hapee-vrrp.cfg. It must be exactly the same as the file on the active load balancer, except for these differences:

In the vrrp_instance block, change the interface value to the name of the network interface that receives traffic on the server.
Change the state value to BACKUP.
Change the priority field to have a lower number than the active node. The load balancer with the highest priority is promoted to be the active node. For example, if the priority on the active node is 101, then set the backup node’s priority to 100.
In the track_interface block, specify the same interface name that is specified in the interface field.

hapee-vrrp.cfg
text
vrrp_instance vrrp_1 {
  interface enp0s8          # Change network interface name
  state BACKUP              # Change to BACKUP
  virtual_router_id 51
  authentication {
    auth_type PASS
    auth_pass 1234 
  }
  priority 100              # A lower priority value than the primary
  virtual_ipaddress_excluded {
    192.168.50.10           # Same IP address as on primary
  }
  track_interface {
    enp0s8 weight -2        # Change network interface name
  }
  track_script {
    chk_sshd
    chk_lb
  }
}

hapee-vrrp.cfg
text
vrrp_instance vrrp_1 {
  interface enp0s8          # Change network interface name
  state BACKUP              # Change to BACKUP
  virtual_router_id 51
  authentication {
    auth_type PASS
    auth_pass 1234 
  }
  priority 100              # A lower priority value than the primary
  virtual_ipaddress_excluded {
    192.168.50.10           # Same IP address as on primary
  }
  track_interface {
    enp0s8 weight -2        # Change network interface name
  }
  track_script {
    chk_sshd
    chk_lb
  }
}

Enable and start the hapee-extras-vrrp service:

nix
sudo systemctl enable hapee-extras-vrrp
sudo systemctl unmask hapee-extras-vrrp
sudo systemctl start hapee-extras-vrrp

nix
sudo systemctl enable hapee-extras-vrrp
sudo systemctl unmask hapee-extras-vrrp
sudo systemctl start hapee-extras-vrrp

Allow the server to bind to the virtual IP address.

Edit the file /etc/sysctl.conf and add the net.ipv4.ip_nonlocal_bind=1 directive. This directive allows the server to accept connections for IP addresses that aren’t bound to any of its interfaces, enabling the use of a floating, virtual IP.
```
sysctl.conf
ini
net.ipv4.ip_nonlocal_bind=1
```
```
sysctl.conf
ini
net.ipv4.ip_nonlocal_bind=1
```

Configure your bind line in the HAProxy Enterprise configuration to use the virtual IP address.

haproxy
frontend myfrontend
  mode http
  bind 192.168.50.10:80
  default_backend web_servers

haproxy
frontend myfrontend
  mode http
  bind 192.168.50.10:80
  default_backend web_servers

Reboot the server.

Verify the setup Jump to heading

To check that the VRRP service is running, use systemctl status:

nix
sudo systemctl status hapee-extras-vrrp --no-pager

nix
sudo systemctl status hapee-extras-vrrp --no-pager

output
text
● hapee-extras-vrrp.service - HAPEE VRRP : VRRP daemon (Keepalived)
     Loaded: loaded (/lib/systemd/system/hapee-extras-vrrp.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-10-18 18:01:59 UTC; 10min ago
   Main PID: 1191 (hapee-vrrp)
      Tasks: 2 (limit: 4555)
     Memory: 1.5M
        CPU: 4.632s
     CGroup: /system.slice/hapee-extras-vrrp.service
             ├─1191 /opt/hapee-extras/sbin/hapee-vrrp -D -f /etc/hapee-extras/hapee-vrrp.cfg -p /var/run/hapee-extras/hapee-vrrp.pid
             └─1192 /opt/hapee-extras/sbin/hapee-vrrp -D -f /etc/hapee-extras/hapee-vrrp.cfg -p /var/run/hapee-extras/hapee-vrrp.pid
Oct 18 18:02:01 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) forcing a new MASTER election
Oct 18 18:02:02 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Transition to MASTER STATE
Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Entering MASTER STATE
Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) setting protocol E-VIPs.
Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Sending gratuitous ARPs on eth0 for 172.1.1.4

output
text
● hapee-extras-vrrp.service - HAPEE VRRP : VRRP daemon (Keepalived)
     Loaded: loaded (/lib/systemd/system/hapee-extras-vrrp.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-10-18 18:01:59 UTC; 10min ago
   Main PID: 1191 (hapee-vrrp)
      Tasks: 2 (limit: 4555)
     Memory: 1.5M
        CPU: 4.632s
     CGroup: /system.slice/hapee-extras-vrrp.service
             ├─1191 /opt/hapee-extras/sbin/hapee-vrrp -D -f /etc/hapee-extras/hapee-vrrp.cfg -p /var/run/hapee-extras/hapee-vrrp.pid
             └─1192 /opt/hapee-extras/sbin/hapee-vrrp -D -f /etc/hapee-extras/hapee-vrrp.cfg -p /var/run/hapee-extras/hapee-vrrp.pid
Oct 18 18:02:01 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) forcing a new MASTER election
Oct 18 18:02:02 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Transition to MASTER STATE
Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Entering MASTER STATE
Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) setting protocol E-VIPs.
Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Sending gratuitous ARPs on eth0 for 172.1.1.4

Manual failover Jump to heading

To activate the standby load balancer while simultaneously putting the active load balancer into standby mode:

Edit the VRRP configuration /etc/hapee-extras/hapee-vrrp.cfg on the active load balancer.
Lower the priority value to less than the standby load balancer’s priority. This will put the active load balancer into the standby state and simultaneously activate the standby load balancer. For example, change it from 101 to 90.
Restart the hapee-extras-vrrp service.

Reverse these actions to restore the original state.

Failover triggers Jump to heading

The following events can trigger a failover:

The active node lowers its weight below one of the backup nodes due to a failed health check.
A backup node is reconfigured with a weight larger than the current active node.
The active node stops emitting its heartbeat packet to the cluster.

VRRP health check scripts Jump to heading

In the VRRP configuration, the vrrp_script blocks define health checks that can trigger a failover. By default, our configuration checks whether the SSH daemon is running and whether HAProxy Enterprise is running. You may want to add more checks.

Consider this block, which checks the status of the SSH daemon every five seconds. If the service is not available, the VRRP instance’s weight (its priority) is reduced by 4 points:

hapee-vrrp.cfg
text
vrrp_script chk_sshd {
  script "pkill -0 sshd"  # pkill -0 is cheaper than pidof
  interval 5              # check every 5 seconds
  weight -4               # remove 4 points of prio if missing
  fall 2                  # check twice before setting down
  rise 1                  # check once before setting up
}

hapee-vrrp.cfg
text
vrrp_script chk_sshd {
  script "pkill -0 sshd"  # pkill -0 is cheaper than pidof
  interval 5              # check every 5 seconds
  weight -4               # remove 4 points of prio if missing
  fall 2                  # check twice before setting down
  rise 1                  # check once before setting up
}

The HAProxy Enterprise health check is similar, except if the hapee-lb process is running, it adds 6 points. The reason is, in case one load balancer has SSH working but not hapee-lb, while the other has hapee-lb but not SSH, the one with hapee-lb will become the active node, even though it is not manageable remotely.

hapee-vrrp.cfg
text
vrrp_script chk_lb {
  script "pkill -0 hapee-lb"  # pkill -0 is cheaper than pidof
  interval 1                  # check every second
  weight 6                    # add 6 points of prio if present
  fall 2                      # check twice before setting down
  rise 1                      # check once before setting up
}

hapee-vrrp.cfg
text
vrrp_script chk_lb {
  script "pkill -0 hapee-lb"  # pkill -0 is cheaper than pidof
  interval 1                  # check every second
  weight 6                    # add 6 points of prio if present
  fall 2                      # check twice before setting down
  rise 1                      # check once before setting up
}

The following fields define a vrrp_script block:

Name	Argument type	Description
`script`	string	Command to run. Running `pkill -0` checks if any process with the given name is running. If it returns 0, the check passes; if it returns 1, the check fails.
`interval`	integer	Interval (in seconds) between two checks.
`weight`	integer	Points to add or remove to instance weight.
`fall`	integer	Number of consecutive negative checks before considered as down.
`rise`	integer	Number of consecutive positive checks before considered as up.

VRRP instance reference Jump to heading

The following fields define a vrrp_instance block:

Name	Argument type	Description
`vrrp_instance`	string	Describes a new VRRP instance.
`interface`	string	Interface managed by the VRRP module.
`state`	string	State at startup until VRRP priority negotiation completes. Can be either `MASTER` or `BACKUP`.
`virtual_router_id`	integer	VRRP instance identifier, which must be common to all nodes of the same cluster. Each cluster must have a unique ID. The value can range from 1 to 255.
`priority`	integer	Weight of local node in this VRRP instance. NOTE This won’t work on cloud provider networks that disable IP multicast.
`virtual_ipaddress`	string	List of virtual IP addresses to add when the state changes to `MASTER` or remove when it changes to `BACKUP`. All VRRP nodes in a cluster must own the same IP. NOTE This won’t work on cloud provider networks that disable IP multicast.
`virtual_ipaddress_excluded`	string	The same as `virtual_ipaddress`, but IPs aren’t announced in the VRRP heartbeat packet, which reduces bandwidth usage when sending packets.
`track_interface`	string	Tracks interface status and updates priority accordingly.
`track_script`	string	Runs the health check scripts and updates priority accordingly.

Troubleshooting Jump to heading

Does a packet capture reveal that packets are reaching the server, but the VRRP service doesn’t receive them?

Check if there is a firewall running that may be blocking the packets. See Prerequisites.

Product Documentation

API Documentation

Configuration Documentation

Active/Standby clustering

Prerequisites Jump to heading

Installation Jump to heading

Active load balancer Jump to heading

Standby load balancer Jump to heading

Verify the setup Jump to heading

Manual failover Jump to heading

Failover triggers Jump to heading

VRRP health check scripts Jump to heading

VRRP instance reference Jump to heading

Troubleshooting Jump to heading

Product Documentation

API Documentation

Configuration Documentation

Prerequisites Jump to heading #

Installation Jump to heading #

Active load balancer Jump to heading #

Standby load balancer Jump to heading #

Verify the setup Jump to heading #

Manual failover Jump to heading #

Failover triggers Jump to heading #

VRRP health check scripts Jump to heading #

VRRP instance reference Jump to heading #

Troubleshooting Jump to heading #

Privacy Settings

Prerequisites Jump to heading

Installation Jump to heading

Active load balancer Jump to heading

Standby load balancer Jump to heading

Verify the setup Jump to heading

Manual failover Jump to heading

Failover triggers Jump to heading

VRRP health check scripts Jump to heading

VRRP instance reference Jump to heading

Troubleshooting Jump to heading