Configuration sync

When you operate two HAProxy ALOHA instances, whether in active-active or active-standby mode, you will often want to synchronize the load balancer configuration between them so that they are identical. Synchronization is a manual process that you must initiate. This allows you to test a configuration on one HAProxy ALOHA instance before pushing it to ther other.

Info

Each load balancer within a cluster can run a different configuration if you choose.
The configuration is not automatically synchronized between peers.
Any load balancer within a cluster can push its configuration to another load balancer.
Each service defined on the Services tab must be synced individually.

Enable the csyncd service Jump to heading

Before you can sync a configuration across load balancers, you must enable the csyncd service, which performs configuration synchronizations between the cluster members.

To enable the csyncd service, follow these steps on each HAProxy ALOHA instance:

In the Services tab, click [advanced mode]. Then edit the csyncd service.

The contents of the /etc/csyncd/csyncd.cfg configuration file displays.

Set each field as described below:

Field	Description
`LOCAL_IP`	Administration IP address of current HAProxy ALOHA instance.
`REMOTE_IP`	Administration IP address of the second instance in the cluster.
`LOCAL_NODE`	Unique ID of the current instance in the cluster (between 1 and 255). By convention, use one of: `1` to indicate the primary instance `2` to indicate the secondary instance.
`REMOTE_NODE`	Unique ID of the remote instance in the cluster (between 1 and 255). Use the ID that you did not use for `LOCAL_NODE`.

Example csyncd configuration on each HAProxy ALOHA instance:

LB1 (Primary)

text
LOCAL_IP=10.0.32.11
REMOTE_IP=10.0.32.12
LOCAL_NODE=1
REMOTE_NODE=2

text
LOCAL_IP=10.0.32.11
REMOTE_IP=10.0.32.12
LOCAL_NODE=1
REMOTE_NODE=2

LB2 (Secondary)

text
LOCAL_IP=10.0.32.12
REMOTE_IP=10.0.32.11
LOCAL_NODE=2
REMOTE_NODE=1

text
LOCAL_IP=10.0.32.12
REMOTE_IP=10.0.32.11
LOCAL_NODE=2
REMOTE_NODE=1

Click OK and then Close.
Click csyncd setup.
Remove the line no autostart.
Click OK and then Close.
To make your changes persistent after a reboot, click the Setup tab. Then click Save under Configuration.

Exchange SSH keys Jump to heading

In order for the HAProxy ALOHA instances to communicate, they must exchange SSH keys.

host keys are exchanged and saved to the /etc/csyncd/ssh_known_hosts file on each server. During a sync, the current server checks this file to verify that it is connecting to a trusted peer.
user keys are exchanged and saved to the /etc/csyncd/csyncd_authorized file on each server. During a sync, the remote server checks this file to verify that it trusts the peer that is connecting to it.

On the primary load balancer instance, stop the csyncd service if it is running.

Ensure the csyncd service is stopped:
Click the genkey button to generate the SSH host and user keys.
Repeat steps 1 and 2 on the secondary load balancer.
On the primary load balancer instance, click the getkey button to fetch the secondary load balancer’s SSH keys.
Click Start on the csyncd service.
Repeat steps 4 and 5 on the secondary load balancer.
To make your changes persistent after a reboot, click the Setup tab. Then click Save under Configuration.

Sync only the load balancer configuration Jump to heading

Follow these steps to synchronize only the configuration related to load balancing.

On the primary HAProxy ALOHA instance, go to either the LB Layer7 tab or the LB Layer4 tab, depending on which one you use for load balancing.
After making configuration changes and applying them, a button labeled Push will appear. Click it to synchronize changes to the remote load balancer.
- If no message appears after a few seconds, everything is fine.
- In there was an issue, the following message may display at the bottom of the page:
To persist the changes on the remote load balancer, go to the Setup tab and click Save under the Remote Configuration section. The Current status indicator should turn green.

Sync other services Jump to heading

You can synchronize the configuration of other services between two HAProxy ALOHA instances, besides the LB Layer7 or LB Layer4 tab load balancer configuration. Each service must be synced individually.

In the Services tab, click [advanced mode] to see the Edit Configuration button for a service.
After you have made changes to the service’s configuration, click Apply and then Push to sync the change to the other load balancer.
To persist the changes on the remote load balancer, go to the Setup tab and click Save under the Remote Configuration section. The Current status indicator should turn green.

Send csyncd traffic over WireGuard Jump to heading

Available since

HAProxy ALOHA 16.5

WireGuard is an open-source VPN tunneling protocol that can create secure UDP connections between two HAProxy ALOHA instances over the internet. Send csyncd traffic over WireGuard if your two HAProxy ALOHA instances are in separate data centers and cannot be on the same layer 2 network.

Enabling the WireGuard service on both of your HAProxy ALOHA instances allows you to perform configuration synchronizations in a layer 4 network, encrypted UDP tunnel. Always test before deploying to a production environment.

In the example illustration below, csyncd traffic is sent between HAProxy ALOHA instances “ALOHA1” and “ALOHA2” with the eth0 network interface. Bad actors can intercept and listen for the unencrypted csyncd traffic, seeing how each HAProxy ALOHA instance is configured.

nix
	                      |
Data Center 1             |             Data Center 2
                          |
   ALOHA1 <<~~~~~~~~~ INTERNET ~~~~~~~~~>> ALOHA2
	  |                   |                  |
    eth0                  |                eth0
192.0.2.10/24             |           198.51.100.10/24
	                      |

nix
	                      |
Data Center 1             |             Data Center 2
                          |
   ALOHA1 <<~~~~~~~~~ INTERNET ~~~~~~~~~>> ALOHA2
	  |                   |                  |
    eth0                  |                eth0
192.0.2.10/24             |           198.51.100.10/24
	                      |

To secure csyncd traffic, you will implement a WireGuard service and network interface (illustration below names it wg0) to encrypt csyncd traffic over WireGuard instead of eth0. The following illustration is an example setup that will be referred to throughout this page:

nix
	                      |
Data Center 1             |             Data Center 2
                          |
   ALOHA1 <<~~~~~~~~~ INTERNET ~~~~~~~~~>> ALOHA2
	  |                   |                  |
    eth0                  |                eth0
192.0.2.10/24             |           198.51.100.10/24
	  |                   |	                 |
	 wg0                  |                 wg0
192.0.2.11/24             |           198.51.100.11/24
	                      |

nix
	                      |
Data Center 1             |             Data Center 2
                          |
   ALOHA1 <<~~~~~~~~~ INTERNET ~~~~~~~~~>> ALOHA2
	  |                   |                  |
    eth0                  |                eth0
192.0.2.10/24             |           198.51.100.10/24
	  |                   |	                 |
	 wg0                  |                 wg0
192.0.2.11/24             |           198.51.100.11/24
	                      |

To enable the WireGuard service, follow these steps on each HAProxy ALOHA instance:

In the web UI, click Launch a terminal in the Tools tab and become a root user in the terminal. Make a new WireGuard directory (the example directory is called wg) and generate public/private keys in that directory.

ALOHA1 terminal

Run the following commands:

nix
root
mkdir -p /etc/wg
cd /etc/wg
wg genkey > /etc/wg/$HOSTNAME.private.key

nix
root
mkdir -p /etc/wg
cd /etc/wg
wg genkey > /etc/wg/$HOSTNAME.private.key

output
text
Warning: writing to world accessible file.
Consider setting the umask to 077 and trying again.

output
text
Warning: writing to world accessible file.
Consider setting the umask to 077 and trying again.

nix
wg pubkey <  $HOSTNAME.private.key > $HOSTNAME.public.key
ll

nix
wg pubkey <  $HOSTNAME.private.key > $HOSTNAME.public.key
ll

example output
text
total 8
drwxr-xr-x  2 root root   80 Nov 22 17:05 ./
drwxr-xr-x 38 root root 1420 Nov 22 17:04 ../
-rw-r--r--  1 root root   45 Nov 22 17:04 ALOHA1.private.key
-rw-r--r--  1 root root   45 Nov 22 17:05 ALOHA1.public.key

example output
text
total 8
drwxr-xr-x  2 root root   80 Nov 22 17:05 ./
drwxr-xr-x 38 root root 1420 Nov 22 17:04 ../
-rw-r--r--  1 root root   45 Nov 22 17:04 ALOHA1.private.key
-rw-r--r--  1 root root   45 Nov 22 17:05 ALOHA1.public.key

Take note of the WireGuard public key’s contents since it will be configured later.

nix
cat ALOHA1.public.key

nix
cat ALOHA1.public.key

example output
text
Kw/i827kdstWTiogkdaRKWnkdsalKkEW2o/284lWas=

example output
text
Kw/i827kdstWTiogkdaRKWnkdsalKkEW2o/284lWas=

ALOHA2 terminal

Run the following commands:

nix
root
mkdir -p /etc/wg
cd /etc/wg
wg genkey > /etc/wg/$HOSTNAME.private.key

nix
root
mkdir -p /etc/wg
cd /etc/wg
wg genkey > /etc/wg/$HOSTNAME.private.key

output
text
Warning: writing to world accessible file.
Consider setting the umask to 077 and trying again.

output
text
Warning: writing to world accessible file.
Consider setting the umask to 077 and trying again.

nix
wg pubkey <  $HOSTNAME.private.key > $HOSTNAME.public.key
ll

nix
wg pubkey <  $HOSTNAME.private.key > $HOSTNAME.public.key
ll

example output
text
total 8
drwxr-xr-x  2 root root   80 Nov 22 17:10 ./
drwxr-xr-x 38 root root 1420 Nov 22 17:09 ../
-rw-r--r--  1 root root   45 Nov 22 17:09 ALOHA2.private.key
-rw-r--r--  1 root root   45 Nov 22 17:10 ALOHA2.public.key

example output
text
total 8
drwxr-xr-x  2 root root   80 Nov 22 17:10 ./
drwxr-xr-x 38 root root 1420 Nov 22 17:09 ../
-rw-r--r--  1 root root   45 Nov 22 17:09 ALOHA2.private.key
-rw-r--r--  1 root root   45 Nov 22 17:10 ALOHA2.public.key

Take note of the WireGuard public key’s contents since it will be configured later.

nix
cat ALOHA2.public.key

nix
cat ALOHA2.public.key

example output
text
OGhxS8Wp0IMcksodOLaowKLnd2wqKDlnskwq93DoeYx

example output
text
OGhxS8Wp0IMcksodOLaowKLnd2wqKDlnskwq93DoeYx

In the web UI on the Tools tab, use the File manager to open the /etc/config.rc file. Create a new network service for WireGuard (the example names it wg0) after all the other configured network services.

ALOHA1 /etc/config.rc file

In ALOHA1’s /etc/config.rc file, edit and save the following configuration replacing ip address with your own.

nix
service network
…
service network wg0
  type            wg
  ip address      192.0.2.11/24 # ALOHA1 WireGuard IP address 
  wg listen_port  51820           # WireGuard's port
  wg private_key  /etc/wg/ALOHA1.private.key
service vrrp
…

nix
service network
…
service network wg0
  type            wg
  ip address      192.0.2.11/24 # ALOHA1 WireGuard IP address 
  wg listen_port  51820           # WireGuard's port
  wg private_key  /etc/wg/ALOHA1.private.key
service vrrp
…

ALOHA2 /etc/config.rc file

In ALOHA2’s /etc/config.rc file, edit and save the following configuration replacing ip address.

nix
service network
…
service network wg0
  type            wg
  ip address      198.51.100.11/24 # ALOHA2 WireGuard IP address 
  wg listen_port  51820           # WireGuard's port
  wg private_key  /etc/wg/ALOHA2.private.key
service vrrp
…

nix
service network
…
service network wg0
  type            wg
  ip address      198.51.100.11/24 # ALOHA2 WireGuard IP address 
  wg listen_port  51820           # WireGuard's port
  wg private_key  /etc/wg/ALOHA2.private.key
service vrrp
…

Start the WireGuard network service (wg0 from the example) on each HAProxy ALOHA instance by navigating to the Services tab and clicking the Start service network button.

From the terminal, confirm that the WireGuard network service is running on both HAProxy ALOHA instances.

ALOHA1 terminal

Here’s an example on ALOHA1 with expected outputs:

nix
ip addr show dev wg0

nix
ip addr show dev wg0

example output
text
6: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
  link/none 
  inet 192.0.2.11/24 scope global wg0
  valid_lft forever preferred_lft forever

example output
text
6: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
  link/none 
  inet 192.0.2.11/24 scope global wg0
  valid_lft forever preferred_lft forever

The state is currently UNKNOWN because you haven’t configured a WireGuard interface yet, but this output tells you that the HAProxy ALOHA instance recognizes a WireGuard network service is running. You will create a WireGuard interface in the next step.

nix
wg

nix
wg

example output
text
interface: wg0
  public key: Kw/i827kdstWTiogkdaRKWnkdsalKkEW2o/284lWas=
  private key: (hidden)
  listening port: 51820

example output
text
interface: wg0
  public key: Kw/i827kdstWTiogkdaRKWnkdsalKkEW2o/284lWas=
  private key: (hidden)
  listening port: 51820

ALOHA2 terminal

Here’s an example on ALOHA2 with expected outputs:

nix
ip addr show dev wg0

nix
ip addr show dev wg0

example output
text
6: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
  link/none 
  inet 198.51.100.11/24 scope global wg0
  valid_lft forever preferred_lft forever

example output
text
6: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
  link/none 
  inet 198.51.100.11/24 scope global wg0
  valid_lft forever preferred_lft forever

nix
wg

nix
wg

example output
text
interface: wg0
  public key: OGhxS8Wp0IMcksodOLaowKLnd2wqKDlnskwq93DoeYx
  private key: (hidden)
  listening port: 51820

example output
text
interface: wg0
  public key: OGhxS8Wp0IMcksodOLaowKLnd2wqKDlnskwq93DoeYx
  private key: (hidden)
  listening port: 51820

Create a UDP tunnel with a WireGuard interface. In the web UI, use the ALOHA File Manager to open the /etc/config.rc file with with its file editor. Create a new service for WireGuard (our example names it “wg”) after all the other configured network services.

ALOHA1 /etc/config.rc file

In ALOHA1’s /etc/config.rc file, edit and save the following configuration replacing public_key, endpoint, and allowed_ips with your own:

nix
service wg
  interface   wg0 # the name of the network service created in the previous step
  public_key  OGhxS8Wp0IMcksodOLaowKLnd2wqKDlnskwq93DoeYx   # ALOHA2's wg public key
  endpoint    198.51.100.10:51820 # ALOHA2's eth0 IP address with wg network service's listen_port
  allowed_ips 192.0.2.0/24
  keepalive   5

nix
service wg
  interface   wg0 # the name of the network service created in the previous step
  public_key  OGhxS8Wp0IMcksodOLaowKLnd2wqKDlnskwq93DoeYx   # ALOHA2's wg public key
  endpoint    198.51.100.10:51820 # ALOHA2's eth0 IP address with wg network service's listen_port
  allowed_ips 192.0.2.0/24
  keepalive   5

ALOHA2 /etc/config.rc file

In ALOHA2’s /etc/config.rc file, edit and save the following configuration replacing public_key, endpoint, and allowed_ips with your own:

nix
service wg
  interface   wg0 # the name of the network service created in the previous step
  public_key  Kw/i827kdstWTiogkdaRKWnkdsalKkEW2o/284lWas=   # ALOHA1's wg public key
  endpoint    192.0.2.10:51820 # ALOHA1's eth0 IP address with wg network service's listen_port
  allowed_ips 198.51.100.0/24
  keepalive   5

nix
service wg
  interface   wg0 # the name of the network service created in the previous step
  public_key  Kw/i827kdstWTiogkdaRKWnkdsalKkEW2o/284lWas=   # ALOHA1's wg public key
  endpoint    192.0.2.10:51820 # ALOHA1's eth0 IP address with wg network service's listen_port
  allowed_ips 198.51.100.0/24
  keepalive   5

Start the WireGuard service (wg from the example) on each HAProxy ALOHA instance by navigating to the Services tab and clicking the Start service button.

Confirm that the WireGuard service is running on both HAProxy ALOHA instances in the terminal.

ALOHA1 terminal

Here’s an example on ALOHA1 with expected outputs:

nix
wg

nix
wg

example output
text
interface: wg0
  public key: Kw/i827kdstWTiogkdaRKWnkdsalKkEW2o/284lWas=
  private key: (hidden)
  listening port: 51820
peer: OGhxS8Wp0IMcksodOLaowKLnd2wqKDlnskwq93DoeYx  
  endpoint: 198.51.100.10:51820
  allowed ips: 192.0.2.0/24
  latest handshake: 21 seconds ago
  transfer: 308 B received, 5.32 KiB sent
  persistent keepalive: every 5 seconds

example output
text
interface: wg0
  public key: Kw/i827kdstWTiogkdaRKWnkdsalKkEW2o/284lWas=
  private key: (hidden)
  listening port: 51820
peer: OGhxS8Wp0IMcksodOLaowKLnd2wqKDlnskwq93DoeYx  
  endpoint: 198.51.100.10:51820
  allowed ips: 192.0.2.0/24
  latest handshake: 21 seconds ago
  transfer: 308 B received, 5.32 KiB sent
  persistent keepalive: every 5 seconds

The peer section shows that ALOHA2 answered the keepalive.

ALOHA2 terminal

Here’s an example on ALOHA2 with expected outputs:

nix
wg

nix
wg

example output
text
interface: wg0
  public key: OGhxS8Wp0IMcksodOLaowKLnd2wqKDlnskwq93DoeYx
  private key: (hidden)
  listening port: 51820
peer: Kw/i827kdstWTiogkdaRKWnkdsalKkEW2o/284lWas= 
  endpoint: 192.0.2.10:51820
  allowed ips: 198.51.100.0/24
  latest handshake: 18 seconds ago
  transfer: 124 B received, 276 B sent
  persistent keepalive: every 5 seconds

example output
text
interface: wg0
  public key: OGhxS8Wp0IMcksodOLaowKLnd2wqKDlnskwq93DoeYx
  private key: (hidden)
  listening port: 51820
peer: Kw/i827kdstWTiogkdaRKWnkdsalKkEW2o/284lWas= 
  endpoint: 192.0.2.10:51820
  allowed ips: 198.51.100.0/24
  latest handshake: 18 seconds ago
  transfer: 124 B received, 276 B sent
  persistent keepalive: every 5 seconds

The peer section shows that ALOHA1 answered the keepalive. Confirmed handshake from both HAProxy ALOHA instances.

Update csyncd configuration with WireGuard’s public IP address. In the web UI, navigate to Tools tab use the ALOHA File Manager to open and edit the /etc/csyncd/csyncd.cfg file with its file editor. Edit and save the appropriate IP addresses and node configuration of the HAProxy ALOHA instances.
ALOHA1 /etc/csyncd/csyncd.cfg

For example, ALOHA1’s csyncd.cfg file is updated to:
```
nix
# csyncd configuration file
LOCAL_IP=192.0.2.11
REMOTE_IP=198.51.100.11
LOCAL_NODE=1
REMOTE_NODE=2
```
```
nix
# csyncd configuration file
LOCAL_IP=192.0.2.11
REMOTE_IP=198.51.100.11
LOCAL_NODE=1
REMOTE_NODE=2
```
ALOHA2 /etc/csyncd/csyncd.cfg

ALOHA2’s csyncd.cfg file is updated to:
```
nix
# csyncd configuration file
LOCAL_IP=198.51.100.11
REMOTE_IP=192.0.2.11
LOCAL_NODE=2
REMOTE_NODE=1
```
```
nix
# csyncd configuration file
LOCAL_IP=198.51.100.11
REMOTE_IP=192.0.2.11
LOCAL_NODE=2
REMOTE_NODE=1
```
Navigate to the LB Admin tab and click Save on both HAProxy ALOHA instances. This change will send csyncd traffic through its local WireGuard interface instead of directly to the remote HAProxy ALOHA instance.

Confirm you are sending csyncd traffic over WireGuard through an encrypted UDP tunnel. The tcpdump package is already installed on HAProxy ALOHA, so you can use it to log each individual packet that is sent to and from WireGuard.

In terminal, run the following command; the -tttt flag adds the date to a timestamp, the -n flag skips hostname lookups, and the -i any flag includes all interfaces.

nix
tcpdump -ttttni any 'udp port 51820'

nix
tcpdump -ttttni any 'udp port 51820'

example output
text
2024-11-26 20:45:15.829578 IP 192.0.2.11.51820 > 198.51.100.11.51820: UDP, length 148
2024-11-26 20:45:15.830338 IP 198.51.100.11.51820 > 192.0.2.11.51820: UDP, length 92
2024-11-26 20:45:15.831375 IP 192.0.2.11.51820 > 198.51.100.11.51820: UDP, length 96
...

example output
text
2024-11-26 20:45:15.829578 IP 192.0.2.11.51820 > 198.51.100.11.51820: UDP, length 148
2024-11-26 20:45:15.830338 IP 198.51.100.11.51820 > 192.0.2.11.51820: UDP, length 92
2024-11-26 20:45:15.831375 IP 192.0.2.11.51820 > 198.51.100.11.51820: UDP, length 96
...

WireGuard reference Jump to heading

Configuration directives for the wg0 network service

Directive	Description
`wg fwmark <mark>`	Mark outgoing WireGuard packets with a specified firewall mark.
`wg listen_port <port>`	Make WireGuard listen on a specified firewall port.
`wg private_key <file>`	The WireGuard private key location to check.

ALOHA terminal commands

Command	Description
`wg`	Display any WireGuard services active on the ALOHA instance.
`wg genkey`	Create a WireGuard private key.
`wg pubkey`	Create a WireGuard public key.

Troubleshooting Jump to heading

If you run into any problems, try these troubleshooting steps:

On the Setup tab, look for the Remote Configuration area. The current status table reports the synchronization capability of the cluster:
- A red indicator means synchronization does not work.
- An orange or green indicator means synchronization works.
If the indicator is red:
- Power on both HAProxy ALOHA instances.
- Check that both HAProxy ALOHA web UIs listen on the same TCP port and on the same protocol scheme (HTTP or HTTPs).
- Check that both HAProxy ALOHA instances can ping each other.
- Check that the csyncd service is started on both HAProxy ALOHA instances, for example, a green indicator on the Services tab.
- Force a csyncd key exchange by clicking the getkey button on each HAProxy ALOHA instance.

Product Documentation

API Documentation

Configuration Documentation

Virtual appliance

Configuration sync

Enable the csyncd service Jump to heading

Exchange SSH keys Jump to heading

Sync only the load balancer configuration Jump to heading

Sync other services Jump to heading

Send csyncd traffic over WireGuard Jump to heading

WireGuard reference Jump to heading

Troubleshooting Jump to heading

Virtual appliance

Product Documentation

API Documentation

Configuration Documentation

Enable the csyncd service Jump to heading #

Exchange SSH keys Jump to heading #

Sync only the load balancer configuration Jump to heading #

Sync other services Jump to heading #

Send csyncd traffic over WireGuard Jump to heading #

WireGuard reference Jump to heading #

Troubleshooting Jump to heading #

Privacy Settings

Enable the csyncd service Jump to heading

Exchange SSH keys Jump to heading

Sync only the load balancer configuration Jump to heading

Sync other services Jump to heading

Send csyncd traffic over WireGuard Jump to heading

WireGuard reference Jump to heading

Troubleshooting Jump to heading