Enterprise modules

Route health injection

On this page

The Route Health Injection (RHI) service interacts with the BIRD Internet Routing Daemon to start or stop the flow of traffic to this HAProxy Enterprise node depending on the health of the network and your load balanced servers. RHI is useful for scaling out an HAProxy Enterprise cluster in an active/active manner, ensuring high availability by relying on well established routing network protocols.

The RHI service adds this HAProxy Enterprise node as a route in BIRD’s configuration using a custom route table named volatile. BIRD then broadcasts these routes to peer routers using the BGP, RIP, or OSPF protocol. If either a frontend or a backend is down, then the RHI service removes the route from the volatile table, notifying BIRD to stop advertising this HAProxy Enterprise node as a route on the network, diverting the flow of traffic to the other HAProxy Enterprise node in the active-active cluster. You can configure ECMP on your router to load balance traffic to both HAProxy Enterprise nodes via the advertised routes.

HAProxy Enterprise Route Health Injection

Configure Route Health Injection Jump to heading

  1. Install the RHI module using your package manager:

    nix
    sudo apt-get install hapee-extras-rhi
    nix
    sudo apt-get install hapee-extras-rhi
    nix
    sudo yum install hapee-extras-rhi
    nix
    sudo yum install hapee-extras-rhi
    nix
    sudo zypper install hapee-extras-rhi
    nix
    sudo zypper install hapee-extras-rhi
    nix
    sudo pkg install hapee-extras-rhi
    nix
    sudo pkg install hapee-extras-rhi

    This installs the hapee-extras-route package too, which is our version of the BIRD Internet Routing Daemon. The daemon is stored as /opt/hapee-extras/sbin/hapee-route.

  2. Create a socket for the Runtime API.

    The RHI service needs to connect to the Runtime API to collect information about the health of your frontends and backends. Add a new stats socket line to the global section of your HAProxy Enterprise configuration. This exposes the Runtime API as the socket /var/run/hapee-extras/hapee-lb.sock:

    haproxy
    global
      stats socket /var/run/hapee-extras/hapee-lb.sock user hapee-lb group hapee mode 660
    haproxy
    global
      stats socket /var/run/hapee-extras/hapee-lb.sock user hapee-lb group hapee mode 660

    The socket path

    The RHI service expects the Runtime API socket to be /var/run/hapee-extras/hapee-lb.sock. However, you can change the path that the RHI service expects by setting the variable HAPEE_LB_SOCKET in the following file:

    • On Debian/Ubuntu: /etc/default/hapee-extras-rhi
    • On RHEL: /etc/sysconfig/hapee-extras-rhi

    For example:

    hapee-extras-rhi
    text
    HAPEE_LB_SOCKET="/var/run/hapee-3.0/hapee-lb.sock"
    hapee-extras-rhi
    text
    HAPEE_LB_SOCKET="/var/run/hapee-3.0/hapee-lb.sock"

    Still, you must ensure that this same path is configured in your HAProxy Enterprise configuration file:

    hapee-lb.cfg
    haproxy
    global
      stats socket /var/run/hapee-3.0/hapee-lb.sock user hapee-lb group hapee mode 660
    hapee-lb.cfg
    haproxy
    global
      stats socket /var/run/hapee-3.0/hapee-lb.sock user hapee-lb group hapee mode 660

  3. Configure the RHI service.

    • Edit the file /etc/hapee-extras/hapee-rhi.cfg.

      The default configuration contains an example:

      hapee-rhi.cfg
      text
      # Inject the 10.200.200.200/32 address into the route daemon if
      # all the backends "be_static" and "be_app" are up.
      10.200.200.200/32 = all(b:be_static,b:be_app)
      hapee-rhi.cfg
      text
      # Inject the 10.200.200.200/32 address into the route daemon if
      # all the backends "be_static" and "be_app" are up.
      10.200.200.200/32 = all(b:be_static,b:be_app)

      This file contains a list of routes that the RHI service should add to BIRD, but only if a given rule returns true; a rule checks the status of one or more frontends or backends to see if they are up or down. A backend is treated as down if all servers fail their health checks or if you manually disable the servers. A frontend is down if you disable it manually.

      For example, the following line uses the all rule to announce the 192.168.1.10/32 IP only when both the be_static and be_app backend are up and running. When the condition is false, the IP is removed from the list of advertised routes:

      hapee-rhi.cfg
      text
      192.168.1.10/32 = all(b:be_static,b:be_app)
      hapee-rhi.cfg
      text
      192.168.1.10/32 = all(b:be_static,b:be_app)

      Or, the following line uses the any rule to advertise the IP if either the be_app or be_app2 backends are up and running:

      hapee-rhi.cfg
      text
      192.168.1.10/32 = any(b:be_app,b:be_app2)
      hapee-rhi.cfg
      text
      192.168.1.10/32 = any(b:be_app,b:be_app2)

    • Save the configuration and then enable and restart the service:

      nix
      sudo systemctl enable hapee-extras-rhi
      sudo systemctl restart hapee-extras-rhi
      nix
      sudo systemctl enable hapee-extras-rhi
      sudo systemctl restart hapee-extras-rhi

  4. Create rules to route traffic to the HAProxy Enterprise socket:

    Create routing rules:

    nix
    sudo iptables -t mangle -N DIVERT
    sudo iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
    sudo iptables -t mangle -A DIVERT -j MARK --set-mark 1
    sudo iptables -t mangle -A DIVERT -j ACCEPT
    sudo ip rule add fwmark 1 lookup 100
    sudo ip route add local 0.0.0.0/0 dev lo table 100
    nix
    sudo iptables -t mangle -N DIVERT
    sudo iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
    sudo iptables -t mangle -A DIVERT -j MARK --set-mark 1
    sudo iptables -t mangle -A DIVERT -j ACCEPT
    sudo ip rule add fwmark 1 lookup 100
    sudo ip route add local 0.0.0.0/0 dev lo table 100

    To learn more about these commands, see the transparent proxy topic in the Linux kernel documentation.

    Make the routing rules changes persistent:

    • To make the iptables changes persistent after reboot, use the iptables-save command. It saves the changes and configures the system to restore them at reboot.

      nix
      sudo apt install iptables-persistent
      sudo su -c 'iptables-save > /etc/iptables/rules.v4'
      nix
      sudo apt install iptables-persistent
      sudo su -c 'iptables-save > /etc/iptables/rules.v4'

    • To make the policy route (ip rule) and route table (ip route) changes persist after reboot, your next step depends on whether or not your system uses netplan. Typically, Ubuntu uses it, but Debian does not.

      • If your system uses netplan, persist the policy route (ip rule) and route table (ip route) changes in the netplan YAML configuration file located in /etc/netplan. The configuration file is probably the one having the lowest number and has a name like 00-installer-config.yaml or 01-netcfg.yaml.

        Edit the netplan YAML file, adding an lo section under the ethernets level:

        yaml
        network:
          ethernets:
            lo:
              routing-policy:
              - to: 0.0.0.0/0
                mark: 1
                table: 100
              routes:
              - to: 0.0.0.0/0
                type: local
                table: 100
        yaml
        network:
          ethernets:
            lo:
              routing-policy:
              - to: 0.0.0.0/0
                mark: 1
                table: 100
              routes:
              - to: 0.0.0.0/0
                type: local
                table: 100

        Then use sudo netplan try and sudo netplan apply before rebooting to make sure the configuration is valid. Ignore warnings about Open vSwitch.

      • If your system does not use netplan, persist the changes by defining the network in /etc/systemd/system/01-static-route.service.

        • Create the systemd service file:

          nix
          sudo touch /etc/systemd/system/01-static-route.service
          sudo vi /etc/systemd/system/01-static-route.service
          nix
          sudo touch /etc/systemd/system/01-static-route.service
          sudo vi /etc/systemd/system/01-static-route.service

        • Add the following lines:

          text
          [Unit]
          Description=Add route table 100
          Wants=network-online.target
          After=network-online.target
          [Service]
          Type=oneshot
          # create the route table and rule
          ExecStart=-/usr/sbin/ip route add local 0.0.0.0/0 dev lo table 100
          ExecStart=-/usr/sbin/ip rule add fwmark 1 lookup 100
          [Install]
          WantedBy=multi-user.target
          text
          [Unit]
          Description=Add route table 100
          Wants=network-online.target
          After=network-online.target
          [Service]
          Type=oneshot
          # create the route table and rule
          ExecStart=-/usr/sbin/ip route add local 0.0.0.0/0 dev lo table 100
          ExecStart=-/usr/sbin/ip rule add fwmark 1 lookup 100
          [Install]
          WantedBy=multi-user.target

        • Set the service to start on boot:

          nix
          sudo systemctl enable 01-static-route.service
          nix
          sudo systemctl enable 01-static-route.service

        • Restart the system.

    The saved settings will be restored after the restart.

    Create routing rules:

    • Add the IP tables rules:

      nix
      sudo firewall-cmd --permanent --direct --add-chain ipv4 mangle DIVERT
      sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -p tcp -m socket -j DIVERT
      sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle DIVERT 0 -j MARK --set-mark 1
      sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle DIVERT 1 -j ACCEPT
      nix
      sudo firewall-cmd --permanent --direct --add-chain ipv4 mangle DIVERT
      sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -p tcp -m socket -j DIVERT
      sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle DIVERT 0 -j MARK --set-mark 1
      sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle DIVERT 1 -j ACCEPT

    • Reload the firewall:

      nix
      sudo firewall-cmd --reload
      nix
      sudo firewall-cmd --reload

    • Add the IP route and rules:

      • Create the systemd service:

        nix
        sudo touch /etc/systemd/system/01-static-route.service
        sudo vi /etc/systemd/system/01-static-route.service
        nix
        sudo touch /etc/systemd/system/01-static-route.service
        sudo vi /etc/systemd/system/01-static-route.service

      • Add the following lines:

        text
        [Unit]
        Description=Add route table 100
        Wants=network-online.target
        After=network-online.target
        [Service]
        Type=oneshot
        # create the route table and rule
        ExecStart=-/usr/sbin/ip route add local 0.0.0.0/0 dev lo table 100
        ExecStart=-/usr/sbin/ip rule add fwmark 1 lookup 100
        [Install]
        WantedBy=multi-user.target
        text
        [Unit]
        Description=Add route table 100
        Wants=network-online.target
        After=network-online.target
        [Service]
        Type=oneshot
        # create the route table and rule
        ExecStart=-/usr/sbin/ip route add local 0.0.0.0/0 dev lo table 100
        ExecStart=-/usr/sbin/ip rule add fwmark 1 lookup 100
        [Install]
        WantedBy=multi-user.target

      • Set the service to start on boot:

        nix
        sudo systemctl enable 01-static-route.service
        nix
        sudo systemctl enable 01-static-route.service

      • Restart the system.

    To verify the IP tables rules, use the iptables command:

    nix
    sudo iptables -L -v -n -t mangle
    nix
    sudo iptables -L -v -n -t mangle
    output
    text
    Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target     prot opt in     out     source               destination
    1941  335K DIVERT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            socket
    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target     prot opt in     out     source               destination
    Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target     prot opt in     out     source               destination
    Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target     prot opt in     out     source               destination
    Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target     prot opt in     out     source               destination
    Chain DIVERT (1 references)
    pkts bytes target     prot opt in     out     source               destination
    1941  335K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK set 0x1
    1941  335K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    output
    text
    Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target     prot opt in     out     source               destination
    1941  335K DIVERT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            socket
    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target     prot opt in     out     source               destination
    Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target     prot opt in     out     source               destination
    Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target     prot opt in     out     source               destination
    Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target     prot opt in     out     source               destination
    Chain DIVERT (1 references)
    pkts bytes target     prot opt in     out     source               destination
    1941  335K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK set 0x1
    1941  335K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

    To verify the rule table, use the ip command:

    nix
    sudo ip rule ls
    nix
    sudo ip rule ls
    output
    text
    ...
    32765:  from all fwmark 0x1 lookup 100
    ...
    output
    text
    ...
    32765:  from all fwmark 0x1 lookup 100
    ...

    To verify the route table, use the ip command:

    nix
    sudo ip route ls table 100
    nix
    sudo ip route ls table 100
    output
    text
    local default dev lo scope host
    output
    text
    local default dev lo scope host

  5. Edit the HAProxy Enterprise configuration file, /etc/hapee-3.0/hapee-lb.cfg:

    • In the frontend section, change the addresses on the bind lines so that they use a new IP address that is not yet assigned to one of the network interfaces.

    • Because the IP addresses are not actually configured on the network interface, configure transparent binding by adding the transparent argument to the bind lines. This indicates that the IP address should be bound even though it does not belong to the local machine. Packets targeting this address will be intercepted as if the address were locally configured. This feature uses the kernel’s TPROXY feature, which has been available since Linux kernel 2.2.

    haproxy
    frontend www
      bind 192.168.1.10:80 name http transparent
      bind 192.168.1.10:443 name https ssl crt site.pem transparent
    haproxy
    frontend www
      bind 192.168.1.10:80 name http transparent
      bind 192.168.1.10:443 name https ssl crt site.pem transparent

  6. Save the changes and then restart the service.

    nix
    sudo systemctl reload hapee-3.0-lb
    nix
    sudo systemctl reload hapee-3.0-lb

    Each HAProxy Enterprise node should be assigned the same IP address.

  7. Configure BIRD for BGP or OSPF, which are used to advertise routes to peer routers.

    • Edit the file /etc/hapee-extras/hapee-route.cfg.

    • Add a section for either BGP or OSPF, depending on which protocol you intend to use for advertising routes to peers. Within it, add an export line that advertises routes from the volatile table, vol1.

      An example BGP configuration section:

      hapee-route.cfg
      text
        protocol bgp r1 {
        local 192.168.0.101 as 65001;
        neighbor 192.168.0.1 as 65001;
        graceful restart on;
        import none;
        
        # advertise the IP route
        export where proto = "vol1";
      }
      hapee-route.cfg
      text
        protocol bgp r1 {
        local 192.168.0.101 as 65001;
        neighbor 192.168.0.1 as 65001;
        graceful restart on;
        import none;
        
        # advertise the IP route
        export where proto = "vol1";
      }

      In this example:

      • the local directive refers to the IP address assigned to this HAProxy Enterprise node’s network interface and assigns the Autonomous System Number 65001.
      • the neighbor directive refers to the layer 3 device, such as the gateway router, with which we are establishing a BGP session.

      An example OSPF configuration section:

      hapee-route.cfg
      text
      protocol ospf anycast {
        tick 2;
        import none;
        
        # advertise the IP route
        export where proto = "vol1";
        
        area 0.0.0.0 {
          stub no;
          interface "eth0" {
             hello 10;
             retransmit 6;
             cost 10;
             transmit delay 5;
             dead count 4;
             wait 50;
             type broadcast;
          };
        };
      }
      hapee-route.cfg
      text
      protocol ospf anycast {
        tick 2;
        import none;
        
        # advertise the IP route
        export where proto = "vol1";
        
        area 0.0.0.0 {
          stub no;
          interface "eth0" {
             hello 10;
             retransmit 6;
             cost 10;
             transmit delay 5;
             dead count 4;
             wait 50;
             type broadcast;
          };
        };
      }

    • Save the configuration and then restart the service:

      nix
      sudo systemctl restart hapee-extras-route
      nix
      sudo systemctl restart hapee-extras-route

    • Repeat these steps for the hapee-extras-route6 service if using IPv6.

  8. Verify that RHI added a route to BIRD by calling the show route command. The IP should display.

    nix
    sudo /opt/hapee-extras/bin/hapee-route-cli show route
    nix
    sudo /opt/hapee-extras/bin/hapee-route-cli show route
    output
    text
    BIRD 1.6.3 ready.
    192.168.1.10/32    dev auto [vol1 18:54:15] * (0)
    output
    text
    BIRD 1.6.3 ready.
    192.168.1.10/32    dev auto [vol1 18:54:15] * (0)

    When you disable all servers in the backend, the command should not return this route.

  9. Configure ECMP on your peer router to distribute traffic to the bound IP.

Volatile table Jump to heading

BIRD stores routes in routing tables, with each table associated with a particular protocol such as BGP or OSPF. The RHI service adds its own table named volatile, which allows it to add routes to BIRD dynamically.

To see the volatile table definition, edit the BIRD configuration.

  1. Edit the file /etc/hapee-extras/hapee-route.cfg.

  2. Scroll down to the protocol volatile section.

    hapee-route.cfg
    text
    protocol volatile vol1 {
      #  gateway <ip>
    }
    hapee-route.cfg
    text
    protocol volatile vol1 {
      #  gateway <ip>
    }

    By default, BIRD announces routes through the gateway configured on the network interface, but you can specify a different network gateway by uncommenting the gateway directive and typing its IP address.

    For example:

    hapee-route.cfg
    text
    protocol volatile vol1 {
      gateway 192.168.1.244
    }
    hapee-route.cfg
    text
    protocol volatile vol1 {
      gateway 192.168.1.244
    }

    You can add more volatile tables to support advertising routes for different frontends:

    hapee-route.cfg
    text
    protocol volatile vol1 {
    }
    protocol volatile vol2 {
    }
    hapee-route.cfg
    text
    protocol volatile vol1 {
    }
    protocol volatile vol2 {
    }

    Then, prefix each route in the RHI service’s configuration, /etc/hapee-extras/hapee-rhi.cfg, with a table’s name:

    hapee-rhi.cfg
    text
    vol1%192.168.1.10/32 = all(b:be_static,b:be_app)
    vol2%192.168.1.11/32 = any(b:k8s_servers)
    hapee-rhi.cfg
    text
    vol1%192.168.1.10/32 = all(b:be_static,b:be_app)
    vol2%192.168.1.11/32 = any(b:k8s_servers)

Rules syntax Jump to heading

This section describes the syntax of the RHI configuration file.

text
<network>[,<network>,[...]] = <agg>(<b:|f:><name>[,<b:|f:><name>,[...]])
text
<network>[,<network>,[...]] = <agg>(<b:|f:><name>[,<b:|f:><name>,[...]])
Argument Description
<network> [%<protoname>]{<ipv4>,<ipv6>}[/<mask>] Specify an IPv4 or IPv6 CIDR subnet or list of several comma-delimited subnets. If you do not specify any subnet mask, RHI applies the /32 mask. For advanced configuration, you can supply the name of a volatile table in the %<protoname> section (default is vol1).
<agg> Aggregation function: all - Returns true if all listed proxies are active. any - Returns true if at least one of the proxies listed is active. never - Always false. For debugging purposes.
<b:|f:> Prefix of either b for backend or f for frontend.
<name> Name of the frontend or backend.

See also Jump to heading

Do you have any suggestions on how we can improve the content of this page?

Previous page Response body injection Next page Send metrics
© 2024 HAProxy Technologies, LLC. All Rights Reserved
Manage Cookie Preferences