Load Balance an Infinite Number of Servers & Never Reload HAProxy

Every load balancer you’ll find on the market must deliver performance, reliability, scalability, and security and do it better than its competitors. Each must solve complex programming challenges that address those needs—choices that will affect the direction of the project for years to come. HAProxy is no different. When evaluating whether you should choose HAProxy or something else, it helps to know how project contributors answered the big architectural questions.

HAProxy’s software architecture reflects the real-world needs of its community. Mainly, it should be fast, secure, and stable. For a load balancer, which serves a critical role within a network, these decisions are make-or-break. If it’s slow, or if it becomes exploited, or if it can’t be relied upon, then that affects all of the services behind it. This philosophy has given HAProxy a unique evolution with long-standing, immutable attributes.

Let’s consider one of HAProxy’s newest features, dynamic servers, which lets you add and remove servers from HAProxy’s load balancing list on-the-fly without ever reloading the process. The benefit of dynamic servers is that it opens the door to very efficient autoscaling and service discovery.

It all started with the decision to give HAProxy a stateless design.

A Stateless Design

Probably one of the most important decisions was to design HAProxy to store zero states on disk. As it runs, it doesn’t write or read any information—such as routing rules, security policies, or server lists—on the filesystem. All information is stored in its running memory.

The HAProxy source code contains a starter guide that describes this:

Stateless design makes it easy to build clusters : by design, HAProxy does its best to ensure the highest service continuity without having to store information that could be lost in the event of a failure.

In other words, we want to avoid storing data in files and consequently losing access to those files if the storage medium catches on fire. By avoiding the state, in the event of a system failure, you should be able to spin up a new instance of HAProxy on a working server and continue right where you left off. If you’re stateless, you can’t lose any state, right? A stateless architecture is all about fault tolerance. In theory, this should work fine because HTTP requests and responses flowing through the load balancer are, likewise, stateless interactions.

However, realistically, a load balancer does have a state, and users want to keep it even if the process or storage is stopped or fails: TLS certificates, map/ACL files, and most importantly, the load balancer’s current configuration can all be counted as a state that can change during the process’s lifetime. Although the design philosophy aims to keep HAProxy stateless, which has led to HAProxy never accessing the filesystem at all, ways of updating and maintaining changes to these files had to be found. The question was, how to do it in a way that aligns with a stateless architecture that had already been ingrained into the software?

An Evolving Solution to Maintaining State

The simple way: Update your files on disk and then reload HAProxy. With a reload command, HAProxy will spawn a new process that picks up the new files while the old process gradually winds down. Reloading is a safe procedure; It loses zero traffic, thanks to HAProxy’s seamless reloads introduced in version 1.8, which is explained in the blog posts Truly Seamless Reloads with HAProxy – No More Hacks! and Hitless Reloads with HAProxy.

However, there is a downside. Having to reload HAProxy every time you make a configuration change incurs some overhead. Having two instances of HAProxy running in parallel, even for a short time, means twice the amount of memory used. If connections last more than a few seconds (e.g. WebSocket, database calls, etc.) and you’re servicing many clients, then winding down the old load balancer process might take more time and this double usage of resources might last a bit longer than you’d like; It could have an impact on your load balancer’s performance if you are reloading often under heavy traffic conditions, which creates a new process each time.

We needed a way to update TLS certificates, Map/ACL files, and the configuration without requiring a full reload.

Six years ago, when version 1.7 was released, HAProxy gained its Runtime API. The API allows users to call command-line operations that change the load balancer’s runtime state without a reload. No reload means no new processes spawned. With the Runtime API, you can enable or disable servers, modify ACL and Map file entries in memory, and manage TLS certificates. Over the years, the Runtime API has gained functions that allow you to update the in-memory representation of each type of state that is likely to change often. These operations update the load balancer without a reload. It’s like fueling up an airplane mid-flight!

The only problem is, those changes made with the Runtime API are not saved on disk. If you reload HAProxy, all that information is lost. Traditionally, users had to find a way to save changes to files while simultaneously calling the Runtime API, which was typically accomplished by writing a custom script that called the Runtime API and then updated the files too.

Then, with HAProxy 2.0, we got the Data Plane API, which offered the best of both worlds. The Data Plane API calls the Runtime API when possible to update HAProxy without requiring a reload, but it also updates the files on disk and, when necessary, reloads HAProxy. Because the most dynamic parts of the configuration are covered by the Runtime API, you still get the benefits of keeping data in memory and avoiding reloads for the load balancer process most of the time, while the Data Plane API takes care of the housekeeping of keeping files in sync.

Does this depart from HAProxy stateless architecture? Maybe a little, but then 20 years ago, load balancer configurations changed less often. HAProxy didn’t support TLS during its earliest days, and so it didn’t need to care about the state of TLS certificates. The list of backend servers was static since on-demand virtual servers weren’t really a thing. Ephemeral microservices, containers, and on-demand virtual servers have changed everything, requiring us to add and remove servers many times throughout the day, and we need to persist in all of those changes.

However, HAProxy's stateless design is still instrumental in giving it speed, stability, and security that other products can’t match. The Runtime API and Data Plane API don’t compromise on that.

You can learn more about the Runtime API and Data Plane API in the blog post The HAProxy APIs.

Adding & Removing Servers Dynamically

With HAProxy 2.5, the Runtime API gained a function for adding and removing servers dynamically. You can now add new servers infinitely, capped only by your operating system’s number of file descriptors and memory, with no reload required.

What’s the ultimate benefit? Better support for service discovery and autoscaling. The Data Plane API is becoming more than just a configuration state updater. It’s now responsible for integrating with service discovery features in Consul and AWS EC2, for instance. These are capabilities that require adding and removing servers from HAProxy’s configuration on-the-fly.

At the implementation level, adding a new server with the Runtime API looks like this:

Issue two more commands to enable the server with health checks:

Remove a server like this:

This opens the door to unlimited scalability. The load balancer can run continuously without a reload, but service discovery and autoscaling can use the Runtime API to update the in-memory list of servers dynamically. Although this feature has not been merged into the Data Plane API yet, it will be in the future.

Conclusion

The HAProxy Runtime API and Data Plane API pave the path to unlimited scalability while integrated with HAProxy’s long-standing, stateless design. However, what happens if the server on which it is running fails? You must have a copy of your load balancer configuration, TLS certificates, and Map/ACL files somewhere else. With HAProxy Enterprise and the HAProxy Fusion Control Plane, you can store copies of these files and keep an entire fleet of load balancers in sync.

Want to stay up to date on similar topics? Follow us on Twitter and join the conversation on Slack.

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.