Announcing HAProxy 3.0

Here we are in our twenty-third year, and open source HAProxy is going strong. HAProxy is the world’s fastest and most widely used software load balancer, with over one billion downloads on Docker Hub. It is the G2 category leader in API management, container networking, DDoS protection, web application firewall (WAF), and load balancing.

HAProxy maintains its edge over alternatives with best-in-class load balancing performance and reliability, the flexibility to support a wide variety of workloads, and a programmable and extensible architecture that fits your workflow. 

Today, HAProxy 3.0 has arrived, and HAProxy Enterprise 3.0 will be released later this year! In this blog post, we'll cover the changes in a short and digestible format. For a deeper dive into what’s new in version 3.0, read our blog post, Reviewing Every New Feature in HAProxy 3.0.

For an on-demand introduction to the new release, watch for our webinar, HAProxy 3.0: Feature Roundup.

How to get HAProxy 3.0

You can install HAProxy version 3.0 in any of the following ways:

Run it as a Docker container. View the Docker installation instructions.

Compile it from source. View the compilation instructions.

Major changes

First, let's cover the most important changes in HAProxy 3.0. These changes substantially modify how things were done in previous versions or introduce entirely new capabilities.

• Loading TLS certificates with the new crt-store section: The new crt-store configuration section provides a flexible way to store and consume SSL certificates. Replacing crt-list, crt-store separates certificate storage from their use in a frontend. The crt-store section allows you to individually specify the locations of each certificate component, for example, certificates files, key files, and OCSP response files. Aliases provide support for human-friendly names for referencing the certificates more easily on bind lines. The ocsp-update argument is now configured in a crt-store instead of a crt-list.

• Limiting glitchy HTTP/2 connections: Some HTTP/2 requests are valid from a protocol perspective but pose problems anyway. For example, sending a single header as a large number of  CONTINUATION frames could cause a denial of service. HAProxy now counts these so-called glitches and allows you to set a limit on them. You can also track them in a stick table to identify buggy applications or misbehaving clients.

• Assigning GUIDs to configuration objects: The new guid directive available in frontend, backend, and listen sections lets you assign a unique identifier to that section. The server directive also gained a guid argument. For now, the main use is for persisting stats after a reload, since only stats associated with objects having a GUID can be restored.

• Persisting stats after a reload: Reloading HAProxy will no longer reset the HAProxy Stats page, as long as you call the new Runtime API command dump stats-file first to save the current state to a file and then load that file with the stats-file configuration directive. Ensure that you've set a GUID on each frontend, backend, listen and server object by using the new guid keywords.

• Load balancing Syslog: The feature for load balancing Syslog messages, which was introduced in version 2.9, has progressed so that you can now set weights on server lines in your mode log backends. Meanwhile, the sticky algorithm, which had been limited to log backends, now applies to mode tcp and mode http backends as well. 

• Log as JSON and CBOR: You can now format log lines as JSON and CBOR. When configuring a custom log format, you will indicate which to use, and then in parentheses set the key for each field.

• More data exposed as fetch methods: New fetch methods expose data previously available only within logs. They include fetches that return the number of open HTTP streams for a backend or frontend, the size of the backend queue, the allowed number of streams, and a value that indicates whether a connection got redispatched because a server was unreachable. 

Noteworthy changes

Beyond the major changes, there are changes that simplify the configuration, improve performance, or extend existing functionality.

• Improving Lua performance: Single-threaded Lua scripts using lua-load will see a performance improvement. This improvement is the result of a change to the loading mechanism, where the maximum number of instructions is now divided by the number of threads. This makes it so that waiting threads have a shorter wait time and share the time slot more evenly. Safeguards are in place to prevent thread contention for threads waiting for the global Lua lock.

• Improving stick table performance: Stick tables have received a performance boost due to a change in the locking mechanism. Stick tables are now sharded over multiple tree heads, each having their own lock, and thus reducing lock contention. This means that on systems with many threads, stick table performance improves greatly. On a system with 80 threads, we measured performance gains of approximately 6x. As for systems with low thread counts, performance could be improved by as much as 2x when using peers.

• Setting default TLS certificates: When using a solitary frontend to load balance multiple websites, you host different TLS certificates for each site, typically by placing all certificates in a directory and letting HAProxy choose the correct one based on TLS SNI. New in this version, you can use the default-crt argument to indicate which certificate to use when no other certificates match. You can also set different defaults to support RSA and ECC algorithms. In a CRT-List, you can designate a default certificate by adding an asterisk after it.

• Controlling which HTTP errors to track: Until now, you could capture in a stick table the count and rate of client HTTP errors (4xx status codes) and server HTTP errors (5xx status codes), but you could not control specifically which status codes were included. This version adds global directives http-err-codes and http-fail-codes that let you set the status codes you care about, allowing you to ignore those that don't matter to you.

• Prioritizing traffic on the frontend and backend: HAProxy can modify the header of an IP packet to include the Differentiated Services (DS) field. This field classifies the packet so that the network stack can prioritize it higher or lower in relation to other traffic on the network. New in this version of HAProxy, you can set this field on connections to backend servers in addition to frontend connections to clients. To set the value, use the set-fc-tos and set-bc-tos actions (referring to the old Type of Service (TOS) header field, which has been superseded by DS).

• Setting a mark on IP packets on the frontend and backend: With HAProxy, you can set the fwmark on an IP packet, which classifies it so that, for example, it can use a specific routing table. HAProxy 3.0 now supports setting an fwmark on connections to backend servers as well as to clients connected on the frontend. Use the set-fc-mark and set-bc-mark actions.

• Creating UUIDv7 identifiers: The uuid fetch method now takes an optional argument that sets the version of the UUID to either 4 or 7. Combine the fetch with the unique-id-format directive and the unique-id fetch method to get an ID that you can attach to log entries.

• Configuring virtual ACL and Map files: ACL and Map files no longer require you to create files on disk. By prefixing the name of the file with @virt on an acl line in the HAProxy configuration, you allow HAProxy to start up and access the ACL and Map files as virtual representations only. Then use the Runtime API to add and delete rows in the virtual files. This is especially useful in containerized environments where the hassle of defining storage volumes and mapping volumes to the container's filesystem can seem like a burden. You can also prefix the filename with @opt, which marks the file as optional. In that case, HAProxy will check for the file on the filesystem, but if it doesn't find the file, it will assume the file is virtual.

• Relaying to the client or server when a gRPC connection has been aborted: Upon abort by the client, the RST_STREAM reason code can be retrieved from the buffer contents using the fetching sample fs.rst_code. The fetching sample fs.aborted returns true when an abort is received from the client. To detect server aborts, use the corresponding fetching samples bs.rst_code for the return code and bs.aborted for the status.

• A change in how servers are mapped in consistent-hash load balancing: When load balancing using a hash-based algorithm, HAProxy must keep track of which server is which. Instead of using numeric IDs to compute hash keys for the servers in a backend, the hash-key directive now supports using the servers’ addresses and ports to compute the hash keys. This is useful in cases where multiple HAProxy processes are balancing traffic to the same set of servers, as each independent HAProxy process will calculate the same hash key and therefore agree on routing decisions, even if its list of servers is in a different order.

Breaking changes

Although this is a major version release, there are only a few breaking changes, as you'll see in the short list below.

• Detecting accidental multiple commands sent to the Runtime API: Previously, it was occasionally possible to successfully issue multiple commands, which had the potential to produce unexpected results for long-running commands that may only partially complete. A warning will now be emitted when a \n is detected in a command, and the command will not be accepted. This change has also been backported to ensure that user scripts that utilize this behavior can be remedied.

• Rejecting the enabled keyword for dynamic servers: When defining a dynamic server, use of the enabled keyword is now rejected with an error, whereas previously it was only silently ignored.

• Stricter parsing of non-standard URIs: Parsing is now more strict during HTTP/1 processing for request target validation. This means that where previously, for compatibility, non-standard-compliant URIs were forwarded as-is for HTTP/1, now some invalid request targets are rejected with a 400-Bad-Request error.

• Renamed tune.ssl.ocsp-update: The tune.ssl.ocsp-update global keyword is now named tune.ocsp-update, as ocsp-update is unrelated to SSL tuning.

Conclusion

In the early days of the HAProxy project, it would have been difficult to foresee the multitude of ways people would use HAProxy, or the vast number of organizations that have adopted it at scale. Today, HAProxy is the market leader in software load balancing. That's thanks to the dedication of our open-source community members who write code, test features, document keywords, help newcomers, and evangelize to their organizations. Thank you to all!

HAProxy 3.0 maintains the strong momentum of our open-source load balancer into 2024 with improvements to simplicity, performance, reliability, observability, and security. This introductory blog post barely scratches the surface!

Subscribe to our blog and stay tuned for further deep dives on the latest updates from HAProxy 3.0. And in case you missed it, catch up with the huge new features we announced earlier this month in HAProxy Enterprise 2.9.

Ready to upgrade to HAProxy 3.0? Here’s how to get started.

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.