Reviewing Every New Feature in HAProxy 3.0

HAProxy 3.0 maintains the strong momentum of our open-source load balancer into 2024 with improvements to simplicity, security, reliability, flexibility, and more. In this blog post, we'll dive into what’s new in version 3.0, providing examples and context. It’s a long list, so get comfortable and maybe bring a snack!

All these improvements (and more) will be incorporated into HAProxy Enterprise 3.0, releasing later this year.

Simplicity

New crt-store configuration section

The new crt-store configuration section provides a flexible way to store and consume SSL certificates. Replacing crt-list, crt-store separates certificate storage from their use in a frontend, and provides better visibility for certificate information by moving it from external files and placing it into the main HAProxy configuration. The crt-store section allows you to individually specify the locations of each certificate component, for example certificates files, key files, and OCSP response files. Aliases provide support for human-friendly names for referencing the certificates more easily on bind lines. The ocsp-update argument is now configured in a crt-store instead of a crt-list.

Consider the following example where a frontend references a crt-list for its certificate information, including OCSP response files. This is how you would use ocsp-update prior to crt-stores:

The contents of crt-list.txt would be:

And contained in site1.pem would be the public certificate, any intermediate certificates, and the private key. Using ocsp-update on, HAProxy knows to look for an .ocsp file in the same directory as the certificates and keys with the same name (site1.ocsp). 

Note that defining the certificate information in this way makes the information less visible, as it is not defined within the HAProxy configuration file.

To remedy this, use a crt-store section. To the HAProxy configuration file, add the following:

Note that in this example, the certificates, keys, and OCSP response files are split into their own files, and may be referenced individually. We are specifying a crt-base; this is the directory where we will place the files.

Now, you can reference these certificate elements in a frontend by their alias:

You must reference the certificates in a crt-store by using @<crt-store name>/<certificate name or alias>, which in this case is @web/site1. Note here that we also gave our certificates for site1 an alias, and are referencing them by the alias (site1).  If you do not provide a name for your crt-store, you can reference its certificates like so: @/site1, leaving out the <crt-store name>.

You can also use crt-store to specify keys separately from their certificates, as is the case in this example:

In this case, for site2, there are separate crt and key files. We specify a crt-base, the location of the certificates, and a key-base, the location of the keys. Once you define your certificates and keys in a crt-store you can then reference multiple certificates on the bind line:

You can also specify the crt-base and key-base in your global settings. Note that including crt-base or key-base in a crt-store take precedence over the global settings. The same is true for using absolute paths when specifying your certificates and keys.

Security

Protocol glitches

Some HTTP/2 requests are valid from a protocol perspective but pose problems anyway. For example, sending a single header as a large number of CONTINUATION frames could cause a denial of service. HAProxy now counts these so-called glitches and allows you to set a limit on them. You can also track them in a stick table to identify buggy applications or misbehaving clients.

OpenSSL security level

A new global keyword ssl-security-level allows you to set globally, that is, on every HAProxy SSL context, the OpenSSL’s internal security level. This enforces the appropriate checks and restrictions per the level specified. Parameters specified, including cipher suite encryption algorithms, supported ECC curves, supported signature algorithms, DH parameter sizes, certificate key sizes, and signature algorithms, inconsistent with the level will be rejected. For more information see: OpenSSL Set Security Level.

Specify a value between 0 and 5 to set the level, where 5 is the most strict:

Dependency on libsystemd removed

HAProxy’s security has been hardened by the removal of the dependency on the libsystemd library. This libsystemd library has many additional dependencies of its own, including the library at fault for the XZ Utils backdoor vulnerability. Removing the dependency on libsystemd means that HAProxy is not exposed to these undesired libraries.

Prevent HAProxy from accepting traffic on privileged ports

Two new global settings, harden.reject-privileged-ports.tcp for TCP connections and harden.reject-privileged-ports.quic for QUIC connections, enable HAProxy to ignore traffic when the client uses a privileged port (0 - 1023) as their source port. Clients using this range of ports are suspicious and such behavior often indicates spoofing, such as to launch DNS/NTP amplification attacks. The benefit in using these settings is that during DNS/NTP amplification attacks, CPU is reserved as HAProxy will drop the packets on these privileged ports instead of parsing them.

Reliability

Persist stats

Reloading HAProxy will no longer reset the HAProxy Stats page, as long as you call the new Runtime API command dump stats-file first to save the current state to a file and then load that file with the stats-file global configuration directive. Upon reload, the new processes set their counters to the previous values (as they appear in the stats file). Only proxy counters are supported which includes values for frontends, backends, servers, and listeners. Ensure that you've set a GUID on each frontend, backend, listen, and server object by using the new guid keyword. Only objects to which you have assigned a GUID will have their stats persisted across the reload.

To enable this behavior, follow these five steps:

Step 1

Assign GUIDs to each frontend, backend, listen, and server object in your configuration. Here is a simple configuration with one frontend that routes to one backend with one server. The frontend, backend, and server will each have a globally unique identifier assigned:

Step 2

Reload HAProxy to apply your configuration changes:

Step 3

Call dump stats-file to create the stats file, redirecting the output to your desired location:

The data in the file will capture the current state of the objects.

Step 4

Add stats-file to your global settings with the path to your stats file location that will be reloaded:

Step 5

Reload HAProxy. It will reload the stats counters from the file.

You can also generate GUIDs for items on bind lines. The guid-prefix should be used in this case to specify a string prefix for automatically generated GUIDs. Each listener on the bind line will have a GUID automatically generated for it with the prefix included. Here's an example:

HTTP/2: Limit the number of streams per connection

You can limit the number of total streams processed per incoming connection for HTTP/2 using the global keyword tune.h2.fe-max-total-streams. Once this limit is reached, HAProxy will send a graceful GOAWAY frame informing the client that it will close the connection after all pending streams have been closed. Usually, this prompts clients to reestablish their connections. This is helpful in situations where there is an imbalance in the load balancing due to clients maintaining long-lived connections.

Diagnostic mode

Diagnostic mode (-dD) will now report on likely problematic ACL pattern values that look like known ACL/sample fetch keywords in addition to its other warnings about suspicious configuration statements. It does not prevent startup. This is helpful for troubleshooting problematic configurations.

Consistent hash server mapping

When load balancing using a hash-based algorithm (balance hash), HAProxy must keep track of which server is which. Instead of using numeric IDs to compute hash keys for the servers in a backend, the hash-key directive now supports using the servers’ addresses and ports to compute the hash keys. This is useful in cases where multiple HAProxy processes are balancing traffic to the same set of servers, as each independent HAProxy process will calculate the same hash key and therefore agree on routing decisions, even if its list of servers is in a different order.

The new hash-key directive allows you to specify how the node keys, that is, the hash for each server, are calculated. This applies to node keys where you have set hash-type to consistent. There are three options for hash-key:

• id: the server’s numeric id if set, or its position in the server list

• addr: the server’s IP address

• addr-port: the server’s IP address and port

If you were to use hash-key id, and have no IDs explicitly set for your servers, the hashes could be inconsistent across load balancers if your server list were in different orders on each server. Using addr or addr-port solves this problem.

Consider the following backend:

This backend will use the hash load balancing algorithm, calculating the hash using the sample expression pathq (the request’s URL path with the query-string). Since we specified the hash-type as consistent, and the servers’ hash-key as addr (the IP address of the server), the servers’ IP addresses are used in the hash key computation. As such, if we had multiple load balancers with the servers listed in different orders, the routing would be consistent across the load balancers.

If multiple instances of HAProxy are running on the same host, for example in Docker containers, using both the IP address and the port (hash-key addr-port) is appropriate for creating the hash key.

Memory limit

To limit the total allocatable memory to a specified number of megabytes across all processes, use -m <limit>. This is useful in situations where resources are constrained, but note that specifying a limit this way may cause connection refusals and slowdowns depending on memory usage requirements. The limiting mechanism now uses RLMIT_DATA instead of RLIMIT_AS, which prevents an issue where the processes were reaching out-of-memory conditions below the configured limit. In master-worker mode, the memory limit is applied separately to the master and its forked worker process, as memory is not shared between processes.

Unix sockets and namespaces

Version 3.0 supports the namespace keyword for UNIX sockets specified on bind and server lines. If a permissions issue arises, HAProxy will log a permissions error instead of “failed to create socket”, since the capability cap_sys_admin is required for using namespaces for sockets.

For example, to specify a UNIX socket with a namespace, use the namespace keyword on your server line:

gRPC

With the gRPC protocol, the client can cancel communication with the server, which should be conveyed to the server so that it can clean up resources and perhaps invoke cancellations of its own to upstream gRPC services. Cancellations are useful for a few reasons. Often, gRPC clients configure deadlines so that a call will be canceled if it runs too long. Or a client might invoke a cancellation if it finishes early and no longer needs the stream. Read more about cancellation in the gRPC documentation.

Prior to HAProxy 3.0, cancellations, which are represented as RST_STREAM frames in HTTP/2 or as STOP_SENDING frames in HTTP/3, were not being properly relayed to the server. This has been fixed. Furthermore, HAProxy 3.0 adds new fetch methods that indicate whether the stream was canceled (aka aborted) and the error code. Below, we include them in a custom log format:

Here's the output when the client cancels:

New global keyword: expose-deprecated-directives

If you want to use deprecated directives you must also use the expose-deprecated-directives global keyword which will silence warnings. This global keyword applies to deprecated directives that do not have alternative solutions.

Emergency memory allocation

The mechanism for emergency memory allocation in low memory conditions has been improved for HTTP/1 and applets. Forward progress in processing is guaranteed as tasks are now queued according to their criticality and which missing buffer they require.

Flexibility

Write HAProxy logs as JSON or CBOR

You can now format log lines as JSON and CBOR. When configuring a custom log format, you will indicate which to use, and then in parentheses set the key for each field.

Here is an example for JSON:

This generates messages in the log file that looks like this:

If you switch that line to start with %{+cbor} instead of %{+json}, then the generated messages will look like this:

Virtual and optional ACL and Map files

In containerized environments, sometimes you don't want to deal with attaching a volume and mapping that volume to the container's filesystem, especially when it comes to storing data that is dynamic. In other words, wouldn't it be nice to remove the need to attach a volume? With HAProxy 3.0, you can now work with virtual ACL and map files. By prefixing your ACL and Map files with virtual@, HAProxy won't search the filesystem for the files but rather creates in-memory representations of them only.

The configuration below sets the stage for adding IP addresses to a denylist, where that denylist is virtual:

Then to add an address to the denylist, use the Runtime API. Below, we deny the IP address 172.20.0.1:

You can also prefix the filename with opt@, which marks the file as optional. In that case, HAProxy will check for the file on the filesystem, but if it doesn't find it, will assume it is virtual. That's useful for later saving the contents to a file so that they persist across reloading HAProxy.

Report the key that matched in a map file

A request matched a row in a map file, but which row? This version of HAProxy adds several, new map_*_key converters that return the key that was matched, rather than the associated value, making it easier to view the reason why the load balancer rerouted a request or took some other action.

In the example below, we use a Map file for path-based routing, where the request's URL path determines which backend to send a request to. By using a converter that ends in _key, in this case map_beg_key, which will match the beginning of the key in the file and then return the key, we can record which key in the Map file matches the request:

Let's assume that paths.map looks like this:

Then when a user requests a URL that begins with /api, they'll be sent to the backend named apiservers. When they request a URL that begins with /cart, they'll be sent to the backend named cartservers. If a request matches neither, it will record a dash in the logs. Our log shows this:

Explicitly set default TLS certificates

HAProxy can proxy traffic for multiple websites through the same frontend. To choose the right TLS certificate, it will compare the TLS Server Name Indication (SNI) value of the incoming connection with the certificates it finds in the certificates directory. If no certificate in your directory matches, it will resort to using a default one, which is the certificate that was loaded first (was first in the directory). But what if you wanted to set multiple defaults? For example, suppose you wanted to default to an ECDSA certificate, if supported, otherwise default to an RSA certificate? Now you have multiple ways to do this:

• Ensure that the first file in the directory is a multi-cert bundle.

• Use the new default-crt argument on a bind line.

• If using crt-list, indicate a default certificate by marking it with an asterisk.

To demonstrate setting one or more default-crt arguments on the frontend's bind line, below we set crt to a directory so that HAProxy will select the correct certificate from that directory based on SNI. But if there is no match, it will instead use one of the default files—either ECDSA or RSA:

Track specific HTTP errors

Until now, you could capture in a stick table the count and rate of client HTTP errors (4xx status codes) and server HTTP errors (5xx status codes), but you could not control specifically which status codes were included. This version adds the global directives http-err-codes and http-fail-codes that let you set the status codes you care about, allowing you to ignore those that don't matter to you. This works for responses from backend servers and for responses from HAProxy.

Tracking client HTTP errors can be useful for discovering misconfigured client applications, such as those that repeatedly use the wrong credentials or that make an unusually high number of requests. Below, we configure a stick table to track only 403 Forbidden errors, but you can also configure HAProxy to track a range of status codes by separating them with commas or indicating a range of codes with a dash:

Then we use the Runtime API command show table to see the number of 403 errors from each client. Here, the client that has the source IP address 172.19.0.10 has had 20 errors:

Error counts show you client-side issues, such as requesting a missing page or a forbidden page. On the other hand, you can set the global directive http-fail-codes to track server HTTP errors, such as 500 Internal Server Error and 503 Service Unavailable. Use it with a stick table that tracks http_fail_rate or http_fail_cnt to track server-side failure rates and counts.

Stick table pointers

Previously, you could use the Runtime API commands clear table [table] [key] and set table [table] [key] to remove or update a record in a stick table based on its key. You can now pass the record's unique ID (its pointer) to remove it or update it. You get the pointer from the show table [table] command.

In the example below, the pointer is 0x7f7de4bb50d8:

To update the record, use its pointer. Below we set the HTTP request count to zero:

Similarly, to remove the record, use its pointer:

Linux capabilities

Version 2.9 added the ability to preserve previously set Linux capabilities after the HAProxy process starts up. Linux capabilities are permissions granted to a process that it would not otherwise have when running as a non-root user. These permissions become relevant since HAProxy does not run as a root user after startup. 

In version 3.0, HAProxy is smarter about checking capabilities and will no longer emit error messages regarding the need for root permissions when running in transparent proxying mode or when binding to a port below 1024 (as is the case when using QUIC), so long as cap_net_admin is set. Additionally, file-system capabilities can now be set on the binary and if you start HAProxy as a root user, then adding setcap in the configuration is enough for adding those capabilities to the process. HAProxy will move the capabilities set on the binary (Permitted set) to its Effective set as long as the capabilities are also present in the setcap keyword list.

As a refresher, you can indicate which Linux capabilities to preserve after startup by using the setcap global keyword, specifying capabilities with a comma between each one:

Note that due to some security restrictions set in place by modules such as SELinux or Seccomp, HAProxy may be unable to set the required capabilities on the process. In this case, you must be also set the capabilities from the command line on the HAProxy binary:

Available capabilities that you can preserve using setcap include:

• cap_net_admin: Used for transparent proxying mode and for binding to privileged ports (lower than 1024, for example, for QUIC).

• cap_net_raw (subset of cap_net_admin): Used for setting a source IP address that does not belong to the system itself, as is the case with transparent proxying mode.

• cap_net_bind_service: Used for binding a socket to a specific network interface. This is required when using QUIC and binding to a privileged port. 

• cap_sys_admin: Used for creating a socket in a specific network namespace.

Set fwmark on packets to clients and servers

With HAProxy, you can set the fwmark on an IP packet, which classifies it so that, for example, it can use a specific routing table. HAProxy 3.0 now supports setting an fwmark on connections to backend servers as well as to clients connected on the frontend. Use the set-fc-mark and set-bc-mark actions. These replace the set-mark action, which had applied only to frontends.

To test this, first, give HAProxy the cap_net_admin capability, which is required for setting marks, by adding the setcap directive to the global section of your configuration:

We'd like to mark packets coming from the servers backend and ensure that they always go out through a specific network interface. Let's set an fwmark with a value of 2 (an arbitrary number) for the current connection. We hardcode the value, but you can also use an expression of fetch methods and converters to set it:

Now that we're marking packets, we just need to tell the network stack to route those packets through the desired interface, which is eth2 in this case:

To verify that the mark was set and that traffic is using the eth2 interface, you can use iptables to log the traffic:

Watch the kernel log. It shows OUT=eth2 and MARK=0x2:

Set traffic priority

HAProxy can modify the header of an IP packet to include the Differentiated Services (DS) field. This field classifies the packet so that the network stack can prioritize it higher or lower in relation to other traffic on the network. New in this version of HAProxy, you can set this field on connections to backend servers in addition to frontend connections to clients. To set the value, use the set-fc-tos and set-bc-tos actions (referring to the old Type of Service (TOS) header field, which has been superseded by DS).

First, give the HAProxy binary the cap_net_admin capability, which is required for setting network priority:

Then in the HAProxy configuration file, add the setcap directive to preserve that capability after HAProxy drops root privileges:

We would like to prioritize video traffic on the backend, so we set the set-bc-tos directive to 26. You can learn more about DSCP in RFC4594, and find common values on the IANA DSCP webpage. Although we're hardcoding the value, you can also use an expression of fetch methods and converters to set it:

New sample fetches

New fetch methods introduced in this version expose data that you can use in ACL expressions. They include fetches that return the number of open HTTP streams for a backend or frontend, the size of the backend queue, the allowed number of streams, and a value that indicates whether a connection was redispatched because a server became unreachable. 

• bc_be_queue – The number of streams de-queued while waiting for a connection slot on the target backend

• bc_glitches – The number of protocol glitches counted on the backend connection

• bc_nb_streams – The number of streams opened on the backend connection

• bc_srv_queue – The number of streams de-queued while waiting for a connection slot on the target server

• bc_settings_streams_limit – The maximum number of streams allowed on the backend connection

• fc_glitches – The number of protocol glitches counted on the frontend connection

• fc_nb_streams – The number of streams opened on the frontend connection

• fc_settings_streams_limit – The maximum number of streams allowed on the frontend connection

• txn.redispatched – True if the connection has experienced redispatch

Weights in log backends

Log backends were introduced in version 2.9. They allow you to set mode log in a backend to load balance the Syslog protocol. You can connect to backend Syslog servers over TCP or UDP.

In version 3.0, you can now set weights on server lines in your mode log backends. The example below demonstrates a log backend that uses weights:

Here, HAProxy listens for incoming log messages on TCP and UDP ports 514. As a simple setup, you can run an NGINX web server as a Docker container, setting up the Docker daemon to forward the container's logs to HAProxy by saving the following as /etc/docker/daemon.json:

Then run the NGINX container and watch the logs come through on your Syslog servers as you make web requests:

Because we've set weights on the servers in HAProxy, the servers will receive different amounts of log traffic.

Support for UUIDv7 identifiers

You can now generate universally unique identifiers that use the UUIDv7 format. UUIDs of this type factor in the current UNIX timestamp and are therefore time sortable, which tells you when a UUID was generated in relation to other UUIDs. In the example below, we set the unique ID format to version 7 and use the unique-id fetch method to get a new UUID to include in the logs:

Log messages will look like this, where the first field is the UUID:

HTTP forward proxy for OCSP updates

HAProxy 2.8 introduced a new and simpler way to configure OCSP stapling. HAProxy periodically contacts your SSL certificate issuer's OCSP server to get the revocation status of your SSL certificate.  In version 3.0, if you're operating in an air-gapped environment where the HAProxy server does not have direct access to the Internet, and therefore can't connect to the OCSP server, you can set an HTTP forward proxy to reach the Internet. Add the ocsp-update.httpproxy global directive to indicate the proxy's address:

Then the HTTP forward proxy can relay the OCSP request to the OCSP server.

Reverse HTTP

Reverse HTTP, which is an experimental feature added in HAProxy 2.9, allows HAProxy load balancers to self-register with an edge HAProxy load balancer and to then fill in as backend servers. In version 2.9, the mechanism for matching client requests with the correct backend server / HAProxy instance was via SNI. A new directive named pool-conn-name provides more flexibility, allowing you to set the name to match with an expression of fetch methods and converters.

Observability

Prometheus

When configuring the Prometheus exporter, you can now include a URL parameter named extra-counters, which enables additional counters that provide information related to the HTTP protocol, QUIC, and SSL. Set this in your Prometheus server's prometheus.yml file, replacing haproxy with your load balancer's IP address:

Decrypt TLS 1.3 packet captures to backend servers

When diagnosing network issues, you will often need to analyze TLS-encrypted traffic to see the underlying application-layer protocol messages. For example, when using Wireshark, you can import a packet capture file that contains the traffic you would like to analyze, but then you need a way to decipher the captured packets. The most common way to do this is to import a key log file into Wireshark, which contains the secrets needed to decipher a TLS session.

While prior versions of HAProxy supported producing a key log file for deciphering traffic between the client and HAProxy, HAProxy 3.0 adds the ability to produce a key log file for deciphering TLS traffic between HAProxy and backend servers, specifically when using TLS 1.3. 

Follow these seven steps:

Step 1

In your HAProxy configuration, set tune.ssl.keylog to on in the global section. This activates the retrieval of the TLS keys you will use for decryption in Wireshark:

Step 2

Force HAProxy and the backend servers to use TLS 1.3 by adding the ssl-min-ver argument to the servers:

Step 3

Define a custom log format that writes TLS session secrets to the access log:

Step 4

After HAProxy connects to a backend server, the access log will contain the keys for that TLS session. The access log will contain lines like this:

Step 5

Copy these lines to a text file. Then import the file into Wireshark via Edit > Preferences > Protocols > TLS > (Pre)-Master-Secret log filename.

Step 6

At the same time, capture traffic between HAProxy and the backend servers. For example, you can run tcpdump on the HAProxy server to get a PCAP file:

Step 7

Open this PCAP file in Wireshark to see the deciphered traffic.

Runtime API

show quic verbosity

The Runtime API's show quic command gained finer-grained levels of verbosity and the ability to filter the output to a specific connection. Previously, the verbosity levels you could specify were online or full. Now you can also specify a comma-delimited list of values that determine the output. Values include tp, sock, pktns, cc, and mux. Also, in this version you can specify the connection's hexadecimal address to see information for just that connection.

Here are examples that show output for the new verbosity levels:

Cookies for dynamic servers

Static cookies for session persistence are now supported for dynamically added servers. Dynamic servers refer to servers that do not have an explicit entry within your HAProxy configuration file. They are dynamic in the sense that you can add them programmatically using Runtime API commands. Dynamic servers are valuable in cases where you may have many servers that scale with traffic: when traffic loads are high, you add more servers, and when traffic loads diminish, you remove servers.

Previous versions of HAProxy did not support adding these dynamic servers and also using static cookies with those servers, but as of version 3.0, you can now use the add server command to add the server and specify its static cookie using just one command. Note that when adding a dynamic server, you must choose a load balancing algorithm for your backend that is dynamic (roundrobin, leastconn, or random).

To add a dynamic server to your backend with a static cookie, issue the add server command, specifying your cookie:

Here's the output:

You can also enable auto-generated names for session persistence cookies.  For more information see our guide for setting a dynamic cookie key  In that case, if you set the cookie argument on your add server command, the static cookie you specify will take precedence over the backend’s setting for dynamic cookies.

As a reminder, servers added via the Runtime API are not persisted after a reload. To ensure that servers you add via the Runtime API persists after a reload, be sure to also add them into your HAProxy configuration file (thus making them static servers).

Del server

Removing a dynamic server with the del server command is faster now that the command can close idle connections, where previously it would wait for the connections to close by themselves. This improvement is made possible by changes to the removal mechanism which allows forceful closure of idle connections.

Wait command

The new wait command for the Runtime API will wait for some specified time before then performing the following command.You could use this to collect metrics on a certain interval. This is also useful in cases where you need to wait until a server becomes removable (the server has been drained of  connections) before running additional commands, such as del server.

The syntax for the command is: wait { -h | <delay> } [<condition> [<args>...]]

If you do not provide any conditions, the command will simply wait for the requested delay (in default milliseconds) time before it continues processing. 

With this release, the only supported condition is srv-removable which will wait until the specified server is removable. When using socat, be sure to extend socat’s timeout to account for the wait time. 

The following example calls show activity, waits 10 seconds, then calls show activity again. Note that socat’s timeout value has been increased to 20 seconds:

Here's the output:

This example disables the server named app/app1, calls shutdown sessions for the server, waits for the server to be removable (using the condition srv-removable), and then once removable, deletes the server:

Finally, here's the output:

Performance

Fast forwarding

Zero-copying forwarding was introduced in Version 2.9 for TCP, HTTP/1, HTTP/2, and HTTP/3. As of version 3.0, applets, such as the cache, can also take advantage of the fast forwarding mechanism which avoids queuing more data when the mux buffer is full. This results in significantly less memory usage and higher performance. This behavior can be disabled by using tune.applet.zero-copy-forwarding for applets only, or tune.disable.zero-copy-forwarding globally.

In regards to QUIC, simplification of the internal send API resulted in removal of one buffer copy. The fast forwarding now considers the flow control, which reduces the number of thread wakeups and optimizes packet filling.

The HTTP/1 mutex now also supports zero-copy forwarding for chunks of unknown size. For example, chunks whose size may be larger than the buffer.

Ebtree update

Performance improved for ebtree on non-x86 machines. This results in approximately 3% faster task switching and approximately 2% faster string lookups.

Server name lookup

Configuration parsing time will see an improvement thanks to a change in server name lookups. The lookups now use a tree, which improves lookup time, whereas before the lookup was a linear operation.

QUIC

QUIC users will see a performance increase when using two new global settings:

• tune.quic.reorder-ratio

  • By adjusting tune.quic.reorder-ratio, you can change how quickly HAProxy detects packet losses. This setting applies to outgoing packets. When HAProxy receives an acknowledgement (ACK) for a packet it sent to the destination and that ACK arrived before other expected ACKs, or in other words it arrived out of sequence, it could indicate that packets never reached the destination. If it happens frequently, it indicates a poor network condition. By lowering the ratio, you're lowering the number of packets that can be missing ACKs before HAProxy takes action. That action is to reduce the packet sending window, which forces HAProxy to slow down its rate of transfer, at the cost of slower throughput. The default value is 50%, which means that the latest ACKed packet was halfway up the range of sent packets awaiting acknowledgements, with packets preceding it not yet ACKed.

• tune.quic.cc-hystart

  • Use this setting to enable use of the HyStart++ (RFC9406) algorithm instead of the Cubic algorithm. This provides an alternative to the TCP slow start phase of  the congestion control algorithm. This algorithm may show better recovery patterns regarding packet loss.

Additionally, the send path for QUIC was improved by cleanup on some of the low level QUIC sending functions. This includes exclusive use of sendmsg() (a system call for sending messages over sockets), optimizations avoiding unnecessary function calls, and avoiding copies where possible.

Traces

An improvement to traces will enable their use on servers with moderate to high levels of traffic without risk of server impact.

The improvement to traces was made possible by the implementation of near lockless rings. Previously, the locking mechanism limited the possibility of using traces in a production environment. Now that the rings are nearly lockless, allowing for parallel writes per group of threads, performance with traces enabled has been increased up to 20x.

A new global setting tune.ring.queues that sets the number of write queues in front of ring buffers can be used for debugging, as changing the value may reveal important behavior during a debugging session. This should only be changed if you are instructed to do so for troubleshooting.

Stick tables

Stick tables have received a performance boost due to a change in the locking mechanism. Previously, when the number of updates was high, stick tables could cause CPU usage to rise, due to the overhead associated with locking. Using peers amplified this issue. Stick tables are now sharded over multiple, smaller tables, each having their own lock, thus reducing lock contention. Also, the interlocking between peers and tables has been significantly reduced.

This means that on systems with many threads, stick table performance improves greatly. On a system with 80 threads, we measured performance gains of approximately 6x. As for systems with low thread counts, performance could be improved by as much as 2x when using peers.

Lua

Single-threaded Lua scripts using lua-load will see a performance improvement. This improvement is the result of a change to the loading mechanism, where the maximum number of instructions is now divided by the number of threads. This makes it so that waiting threads have a shorter wait time and share the time slot more evenly. Safeguards are in place to prevent thread contention for threads waiting for the global Lua lock.

Use tune.lua.forced-yield to tune the thread yielding behavior. For scripts that use lua-load, the optimal (and default) value was found to be the maximum of either 500 instructions or 10000 instructions divided by the number of threads. As for scripts loaded using lua-load-per-thread, in cases where more responsiveness is required, the value can be lowered from the default of 10000 instructions. In cases where the results of the Lua scripts are mandatory for processing the data, the value can be increased, but with caution, as an increase could cause thread contention.

Breaking Changes

Multiple CLI commands no longer supported

Previously, it was occasionally possible to successfully issue multiple commands by separating them with newlines, which had the potential to produce unexpected results for long-running commands that may only partially complete. A warning will now be emitted when a \n is detected in a command and the command will not be accepted. This change has also been backported to ensure that user scripts that utilize this behavior can be remedied.

Enabled keyword rejected for dynamic servers

When defining a dynamic server, use of the enabled keyword is now rejected with an error, whereas previously it was only silently ignored. Here's a sample input:

This produces the following output:

Invalid request targets rejected for non-compliant URIs

Parsing is now more strict during H1 processing for request target validation. This means that where previously, for compatibility, non-standard-compliant URIs were forwarded as-is for HTTP/1, now some invalid request targets are rejected with a 400 Bad Request error. The following rules now apply:

• The asterisk-form is now only allowed for OPTIONS and OTHER methods. There must now be only one asterisk and nothing more. 

• The CONNECT method must have a valid authority-form. All other forms are rejected. 

• The authority-form is now only supported for the CONNECT method. Origin-form is only checked for the CONNECT method.

• Absolute-form must have a scheme and a valid authority.

Tune.ssl.ocsp-update renamed to oscp-update

The tune.ssl.oscp-update global keyword is now named oscp-update, as ocsp-update is unrelated to SSL tuning.

Development Improvements

This release brings with it some major improvements for developers and contributors, as well as aids in saving time diagnosing issues and speeding up recovery:

• The internal API for applets has been simplified, with new applet code having its own buffers, and keyword handlers for the Runtime API now have their own buffers as well. 

• Updates to the makefile improve ease of use for packagers, including improved warnings plus easier passing of CFLAGS and LDFLAGS. Unused build options will produce a warning which will assist in debugging typos for build options with long names. 

• A new debugging feature has been added to the SSL and HTTP cache that allows assignment of a name to some memory areas so that it is more easily identified in the process map (using /proc/$pid/maps or using pmap on Linux versions 5.17 or greater). This makes it so that you can more easily determine where and why memory is being used. Future iterations will include more places where this debugging feature is implemented, further improving troubleshooting.

• By default HAProxy tries hard to prevent any thread and process creation after it starts. This is particularly important when running HAProxy’s own test suite, when executing Lua files of uncertain origin, and when experimenting with development versions, which may still contain bugs whose exploitability is uncertain. Generally speaking, it's a best practice to make sure that no unexpected background activity can be triggered by traffic. However, this may prevent external checks from working, and it may break some very specific Lua scripts which actively rely on the ability to fork. This global option insecure-fork-wanted disables this protection. As of version 3.0, you can also activate this option by using -dI (-d uppercase “i”) on the HAProxy command line. Note that it is a bad idea to disable it, as a vulnerability in a library or within HAProxy itself may be easier to exploit once disabled. In addition, forking from Lua, or anywhere else, is not reliable, as the forked process could embed a lock set by another thread and cause operations to never cease execution. As such, we recommend that you use this option with extreme caution, and that you move any workload requiring such a fork to a safer solution (such as using agents instead of external checks).  

• The DeviceAtlas module has been updated to support the new version of DeviceAtlas. 

• Support for FreeBSD 14 (and its new sched_setaffinity() system call) has been added.

Conclusion

HAProxy 3.0 was made possible through the work of contributors that pour immense effort into open-source projects like this one. This work includes participating in discussions, bug reporting, testing, documenting, providing help, writing code, reviewing code, and hosting packages.

While it's sadly impossible to include every contributor name here, all of you are invaluable members of the HAProxy community! Thank you.

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.