Reviewing Every New Feature in HAProxy 3.1

HAProxy 3.1 makes significant gains in performance and usability, with better capabilities for troubleshooting. In this blog post, we list all of the new features and changes.

All these improvements (and more) will be incorporated into HAProxy Enterprise 3.1, releasing Spring 2025.

Watch our webinar HAProxy 3.1: Feature Roundup and listen to our experts as we examine new features and updates and participate in the live Q&A. 

Log profiles

The way that HAProxy emits its logs is more flexible now with the introduction of log profiles, which let you assign names to your log formats. By defining log formats with names, you can choose the one best suited for each log server and even emit logs to multiple servers at the same time, each with its own format.

In the example below, we define a log profile named syslog that uses the syslog format and another profile named json that uses JSON. For syslog, we set the log-tag directive inside to change the syslog header's tag field, to give a hint to the syslog server about how to process the message. Notice that we also get to choose when to emit the log message. We're emitting the log message on the close event, when HAProxy has finalized the request-response transaction and has access to all of the data:

Our frontend uses both log profiles. By setting the profile argument on each log line, the frontend will send syslog to one log server and JSON to another:

By default, HAProxy emits a log message when the close event fires, but you can emit messages on other events, too. By tweaking the syslog profile to include more on lines, we have logged a message at each step of HAProxy's processing:

To enable these extra messages, set the log-steps directive to all or to a comma-separated list of steps:

Log profiles present plenty of opportunities:

• Create a log profile for emitting timing information to see how long HAProxy took to handle a request.

• Create another log profile containing every bit of information you can squeeze out of the load balancer to aid debugging.

• Switch the log format just by changing the profile argument on the log line.

• Reuse profiles across multiple frontends.

• Decide whether you want to emit messages for every step defined in a profile or for only some of them by setting the log-steps directive. 

do-log action

With the new do-log action, you can emit custom log messages throughout the processing of a request or response, allowing you to add debug statements that help you troubleshoot issues. Add the do-log action at various points of your configuration. In the example below, we set a variable named req.log_msg just before invoking a do-log directive:

Update your syslog log-profile section (see the section on log profiles) so that it includes the line on http-req, which defines the log format to use whenever http-request do-log is called. Notice that this log format prints the value of the variable req.log_msg:

Your log will show the custom log message:

The do-log action works with other directives too. Each matches up with a step in the log-profile section: 

• http-response do-log matches the step http-res.

• http-after-response do-log matches the step http-after-res.

• quic-initial do-log matches the step quic-init.

• tcp-request connection do-log matches the step tcp-req-conn.

• tcp-request session do-log matches the step tcp-req-sess.

• tcp-request content do-log matches the step tcp-req-cont.

• tcp-response content do-log matches the step tcp-res-cont.

set-retries action

The tcp-request content and http-request directives have a new action named set-retries that dynamically changes the number of times HAProxy will try to connect to a backend server if it fails to connect initially. Because HAProxy supports layer 7 retries via the retry-on directive, this new action also lets you retry on several other failure conditions.

In the example below, we use the set-retries action to change the number of retries from 3 to 10 when there's only one server up. In other words, when all the other servers are down and we've only got one server left, we make more connection attempts.

quic-initial directive

The new quic-initial directive, which you can add to frontend, listen, and named defaults sections, gives you a way to deny QUIC (Quick UDP Internet Connections) packets early in the pipeline to waste no resources on unwanted traffic. You have several options, including: 

• reject, which closes the connection before the TLS handshake and sends a CONNECTION_REFUSED error code to the client.

• dgram-drop, which silently ignores the reception of a QUIC initial packet, preventing a QUIC connection in the first place.

• send-retry, which sends a Retry packet to the client.

• accept, which allows the packet to continue.

Here's an example that rejects the initial QUIC packet from all source IP addresses, essentially disabling QUIC on this frontend:

You can test it with the HTTP/3 enabled curl command. Below, the client's connection is rejected:

After failing to connect via HTTP/3 over QUIC, the client (browser) will typically fall back to using HTTP/2 over TCP. So, if you want to block the client completely, you need to add additional rules that block the TCP traffic.

Server initial state

Add the new init-state argument to a server directive or server-template directive to control how quickly each server can return to handling traffic after restarting, coming out of maintenance mode, or adding the server through service discovery. The default setting, up, optimistically marks the server as ready to receive traffic immediately. But it will be marked as down if it fails its initial health check. Available options include:

• up - up immediately, but it will be marked as down if it fails the initial health check.

• fully-up - up immediately, but it will be marked as down if it fails all of its health checks.

• down - down initially and unable to receive traffic until it has passed the initial health check.

• fully-down - down initially and unable to receive traffic until it has passed all of its health checks.

In the example below, we use fully-down so that the server remains unavailable after coming out of maintenance mode until it has passed all ten of its health checks. In this case, the health checks happen five seconds apart.

Use the Runtime API's set server command to put servers into and out of maintenance mode:

SPOE

The Stream Processing Offloading Engine (SPOE) filter forwards streaming load balancer data to an external program. It enables you to implement custom functions at the proxy layer using any programming language to extend HAProxy.

What's new? A multiplexer-based implementation that allows idle connection sharing between threads and load balancing, queueing, and stickiness per request instead of per connection.This greatly improves reliability as the engine is no longer applet-based and is better aligned with the other proven mux-based mechanisms. This mux-based implementation allows for management of SPOP (Stream Processing Offload Protocol) through a new backend mode called spop. It also adds flexibility to SPOE, optimizes traffic distribution among servers, improves performance, and will ultimately make the entire system more reliable, as future changes to the SPOE engine will only affect pieces specific to SPOE.

In a configuration file, specify the mode for your backend as spop. This mode is now mandatory and automatically set for backends referenced by SPOEs. Configuring your backend in this way means that you are no longer required to use a separate configuration file for SPOE.

When an SPOE is used on a stream, a separate stream is created to handle the communication with the external program. The main stream is now the "parent" stream of this newly created "child" stream, which allows you to retrieve variables from it and perform some processing in the child stream based on the properties of the parent stream.

The following SPOE parameters were removed in this version and are silently ignored when present in the SPOE configuration: 

• maxconnrate

• maxerrrate 

• max-waiting-frames 

• timeout hello

• timeout idle

Variables for SPOA child streams

You can now pass variables from the main stream that's processing a request to the child stream of a Stream Processing Offload Agent (SPOA). Passing data like the source IP address to the agent was never a problem; that's already supported. What was missing was the ability to pass variables to the backend containing the agent servers. That prevented users from configuring session stickiness for agent servers or selecting a server based on a variable.

In the example below, we try to choose an agent server based on a URL parameter named target_server. The variable req.target_server gets its value from the URL parameter. Then, we check the value in the backend to choose which server to use. However, this method fails because the agents backend can't access the variables from the frontend. The agents backend is running in a child stream, not the stream that's processing the request, so it can't access the variables.

But in this version of HAProxy, you can solve this by prefixing the variable scope with the letter p for parent stream. Here, req becomes preq:

This works for these scopes: psess, ptxn, preq, and pres. Use this feature for session stickiness based on the client's source IP or other scenarios that require reading variables set by the parent stream.

TCP log supports CLF

HAProxy 3.1 updates the option tcplog directive to allow an optional argument: clf. When enabled, CLF (Common Log Format) sends the same information as the non-CLF option, but in a standardized format that CLF log servers can parse.

It's equivalent to the following log-format definition:

Send a host header with option httpchk

As of version 2.2, you can send HTTP health checks to backend servers like this:

Before version 2.2, the syntax for performing HTTP health checks was this:

If you prefer the traditional way, this version of HAProxy allows you to pass a host header to backend servers without having to specify carriage return and newline characters, and you don’t have to escape spaces with backslashes. Just add it as the last parameter on the option httpchk line, like this:

Size unit suffixes

Many size-related directives now correctly support unit suffixes. For example, a ring buffer size set to 10g will now be understood as 1073741824 bytes, instead of incorrectly interpreting it as 10 bytes.

New address family: abnsz

To become compatible with other software that supports Linux abstract namespaces, this version of HAProxy adds a new address family, abnsz, which stands for zero-terminated abstract namespace. So HAProxy can interconnect with software that determines the length of the namespace's name by the length of the string, terminated by a null byte. In contrast, the abns address family, which continues to exist, expects that the name is always 108 characters long, with null bytes filling in the trailing spaces.

The syntax when using abnsz is the same as with abns:

New address family: mptcp

MultiPath Transmission Control Protocol (MPTCP) is an extension of TCP and is described in RFC 8684. MPTCP, according to its RFC, "enables a transport connection to operate across multiple paths simultaneously". MPTCP improves resource utilization, increases throughput, and responds quicker to failures. MPTCP addresses can be explicitly specified using the following prefixes: mptcp@, mptcp4@, and mptcp6@.

• If you declare mptcp@<address>[:port1[-port2]] in your configuration file, the IP address is considered as an IPv4 or IPv6 address depending on its syntax. 

• If you declare mptcp4@<address>[:port1[-port2]] in your configuration file, the IP address will always be considered as an IPv4 address.

• If you declare mptcp6@<address>[:port1[-port2]] in your configuration file, the IP address will always be considered as an IPv6 address.

With all three MPTCP prefixes, the socket type and transport method is forced to "stream" with MPTCP. Depending on the statement using this MPTCP address, a port or a port range must be specified.

New sample fetches

HAProxy 3.1 adds new sample fetch methods related to SSL/TLS client certificates:

• ssl_c_san - Returns a string of comma-separated Subject Alt Name fields contained in the client certificate.

• ssl_fc_sigalgs_bin - Returns the content of the signatures_algorithms (13) TLS extension presented during the Client Hello.

• ssl_fc_supported_versions_bin - Returns the content of the supported_versions (43) TLS extension presented during the Client Hello.

New converters

This version introduces new converters. Converters transform the output from a fetch method.

• date - Converts an HTTP date string to a UNIX timestamp.

• rfc7239_nn - Converts an IPv4 or IPv6 address to a compliant address that you can use in the from field of a Forwarded header. The nn here stands for node name. You can use this converter to build a custom Forwarded header.

• rfc7239_np - Converts an integer into a compliant port that you can use in the from field of a Forwarded header. The np here stands for node port. You can use this converter to build a custom Forwarded header.

HAProxy Runtime API

This version of HAProxy updates the Runtime API with new commands and options.

debug counters

A new Runtime API command debug counters shows all internal counters placed in the code. Primarily aimed at developers, these debug counters provide insight for analyzing glitch counters and counters placed in the code using the new COUNT_IF() macro. Developers can use this macro during development to place arbitrary event counters anywhere in the code and check the counters' values at runtime using the Runtime API. For example, glitch counters can provide useful information when they are increasing even though no request is instantiated or no log is produced.

While diagnosing a problem, you might be asked by a developer to run the command debug counters show or debug counters all to list all available counters. The counters are listed along with their count, type, location in the code (file name and line number), function name, the condition that triggered the counter, and any associated description. Here is an example for debug counters all:

Please note that the format and contents of this output may change across versions and should only be used when requested during a debugging session.

dump ssl cert

The new dump ssl cert command for the Runtime API will display an SSL certificate directly in PEM format; useful for placing delimiters and saving certificates when it was updated on the CLI and not on the filesystem yet. You can also dump a transaction by prefixing the filename with an asterisk. This command is restricted and can only be issued on sockets configured for level admin.

The syntax for the command is: 

echo

The echo command with syntax echo <text> will print what's contained in <text> to the console output; it's useful for writing comments in between multiple commands. For example:

show dev

This version improves the show dev Runtime API command  by printing more information about arguments provided on the command line as well as the Linux capabilities set at process start and the current capabilities (the ability to preserve capabilities was introduced in Version 2.9 and improved in Version 3.0). This information is crucial for engineers troubleshooting the product.

To view this development and debug information, issue the the show dev command:

You can see in the output that the command-line arguments and capabilities are present:

Note that the format and contents of this output may change per version, and is most useful for providing current system status to developers that are diagnosing issues.

show env

The command show env dumps environment variables known to the process, and you can specify which environment variable you would like to see as well:

Here's an example output:

show sess

The new show-uri option for command show sess dumps to the console output a list of active streams and displays the transaction URI, if available and captured during the request analysis.

show quic

The show quic command produces more internal information about the internal state of the congestion control algorithm and other dynamic metrics (such as window size, bytes in flight, and counters).

show info

The show info command will now report the current and total number of streams. It can help quickly detect if a slowdown is caused on the client side or the server side and facilitate the export of activity metrics. Here's an example output that shows the new CurrStreams and CumStreams:

Troubleshooting 

This release includes a number of troubleshooting and debugging improvements in order to reduce the number of round trips between developers and users and to provide better insights for debugging. The aim is to minimize impact to the user while also being able to gather crucial logs, traces, and core dumps. Improvements here include new log fetches, counters, and converters, improved log messages in certain areas, improved verbosity and options for several Runtime API commands, the new traces section, and improvements to the thread watchdog. 

Traces

Starting in version 3.1, traces get a dedicated configuration section named traces, providing a better user experience compared to previous versions. Traces report more information than before, too.

Traces let you see events as they happen inside the load balancer during the processing of a request. They're useful for debugging, especially since you can enable them on a deployed HAProxy instance. Traces were introduced in version 2.1, but at that time you had to configure them through the Runtime API. In version 2.7, you could configure traces from the HAProxy configuration file, but the feature was marked as experimental. The new traces section, which is not experimental, offers better separation from other process-level settings and a more straightforward syntax. Use traces cautiously, as it could impact performance.

To become familiar with them, read through the Runtime API documentation on traces. Then, try out the new syntax in the configuration file. In the following configuration example, we trace HTTP/2 requests:

We restarted HAProxy and used the journalctl command to follow the output of this trace:

The output shows the events happening inside the load balancer:

You can list multiple trace statements in a traces section to trace various requests simultaneously. Also new to traces is the ability to specify an additional source to follow along with the one you are tracing; this is useful for tracing backend requests while also tracing their associated frontend connections, for example.

Major improvements to the underlying muxes' debugging and troubleshooting information make all of this possible. Thanks to these improvements, traces for H1, H2, and H3/QUIC now expose much more internal information. This aids in more easily piecing together requests through their entire path through the system, which was not possible previously. 

when() converter

Consider a case where you may want to log some information or pass data to a converter only when certain conditions are met. Thanks to the new when() converter, you can! The new when() converter enables you to pass data, such as debugging information, only when a condition is met, such as an error condition. 

Along with the when() converter, there are several new fetches as well that can produce data related to debugging and troubleshooting. The first new fetches are the debug string fetches, fs.debug_str for a frontend stream and bs.debug_str for a backend stream. These two fetches return debugging information from the lower layers of the stream and connection. The next set of fetches are the entity fetches last_entity and waiting_entity where the former returns the ID of the last entity that was evaluated during stream analysis and the former returns the ID of the entity that was waiting to continue its processing when an error or timeout occurred. In this context, entity refers to a rule or filter.

You can use these fetches on their own to always print this debug information, which may be too verbose to log on every request, or you can use these fetches with the when() converter as follows to log this information only when an error condition occurs, so as to avoid flooding the logs:

For the debug string fetches, you can provide the when() converter with a condition that tells HAProxy to log the debug information only when there is an error. The when() converter is flexible in terms of the conditions you are able to provide to it, and you can prefix a condition with ! to negate it. You can also specify an ACL to evaluate. The available conditions are listed here:

• error: returns true when an error was encountered during stream processing

• forwarded: returns true when the request was forwarded to a backend

• normal: returns true when no error occurred

• processed: returns true when the request was either forwarded to a backend server or processed by an applet

• stopping: returns true if the process is currently stopping

• acl: returns true when the ACL condition evaluates to true. Use this condition like so, specifying the ACL condition and ACL name separated by a comma: when(acl,<acl_name>).

Note that if the condition evaluates to false, then the fetch or converter associated with it will  not be called. This may be useful in cases where you want to customize when certain items are logged or you want to call a converter only when some condition is met.

For example, to log upon error in a frontend, add a log format statement like this to your frontend, using the condition normal and prefix it with ! to negate the condition:

That is to say "log the frontend debug string only when the results of the expression are not normal." When this condition is met, HAProxy will log a message that contains the content of the debug string:

You can do the same for a backend, replacing fs.debug_str with bs.debug_str.

As for the last_entity and waiting_entity fetches, you can use them with when() to log the ID of the last entity or the waiting entity only when an error condition is met. In this case, you can set the condition for when() to error, which means it will log the entity ID only when there is an error. You can add a log format line as follows, specifying which entity's, last or waiting, ID to log:

If the condition for logging is not met, a dash "-" is logged in the message instead. 

fc/bc_err fetches

As of version 2.5, you can use the sample fetches fc_err for frontends and bc_err for backends to help determine the cause of an error on the current connection. In this release, these fetches have been enhanced to include connection-level errors that occur during data transfers. This is useful for detecting network misconfigurations at the OS level, for example incorrect firewall rules, resource limits of the TCP stack, or a bug in the kernel, as would be indicated by an error such as ERESET or ECONNRESET. 

You can use the intermediary fetches fc_err_name and bc_err_name to get the short name of the error instead of just the error code (as would be returned from fc_err or bc_err) or the long error message returned by fc_err_str or bc_err_str. As with the fc_err and bc_err sample fetches, use the intermediary fetches prefixed with fc_* for frontends and bc_* for backends.

Post_mortem structure for core dumps

The system may produce a core dump on a fatal error or when the watchdog fires, which detects deadlocks. While crucial to diagnosing issues, sometimes these files are truncated or can be missing information vital to analysis. This release includes an internal post_mortem structure to be included in core dumps, which contains pointers to the most important internal structures. This structure, present in all core dumps, allows developers to more easily navigate the process's memory, reducing analysis time, and prevents the user from needing to change their settings to produce different debug output. Additionally, more hints have been added to the crash output to help in decoding the core dump. To view this debugging information without producing a core dump, use the improved show dev command. 

Improved thread dump

In previous versions, sometimes stderr outputs of the thread backtraces in core dumps would be missing, or only the last one was present due to the reuse of the same output buffer for each thread. Core dumps now include backtraces for all threads, as each thread's backtrace is now dumped in its own buffer. Also present in core dumps as of this version are the output messages for each thread, which assists developers in determining the causes of issues even when debug symbols are not present. 

Watchdog and stuck threads

This version includes improvements to HAProxy's watchdog, which detects deadlocks and kills runaway processes. The watchdog will now watch for stuck threads more often, by default every 100ms, and it will emit warnings regarding a stuck thread's backtrace before killing it. It will stop the thread if after the first warning the thread makes no progress for one second. In this way, you should see ten warnings about a stuck thread before the watchdog kills it. 

Note that you can adjust the time delay after which HAProxy will emit a warning for a stuck thread using the global debugging directive warn-blocked-traffic-after. We do not advise that you change this value, but changing it may be necessary during a debugging session. 

Also note that you may see this behavior where the watchdog warns about a thread when you are doing computationally-heavy operations, such as Lua parsing loops in sample fetches or while using map_reg or map_regm. 

An issue regarding the show threads Runtime API command that caused it to take action on threads sooner than expected has also been remedied. 

GDB core inspection scripts

This release includes  GDB (GNU debugger) scripts that are useful for inspecting core dumps. You can find them here: /haproxy/haproxy/tree/v3.1.0/dev/gdb

Memory profiling

This version enhances the accuracy of the memory profiler by improving the tracking of the association between memory allocations and releases and by intercepting more calls such as strdup() as well as non-portable calls such as strndup() and memalign(). This improvement in accuracy applies to the per-DSO (dynamic shared object) summary as well, and should fix some rare occurrences where it incorrectly appeared that there was more memory free than allocated. New to this version, a summary is provided per external dependency, which can help to determine if a particular library is leaking memory and where.

Logged server status

In this version, HAProxy now logs the correct server status after an L7 retry occurs. Previously it reported only the first code that triggered the retry.

Short timeouts

Under high load, unexpected behavior may arise due to extremely short timeouts. Given that the default unit for timeouts is milliseconds, it is not so obvious that the timeout value you specify may be too small if you do not also specify the unit. HAProxy will now emit a warning for a timeout value less than 100ms if you do not provide a unit with the timeout value. The warning will suggest how to configure the directive to avoid the warning, typically by appending "s" if you are specifying a value in seconds or "ms" for milliseconds.

File descriptor limits

A new global directive fd-hard-limit sets the maximum number of file descriptors the process can use. By default, it is set to 1048576 (roughly one million, the long-standing default for most operating systems). This value is used to remedy an issue that can be caused by a new operating system default declaring that the process can have up to one billion file descriptors, thus resulting in either slow boot times or failing on an out-of-memory exception. HAProxy uses the value of this directive to set the maximum number of file descriptors and to determine a reasonable limit based on the available resources (for example RAM size). If you require a custom maximum number of file descriptors, use this global directive as follows:

Time jumping

To remedy an issue some users have been facing regarding incorrect rate counters as a result of time jumps, that is, a sudden, significant jump forward or backwards in the system time, HAProxy will now use the precise monotonic clock as the main clock source whenever the operating system supports it. In previous versions, measures were put in place to detect and correct these jumps, leaving a few hard-to-detect cases, but now the use of the precise monotonic clock helps to better detect small time jumps and to provide a finer time resolution.

Log small H2/QUIC anomalies

HAProxy 3.0 introduced the ability to track protocol glitches, or those requests that are valid from a protocol perspective but have potential to pose problems anyway. This version enables the HTTP/2 and QUIC multiplexers to count small anomalies that could force a connection to close. You can capture and examine this information in the logs. These could help to identify to what level a request is suspicious.

Performance

HAProxy 3.1 improved performance in the following ways.

H2 

The H2 mux is significantly more performant in this version. This was accomplished by optimizing the H2 mux to wake up only when there are requests ready to process, saving CPU cyles, and resulting in using 30% fewer instructions on average when downloading. The POST upload performance has been increased up to 24x with default settings and it now also avoids head-of-line blocking when downloading from H2 servers. 

Two new global directives, tune.h2.be.rxbuf and tune.h2.fe.rxbuf allow for further tuning of this behavior. Specify a buffer size in bytes using tune.h2.fe.rxbuf for incoming connections and tune.h2.be.rxbuf for outgoing connections. For both uploads and for downloads, one buffer is granted to each stream and 7/8 of the unused buffers is shared between streams that are uploading / downloading, which is the mechanism that significantly improves performance.

QUIC

New to this version are two new global directives for tuning QUIC performance. The first, tune.quic.cc.cubic.min-losses takes a number that defines a threshold for how many packets
must be missed before the Cubic congestion control algorithm determines that a loss has occurred. This setting allows the algorithm to be slightly more tolerant to false losses, though you should exercise caution when changing the value from the default value of 1. A value of 2 may prove to show some performance improvement, though we do not recommend running this way for extended periods of time, only for analysis, and you should avoid providing a value larger than 2. 

As for tune.quic.frontend.default-max-window-size, you can use this global directive to define the default maximum window size for the congestion controller of a single QUIC connection, by specifying an integer value between 10k and 4g, with a suffix of "k", "m" or "g".

This version sees an efficiency improvement in regards to the QUIC buffer allocator and using this tunable, you are able to vary the size of the memory required per-connection, thus reducing overallocation.

Regarding the transmission path for QUIC, its performance has been significantly improved in this version so that it will now adapt to the current send window size and will use Generic Send Offload to let the kernel send multiple packets in a single system call. This offloads processing from HAProxy and the kernel and places it onto the hardware. This is especially meaningful when used on virtual machines where system calls have potential to be expensive.

Process priorities

To help improve performance in the case of large configurations that consume a lot of CPU on reload, two new global configuration directives tune.renice.startup and tune.renice.runtime are new to this version. These global directives take a value between -20 and 19 to apply a scheduling priority to configuration parsing. A lower value will lower the priority of the parsing, for example, a priority value of 10 will be scheduled before a priority value of 8. These values correspond to the scheduling priority values accepted by the setpriority() Linux system call. Once the parsing is complete, the priority of the parsing returns to its previous value, or to the value of tune.renice.runtime, if also present in the configuration. See the Linux manual page on scheduling priority (sched()) for more information. 

TCP logs

TCP logs saw a 56% performance gain in this version thanks to the implementation of the line-by-line parser into the TCP log forwarder. In regards to log servers, the ring sending mechanism sees improvement in this version, as the load is better balanced across available threads, assigning new server connections to threads with the least load. You can now use the max-reuse directive for TCP connections served by rings. When used for this reason, the sink TCP connection processors will not reuse a server connection more times than the indicated maximum. This means that connections to the servers will be forcefully removed and re-created, which helps to better distribute the load across available threads, thus increasing performance. Make sure that when using this directive that the connections are not closed more than a couple of times per second.

Pattern cache

In previous versions, some users may have seen intense CPU usage by the pattern LRU cache when performing lookups with low cardinality. To remedy this, in this version the cache will be skipped for maps or expressions with patterns with low cardinality, that is, less than 5 for regular expressions, less than 20 for others. Depending on your setup, you could see a savings of 5-15% CPU in these cases.

Config checking

As of this version, configured servers for backends are now properly indexed, which saves time in detecting duplicate servers. As such, the startup time for a configuration with a large number of servers could see a reduction of up to a factor of 4.

Variables

Variables have been moved from a list to a tree, resulting in a 67% global performance gain for a configuration including 100 variables.

Expressions

We saw a performance gain of, on average, 7% regarding arithmetic and string expressions by removing the need for trivial casts samples and converters of the same types. 

Lua

The Lua function core.set_map() has doubled its performance in speed by avoiding duplicate lookups.

QUIC buffer

Small frames for the QUIC buffer handling now use small buffers. This improves both the memory and CPU usage, as the buffers are now more appropriately sized and do not require realignment.

QUIC will always send a NEW_TOKEN frame to new clients for reuse in the next connection. This behavior permits clients to reconnect after being validated without going through the address validation process again on the next connection. In other words, the next established connection will improve network performance when a listener is attacked or when dealing with a lossy network.

File descriptors

This version includes a performance gain regarding smoother reloads for large systems, that is, systems requiring a large number of file descriptors and a large number of threads. This gain is due to how file descriptors are handled on boot, shortening initialization time from 1.6s to 10ms for a setup with 2M configured file descriptors. 

Master-worker

HAProxy's master-worker mode was heavily reworked in this version to improve stability and maintainability. Its previous architecture model proved difficult in maintaining forward compatibility for seamless upgrades; the rework aims to remedy this problem. Per the new model, the master process does nothing after starting until it confirms the worker is ready, and it no longer re-executes itself to read the configuration, which greatly reduces the number of potential race conditions. The configuration is now buffered once for both the master and worker and as such will be identical for both. As such, environment variables shared by both will be more consistent, and the worker will be isolated from variables applicable to the master only. This all improves the separation between the processes. An additional improvement is that this rework will reduce file descriptor leaks across the processes as they are now better separated. All of this to say: you should not notice anything as a result of this change except for improved reliability.

HAProxy test suite

An additional milestone regarding reliability that is worth a mention is that the regtests, that is, HAProxy's test suite, have now exceeded 5000 expect rules, spread over 216 files. These tests are set to strict evaluation, which means that, when run, any warning will produce an error. Know that reliability is a top priority, and these tests are executed on 20-30 platform combinations on every push, and are run locally by developers on each commit. This ensures that HAProxy continues to shine in regards to reliability and robustness.

Deprecation

The program section is deprecated in HAProxy 3.1 and will no longer be supported starting HAProxy 3.3. To replace them, we suggest using process managers such as Systemd, SysVinit, Supervisord, or Docker s6-overlays. The program section will also behave differently in HAProxy 3.1. During a reload of HAProxy, the master load balancer process will start a configured program, but a worker process will execute the rest of the program instead. A program can execute even if the worker process has a faulty configuration at reload.

The configuration options accept-invalid-http-request and accept-invalid-http-response are deprecated. Instead, use accept-unsafe-violations-in-http-request and accept-unsafe-violations-in-http-response. The accept-unsafe-violations-in-http-request will enable or disable relaxing of HTTP request parsing, while accept-unsafe-violations-in-http-response will enable or disable relaxing of HTTP response parsing. 

Duplicate names in various families of proxies—for example frontend, listen, backend, defaults, and log-forward sections—and between servers are detected and reported with a deprecation warning, specifying that the duplicate names will not be supported in HAProxy 3.3. Update your configurations as the deprecation warnings appear before upgrading to HAProxy 3.3. Addressing these deprecation warnings will result in faster configuration parsing times, better visibility in logs since there are no duplicate names, and a reliable configuration at the end of the day.

The legacy C-based mailers are deprecated and will be removed in HAProxy 3.3. Set up mailers using Lua mailers instead.

Breaking changes

Visit /haproxy/wiki/wiki/Breaking-changes to see the latest on upcoming breaking changes in HAProxy and which releases they are planned for. The breaking changes here aids users upgrading from older versions of HAProxy to newer versions.

Conclusion

HAProxy 3.1 was made possible through the work of contributors who pour immense effort into open-source projects like this one. This work includes participating in discussions, bug reporting, testing, documenting, providing help, writing code, reviewing code, and hosting packages.

While it's impossible to include every contributor's name here, you are all invaluable members of the HAProxy community. Thank you for contributing!

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.