Companies are finding that AI-based applications, just like conventional applications, have deliverability concerns. For most conventional applications, an API gateway is a vital tool for deliverability. Something very similar is needed for AI applications, but some of the specifics are different. So, a new form of API gateway, called an AI gateway, is coming to the fore. HAProxy is one of the companies pioneering the development and delivery of this new type of gateway. 

What’s new with AI gateways?

One key difference between a conventional API gateway and an AI gateway is in the area of rate limiting. API gateways implement rate limiting for requests based on the number of requests per IP address, which contributes to balanced service delivery and is an initial step in limiting the impact of malfunctioning or malicious software.

While AI gateways also require rate limiting, the limits imposed should be based on the API key and the number of tokens used rather than on the requests per IP address. This actually provides a higher level of control than is possible with conventional rate limiting, since the API key and token counts are specific to a given machine, whereas IP addresses don’t always represent a single machine. 

Other needs such as data loss prevention, API key management, retry support, and caching are more or less the same. However, the way these similar requirements are implemented introduces some differences, which we'll discuss later.

Implementing an AI gateway in HAProxy Enterprise

In part 1 of this guide, we'll implement the basics:

• Creating a gateway in front of OpenAI and providing rate limiting per API key

In part 2 (coming later), we'll tackle more advanced steps:

• Enhancing the configuration for a gateway using multiple API keys, in front of vLLM

• Adding API key encryption

• Implementing security for personally identifiable information (PII) protection and data quality at the gateway level

To make all this happen, we'll use HAProxy Enterprise with HAProxy Fusion Control Plane (included with HAProxy Enterprise). Together, these products make it easy to implement an AI gateway that's performant and scalable. Let's first review a few details to show you how everything will fit together.

Challenges facing AI applications

We won’t have a full picture of the unique challenges AI applications face until the industry has more experience creating and delivering them at scale. However, at this early stage, some important concerns have already arisen:

• Cost control is paramount – Developers usually aren’t (and maybe shouldn’t be) aware of the full expense and large per-token costs of running an LLM model.

• API keys get compromised – LLM platforms such as OpenAI let you create different API keys per developer—a basic security measure. These keys, like any other API keys, can still be compromised or stolen. However, the urgency to enforce sophisticated key management and protection isn't quite keeping pace with today's usage trends. HAProxy can help you to bridge the gap. 

• API key quotas get used up – Some LLM platforms let you set rate limits for tokens. However, these limits are set globally and not per API key. Wherever developer-specific APIs are available, a single developer can use up most or all of your daily token quota. 

• Security and PII concerns must be addressed – Users' prompts must not include PII, such as social security numbers or credit card information. 

You can begin to address these concerns effectively using the AI gateway that we'll show you how to create here, improving your delivery of LLM-powered AI applications. 

New to HAProxy Enterprise?

HAProxy is the world’s fastest and most widely used software load balancer and the G2 category leader in API management, container networking, DDoS protection, web application firewall (WAF), and load balancing. HAProxy Enterprise elevates the experience with authoritative support, robust multi-layered security, and centralized management, monitoring, and automation with HAProxy Fusion. HAProxy Enterprise and HAProxy Fusion provide a secure application delivery platform for modern enterprises and applications.

To learn more, contact our sales team for a demonstration or request a free trial.

Key concepts before getting started

Storing and encrypting API keys

Storing unencrypted API keys on a load balancer is never a good idea, even though you can use them as keys for rate limits and more.

Since we're just implementing the basics needed for an AI gateway in this guide, we'll simply accept a key for OpenAI, hash it, and use the result as a key for rate limiting.

In part 2, we'll demonstrate how to encrypt your keys and create an intermediate key, so even your application developers can't access real API key values—a production-ready approach.

Quotas and global rate limiting

HAProxy includes stick tables that can be used as counters for rate limiting. Our use case requires a scalable active/active AI gateway, so we need all HAProxy instances to aggregate—and be aware of—each other's rates.

Our Global Profiling Engine (GPE), which comes pre-installed and integrated with HAProxy Fusion, provides exactly that. This feature automatically aggregates all token rates across every HAProxy Enterprise instance. GPE will later include ranking capabilities out of the box, enabling it to determine the most-used keys within your organization, as well as a convenient web endpoint for integration with any number of other systems. GPE is also available as a standalone module. Future HAProxy Fusion Control Plane releases will also provide similar metrics within customizable dashboards, enhancing observability for your AI-powered applications.

Uniquely, GPE can be configured to aggregate historical rates. Static token limits are really hard to define correctly in unpredictable environments, so we'll dynamically rate limit users based on their usage. In our example, we will impose rate limits when current usage exceeds twice the 90th percentile of the general usage during the same time period on the previous day. You can implement different and more sophisticated controls in your own AI gateway implementation.

Metrics and statistics

We also want to use HAProxy's extensive logging capabilities to collect API usage metrics. Specifically, we'll log the total amount of prompt tokens and completion tokens consumed by all users. 

We have two flexible collection options, which can be used separately or jointly:

• Logging the statistics into the standard HAProxy log, then parsing the logs

• Asynchronous, real-time funneling of token metrics into an external endpoint such as Grafana, an HTTP endpoint for TimescaleDB, or others (not covered in part 1)

PII protection

We want to detect social security numbers, credit card numbers, and other types of potentially sensitive data.

How to implement HAProxy Enterprise as an AI gateway

This guide describes how to use HAProxy Enterprise, which has built-in API gateway functionality, as an AI gateway instead. The concept is the same, and we'll explore tokens, API key-based limiting, and other differences as they arise. 

Step 1: API key implementation

Let’s look at implementing the API key authorization. In this first version of our AI gateway, your users will continue using standard OpenAI keys, but we'll implement additional controls on top of those keys. These include the following:

• Denylists to outright block any compromised keys

• A quota or a rate limit per key

We never recommend storing unencrypted OpenAI keys (or keys of any kind) on your load balancer, so we'll use hashing. We will only store the hashes on each request. HAProxy will receive the OpenAI key, hash it using SHA-256, and compare the hash to the stored data.

Hashing API keys

Let’s say your requests contain an HTTP Authorization header with the OpenAI key. We can get the value of the key, apply a SHA-2 256 digest, then return the value to a variable inside HAProxy:

Let’s say you have a client with an OpenAI key (hashed as 5fd924625a10e0baacdb8) that’s been compromised and must be blocked at the load balancer layer. We're going to create a .map file called denylist.acl with a single compromised API key per line:

For security reasons, HAProxy Enterprise is designed not to use any I/O once loaded. HAProxy Fusion also ensures that HAProxy Enterprise always has an up to date denylist.acl file in memory. When needed, it’s easy to write a script to make changes to this file using HAProxy Fusion’s native API.

You can now block any key on denylist.acl as follows in your HAProxy configuration:

Step 2: Quotas and rate limits

Next, we want to implement quotas or rate limits per key. Enterprise-grade quotas require the following properties:

• Quotas or rate limits should be persisted across all load balancer instances, whether they are configured as Active/Active or Active/Passive.

• Quotas or rate limits should be flexible and configurable to any specification. In this example, we'll use per-minute limits and daily limits for both prompt tokens and completion tokens. This matches OpenAI's implementation while also letting you control usage per individual API key, instead of only at the account level. 

We'll start with a .map file to define our limits. I’ve chosen the following format:

We absolutely want to support quotas or rate limits across all load balancers in an Active/Active configuration. By default, HAProxy Enterprise's stick tables are used for rate limiting per instance—each instance of HAProxy can configure a local stick table. To comply with the above limits (for example 100 prompt tokens per minute), you’d need to set the actual limit to be 100 divided by the number of HAProxy instances in your cluster. However, you can't easily autoscale, and you would need to constantly update calculations. Mathematical errors and other issues would be hard to detect.

It's Global Profiling Engine (GPE) to the rescue! In an Active/Active load balancer configuration, where traffic is spread across two or more load balancers in a round-robin rotation, GPE ensures that each load balancer receives a client's aggregated requests. This is true even when those requests were routed to another load balancer in the cluster.

I'm going to perform a director's cut here: if you're using HAProxy Fusion, you're already using GPE by default. It's automatically configured for each cluster of load balancers you create. If you aren't using HAProxy Fusion, you can easily install the GPE module by following the instructions in our documentation. 

Defining stick tables

Let’s focus solely on the configuration and define HAProxy stick tables to hold our prompt rates. We'll need four tables with two variants—or eight total tables. The variants will be local and aggregate. It's important that on the HAProxy instances we'll write (or track) our data into the local tables, yet read from the aggregate tables. The Global Profiling Engine supplies these aggregate tables, which contain all aggregated rates across all instances.

To get started, we'll need two of each of the following:

• Per-minute prompt tokens tables

• Per-day prompt tokens tables

• Per-minute completion tokens tables

• Per-day completion tokens tables

Here's a sample definition of all of them:

Two differences exist between our per-minute and per-day tables:

• The per-minute table records store rates per minute and expires every 60 seconds.

• The per-day tables store daily rates and expire every 24 hours.

Fetching rate limits from the map file

Next, we'll fetch the rate limits for each OpenAI key by looking up each hash in the .map file, and requesting the fields that we can use to set the maxrate variables.

The field(1,:) variable will return the first number from the .map file for the key, delimited by a colon:

Tracking current requests

Once we've fetched our rate limits from the .map file we must carefully track the current requests using a sticky counter:

Next, request the current rates from the associated aggregated tables:

Calculating if over the limit

Finally, let's determine if the current request exceeds the rate limit we've set. We're effectively making this comparison in pseudo-code:

1. To initially deny the request if the customer exceeds their limit

2. To add the amount of tokens returned into the stick table after getting a response

Here's how we'll deny requests if they exceed the token limit, which subsequently triggers the 429 Too Many Requests error code:

You could run a tokenizer on each request to improve consistency, but doing so would be slow. Eventually, consistent Active/Active rate limits will work the best.

Getting token usage from JSON responses

We're finally nearing the finish line! The OpenAI HTTP response will contain information about per-request token consumption for prompt and completion. We can use HAProxy’s JSON parser to get the details:

In this blog, we've introduced the concept of an AI gateway (quite similar to widely used API gateways) while implementing API key checking and token-based rate limiting. This element is unique to the AI gateway concept, where we care much more about rate limiting based on tokens as opposed to requests. This includes both incoming (prompt) and outgoing (completion) tokens.

In part 2 (coming soon), we'll implement an AI gateway in front of vLLM and/or Ray, while enforcing PII protection and more. Subscribe to our blog to make sure you stay updated!

This complexity is particularly evident when Kubernetes, a system designed to automate the deployment, scaling, and management of containerized applications, is integrated into AWS environments. Despite the fact that its intended effect is to simplify networking, Kubernetes often adds layers of complexity to AWS load balancing, especially in multi-cluster, multi-region setups. Businesses find themselves facing the daunting task of managing application and load balancer sprawl, a challenge HAProxy is uniquely equipped to address.

HAProxy is sometimes referred to as a load balancer, but its capabilities go well beyond that. It serves as a solution for many of the challenges faced when using modern web infrastructure, in areas that include security and support for agile infrastructure. 

This blog post describes how HAProxy supports scalability and security along with load balancing. This blog post summarizes Jakub Suchy’s talk on "Scalable load balancing and security on AWS with HAProxy Fusion" from AWS re:Invent 2023, demonstrating in detail how HAProxy is revolutionizing AWS load balancing.

AWS load balancing in complex environments

Initially, deploying an application on AWS seems straightforward, but as businesses scale, complexity escalates. 

Managing multiple applications across various AWS regions and legacy data centers introduces a new layer of complexity, necessitating advanced setups, enhanced security, and seamless cross-region operability.

The challenge of Kubernetes in AWS load balancing

Kubernetes is often introduced into an already complex environment to help with scaling. While serving this purpose, it can inadvertently add complexity to AWS load balancing across multi-cluster and multi-region setups. 

Many users aim to have a Kubernetes cluster in every AWS region, so as to seamlessly move traffic between them during failures or upgrades. This leads to a profusion of load balancer instances. What starts as a manageable setup can quickly grow into thousands of load balancer, application, and database instances, significantly complicating the request path.

HAProxy solves this problem by streamlining the request path through applications and load balancers. Our solutions adapt to the complexities of Kubernetes environments, making it easier to handle the challenges of large-scale load balancing in AWS.

Simplification and scaling

HAProxy stands out for its dual capability to simultaneously scale network infrastructure and simplify it.  

We achieve simplicity by consolidating the request path across multiple supportive capabilities, such as a WAF, rate limiting, and access control list (ACL) functionality, into a single layer led by load balancing capability. This approach significantly reduces complexity, merging various load balancing-related capabilities into one cohesive HAProxy Enterprise layer.

This simplification is complemented by HAProxy's scalability, which efficiently handles traffic spikes and traffic growth while providing features such as EC2 auto-scaling. This blend of simplification and scalability makes HAProxy a powerful tool for handling the demands of complex network infrastructures. 

Multi-layered security with HAProxy Enterprise

Integrating multiple functionalities into a single layer doesn't just simplify operations, it also enhances security. In environments where web traffic demands are high, HAProxy Enterprise stands out for its robust security measures. 

The simple act of consolidating functions enhances security. There are fewer vendors to deal with, fewer interfaces between different services, and less traffic on the network. 

HAProxy Enterprise employs advanced techniques such as IP anycast and Route 53 DNS for smart traffic routing, while also offering strong DDoS protection and rate limiting crucial for maintaining service availability and performance under attack - all within the load balancer. 

The built-in next-generation HAProxy Enterprise WAF provides ultra-low latency protection against application attacks, with exceptional balanced accuracy that virtually eliminates the security impact of false negatives and the noise of false positives. At the same time, HAProxy Enterprise’s fingerprinting and bot management capabilities help identify and mitigate sophisticated attacks. Rate limiting and IP-based access control further bolster security, managing traffic flow and ensuring that only authorized access to web resources is allowed. 

This comprehensive suite of features ensures protection in demanding web traffic environments, all of which contribute to reduced latency and faster processing speeds.

Flexible, customizable, and extensible

HAProxy is not a fixed solution; it can be fine-tuned to fit the demands of any network environment.

Use case: rate limiting across VPC regions using VPC ID identification

Many AWS users face challenges when managing web traffic across different virtual private clouds (VPCs) in various regions. VPCs with overlapping IP addresses make it difficult to identify the origin of incoming traffic. Traditional load balancing approaches, which typically rely on the source IP address for rate limiting, are inadequate. 

HAProxy Enterprise provides a solution, using the VPC ID to identify each request. This is made possible by leveraging a feature from AWS's Network Load Balancer (NLB) that transmits the VPC ID alongside the traffic, utilizing the PROXY protocol. By rate limiting based on the VPC ID, HAProxy Enterprise achieves a more accurate and efficient traffic management system, overcoming the challenges posed by overlapping IP addresses. (The basics of rate limiting on AWS are described in our blog post.) 

HAProxy's architecture allows for high customizability and flexibility, making it adaptable to a wide range of network environments. Unlike rigid, one-size-fits-all solutions, HAProxy can be precisely tailored to meet the specific demands and challenges of any setup, ensuring optimal performance and security tailored to each unique scenario.

Centralized management with HAProxy Fusion Control Plane

The HAProxy Fusion Control Plane provides centralized management, monitoring, and automation for multiple clusters of HAProxy Enterprise instances in a distributed load balancer layer. It simplifies the task of overseeing numerous load balancers spread across different regions by enabling auto-scaling and facilitating automatic configuration inheritance. 

This centralization reduces the burden of management, promotes consistency, and enhances the efficiency of traffic handling across the network.

Promoting infrastructure flexibility and observability

HAProxy strongly advocates the concept of infrastructure immutability, emphasizing the idea of replacing components rather than upgrading or repairing them. This approach, similar to modern practices observed in Kubernetes environments, ensures that the infrastructure remains agile and capable of quick adaptation to changing needs without being bogged down by legacy issues.

In addition to promoting flexibility, HAProxy plays a crucial role in enhancing the observability of complex systems. It aids in the easier tracking of traffic flows and provides insights into system performance in real-time. This level of observability is essential for maintaining an efficient and responsive infrastructure, allowing for immediate identification and resolution of issues in real-time.

HAProxy is a game-changer in AWS load balancing

HAProxy goes beyond being just a load-balancing tool. It serves as a complete solution to the complex challenges faced by modern web infrastructures. By effectively managing traffic at scale, ensuring robust security, simplifying traffic flows, and fostering agile infrastructure, HAProxy Enterprise and HAProxy Fusion present a game-changing option for AWS users.

Watch Jakub’s talk on "Scalable load balancing and security on AWS with HAProxy Fusion" from AWS re:Invent 2023 to dive deeper into how HAProxy is revolutionizing AWS load balancing.

A vulnerability in sshd’s SIGALRM handler permits unauthenticated remote code execution as root. This allows remote attackers to cause a denial of service (DoS), and possibly execute arbitrary code.

If you are using an affected product, you should upgrade to the fixed version as soon as possible. There is no workaround available.

Affected Versions & Remediation

HAProxy Technologies released new versions of HAProxy Enterprise and HAProxy ALOHA on Thursday, 4 July 2024. These releases patch the vulnerability described in CVE-2024-6387 (CVSSv3 score of 8.1). 

Users of the affected products should upgrade to the fixed version as soon as possible by following the instructions below.

• Update HAProxy Enterprise public/private cloud images using your Linux distribution’s regular package management operation, for example by using apt or yum

• Update HAProxy ALOHA

Amazon AMIs and Azure VHDs are available.

Support

If you are an HAProxy Enterprise or HAProxy ALOHA customer and have questions about upgrading to the latest version, please get in touch with the HAProxy support team.

CVE-2024-24791 exposes a denial of service (DoS) vulnerability in Go's net/http client. The client misinterprets a server's "Expect: 100-continue" header with a non-informational response (like a 200 OK). This leaves the connection unusable, causing subsequent requests to fail. Attackers can exploit this by sending "Expect: 100-continue" requests to overwhelm the proxy with unusable connections.

If you are using an affected product, you should upgrade to the fixed version as soon as possible. There is no workaround available.

Affected Versions & Remediation

HAProxy Technologies released new versions of HAProxy Fusion, HAProxy Enterprise Verify Crawler Module, HAProxy ALOHA, HAProxy Kubernetes Ingress Controller, HAProxy Enterprise Kubernetes Ingress Controller, Data Plane API, and Data Plane API Enterprise on Thursday, 4 July 2024. These releases patch the vulnerability described in CVE-2024-24791 (CVSSv3 score of 7.5). 

Users of the affected products should upgrade to the fixed version as soon as possible by following the instructions below.

• Update HAProxy Fusion

• Update HAProxy Enterprise to patch the Verify Crawler Module

  • Update HAProxy Enterprise as a Docker container

• Update HAProxy ALOHA Management Package

• Update HAProxy Kubernetes Ingress Controller

• Update HAProxy Enterprise Kubernetes Ingress Controller

• Update Data Plane API by installing the fixed version from the GitHub repository

• Update Data Plane API Enterprise

Support

If you are a customer and have questions about upgrading to the latest version, please get in touch with the HAProxy support team.

All these improvements (and more) will be incorporated into HAProxy Enterprise 3.0, releasing later this year.

Simplicity

New crt-store configuration section

The new crt-store configuration section provides a flexible way to store and consume SSL certificates. Replacing crt-list, crt-store separates certificate storage from their use in a frontend, and provides better visibility for certificate information by moving it from external files and placing it into the main HAProxy configuration. The crt-store section allows you to individually specify the locations of each certificate component, for example certificates files, key files, and OCSP response files. Aliases provide support for human-friendly names for referencing the certificates more easily on bind lines. The ocsp-update argument is now configured in a crt-store instead of a crt-list.

Consider the following example where a frontend references a crt-list for its certificate information, including OCSP response files. This is how you would use ocsp-update prior to crt-stores:

Note that defining the certificate information in this way makes the information less visible, as it is not defined within the HAProxy configuration file.

To remedy this, use a crt-store section. To the HAProxy configuration file, add the following:

Now, you can reference these certificate elements in a frontend by their alias:

You can also use crt-store to specify keys separately from their certificates, as is the case in this example:

Security

Some HTTP/2 requests are valid from a protocol perspective but pose problems anyway. For example, sending a single header as a large number of CONTINUATION frames could cause a denial of service. HAProxy now counts these so-called glitches and allows you to set a limit on them. You can also track them in a stick table to identify buggy applications or misbehaving clients.

OpenSSL security level

A new global keyword ssl-security-level allows you to set globally, that is, on every HAProxy SSL context, the OpenSSL’s internal security level. This enforces the appropriate checks and restrictions per the level specified. Parameters specified, including cipher suite encryption algorithms, supported ECC curves, supported signature algorithms, DH parameter sizes, certificate key sizes, and signature algorithms, inconsistent with the level will be rejected. For more information see: OpenSSL Set Security Level.

Specify a value between 0 and 5 to set the level, where 5 is the most strict:

HAProxy’s security has been hardened by the removal of the dependency on the libsystemd library. This libsystemd library has many additional dependencies of its own, including the library at fault for the XZ Utils backdoor vulnerability. Removing the dependency on libsystemd means that HAProxy is not exposed to these undesired libraries.

Prevent HAProxy from accepting traffic on privileged ports

Two new global settings, harden.reject-privileged-ports.tcp for TCP connections and harden.reject-privileged-ports.quic for QUIC connections, enable HAProxy to ignore traffic when the client uses a privileged port (0 - 1023) as their source port. Clients using this range of ports are suspicious and such behavior often indicates spoofing, such as to launch DNS/NTP amplification attacks. The benefit in using these settings is that during DNS/NTP amplification attacks, CPU is reserved as HAProxy will drop the packets on these privileged ports instead of parsing them.

Reliability

Persist stats

Reloading HAProxy will no longer reset the HAProxy Stats page, as long as you call the new Runtime API command dump stats-file first to save the current state to a file and then load that file with the stats-file global configuration directive. Upon reload, the new processes set their counters to the previous values (as they appear in the stats file). Only proxy counters are supported which includes values for frontends, backends, servers, and listeners. Ensure that you've set a GUID on each frontend, backend, listen, and server object by using the new guid keyword. Only objects to which you have assigned a GUID will have their stats persisted across the reload.

To enable this behavior, follow these five steps:

Step 1

Assign GUIDs to each frontend, backend, listen, and server object in your configuration. Here is a simple configuration with one frontend that routes to one backend with one server. The frontend, backend, and server will each have a globally unique identifier assigned:

Reload HAProxy to apply your configuration changes:

Call dump stats-file to create the stats file, redirecting the output to your desired location:

Step 4

Add stats-file to your global settings with the path to your stats file location that will be reloaded:

Reload HAProxy. It will reload the stats counters from the file.

You can also generate GUIDs for items on bind lines. The guid-prefix should be used in this case to specify a string prefix for automatically generated GUIDs. Each listener on the bind line will have a GUID automatically generated for it with the prefix included. Here's an example:

You can limit the number of total streams processed per incoming connection for HTTP/2 using the global keyword tune.h2.fe-max-total-streams. Once this limit is reached, HAProxy will send a graceful GOAWAY frame informing the client that it will close the connection after all pending streams have been closed. Usually, this prompts clients to reestablish their connections. This is helpful in situations where there is an imbalance in the load balancing due to clients maintaining long-lived connections.

Diagnostic mode

Diagnostic mode (-dD) will now report on likely problematic ACL pattern values that look like known ACL/sample fetch keywords in addition to its other warnings about suspicious configuration statements. It does not prevent startup. This is helpful for troubleshooting problematic configurations.

Consistent hash server mapping

When load balancing using a hash-based algorithm (balance hash), HAProxy must keep track of which server is which. Instead of using numeric IDs to compute hash keys for the servers in a backend, the hash-key directive now supports using the servers’ addresses and ports to compute the hash keys. This is useful in cases where multiple HAProxy processes are balancing traffic to the same set of servers, as each independent HAProxy process will calculate the same hash key and therefore agree on routing decisions, even if its list of servers is in a different order.

The new hash-key directive allows you to specify how the node keys, that is, the hash for each server, are calculated. This applies to node keys where you have set hash-type to consistent. There are three options for hash-key:

• id: the server’s numeric id if set, or its position in the server list

• addr: the server’s IP address

• addr-port: the server’s IP address and port

If you were to use hash-key id, and have no IDs explicitly set for your servers, the hashes could be inconsistent across load balancers if your server list were in different orders on each server. Using addr or addr-port solves this problem.

Consider the following backend:

If multiple instances of HAProxy are running on the same host, for example in Docker containers, using both the IP address and the port (hash-key addr-port) is appropriate for creating the hash key.

Memory limit

To limit the total allocatable memory to a specified number of megabytes across all processes, use -m <limit>. This is useful in situations where resources are constrained, but note that specifying a limit this way may cause connection refusals and slowdowns depending on memory usage requirements. The limiting mechanism now uses RLMIT_DATA instead of RLIMIT_AS, which prevents an issue where the processes were reaching out-of-memory conditions below the configured limit. In master-worker mode, the memory limit is applied separately to the master and its forked worker process, as memory is not shared between processes.

Unix sockets and namespaces

Version 3.0 supports the namespace keyword for UNIX sockets specified on bind and server lines. If a permissions issue arises, HAProxy will log a permissions error instead of “failed to create socket”, since the capability cap_sys_admin is required for using namespaces for sockets.

For example, to specify a UNIX socket with a namespace, use the namespace keyword on your server line:

With the gRPC protocol, the client can cancel communication with the server, which should be conveyed to the server so that it can clean up resources and perhaps invoke cancellations of its own to upstream gRPC services. Cancellations are useful for a few reasons. Often, gRPC clients configure deadlines so that a call will be canceled if it runs too long. Or a client might invoke a cancellation if it finishes early and no longer needs the stream. Read more about cancellation in the gRPC documentation.

Prior to HAProxy 3.0, cancellations, which are represented as RST_STREAM frames in HTTP/2 or as STOP_SENDING frames in HTTP/3, were not being properly relayed to the server. This has been fixed. Furthermore, HAProxy 3.0 adds new fetch methods that indicate whether the stream was canceled (aka aborted) and the error code. Below, we include them in a custom log format:

If you want to use deprecated directives you must also use the expose-deprecated-directives global keyword which will silence warnings. This global keyword applies to deprecated directives that do not have alternative solutions.

Emergency memory allocation

The mechanism for emergency memory allocation in low memory conditions has been improved for HTTP/1 and applets. Forward progress in processing is guaranteed as tasks are now queued according to their criticality and which missing buffer they require.

Flexibility

You can now format log lines as JSON and CBOR. When configuring a custom log format, you will indicate which to use, and then in parentheses set the key for each field.

Here is an example for JSON:

In containerized environments, sometimes you don't want to deal with attaching a volume and mapping that volume to the container's filesystem, especially when it comes to storing data that is dynamic. In other words, wouldn't it be nice to remove the need to attach a volume? With HAProxy 3.0, you can now work with virtual ACL and map files. By prefixing your ACL and Map files with virtual@, HAProxy won't search the filesystem for the files but rather creates in-memory representations of them only.

The configuration below sets the stage for adding IP addresses to a denylist, where that denylist is virtual:

Report the key that matched in a map file

A request matched a row in a map file, but which row? This version of HAProxy adds several, new map_*_key converters that return the key that was matched, rather than the associated value, making it easier to view the reason why the load balancer rerouted a request or took some other action.

In the example below, we use a Map file for path-based routing, where the request's URL path determines which backend to send a request to. By using a converter that ends in _key, in this case map_beg_key, which will match the beginning of the key in the file and then return the key, we can record which key in the Map file matches the request:

HAProxy can proxy traffic for multiple websites through the same frontend. To choose the right TLS certificate, it will compare the TLS Server Name Indication (SNI) value of the incoming connection with the certificates it finds in the certificates directory. If no certificate in your directory matches, it will resort to using a default one, which is the certificate that was loaded first (was first in the directory). But what if you wanted to set multiple defaults? For example, suppose you wanted to default to an ECDSA certificate, if supported, otherwise default to an RSA certificate? Now you have multiple ways to do this:

• Ensure that the first file in the directory is a multi-cert bundle.

• Use the new default-crt argument on a bind line.

• If using crt-list, indicate a default certificate by marking it with an asterisk.

To demonstrate setting one or more default-crt arguments on the frontend's bind line, below we set crt to a directory so that HAProxy will select the correct certificate from that directory based on SNI. But if there is no match, it will instead use one of the default files—either ECDSA or RSA:

Until now, you could capture in a stick table the count and rate of client HTTP errors (4xx status codes) and server HTTP errors (5xx status codes), but you could not control specifically which status codes were included. This version adds the global directives http-err-codes and http-fail-codes that let you set the status codes you care about, allowing you to ignore those that don't matter to you. This works for responses from backend servers and for responses from HAProxy.

Tracking client HTTP errors can be useful for discovering misconfigured client applications, such as those that repeatedly use the wrong credentials or that make an unusually high number of requests. Below, we configure a stick table to track only 403 Forbidden errors, but you can also configure HAProxy to track a range of status codes by separating them with commas or indicating a range of codes with a dash:

Stick table pointers

Previously, you could use the Runtime API commands clear table [table] [key] and set table [table] [key] to remove or update a record in a stick table based on its key. You can now pass the record's unique ID (its pointer) to remove it or update it. You get the pointer from the show table [table] command.

In the example below, the pointer is 0x7f7de4bb50d8:

Version 2.9 added the ability to preserve previously set Linux capabilities after the HAProxy process starts up. Linux capabilities are permissions granted to a process that it would not otherwise have when running as a non-root user. These permissions become relevant since HAProxy does not run as a root user after startup. 

In version 3.0, HAProxy is smarter about checking capabilities and will no longer emit error messages regarding the need for root permissions when running in transparent proxying mode or when binding to a port below 1024 (as is the case when using QUIC), so long as cap_net_admin is set. Additionally, file-system capabilities can now be set on the binary and if you start HAProxy as a root user, then adding setcap in the configuration is enough for adding those capabilities to the process. HAProxy will move the capabilities set on the binary (Permitted set) to its Effective set as long as the capabilities are also present in the setcap keyword list.

As a refresher, you can indicate which Linux capabilities to preserve after startup by using the setcap global keyword, specifying capabilities with a comma between each one:

• cap_net_admin: Used for transparent proxying mode and for binding to privileged ports (lower than 1024, for example, for QUIC).

• cap_net_raw (subset of cap_net_admin): Used for setting a source IP address that does not belong to the system itself, as is the case with transparent proxying mode.

• cap_net_bind_service: Used for binding a socket to a specific network interface. This is required when using QUIC and binding to a privileged port. 

• cap_sys_admin: Used for creating a socket in a specific network namespace.

Set fwmark on packets to clients and servers

With HAProxy, you can set the fwmark on an IP packet, which classifies it so that, for example, it can use a specific routing table. HAProxy 3.0 now supports setting an fwmark on connections to backend servers as well as to clients connected on the frontend. Use the set-fc-mark and set-bc-mark actions. These replace the set-mark action, which had applied only to frontends.

To test this, first, give HAProxy the cap_net_admin capability, which is required for setting marks, by adding the setcap directive to the global section of your configuration:

HAProxy can modify the header of an IP packet to include the Differentiated Services (DS) field. This field classifies the packet so that the network stack can prioritize it higher or lower in relation to other traffic on the network. New in this version of HAProxy, you can set this field on connections to backend servers in addition to frontend connections to clients. To set the value, use the set-fc-tos and set-bc-tos actions (referring to the old Type of Service (TOS) header field, which has been superseded by DS).

First, give the HAProxy binary the cap_net_admin capability, which is required for setting network priority:

New fetch methods introduced in this version expose data that you can use in ACL expressions. They include fetches that return the number of open HTTP streams for a backend or frontend, the size of the backend queue, the allowed number of streams, and a value that indicates whether a connection was redispatched because a server became unreachable. 

• bc_be_queue – The number of streams de-queued while waiting for a connection slot on the target backend

• bc_glitches – The number of protocol glitches counted on the backend connection

• bc_nb_streams – The number of streams opened on the backend connection

• bc_srv_queue – The number of streams de-queued while waiting for a connection slot on the target server

• bc_settings_streams_limit – The maximum number of streams allowed on the backend connection

• fc_glitches – The number of protocol glitches counted on the frontend connection

• fc_nb_streams – The number of streams opened on the frontend connection

• fc_settings_streams_limit – The maximum number of streams allowed on the frontend connection

• txn.redispatched – True if the connection has experienced redispatch

Weights in log backends

Log backends were introduced in version 2.9. They allow you to set mode log in a backend to load balance the Syslog protocol. You can connect to backend Syslog servers over TCP or UDP.

In version 3.0, you can now set weights on server lines in your mode log backends. The example below demonstrates a log backend that uses weights:

Support for UUIDv7 identifiers

You can now generate universally unique identifiers that use the UUIDv7 format. UUIDs of this type factor in the current UNIX timestamp and are therefore time sortable, which tells you when a UUID was generated in relation to other UUIDs. In the example below, we set the unique ID format to version 7 and use the unique-id fetch method to get a new UUID to include in the logs:

HAProxy 2.8 introduced a new and simpler way to configure OCSP stapling. HAProxy periodically contacts your SSL certificate issuer's OCSP server to get the revocation status of your SSL certificate.  In version 3.0, if you're operating in an air-gapped environment where the HAProxy server does not have direct access to the Internet, and therefore can't connect to the OCSP server, you can set an HTTP forward proxy to reach the Internet. Add the ocsp-update.httpproxy global directive to indicate the proxy's address:

Reverse HTTP

Reverse HTTP, which is an experimental feature added in HAProxy 2.9, allows HAProxy load balancers to self-register with an edge HAProxy load balancer and to then fill in as backend servers. In version 2.9, the mechanism for matching client requests with the correct backend server / HAProxy instance was via SNI. A new directive named pool-conn-name provides more flexibility, allowing you to set the name to match with an expression of fetch methods and converters.

Observability

When configuring the Prometheus exporter, you can now include a URL parameter named extra-counters, which enables additional counters that provide information related to the HTTP protocol, QUIC, and SSL. Set this in your Prometheus server's prometheus.yml file, replacing haproxy with your load balancer's IP address:

When diagnosing network issues, you will often need to analyze TLS-encrypted traffic to see the underlying application-layer protocol messages. For example, when using Wireshark, you can import a packet capture file that contains the traffic you would like to analyze, but then you need a way to decipher the captured packets. The most common way to do this is to import a key log file into Wireshark, which contains the secrets needed to decipher a TLS session.

While prior versions of HAProxy supported producing a key log file for deciphering traffic between the client and HAProxy, HAProxy 3.0 adds the ability to produce a key log file for deciphering TLS traffic between HAProxy and backend servers, specifically when using TLS 1.3. 

Follow these seven steps:

Step 1

In your HAProxy configuration, set tune.ssl.keylog to on in the global section. This activates the retrieval of the TLS keys you will use for decryption in Wireshark:

Force HAProxy and the backend servers to use TLS 1.3 by adding the ssl-min-ver argument to the servers:

Define a custom log format that writes TLS session secrets to the access log:

After HAProxy connects to a backend server, the access log will contain the keys for that TLS session. The access log will contain lines like this:

Copy these lines to a text file. Then import the file into Wireshark via Edit > Preferences > Protocols > TLS > (Pre)-Master-Secret log filename.

Step 6

At the same time, capture traffic between HAProxy and the backend servers. For example, you can run tcpdump on the HAProxy server to get a PCAP file:

Open this PCAP file in Wireshark to see the deciphered traffic.

Runtime API

show quic verbosity

The Runtime API's show quic command gained finer-grained levels of verbosity and the ability to filter the output to a specific connection. Previously, the verbosity levels you could specify were online or full. Now you can also specify a comma-delimited list of values that determine the output. Values include tp, sock, pktns, cc, and mux. Also, in this version you can specify the connection's hexadecimal address to see information for just that connection.

Here are examples that show output for the new verbosity levels:

Static cookies for session persistence are now supported for dynamically added servers. Dynamic servers refer to servers that do not have an explicit entry within your HAProxy configuration file. They are dynamic in the sense that you can add them programmatically using Runtime API commands. Dynamic servers are valuable in cases where you may have many servers that scale with traffic: when traffic loads are high, you add more servers, and when traffic loads diminish, you remove servers.

Previous versions of HAProxy did not support adding these dynamic servers and also using static cookies with those servers, but as of version 3.0, you can now use the add server command to add the server and specify its static cookie using just one command. Note that when adding a dynamic server, you must choose a load balancing algorithm for your backend that is dynamic (roundrobin, leastconn, or random).

To add a dynamic server to your backend with a static cookie, issue the add server command, specifying your cookie:

As a reminder, servers added via the Runtime API are not persisted after a reload. To ensure that servers you add via the Runtime API persists after a reload, be sure to also add them into your HAProxy configuration file (thus making them static servers).

Del server

Removing a dynamic server with the del server command is faster now that the command can close idle connections, where previously it would wait for the connections to close by themselves. This improvement is made possible by changes to the removal mechanism which allows forceful closure of idle connections.

Wait command

The new wait command for the Runtime API will wait for some specified time before then performing the following command.You could use this to collect metrics on a certain interval. This is also useful in cases where you need to wait until a server becomes removable (the server has been drained of  connections) before running additional commands, such as del server.

The syntax for the command is: wait { -h | <delay> } [<condition> [<args>...]]

If you do not provide any conditions, the command will simply wait for the requested delay (in default milliseconds) time before it continues processing. 

With this release, the only supported condition is srv-removable which will wait until the specified server is removable. When using socat, be sure to extend socat’s timeout to account for the wait time. 

The following example calls show activity, waits 10 seconds, then calls show activity again. Note that socat’s timeout value has been increased to 20 seconds:

Zero-copying forwarding was introduced in Version 2.9 for TCP, HTTP/1, HTTP/2, and HTTP/3. As of version 3.0, applets, such as the cache, can also take advantage of the fast forwarding mechanism which avoids queuing more data when the mux buffer is full. This results in significantly less memory usage and higher performance. This behavior can be disabled by using tune.applet.zero-copy-forwarding for applets only, or tune.disable.zero-copy-forwarding globally.

In regards to QUIC, simplification of the internal send API resulted in removal of one buffer copy. The fast forwarding now considers the flow control, which reduces the number of thread wakeups and optimizes packet filling.

The HTTP/1 mutex now also supports zero-copy forwarding for chunks of unknown size. For example, chunks whose size may be larger than the buffer.

Ebtree update

Performance improved for ebtree on non-x86 machines. This results in approximately 3% faster task switching and approximately 2% faster string lookups.

Server name lookup

Configuration parsing time will see an improvement thanks to a change in server name lookups. The lookups now use a tree, which improves lookup time, whereas before the lookup was a linear operation.

QUIC

QUIC users will see a performance increase when using two new global settings:

• tune.quic.reorder-ratio

  • By adjusting tune.quic.reorder-ratio, you can change how quickly HAProxy detects packet losses. This setting applies to outgoing packets. When HAProxy receives an acknowledgement (ACK) for a packet it sent to the destination and that ACK arrived before other expected ACKs, or in other words it arrived out of sequence, it could indicate that packets never reached the destination. If it happens frequently, it indicates a poor network condition. By lowering the ratio, you're lowering the number of packets that can be missing ACKs before HAProxy takes action. That action is to reduce the packet sending window, which forces HAProxy to slow down its rate of transfer, at the cost of slower throughput. The default value is 50%, which means that the latest ACKed packet was halfway up the range of sent packets awaiting acknowledgements, with packets preceding it not yet ACKed.

• tune.quic.cc-hystart

  • Use this setting to enable use of the HyStart++ (RFC9406) algorithm instead of the Cubic algorithm. This provides an alternative to the TCP slow start phase of  the congestion control algorithm. This algorithm may show better recovery patterns regarding packet loss.

Additionally, the send path for QUIC was improved by cleanup on some of the low level QUIC sending functions. This includes exclusive use of sendmsg() (a system call for sending messages over sockets), optimizations avoiding unnecessary function calls, and avoiding copies where possible.

Traces

An improvement to traces will enable their use on servers with moderate to high levels of traffic without risk of server impact.

The improvement to traces was made possible by the implementation of near lockless rings. Previously, the locking mechanism limited the possibility of using traces in a production environment. Now that the rings are nearly lockless, allowing for parallel writes per group of threads, performance with traces enabled has been increased up to 20x.

A new global setting tune.ring.queues that sets the number of write queues in front of ring buffers can be used for debugging, as changing the value may reveal important behavior during a debugging session. This should only be changed if you are instructed to do so for troubleshooting.

Stick tables

Stick tables have received a performance boost due to a change in the locking mechanism. Previously, when the number of updates was high, stick tables could cause CPU usage to rise, due to the overhead associated with locking. Using peers amplified this issue. Stick tables are now sharded over multiple, smaller tables, each having their own lock, thus reducing lock contention. Also, the interlocking between peers and tables has been significantly reduced.

This means that on systems with many threads, stick table performance improves greatly. On a system with 80 threads, we measured performance gains of approximately 6x. As for systems with low thread counts, performance could be improved by as much as 2x when using peers.

Lua

Single-threaded Lua scripts using lua-load will see a performance improvement. This improvement is the result of a change to the loading mechanism, where the maximum number of instructions is now divided by the number of threads. This makes it so that waiting threads have a shorter wait time and share the time slot more evenly. Safeguards are in place to prevent thread contention for threads waiting for the global Lua lock.

Use tune.lua.forced-yield to tune the thread yielding behavior. For scripts that use lua-load, the optimal (and default) value was found to be the maximum of either 500 instructions or 10000 instructions divided by the number of threads. As for scripts loaded using lua-load-per-thread, in cases where more responsiveness is required, the value can be lowered from the default of 10000 instructions. In cases where the results of the Lua scripts are mandatory for processing the data, the value can be increased, but with caution, as an increase could cause thread contention.

Breaking Changes

Multiple CLI commands no longer supported

Previously, it was occasionally possible to successfully issue multiple commands by separating them with newlines, which had the potential to produce unexpected results for long-running commands that may only partially complete. A warning will now be emitted when a \n is detected in a command and the command will not be accepted. This change has also been backported to ensure that user scripts that utilize this behavior can be remedied.

Enabled keyword rejected for dynamic servers

When defining a dynamic server, use of the enabled keyword is now rejected with an error, whereas previously it was only silently ignored. Here's a sample input:

Parsing is now more strict during H1 processing for request target validation. This means that where previously, for compatibility, non-standard-compliant URIs were forwarded as-is for HTTP/1, now some invalid request targets are rejected with a 400 Bad Request error. The following rules now apply:

• The asterisk-form is now only allowed for OPTIONS and OTHER methods. There must now be only one asterisk and nothing more. 

• The CONNECT method must have a valid authority-form. All other forms are rejected. 

• The authority-form is now only supported for the CONNECT method. Origin-form is only checked for the CONNECT method.

• Absolute-form must have a scheme and a valid authority.

Tune.ssl.ocsp-update renamed to oscp-update

The tune.ssl.oscp-update global keyword is now named oscp-update, as ocsp-update is unrelated to SSL tuning.

Development Improvements

This release brings with it some major improvements for developers and contributors, as well as aids in saving time diagnosing issues and speeding up recovery:

• The internal API for applets has been simplified, with new applet code having its own buffers, and keyword handlers for the Runtime API now have their own buffers as well. 

• Updates to the makefile improve ease of use for packagers, including improved warnings plus easier passing of CFLAGS and LDFLAGS. Unused build options will produce a warning which will assist in debugging typos for build options with long names. 

• A new debugging feature has been added to the SSL and HTTP cache that allows assignment of a name to some memory areas so that it is more easily identified in the process map (using /proc/$pid/maps or using pmap on Linux versions 5.17 or greater). This makes it so that you can more easily determine where and why memory is being used. Future iterations will include more places where this debugging feature is implemented, further improving troubleshooting.

• By default HAProxy tries hard to prevent any thread and process creation after it starts. This is particularly important when running HAProxy’s own test suite, when executing Lua files of uncertain origin, and when experimenting with development versions, which may still contain bugs whose exploitability is uncertain. Generally speaking, it's a best practice to make sure that no unexpected background activity can be triggered by traffic. However, this may prevent external checks from working, and it may break some very specific Lua scripts which actively rely on the ability to fork. This global option insecure-fork-wanted disables this protection. As of version 3.0, you can also activate this option by using -dI (-d uppercase “i”) on the HAProxy command line. Note that it is a bad idea to disable it, as a vulnerability in a library or within HAProxy itself may be easier to exploit once disabled. In addition, forking from Lua, or anywhere else, is not reliable, as the forked process could embed a lock set by another thread and cause operations to never cease execution. As such, we recommend that you use this option with extreme caution, and that you move any workload requiring such a fork to a safer solution (such as using agents instead of external checks).  

• The DeviceAtlas module has been updated to support the new version of DeviceAtlas. 

• Support for FreeBSD 14 (and its new sched_setaffinity() system call) has been added.

Conclusion

HAProxy 3.0 was made possible through the work of contributors that pour immense effort into open-source projects like this one. This work includes participating in discussions, bug reporting, testing, documenting, providing help, writing code, reviewing code, and hosting packages.

While it's sadly impossible to include every contributor name here, all of you are invaluable members of the HAProxy community! Thank you.

In this blog post, we'll share a quick note on updated naming conventions before diving deeper into HAProxy Kubernetes Ingress Controller's headlining features.

Version compatibility with HAProxy

HAProxy Kubernetes Ingress Controller 3.0 is built with HAProxy version 3.0 and has now jumped from version 1.11 to version 3.0. Starting with this release, Kubernetes Ingress Controller's version number will match the version of HAProxy it uses. We hope this clarifies the link between HAProxy Kubernetes Ingress Controller and its baseline version of HAProxy, moving forward.

Custom Resource Definitions: TCP

Until now, mapping for TCP services was available through a custom ConfigMap using the --configmap-tcp-services flag. While this worked as expected, there were a few limitations we needed to address. 

For example, ConfigMap alone doesn't have a standardized structure nor validation. Therefore, keeping a larger list of services tidy can be challenging. Additionally, only some HAProxy options (such as service, port, and SSL/TLS offloading) were available for those types of services. 

The tcps.ingress.v1.haproxy.org definition, conversely, lets us define and use more HAProxy options than we could with ConfigMap.

Installing and getting to know TCP CRDs

If you're using Helm, the TCP services definition will be installed automatically. Otherwise, it's available as a raw YAML file via GitHub.

TCP custom resources (CRs) are namespaced and you can deploy several of them in a shared namespace. HAProxy will apply them all.

A TCP CR contains a list of TCP service definitions. Each service definition has:

• A name

• A frontend section containing two permitted components:

  • Any setting from client-native frontend model 

  • A list of binds coupled with any settings from client-native bind models 

• A service definition that's a Kubernetes upstream Service/Port (the K8s Service and the deployed TCP CR must be in the same namespace).

Here's a simple example of a TCP service:

Mitigating TCP collisions

TCP services are tricky since they allow for unwanted naming and configuration duplications. This overlap can cause transmission delays and other performance degradations while impacting reliability. 

Luckily, HAProxy can detect and manage two types of collisions:

• Collisions on frontend names

• Collisions on bind addresses and ports

If several TCP services across all namespaces encounter these collisions, HAProxy will only apply the one that was created first based on the older CreationTimestamp of the custom resource. This will generate a message in the log.

SSL/TLS in a TCP custom resource

Here's a quick example of a TCP service with SSL/TLS enabled:

• The name of a Kubernetes Secret (in the same namespace as the TCP CR) containing the certificate and key

• A folder or filename on the pod's local filesystem, which was mounted as a Secret Volume

For example, you can mount an SSL/TLS Secret in the Ingress Controller Pod on a volume and reference the volume mount path in ssl_certificate. Without changing the Pod (or deployment manifest), you can instead use a Secret name within the ssl_certificate configuration. As a result, the certificate and key will be written in the Pod's filesystem at the etc/haproxy/certs/tcp path.

Breaking changes

If you're using --configmap-tcp-services, this release changes the default backend configuration for a TCP Service defined via annotation in your ConfigMap. Previously, any backend options defined in the ConfigMap (such as maxconn or server-slots) didn't apply to TCP backends. These options now apply to TCP backends defined via annotation in ConfigMap.

Contributions

Conclusion

HAProxy Kubernetes Ingress Controller 3.0 represents our commitment to delivering a flexible and efficient platform for managing ingress traffic. By extending our prior CRD support to include TCP CRDs, our Kubernetes solutions can meet even more use cases with less complexity. 

To learn more about HAProxy Kubernetes Ingress Controller, follow our blog and browse our Ingress Controller documentation. If you want to see how HAProxy Technologies also provides external load balancing and multi-cluster routing alongside our ingress controller, check out our Kubernetes solutions and our K8s webinar.

HAProxy maintains its edge over alternatives with best-in-class load balancing performance and reliability, the flexibility to support a wide variety of workloads, and a programmable and extensible architecture that fits your workflow. 

Today, HAProxy 3.0 has arrived, and HAProxy Enterprise 3.0 will be released later this year! In this blog post, we'll cover the changes in a short and digestible format, leaving the longer-format configuration examples and deep dives for follow-up blog posts.

For a live introduction to the new release, register for our webinar HAProxy 3.0: Feature Roundup. Join our experts as we examine new features and updates and participate in the live Q&A. 

How to get HAProxy 3.0

You can install HAProxy version 3.0 in any of the following ways:

Run it as a Docker container. View the Docker installation instructions.

Compile it from source. View the compilation instructions.

Major changes

First, let's cover the most important changes in HAProxy 3.0. These changes substantially modify how things were done in previous versions or introduce entirely new capabilities.

• Loading TLS certificates with the new crt-store section: The new crt-store configuration section provides a flexible way to store and consume SSL certificates. Replacing crt-list, crt-store separates certificate storage from their use in a frontend. The crt-store section allows you to individually specify the locations of each certificate component, for example, certificates files, key files, and OCSP response files. Aliases provide support for human-friendly names for referencing the certificates more easily on bind lines. The ocsp-update argument is now configured in a crt-store instead of a crt-list.

• Limiting glitchy HTTP/2 connections: Some HTTP/2 requests are valid from a protocol perspective but pose problems anyway. For example, sending a single header as a large number of  CONTINUATION frames could cause a denial of service. HAProxy now counts these so-called glitches and allows you to set a limit on them. You can also track them in a stick table to identify buggy applications or misbehaving clients.

• Assigning GUIDs to configuration objects: The new guid directive available in frontend, backend, and listen sections lets you assign a unique identifier to that section. The server directive also gained a guid argument. For now, the main use is for persisting stats after a reload, since only stats associated with objects having a GUID can be restored.

• Persisting stats after a reload: Reloading HAProxy will no longer reset the HAProxy Stats page, as long as you call the new Runtime API command dump stats-file first to save the current state to a file and then load that file with the stats-file configuration directive. Ensure that you've set a GUID on each frontend, backend, listen and server object by using the new guid keywords.

• Load balancing Syslog: The feature for load balancing Syslog messages, which was introduced in version 2.9, has progressed so that you can now set weights on server lines in your mode log backends. Meanwhile, the sticky algorithm, which had been limited to log backends, now applies to mode tcp and mode http backends as well. 

• Log as JSON and CBOR: You can now format log lines as JSON and CBOR. When configuring a custom log format, you will indicate which to use, and then in parentheses set the key for each field.

• More data exposed as fetch methods: New fetch methods expose data previously available only within logs. They include fetches that return the number of open HTTP streams for a backend or frontend, the size of the backend queue, the allowed number of streams, and a value that indicates whether a connection got redispatched because a server was unreachable. 

Noteworthy changes

Beyond the major changes, there are changes that simplify the configuration, improve performance, or extend existing functionality.

• Improving Lua performance: Single-threaded Lua scripts using lua-load will see a performance improvement. This improvement is the result of a change to the loading mechanism, where the maximum number of instructions is now divided by the number of threads. This makes it so that waiting threads have a shorter wait time and share the time slot more evenly. Safeguards are in place to prevent thread contention for threads waiting for the global Lua lock.

• Improving stick table performance: Stick tables have received a performance boost due to a change in the locking mechanism. Stick tables are now sharded over multiple tree heads, each having their own lock, and thus reducing lock contention. This means that on systems with many threads, stick table performance improves greatly. On a system with 80 threads, we measured performance gains of approximately 6x. As for systems with low thread counts, performance could be improved by as much as 2x when using peers.

• Setting default TLS certificates: When using a solitary frontend to load balance multiple websites, you host different TLS certificates for each site, typically by placing all certificates in a directory and letting HAProxy choose the correct one based on TLS SNI. New in this version, you can use the default-crt argument to indicate which certificate to use when no other certificates match. You can also set different defaults to support RSA and ECC algorithms. In a CRT-List, you can designate a default certificate by adding an asterisk after it.

• Controlling which HTTP errors to track: Until now, you could capture in a stick table the count and rate of client HTTP errors (4xx status codes) and server HTTP errors (5xx status codes), but you could not control specifically which status codes were included. This version adds global directives http-err-codes and http-fail-codes that let you set the status codes you care about, allowing you to ignore those that don't matter to you.

• Prioritizing traffic on the frontend and backend: HAProxy can modify the header of an IP packet to include the Differentiated Services (DS) field. This field classifies the packet so that the network stack can prioritize it higher or lower in relation to other traffic on the network. New in this version of HAProxy, you can set this field on connections to backend servers in addition to frontend connections to clients. To set the value, use the set-fc-tos and set-bc-tos actions (referring to the old Type of Service (TOS) header field, which has been superseded by DS).

• Setting a mark on IP packets on the frontend and backend: With HAProxy, you can set the fwmark on an IP packet, which classifies it so that, for example, it can use a specific routing table. HAProxy 3.0 now supports setting an fwmark on connections to backend servers as well as to clients connected on the frontend. Use the set-fc-mark and set-bc-mark actions.

• Creating UUIDv7 identifiers: The uuid fetch method now takes an optional argument that sets the version of the UUID to either 4 or 7. Combine the fetch with the unique-id-format directive and the unique-id fetch method to get an ID that you can attach to log entries.

• Configuring virtual ACL and Map files: ACL and Map files no longer require you to create files on disk. By prefixing the name of the file with @virt on an acl line in the HAProxy configuration, you allow HAProxy to start up and access the ACL and Map files as virtual representations only. Then use the Runtime API to add and delete rows in the virtual files. This is especially useful in containerized environments where the hassle of defining storage volumes and mapping volumes to the container's filesystem can seem like a burden. You can also prefix the filename with @opt, which marks the file as optional. In that case, HAProxy will check for the file on the filesystem, but if it doesn't find the file, it will assume the file is virtual.

• Relaying to the client or server when a gRPC connection has been aborted: Upon abort by the client, the RST_STREAM reason code can be retrieved from the buffer contents using the fetching sample fs.rst_code. The fetching sample fs.aborted returns true when an abort is received from the client. To detect server aborts, use the corresponding fetching samples bs.rst_code for the return code and bs.aborted for the status.

• A change in how servers are mapped in consistent-hash load balancing: When load balancing using a hash-based algorithm, HAProxy must keep track of which server is which. Instead of using numeric IDs to compute hash keys for the servers in a backend, the hash-key directive now supports using the servers’ addresses and ports to compute the hash keys. This is useful in cases where multiple HAProxy processes are balancing traffic to the same set of servers, as each independent HAProxy process will calculate the same hash key and therefore agree on routing decisions, even if its list of servers is in a different order.

Breaking changes

Although this is a major version release, there are only a few breaking changes, as you'll see in the short list below.

• Detecting accidental multiple commands sent to the Runtime API: Previously, it was occasionally possible to successfully issue multiple commands, which had the potential to produce unexpected results for long-running commands that may only partially complete. A warning will now be emitted when a \n is detected in a command, and the command will not be accepted. This change has also been backported to ensure that user scripts that utilize this behavior can be remedied.

• Rejecting the enabled keyword for dynamic servers: When defining a dynamic server, use of the enabled keyword is now rejected with an error, whereas previously it was only silently ignored.

• Stricter parsing of non-standard URIs: Parsing is now more strict during HTTP/1 processing for request target validation. This means that where previously, for compatibility, non-standard-compliant URIs were forwarded as-is for HTTP/1, now some invalid request targets are rejected with a 400-Bad-Request error.

• Renamed tune.ssl.ocsp-update: The tune.ssl.ocsp-update global keyword is now named tune.ocsp-update, as ocsp-update is unrelated to SSL tuning.

Conclusion

In the early days of the HAProxy project, it would have been difficult to foresee the multitude of ways people would use HAProxy, or the vast number of organizations that have adopted it at scale. Today, HAProxy is the market leader in software load balancing. That's thanks to the dedication of our open-source community members who write code, test features, document keywords, help newcomers, and evangelize to their organizations. Thank you to all!

HAProxy 3.0 maintains the strong momentum of our open-source load balancer into 2024 with improvements to simplicity, performance, reliability, observability, and security. This introductory blog post barely scratches the surface!

Subscribe to our blog and stay tuned for further deep dives on the latest updates from HAProxy 3.0. And in case you missed it, catch up with the huge new features we announced earlier this month in HAProxy Enterprise 2.9.

Ready to upgrade to HAProxy 3.0? Here’s how to get started.

New to HAProxy ALOHA?

HAProxy is the world’s fastest and most widely used software load balancer and the G2 category leader in API management, container networking, DDoS protection, web application firewall (WAF), and load balancing. HAProxy ALOHA makes application delivery simple with premium support, robust multi-layered security, and an intuitive interface—all in a convenient hardware or virtual appliance. HAProxy ALOHA provides an all-in-one application delivery controller with everything you need out-of-the-box.

What’s new?

HAProxy ALOHA 16 includes exclusive new enterprise features, plus all the features from the community version, HAProxy 2.9. For a full list of features, read the release notes for HAProxy ALOHA 16.

New in HAProxy ALOHA 16 are the following important features:

• The next-generation HAProxy Enterprise WAF powered by our unique Intelligent WAF Engine provides exceptional accuracy, zero-day threat detection, ultra-low latency, and simple management with optional OWASP Core Rule Set (CRS) compatibility. Our industry-leading WAF performance virtually eliminates the security impact of false negatives and the noise of false positives, with a balanced accuracy of 98.53% measured in testing based on open source WAF benchmark data.

• Four new Layer 4 load balancing algorithms expand HAProxy ALOHA’s flexibility in distributing traffic using a wider range of load balancing techniques. The four new algorithms include Weighted Overflow, Weighted Failover, Maglev Hashing, and Weighted Two Random Choices.

• HAProxy ALOHA’s upgraded Linux kernel to version 6.1 lengthens the duration of support from Linux security fixes.

• Better isolation of admin services makes it easier to segregate administrative services, strengthening security and enabling stricter access control.

We announced the release of HAProxy 2.9 in December 2023, which included faster performance, more flexibility, and better observability. The features from HAProxy 2.9 are now incorporated in HAProxy ALOHA 16.

Ready to upgrade?

To start the upgrade procedure, visit the installation instructions for HAProxy ALOHA 16.

One thing customers love about HAProxy ALOHA is that it comes bundled with its own web application firewall (WAF). We’re happy to share that the WAF in HAProxy ALOHA 16 is even better than before. The next-generation HAProxy Enterprise WAF brings industry-leading accuracy, performance, and simplicity.

Why a next-generation WAF?

Previously, HAProxy ALOHA included multiple WAF options, including an Advanced WAF and a ModSecurity WAF based on the OWASP Core Rule Set (CRS).

• Customers using the Advanced WAF found it to be extremely fast and powerful, but the skill requirements were relatively high. 

• Meanwhile, customers using the ModSecurity WAF appreciated the simplicity and industry-standard CRS compatibility, but the open source ModSecurity WAF introduced more latency and higher false positives than many customers were comfortable with. 

As we looked at the other WAF options on the market we also realized that low accuracy was a common problem, leaving users struggling to manage many false positives and to mitigate the damage caused by application attacks that slip through a WAF undetected. 

We wanted to give customers a WAF experience that combined the speed of the Advanced WAF, the simplicity of the ModSecurity WAF, and unprecedented accuracy to strengthen security and eliminate the noise. This goal led us to create the next-generation HAProxy Enterprise WAF, which delivers secure application delivery without compromise.

​What can you do with HAProxy Enterprise WAF?

Out-of-the-box, the HAProxy Enterprise WAF provides ultra-low latency protection against application attacks. This includes common attacks such as SQL Injection, Cross Site Scripting (XSS), Remote Code Execution (RCE), and Local File Inclusion (LFI), as well as emerging and zero-day threats. You can optionally use the industry-standard OWASP Core Rule Set (CRS) compatibility mode to maximize compatibility and transparency where needed.

Why should you use HAProxy Enterprise WAF?

Three reasons:

• Stronger security with exceptional balanced accuracy measured using open source WAF benchmark data, virtually eliminating the security impact of false negatives and the noise of false positives.

• Higher performance ensures ultra-low latency threat detection and traffic filtering while keeping resource use and operational costs low. 

• Simple to set up and manage with out-of-the-box behavior suitable for most deployments.

The next-generation HAProxy Enterprise WAF powered by the unique Intelligent WAF Engine brings industry-leading efficacy and performance. The Intelligent WAF Engine is a single low-latency process based on the company’s unique data science, security analytics, and real-world datasets. It identifies security threats using a non-signature-based detection system capable of blocking emerging and zero-day threats without requiring users to create and manage long or complex lists of rules.

Let’s talk accuracy. WAF accuracy can be calculated by measuring the true positive rate and the true negative rate:

• True positive rate refers to the proportion of dangerous traffic correctly identified by the WAF. Dangerous traffic incorrectly identified as safe is a “false negative”.

• True negative rate refers to the proportion of safe traffic correctly identified by the WAF. Safe traffic incorrectly identified as dangerous is a “false positive”.

The average of these two values is called “balanced accuracy”. The vast majority of WAFs on the market do well at one metric but not the other, resulting in poor scores for balanced accuracy (generally below 90%). Naturally, we wanted to have a go ourselves, so we followed the same methodology with the new HAProxy Enterprise WAF.

HAProxy Enterprise WAF powered by the Intelligent WAF Engine achieved: 

• a true-positive rate of 99.61%

• a true-negative rate of 97.45%

• a resulting balanced accuracy rate of 98.53%, 

This result comfortably beats the category average. It means that false positives are a thing of the past, reducing the impact on legitimate users and the operational burden of monitoring security alerts. It also means that false negatives are virtually eliminated, reducing the risk that malicious traffic will cause downtime, data loss, fraud, and more.

When using the optional OWASP CRS compatibility mode, we measured an impressively low false-positive rate of 1.78% at paranoia level 2 (compared with 28.36% for the ModSecurity WAF at the same paranoia level) resulting in reduced noise and a better user experience.

How about performance? WAF performance can be evaluated by the latency (the time taken to process each request) incurred with a variety of attack payloads and traffic volumes. The HAProxy Enterprise WAF provides high-performance threat detection and filtering with latency below measurable thresholds for the majority of attack payloads, meaning no performance penalty for security and virtually zero impact on legitimate traffic.

Performance is also improved significantly when using the optional OWASP CRS compatibility mode. With a realistic mix of safe and suspicious traffic (approximately 5% suspicious), the HAProxy Enterprise WAF achieves on average 15X lower latency than the ModSecurity WAF using the OWASP CRS.

This incredible accuracy and performance is available out-of-the-box to users of HAProxy ALOHA 16. You won’t need to write and maintain your own custom WAF rules. With the power of the Intelligent WAF Engine, it just works. This industry-leading performance in a simple package helps customers protect their business and reputation, simplify security, and reduce the impact on application performance and user experience.

HAProxy ALOHA 16 introduces support for four new Layer 4 load balancing algorithms. This enhanced support expands HAProxy ALOHA’s flexibility in distributing traffic using a wider range of load balancing techniques.

Overflow Connection

The Overflow Connection scheduling algorithm implements "overflow" load balancing according to the number of active connections. This algorithm keeps all connections on the server with the highest weight and overflows to the next server if the number of connections exceeds the server's weight.

Overflow Connection offers predictable behavior in how connections are distributed, advantageous for those seeking consistent performance and resource utilization. However, considering that it uses active connections in its distribution, it may not be suitable for UDP.

Weighted Failover

The Weighted Failover scheduling algorithm offers a simple failover solution to ensure service continuity and minimize downtime. When initiating failover, this algorithm redirects connections to the next available server in line with the highest weight.

By prioritizing servers based on their weights, Weighted Failover ensures that your services remain accessible to users. This fast response to failures mitigates downtime and helps maintain availability, enhancing the overall reliability of your application delivery.

Maglev Hashing

Maglev Hashing scheduler provides consistent hashing but with minimal disruption, each destination receiving an almost equal amount of connections. Maglev Hashing hashes incoming requests, which is ideal for efficiently distributing identical requests to the same backend server.

Maglev Hashing improves consistent hashing by leveraging a table to cache which server handles a hash, avoiding the shuffling of buckets. This consistency is critical for preserving session affinity in application delivery.

Weighted Two Random Choices

This algorithm decides which server will respond to each request by picking two random servers based on weighting and choosing the one with the fewest active connections. 

Weighted Two Random Choices improves the Power of Two Random Choices algorithm by considering the weights assigned to each server. This algorithm ensures that requests are routed to servers with ample capacity and resources, reducing response times and improving system performance.

In HAProxy ALOHA 16, some services now support specifying the network interface on which to run. This makes it easier to isolate administrative services onto a specific subnet.

This is especially helpful when HAProxy ALOHA is attached to two networks. By specifying that the admin services are on one network while web traffic is on the other, customers are able to enhance their security and enable stricter access control. This helps limit the attack surface for security threats because attackers would need to bypass additional security measures to gain unauthorized access.

Customers can bind admin services on a dedicated interface using the new @iface keyword for the following services:

• collectd

• httpd

• notify

• ntpd

• snmpd

• sshd

• syslog

Upgraded Linux kernel

This release upgrades HAProxy ALOHA’s Linux kernel to version 6.1. Users will benefit from a lengthened duration of support from Linux.org for essential security fixes.

Conclusion

For a complete list of features and changes, read the HAProxy ALOHA 16 release notes.

Upgrade to HAProxy ALOHA 16 to enhance your application's performance, flexibility, and security with the latest features and changes.

New to HAProxy Enterprise?

HAProxy is the world’s fastest and most widely used software load balancer and the G2 category leader in API management, container networking, DDoS protection, web application firewall (WAF), and load balancing. HAProxy Enterprise elevates the experience with premium support, robust multi-layered security, and centralized management, monitoring, and automation with HAProxy Fusion. HAProxy Enterprise and HAProxy Fusion provide a secure application delivery platform for modern enterprises and applications.

To learn more, contact our sales team for a demonstration or request a free trial.

What’s new?

HAProxy Enterprise 2.9 includes exclusive new enterprise features plus all the features from the community version of HAProxy 2.9. For the full list of features, read the release notes for HAProxy Enterprise 2.9.

New and exclusive in HAProxy Enterprise 2.9 are the following important features:

• The next-generation HAProxy Enterprise WAF powered by our unique Intelligent WAF Engine provides exceptional accuracy, zero-day threat detection, ultra-low latency, and simple management with optional OWASP Core Rule Set compatibility. Our industry-leading WAF performance virtually eliminates the security impact of false negatives and the noise of false positives, with a balanced accuracy of 98.53% measured in testing based on open source WAF benchmark data.

• The new HAProxy Enterprise Bot Management Module provides fast, reliable, and flexible identification and categorization of bots attempting to access websites or applications, with 100% local processing for low latency and no external dependencies. Our proven real-world performance shows the ability to catch previously unidentified bots while supporting 2X more requests per second on one instance than the nearest competitor.

• The new HAProxy Enterprise UDP Module provides fast and reliable UDP proxying and load balancing alongside HAProxy’s existing broad protocol support. Our best-in-class UDP performance is capable of reliably handling 3.8 million Syslog messages per second (46Gb/s) – 4.6X faster than the nearest enterprise competitor. 

We announced the release of HAProxy 2.9 in December 2023, which included faster performance, more flexibility, and better observability. The features from HAProxy 2.9 are now available in HAProxy Enterprise 2.9.

For an introduction to the features listed above, watch our HAProxy Enterprise 2.9: Next-Generation WAF, New Bot Management, and UDP Load Balancing webinar.

Ready to upgrade?

When you are ready to start the upgrade procedure, go to the upgrade instructions for HAProxy Enterprise.

One reason why HAProxy Enterprise is so popular is the bundled web application firewall (WAF). In HAProxy Enterprise 2.9, security is better than ever. The next-generation HAProxy Enterprise WAF brings industry-leading accuracy, performance, and simplicity.

Why a next-generation WAF?

Previously, HAProxy Enterprise included multiple WAF options including an Advanced WAF and a ModSecurity WAF based on the OWASP Core Rule Set (CRS). 

• Customers using the Advanced WAF found it to be extremely fast and powerful, but the skill requirements were relatively high. 

• Meanwhile, customers using the ModSecurity WAF appreciated the simplicity and industry-standard CRS compatibility, but the open source ModSecurity WAF introduced more latency and higher false positives than many customers were comfortable with. 

As we looked at the other WAF options on the market we also realized that low accuracy was a common problem, leaving users struggling to manage many false positives and to mitigate the damage caused by application attacks that slip through a WAF undetected. 

We wanted to give customers a WAF experience that combined the speed of the Advanced WAF, the simplicity of the ModSecurity WAF, and unprecedented accuracy to strengthen security and eliminate the noise. This goal led us to create the next-generation HAProxy Enterprise WAF, which delivers secure application delivery without compromise.

What can you do with HAProxy Enterprise WAF?

Out-of-the-box, the HAProxy Enterprise WAF provides ultra-low latency protection against application attacks. This includes common attacks such as SQL Injection, Cross Site Scripting (XSS), Remote Code Execution (RCE), and Local File Inclusion (LFI), as well as emerging and zero-day threats. You can optionally use the industry-standard OWASP Core Rule Set (CRS) compatibility mode to maximize compatibility and transparency where needed.

HAProxy Enterprise WAF is part of HAProxy Enterprise’s multi-layered security, which also includes the new Bot Management Module and Global Rate Limiting (powered by the Global Profiling Engine). You can combine accurate WAF detection and blocking with the other powerful layers in the security suite to create highly customizable threat management strategies – from simple to advanced.

Why should you use HAProxy Enterprise WAF?

Three reasons:

1. Stronger security with exceptional balanced accuracy measured using open source WAF benchmark data, virtually eliminating the security impact of false negatives and the noise of false positives.

2. Higher performance ensures ultra-low latency threat detection and traffic filtering while keeping resource use and operational costs low. 

3. Simple to set up and manage with out-of-the-box behavior suitable for most deployments.

The next-generation HAProxy Enterprise WAF powered by the unique Intelligent WAF Engine brings industry-leading efficacy and performance. The Intelligent WAF Engine is a single low-latency process based on the company’s unique data science, security analytics, and real-world datasets. It identifies security threats using a non-signature-based detection system capable of blocking emerging and zero-day threats without requiring users to create and manage long or complex lists of rules.

Let’s talk accuracy. WAF accuracy can be calculated by measuring the true positive rate and the true negative rate:

• True positive rate refers to the proportion of dangerous traffic correctly identified by the WAF. Dangerous traffic incorrectly identified as safe is a “false negative”.

• True negative rate refers to the proportion of safe traffic correctly identified by the WAF. Safe traffic incorrectly identified as dangerous is a “false positive”.

The average of these two values is called “balanced accuracy”. The vast majority of WAFs on the market do well at one metric but not the other, resulting in poor scores for balanced accuracy (generally below 90%). Naturally, we wanted to have a go ourselves, so we followed the same methodology with the new HAProxy Enterprise WAF.

HAProxy Enterprise WAF powered by the Intelligent WAF Engine achieved: 

• a true-positive rate of 99.61%

• a true-negative rate of 97.45%

• a resulting balanced accuracy rate of 98.53%, 

This result comfortably beats the category average. It means that false positives are a thing of the past, reducing the impact on legitimate users and the operational burden of monitoring security alerts. It also means that false negatives are virtually eliminated, reducing the risk that malicious traffic will cause downtime, data loss, fraud, and more.

When using the optional OWASP CRS compatibility mode, we measured an impressively low false-positive rate of 1.78% at paranoia level 2 (compared with 28.36% for the ModSecurity WAF at the same paranoia level) resulting in reduced noise and a better user experience.

How about performance? WAF performance can be evaluated by the latency (the time taken to process each request) incurred with a variety of attack payloads and traffic volumes. The HAProxy Enterprise WAF provides high-performance threat detection and filtering with latency below measurable thresholds for the majority of attack payloads, meaning no performance penalty for security and virtually zero impact on legitimate traffic.

Performance is also improved significantly when using the optional OWASP CRS compatibility mode. With a realistic mix of safe and suspicious traffic (approximately 5% suspicious), the HAProxy Enterprise WAF achieves on average 15X lower latency than the ModSecurity WAF using the OWASP CRS.

This incredible accuracy and performance is available out-of-the-box to users of HAProxy Enterprise 2.9. You won’t need to write and maintain your own custom WAF rules. With the power of the Intelligent WAF Engine, it just works. This industry-leading performance in a simple package helps customers protect their business and reputation, simplify security, and reduce the impact on application performance and user experience.

Our customers have implemented some impressive bot management strategies using HAProxy’s tools for traffic profiling, tracking, and filtering. Now, it’s even easier to use HAProxy Enterprise as a powerful alternative to a separate bot management solution. The new Bot Management Module provides fast, reliable, and flexible bot identification and categorization with low latency and deep integration with HAProxy Enterprise’s multi-layered security controls. 

Why bot management?

HAProxy Enterprise has long enabled users to identify bot traffic using a combination of tools, such as:

• the Verify Crawler module, which verifies the identity of bots claiming to be valid crawlers, such as the Googlebot web crawler,

• the Fingerprint modules, which use multiple data points to accurately identify clients and requests,

• the Global Profiling Engine, which provides a comprehensive and up-to-date view of client behavior across a cluster and enables Global Rate Limiting.

From DoS attacks to content scraping, the risks from bot traffic are growing yearly. Failure to identify and block malicious bots could result in downtime, data theft, fraud, and more, affecting an organization’s reputation and revenue. Additionally, bot traffic can significantly increase resource use, which increases operational costs and could affect application performance for legitimate human users. 

To combat the rising risks, we wanted to make effective bot management more accessible and more powerful. In HAProxy Enterprise 2.9, customers have a new weapon in their arsenal against bots. The new HAProxy Enterprise Bot Management Module is simple to set up and uses HAProxy Enterprise’s unique advantages to make it faster, more reliable, and more flexible than the market-leading alternatives.

What can you do with the HAProxy Enterprise Bot Management Module?

HAProxy Enterprise’s Bot Management Module works out-of-the-box to identify traffic accurately, categorizing it as human, suspicious, bot, verified crawler (search engines), or verified bot/tool/app (non-browser). 

You can combine accurate bot identification with the other powerful layers in the security suite (including the next-generation HAProxy Enterprise WAF and Global Rate Limiting) to create customizable, high-performance, and low latency bot management and rate limiting strategies – from simple to advanced.

Why should you use the HAProxy Enterprise Bot Management Module?

Three reasons:

• Fast performance eliminates latency and ensures rapid bot identification and enforcement of bot management policies even under heavy load (eg. DoS attack). 

• Reliable bot management with a simple architecture reduces complexity and keeps your data local and secure.

• Flexible and customizable bot management shares intelligence with other powerful security layers for smarter, more holistic decision-making and enforcement. 

For most users, we expect the simple answer to be: why wouldn’t you use it? 🙂 You can enable it in moments, and since it’s built into HAProxy Enterprise – the world’s fastest software load balancer – it works quickly and efficiently even under heavy load. 

But the real question for many customers is: why use this instead of one of the market-leading bot management solutions? 

Unfortunately, bot management solutions often come with significant compromises (not even counting the extra cost).

• Latency: solutions that pass requests through an additional layer, sometimes in a different network location, add latency (in addition to the often-quoted processing time) that affects the user experience.

• Complexity: solutions that require a constant or frequent connection to the vendor’s cloud (for example, for automatic updates to the detection algorithm) introduce complexity and an additional point of failure, putting reliability and data privacy at risk. 

• Lack of integration: solutions without deep integration with other security layers, such as with the WAF and anomaly detection layers, make decisions with incomplete information and do not give users the flexibility to enhance and customize their bot management strategy.

HAProxy Enterprise’s Bot Management Module uses reputational signals and scoring based on HAProxy Technologies’ security expertise, data science, and large real-world datasets to identify traffic accurately. Importantly, all the detection, processing, and enforcement is local to the HAProxy Enterprise instance. It does not add additional layers to the request path and does not require an external connection. This minimizes latency, maximizes reliability, and gives you the flexibility to deploy anywhere you like – such as in air-gapped environments.

With deep integration with HAProxy Enterprise’s multi-layered security, you can customize your organization’s bot management to meet your unique needs and traffic profile. You can customize your enforcement policies with options including blocking, tarpitting, challenging, and rate limiting.

But how good is it at identifying bots? While this is hard to test in a benchmark scenario, in real-world deployments with early adopters the HAProxy Enterprise Bot Management Module helped a top eCommerce website handling 300,000 requests per second identify heavy amounts of suspicious traffic and avoid crippling outages. As much as 20% of traffic was identified as anomalous, which their previous system had accepted without raising any security concerns. HAProxy Enterprise’s efficiency also resulted in cost savings, supporting 2X more requests per second on one instance than the nearest competitor.

This fast, reliable, and flexible bot management solution helps customers protect their business and reputation and reduce the resource cost of serving requests from unwanted bots.

We heard from a few of you that you wanted UDP in HAProxy Enterprise, to supplement the existing support for TCP, QUIC, HTTP, and everything else. And by “a few”, I mean this was our most requested feature. Well, the wait is over and we promise it was worth it. HAProxy Enterprise’s UDP Module delivers best-in-class performance for software load balancers, capable of reliably handling 3.8 million Syslog messages per second.

Why UDP?

Adding UDP proxying and load balancing to HAProxy Enterprise is a critical move to simplify application delivery infrastructure. Previously, those with UDP applications might have used another load balancing solution alongside HAProxy Enterprise, introducing inconsistent performance, management, and deployment form factors. No one wants to deal with that extra complexity. By including UDP support in HAProxy Enterprise, alongside support for TCP, QUIC, SSL, HTTP and everything else HAProxy is known for, we provide customers with a simple, unified solution with the flexibility to proxy and load balance many more applications.

What can you do with the HAProxy Enterprise UDP Module?

HAProxy Enterprise’s UDP Module supports proxying and load balancing for time-sensitive UDP applications including – but not limited to – DNS, NTP, RADIUS, and Syslog traffic. 

You can now combine UDP proxying and load balancing with HAProxy Enterprise’s powerful features such as health checks and monitoring, extending HAProxy Enterprise’s reliability to more applications – while avoiding the complexity and overhead of managing multiple products.

Why should you use the HAProxy Enterprise UDP Module?

It’s fast. It wouldn’t be HAProxy if it wasn’t.

HAProxy Enterprise customers have long benefited from HAProxy’s legendary performance in load balancing TCP/HTTP traffic. When we decided to add UDP support, we resolved that UDP would not be treated as a second-class citizen in HAProxy Enterprise. It had to meet our high standards for performance and reliability.

Customers using HAProxy Enterprise’s UDP Module benefit from faster and more reliable UDP load balancing compared with other software load balancers. We measured excellent throughput and reliability when testing HAProxy Enterprise’s UDP Module with Syslog traffic.

Test parameters:

• A single instance of HAProxy Enterprise 2.9.

• A network with 100Gb/s bandwidth.

• Log servers with a bandwidth of 40Gb/s capable of receiving 3.3M messages per second.

• A log line message size of 1,472 Bytes. 

• A payload of 2 million log messages sent via the load balancer to the log servers.

In our tests, we saw that HAProxy Enterprise’s UDP Module is capable of processing 3.8 million messages per second – up to 4.6X faster than the nearest enterprise competitor. 

Reliability was also excellent. UDP is a connectionless transport protocol where some packet loss is expected due to a variety of network conditions and, when it happens, is uncorrected because (unlike TCP) there is no client-server connection to identify and correct packet loss. Despite this, we saw that HAProxy Enterprise’s UDP Module achieved a very high delivery rate of 99.2% when saturating the log server’s 40Gb’s bandwidth – 4X more reliable message delivery than the nearest enterprise competitor. 

This best-in-class UDP performance compared with other software load balancers helps customers scale higher, eliminate performance bottlenecks, reduce resource utilization on servers and cloud compute, and decrease overall costs.

Try HAProxy Enterprise 2.9

The world’s leading companies and cloud providers trust HAProxy Technologies to protect their applications and APIs. High-performing teams delivering mission-critical applications and APIs need the most secure, reliable, and efficient application delivery engine available. HAProxy Enterprise’s no-compromise approach to secure application delivery empowers organizations to deliver next-level enterprise scale and innovation.

There has never been a better time to start using HAProxy Enterprise. Request a free trial of HAProxy Enterprise and see for yourself.

We've covered the headlining features in our HAProxy Fusion Control Plane 1.2 LTS release blog. But while service discovery, external load balancing, and multi-cluster routing are undeniably beneficial, context helps us understand their impact. 

Here are some key takeaways from our latest Kubernetes developments and why they matter.

HAProxy already has an Ingress Controller. Why start building K8s routing around HAProxy Fusion Control Plane? 

While we've long offered HAProxy Enterprise Kubernetes Ingress Controller to help organizations manage load balancing and routing in Kubernetes, some use cases have required technical workarounds. Some users have also desired a K8s solution that mirrors their familiar experiences with load balancers. Resultantly, we've redefined Kubernetes load balancing with HAProxy Fusion for the following reasons:

• The Ingress API wasn't designed to functionally emulate a typical load balancer and doesn't support external load balancing.

• Ingress-based solutions can have a steeper learning curve than load balancers, and users don't want to manage individualized routing for hundreds of services.  

• Organizations that favor flexibility and choose to run K8s in public clouds (due to Ingress limitations) are often forced to use restrictive, automatically provisioned load balancers. Those who also need to deploy on-premises face major hurdles.

External load balancing for public clouds is relatively simple—with public cloud service integration being a key advantage—but instantly becomes more complicated in an on-premises environment. Those integrations simply didn’t exist until now.

Few (if any) solutions were available to tackle external load balancing for bare metal K8s. And if you wanted to bring your external load balancer to the public cloud, the need for an ingress controller plus an application load balancer (ALB) quickly inflated operational costs.

All users can now manage traffic however they want with HAProxy Fusion 1.2. Our latest release brings K8s service discovery, external load balancing, and multi-cluster routing to HAProxy Enterprise. You no longer need an ingress controller to optimize K8s application delivery—and in some cases, HAProxy Fusion-managed load balancing is easier.

Easing the pain of common K8s deployment challenges

Here's what Spectro Cloud's 2023 State of Production Kubernetes report found:

• 98% of the report's 333 IT and AppDev stakeholders face one or more of K8s' most prominent deployment and management challenges.

• Most enterprises operate 10+ Kubernetes clusters in multiple hosting environments, and 14% of those enterprises manage over 100 clusters!

• 83% of interviewees had two or (many) more distributions across different services and vendors (including AWS EKS-D, Red Hat OpenShift, etc.).

The takeaway? Kubernetes remains universally challenging to effectively deploy and manage. Meanwhile, the increased scale and complexity of K8s distributions is magnifying those issues organizations are grappling with.

Plus, Kubernetes adoption is through the roof as containerization gathers steam. The value of developing a multi-cluster, deployment-agnostic K8s load balancing solution is immense—as is the urgency.

HAProxy helps solve common challenges around bring-your-own external load balancing practices, network bridging, pod management, and testing.

It all starts with service discovery

Without successfully exposing your pod services (or understanding your deployment topology), it's tough to set up traffic routing and load balancing. This also prevents services from being dynamically aware of one another without hard coding or tedious endpoint configuration. A dynamic approach is crucial since K8s pods and their IP addresses are ephemeral—or short-lived. Service discovery solves these issues, but not all approaches are equal.

Kubernetes setups without a load balancer commonly enable service discovery through a service registry. This happens on the client side and can complicate the logic needed for pod and container awareness. HAProxy server discovery is server-side since the load balancer does the work of connecting to services and retrieving information on active pods.

Understanding the HAProxy advantage 

• Layer 4 (TCP) and Layer 7 (HTTP) load balancing without having to separately manage Ingress services or Gateway API services

• Centralized management and observability

• Easier configuration language without complicated annotations

• Multi-cluster routing

HAProxy Enterprise can now perform external load balancing for on-premises Kubernetes applications, configured via HAProxy Fusion Control Plane. HAProxy Fusion is aware of your Kubernetes infrastructure, and HAProxy Enterprise can sit inside or outside of your Kubernetes cluster.

More traffic, fewer problems

HAProxy Enterprise also treats TCP traffic as a first-class citizen and includes powerful multi-layered security features:

• Web application firewall

• Bot management

• Rate limiting

• …and more

External load balancing (and multi-cluster routing) leverages normal HAProxy Enterprise instances for traffic management. We can now automatically update load balancer configurations for backend pods running behind them. 

Aside from external load balancing, HAProxy Fusion Control Plane and HAProxy Enterprise play well with cloud-based Kubernetes clusters. In instances where you're otherwise paying for multiple load balancing services (as with AWS, for example), this tandem can help cut costs. Greater overall simplicity, speed, and consolidation are critical wins for users operating within a complex application environment.

Leverage external load balancing and multi-cluster routing in any environment

Running a single K8s cluster has always been cumbersome for organizations that value high availability, A/B testing, blue/green deployments, or multi-region flexibility. Our earlier statistics also support just how pervasive multi-cluster setups are within Kubernetes. 

HAProxy’s multi-cluster routing is based around one central requirement: you need to load balance between multiple K8s clusters that are active/active, active/passive, or spread across multiple regions. Here's how HAProxy Fusion and HAProxy Enterprise support a few important use cases.

Multi-cluster routing example #1: multiple simultaneous clusters

Organizations often want to balance network traffic across multiple clusters. These clusters could be running the same version of an application within clusters across different availability zones. HAProxy Enterprise and HAProxy Fusion let you run your load balancer instances in either active/active or active/passive mode, depending on your needs.

This setup is pretty straightforward: 

1. Your HAProxy Enterprise instance and Kubernetes clusters are contained within one region. 

2. HAProxy Enterprise routes traffic between two or more clusters and their pods using standard load-balancing mechanisms.

Organizations often use A/B testing to compare two versions of something to see what performs better. For applications, this involves sending one portion of users to Cluster 1 (for Test A) and Cluster 2 (for Test B) where different app versions are waiting for them. 

Blue/green deployments work quite similarly, but we're transitioning traffic gradually from one application version to another. This only happens once the second cluster is ready to accept traffic. As a result, you can avoid downtime and switch between applications as needed.

Having a global Kubernetes infrastructure is highly desirable, but stretching a single cluster across multiple regions isn't readily possible. Networking, storage, and other factors can complicate deployments—highlighting the need for a solution. Having more clusters and pods at your disposal means unwavering uptime, which is exactly what this setup is geared towards.

Each region can run one or more Kubernetes clusters with HAProxy instances in front of them. Should one of the Kubernetes clusters fail, HAProxy can automatically send traffic to the other cluster, without disrupting user traffic. The one tradeoff is slightly higher latency before you recover your services.

Thanks to automated service discovery in HAProxy Fusion and seamless integration with HAProxy Enterprise, we can now address many common pain points associated with Kubernetes deployments. On-premises external load balancing must be as easy as it is for public clouds, and load balancing between clusters is critical for uptime, scalability, and testing purposes. Our latest updates deliver these capabilities to our customers. 

External load balancing and multi-cluster routing are standard in HAProxy Enterprise. HAProxy Fusion ships with HAProxy Enterprise at no added cost, unlocking these powerful new functions. 

However, we have so much more to talk about! Check out our webinar to dive even deeper.