Easily Remove Existing HAProxy Connections Made via Client Authentication

Most load balancers only check a client certificate when the client first connects. However, this can be problematic if a client stays connected for an extended period of time. Staying connected would allow clients to continually send and receive data.

Imagine you have an employee whose certificate and key were stolen by an adversary. If you are using TLS client authentication, that adversary can connect to your infrastructure and maintain illegal access. And even after you revoke that certificate, that adversary will remain connected until they themselves close that connection. 

For many vendors, the only fix for this is to clear all existing connections and force everyone to re-connect. This is extremely disruptive and can lead to lost revenue since it also kills legitimate connections. 

Luckily, HAProxy Enterprise lets you immediately drop connections and remove the client if their certificate is revoked. Meanwhile, other customers and users aren't affected. Any subsequent attempt to reconnect using that revoked certificate will fail.

Introduction to client authentication

You can use a certificate authority (CA) to generate a client certificate and key. That certificate and key function is like a virtual ID badge—serving as authentication for that client while enabling the server to verify that the client is authorized to access a given resource. 

Since TLS exists at OSI Layer 6, you can use TLS client authentication to secure and authenticate for a variety of applications. One of the most common use cases is securing HTTP traffic. However, you can use TLS authentication to add a layer of security to other protocols, such as SSH. 

The mTLS authentication configuration for either of these protocols can be identical from an HAProxy perspective, which is why we've used a generic example below using a random port. You can use this same configuration with HTTP—just remember to keep the frontend and backend in mode tcp for that configuration, too:

There are three important things to note about this configuration:

1. This module currently only works with mode tcp and not http mode.

2. If you're sending more than one certificate authority (CA) you MUST have a CRL list for each.

3. If you update the file using the CLI, the global option tune.bufsize needs to be larger than the size of that file (the default is 16Kb).

Understanding the configuration

For those new to the HAProxy Enterprise configuration file, the code sample above might seem complex. Let's break down each section and its key components to highlight their importance.

global

Generally appearing at the top of the configuration file, the global section defines process-level directives such as the maximum number of concurrent connections, where to store logs, and which user and group the process should run under. Here's why each line from our sample matters: 

• The log line helps if you need to troubleshoot.

• stats socket is where you'll post your changes, also called the Runtime API.

• module-path is the path where all the modules reside by default.

• module-load references the module we need.

defaults

The defaults section contains common settings which will be inherited by frontend,backend and listen sections after it. It helps shorten longer configurations by reducing duplicated lines. Our sample configuration contains some generic values that are helpful to define. 

For example, client, server, and connection timeout values default to infinite if not set explicitly. Choosing a value for each helps avoid keeping open connections that were silently closed or discarded by one endpoint.

frontend

The frontend section defines which IP addresses and ports clients can connect to. This helps expose applications or websites to the internet. Our sample configuration contains many important lines that impact SSL/TLS enablement and certificate management. 

First, you must be in mode tcp since mode http won't work effectively at the time of this writing but should exist in the future. 

Second, we've set up a bind on the randomly chosen port 76. Make sure you modify this to be the port that your application uses. This particular line also contains some important pieces that influence SSL and certificate-related behaviors:  

• Together, the ssl and crt portions enable SSL in HAProxy Enterprise while specifying what certificate to present to the client in the server hello. 

• ca-file specifies which certificate authority to present. This file can contain more than one CA if needed, but requires a CRL for each in this instance. 

• verify required states that the client's certificate must be valid or the connection will be rejected.

• crl-file refers to your certificate revocation list.

• Lastly, filter sslcrl enables the ssl-crl module.

backend

The backend section defines a server pool to which HAProxy Enterprise will route requests. You can normally add as many backend sections as you need to your configuration, but our example contains a generic configuration for the purposes of this walkthrough. 

With this overall configuration, you can now configure your client software to send a certificate. HAProxy Enterprise will validate this certificate and accept it if it's valid. If you'd like to set up your own test certificates, this blog post from Didier Stevens (unaffiliated with HAProxy Technologies) outlines some great how-to steps. The creation of these files is outside the scope of this blog, however.

Using the CLI to view and revoke

Once the configuration is up and running it's quite simple to revoke a certificate with the command line interface (CLI). Here's a brief overview.

First, we recommend viewing your current CRL file. This is also helpful to run before and after your updates if you want to validate the changes. Run the following command:

The second step comes after you revoke a certificate and get an updated CRL. You'll then update the CRL file in memory with that updated CRL—which adds another certificate to revoke (this appends the existing list in memory)—using the following command:

Finally, commit that change after updating your CRL file:

Once you issue this commit command, HAProxy Enterprise will update the CRL file in memory. HAProxy Enterprise will then promptly update its list of revoked client certificates and close any client connection(s) using that now-revoked certificate.

Here's a sample output from our earlier configuration:

Basic troubleshooting

To verify that the certificate is valid and usable with your certificate authority, you can use OpenSSL to check the client certificate:

If that returns "OK" then the certificate and CA are valid for use together.

Lastly, you can then use openssl to open a connection to your frontend and use the client certificate. Also confirm that www.host.com resolves properly to the IP address that your frontend is listening on:

Conclusion

Intelligent certificate management is essential in cases where certificates and keys are lost, stolen, or intentionally abused to launch cyber attacks. HAProxy Enterprise makes this complex process even easier and more flexible—letting you enable SSL, define certificate-related behaviors, and leverage certificate revocation lists to bolster infrastructure security. 

HAProxy Enterprise removes the burden, frustration, and revenue loss that can result from dropping all existing client connections to counteract suspicious behavior. Harness real-time revocation to keep your systems safe and your customers happy. To learn more, check out our HAProxy Enterprise SSL-CRL documentation or our HAProxy and Let's Encrypt: Improved Support in acme.sh blog post.

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.