Zero-Trust mTLS Automation with HAProxy and SPIFFE/SPIRE

Whether you’re running a service mesh composed of HAProxy instances or facilitating communication between multiple systems, ensuring the authentication of traffic between your services is critical. This zero-trust security model operates under the assumption that you should not extend trust without verification, even within your own systems. By verifying every interaction, you mitigate the risks that arise when third parties imitate your systems.

Building on our prior discussion on securing traffic between systems using mTLS certificates, this blog post describes how to deliver the same capability using SPIFFE and SPIRE, both CNCF projects, to automatically generate and renew identities that include mTLS certificates. 

While the original approach to achieving mTLS is a fine solution, manually generating certificates is not ideal for zero-trust networks. Instead, automatic generation of identities, including the needed mTLS certificates, will ensure that certificate management follows best practices—that certificates are short-lived, and that certificate generation does not result in inevitable failure.

The end result of automating certificate generation? A service mesh composed of HAProxy instances, all securely communicating with each other.

By following this guide, you will implement:

• A central system that you can use to register new workloads.

• HAProxy instances that automatically receive a SPIFFE SVID (a workload ID that includes a mTLS certificate).

• HAProxy instances that communicate with one another using the appropriate SVID.

Key terms

Before diving into configuration, let’s introduce key concepts related to the SPIFFE protocol and its SPIRE implementation.

SPIFFE

Secure Production Identity Framework For Everyone (SPIFFE) is a set of open standards aimed at securely identifying software systems in diverse and changing environments.

SPIRE

SPIFFE Runtime Environment (SPIRE) implements the SPIFFE specification, a framework for issuing and managing SPIFFE identities.

mTLS

Mutual Transport Layer Security (mTLS) is a security protocol for authenticating the client and server in a session, providing encryption and ensuring the secure exchange of data.

Workload

A workload is the collection of resources communicating with each other, each using the SVID issued to it.

SVID

A SPIFFE verifiable Identity Document (SVID) is a cryptographic document attesting to the identity of a workload, used to verify authenticity within a SPIFFE environment.

SPIRE Server

A SPIRE Server holds your workload identities, issues the right SVIDs, renews them, and serves them to SPIRE Agents.

In the most basic deployment, you'll run one server. It's a good idea to run more in production to support high availability.

SPIRE Agent

Every HAProxy instance will run a SPIRE Agent, a software instance that carries out several functions needed for secure communications. The agent attests itself (registers) to a SPIRE Server through one of the available methods—in our case, a join token.

When a new SVID is issued, the agent receives it and caches it. We can then instruct the agent to output the two needed mTLS certificates, which are sometimes called an SVID certificate and a Bundle certificate.

The SVID certificate is the client certificate that an HAProxy node should use on the backend to connect to the other HAProxy nodes. The Bundle certificate is the server certificate that each HAProxy node will use on its bind line to verify the incoming requests (which use the SVID certificate).

The official SPIRE documentation has good instructions to get started, but in the below, we have adjusted them for the specific HAProxy use case.

Guide: Setting up automatic generation of mTLS certificates using SPIFFE/SPIRE

Part one: Installing the SPIRE Server

First, install a SPIRE Server on a typical Linux distribution using the latest SPIRE release:

You will get a directory called spire-1.9.0.

If you look into spire-1.9.0/conf/server/server.conf, you should see a configuration similar to the following:

There will also be configurations related to plugins that you can leave in place for now. Most importantly, set the:

• bind_address and bind_port: these are addresses the SPIRE server will listen on

• trust_domain: This can be, but does not have to be, a DNS name. It identifies the trust “circle” for the workloads. SPIRE supports advanced federation, which is not part of this blog post; just know that, in this case, every node in this trust domain will be able to talk to other nodes in the same domain

Next, start the SPIRE Server:

Then, generate a join token for each of your nodes. Each join token can only be used once, so we must create one for every node in your network. Each token has a spiffeID, which identifies the specific token. You can use the same spiffeID for your workload or give each its own ID.

For the first node:

For the next node:

You will get a unique token for each HAProxy node. 

Now log in to your HAProxy nodes, download the same SPIRE release as your server, and edit your conf/agent/agent.conf file:

It’s important to note:

• trust_domain must be the same as on your server.

• Set server_address and port to the IP address of the SPIRE Server and the port that you bound SPIRE to.

Finally, start the agent on every node and use the correct join token you obtained previously:

Part two: Generate workload entries

Back on the SPIRE Server, you can now create a workload entry for each one of your nodes. This entry is what will generate the SVIDs on the agent.

Notice that the parentID is the same spiffeID from the node that we used with the join token. The -spiffeID argument identifies this workload as haproxy on the node.

Part three: Get the certificate on the agent and put it together

To validate that everything works, run the following command on the agent to fetch the certificate:

You should now see the following files in your /tmp directory:

• svid.0.key (the certificate key)

• svid.0.pem (the client certificate)

• bundle.0.pem (the server certificate)

Following our original instructions, you can now use these files in your HAProxy configuration. 

Just remember: bundle.0.pem is the certificate you should use on your bind line, and svid.0.key and svid.0.pem are the certificates for the client or backend server line (in the Send a Client Certificate to a Backend Server section).

As long as the spire-agent is running, it will automatically fetch and cache the new certificates on your nodes, but it doesn’t actually save them in your filesystem to be used. To do that, you could continuously run the api fetch command, or use a utility called spiffe-helper.

Here is an example script you could run in cron to fetch the certificates and send them to HAProxy:

Conclusion

Adopting a zero-trust security model for your HAProxy instances is necessary in an environment where internal systems cannot be trusted. 

By following the steps outlined in this blog post, you can leverage SPIFFE and SPIRE to automatically generate mTLS certificates for your service mesh. With this approach, your HAProxy instances can securely communicate with each other, strengthening your security posture while simplifying certificate management by removing the manual work and risks involved with renewals.

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.