Dynamic Configuration With the HAProxy Runtime API

In this blog post, you’ll learn how to dynamically configure HAProxy using the Runtime API.

Over the past 15 years, HAProxy has become well known for its reliability, superior performance, extensible features, and advanced security. It is relatively less known that one of HAProxy’s core building blocks is the Runtime API which provides very powerful dynamic configuration capabilities with no service reloads or restarts.

The early improvements to the Runtime API’s dynamic configuration capabilities were driven by requests from advanced HAProxy users such as Airbnb and Yelp through their development of SmartStack, a pioneering automated service discovery and registration platform. Since then, we have been strongly committed to evolving the Runtime API based on feedback from our users, as exemplified by our recent blog post Dynamic Scaling for Microservices with the HAProxy Runtime API and the release of HAProxy version 1.8.

In this blog post, we are going to take you on a tour of the HAProxy Runtime API and some of its key features, such as the ability to dynamically configure ACLs, stick tables, and TLS ticket keys. Beyond that, it allows for improved integration with service discovery tools and orchestration systems, enhances geolocation functionality, enables adaptive security policies, increases the efficiency of SSL/TLS clusters, and much more.

Related Blog: 

• The Four Essential Sections of an HAProxy Configuration

Getting Started

The HAProxy Runtime API traces its origins back to our wishes to create a complete configuration and statistics API for HAProxy, whose commands would all take effect immediately, during runtime. One of our early features in this API was of course the ability to retrieve detailed real-time statistics. Also, unlike typical newer APIs which only support HTTP, the HAProxy Runtime API was always accessible through TCP and Unix sockets. That is why we sometimes still refer to it as the HAProxy stats socket or just socket, and why the configuration directive for enabling the Runtime API bears the same name.

The Runtime API is enabled in the HAProxy configuration by using the following example:

If you are not using HAProxy Enterprise, the default Unix socket name in your installation might be named /var/run/haproxy.sock instead of /var/run/hapee-lb.sock.

Then, for testing or executing commands interactively, the Runtime API can be conveniently accessed using the interactive command prompt provided by socat:

Please note that for the above to work, your version of socat needs to be compiled with GNU Readline support. This could be verified by running socat -V | grep READLINE. If the output of socat -V doesn’t mention READLINE, or it mentions under READLINE or running the actual command produces an error unknown device/address “readline”, which means you have a version without readline support.

For accessing the Runtime API from scripts, or as an alternative to the interactive use shown above, the following command could be used:

And finally, as an alternative to socat altogether, command nc from the package netcat could be used. In that case, please use the netcat-openbsd version of netcat; it also supports the option nc -U for connecting to Unix domain sockets.

The best way to start with the Runtime API is to execute a simple request returning a list of all available commands. This is done by sending help or any unknown command to the API. Here is the actual output from “help”, executed on HAProxy version 1.8-dev2. If your HAProxy is missing any particular commands or options, please make sure you are using a recent HAProxy or HAProxy Enterprise release.

Retrieving Statistics

For a light introduction before going into the Runtime API’s more powerful commands, let’s demonstrate the method for obtaining real-time statistics from HAProxy.

One of the already well known methods for retrieving real-time statistics is through HAProxy’s built-in, HTML web page. These statistics, formatted in a simple and straightforward way, can be obtained after configuring the stats uri in HAProxy and visiting the statistics URL.

The same complete and raw information can also be obtained through the HAProxy Runtime API. The API command is named show stat:

The command’s output will be provided in CSV format. Support for JSON output has been included in HAProxy version 1.8 and HAProxy Enterprise version 1.7r2.

The statistics output contains more than 80 different metrics. To quickly convert it to a shorter and human readable output, we could use standard command line tools. Here is an example that shows a chosen subset of data and refreshes it every two seconds:

By using the command cut above, we have narrowed the selection down to the fields useful for quick and summarized monitoring:

• scur: current number of sessions

• smax: highest number of sessions seen since HAProxy was reloaded/restarted or had its statistics reset

• slim: configured upper limit on the number of sessions

• stot: cumulative number of sessions processed so far

• bin: bytes in

• bout: bytes out

• rate: average number of sessions per second, calculated based on the 1 second

• rate_lim: configured upper limit on new sessions per second

• rate_max: highest number of new sessions per second seen since HAProxy was reloaded/restarted or had its statistics reset

For a listing and description of all the fields available in the show stat output, please refer to the HAProxy Management Guide, section 9.1.

Administering Servers and Proxies

Using the Runtime API, you’re able to disable and enable servers or gradually drain traffic away from them. You can also enable and disable health checks, enable and disable frontends, and change the load balancing weight assigned to servers. Let’s take a look at a few examples.

If you want to gradually drain traffic away from a particular server, use the set server command with the state argument set to drain:

When a server is in drain mode, HAProxy will still send it regular health checks. You can stop them too by setting the server’s state to maint instead of drain. Draining a server gradually takes it out of the load balancing rotation, while putting it into maintenance mode is the same as disabling it immediately.

If you only want to stop the health checks, then use the disable health command, like so:

Or, if you’d like to send more or less traffic to a server, you can change its weight. Use the set server command with the weight argument:

Something else that you can do is stop all traffic to a frontend proxy. Use the disable frontend command to release a port binding from a frontend. You might use this to associate that port with a different frontend, if the original is causing trouble. Here’s an example:

There’s more to see, including changing a server’s maxconn value, changing a server’s assigned IP address, and changing the health check port. See the official documentation for more information.

Updating Stick Tables

Stick tables are yet another very powerful and very applicable, but simple to use feature in HAProxy. Stick tables allow keeping track of arbitrary client information across requests, so they are used for purposes such as creating “sticky” sessions, implementing security protections, setting limits, and maintaining counters and lists.

Stick tables (along with timers, schedulers, ACLs, and many other features and tools) are based on high-performance Elastic Binary Trees, invented by our HAProxy Technologies CTO, Willy Tarreau. Ebtrees allow these tables to contain millions of records while maintaining extremely fast speed of access. We have written about stick tables and their uses in many of our previous blog posts, including:

• Load balancing, Affinity, Persistence, and Sticky Sessions

• Microsoft Remote Desktop Services (RDS) Load-Balancing

• Load Balancer as First Row Defense Against DDoS

• WordPress CMS Brute Force Protection with HAProxy

The Runtime API allows stick table entries to be added, removed, searched, and updated during runtime. In the following example, we are going to create a stick table of 1 million entries to keep track of the request rates coming from individual client IPs. We will impose a limit of a maximum 5 requests per second, and the IP addresses which keep sending requests with a rate above that limit over a period of 10 seconds will be denied further requests for a period of 15 minutes.

As mentioned, the above stick table can store up to 1 million entries, and each entry will contain 3 values: an IP address, the determined average HTTP request rate over 10 seconds, and a general purpose tag “GPT0”, which will have a value of “1” if the requests are to be denied. Entries in the table will be removed to not occupy space if the client IP makes no further requests within the configured expiration period (in our example, 15 minutes). Requests coming from the whitelisted IPs will be passed through without imposing any restrictions on them.

All of this will work automatically and require no manual intervention, thanks just to the configuration lines above.

However, we might still want to be able to manipulate the stick table during runtime, for example to change the entries’ GPT0 flag, to remove entries altogether, or to list currently blocked IPs. This can again be done using the Runtime API:

Updating ACLs

To extend the above example of rate-limiting the incoming HTTP requests, let’s assume we would now want to change the existing rate limit from 10 to 60 requests per second.

To do so, the HAProxy Runtime API will certainly allow us to change the ACLs during runtime. We will execute the command “show acl” to see the configured ACLs, then identify the one we want to modify, and finally perform the actual modification. The modification will consist of adding a new value and then deleting the old one to complete the transition.

A complete session could look like the following:

Updating Whitelists and Blacklists

Again extending our original rate limiting example, please notice that we have used the following configuration directive to provision for whitelisted IP addresses:

This configuration line tells HAProxy to fetch the source IP, check it against the list of IP addresses that were loaded from whitelist.lst, and pass the request through without further checks if the IP address is found in the list.

In this example we are going to modify the contents of the whitelist by using the Runtime API – we will be changing an IP address entry from “192.168.1.6” to “192.168.1.9”. As usual, to do so we are going to list the ACLs, identify the entries we want to modify, and then perform the actual modification. The modification will consist of adding the new value and then deleting the old one to complete the transition.

A complete session could look like the following:

SSL/TLS

Updating TLS tickets

HAProxy uses tls-ticket-keys to avoid the expensive key renegotiation when an existing client wants to start a new session after closing the previous one. If a client supports session tickets, HAProxy will send it a new session ticket record containing all of the negotiated session data (cipher suite, master secret, etc.). The record will be encrypted with a secret key only known to the HAProxy process, ensuring that the data won’t be read or modified by the client.

The secrets used to encrypt the TLS tickets are generated on HAProxy startup. If an active-active HAProxy cluster is set up and a client moves between nodes, it will have to renegotiate the keys often because the tickets, encrypted with node-specific keys, will not be understood by other nodes in the cluster. In order to avoid this, it is possible to place secrets in a file and also periodically rotate them to maintain perfect forward secrecy. That way all nodes in the cluster will use exactly the same set of secrets and clients will be able to move between HAProxy nodes without any side effects.

This could be done by updating the file containing the keys and reloading HAProxy manually, but it can also be done during runtime by using the HAProxy Runtime API.

A complete session could look like the following:

Configuring OCSP responses

HAProxy also supports the TLS Certificate Status Request extension, also known as “OCSP stapling”. For more information on configuring the OCSP responses, please see the documentation section 5.1 – crt. OCSP stapling allows HAProxy to send certificate status information to the clients. The information is a signed OCSP response and will avoid the need for a client to send a separate request to the OCSP service hosted by the issuing Certificate Authority. Here is an example of a command used to display the OCSP response from a server:

Near the end of the output, we can see the information “Cert Status: Good”. We can also see in the last line of output that the next update is expected in 10 days.

Now, if we would want to update the OCSP response in order to change the next update time from 10 to 20 days, we could prepare the response and then load it into the running HAProxy instance by using the Runtime API:

And using the same approach as above, we can verify that HAProxy is now serving the updated response:

Troubleshooting

When it comes to troubleshooting the configuration or observed application behavior, looking into the HAProxy log files is probably one of the most common courses of action. And it is certainly a good one – HAProxy logs contain all sorts of information, including data about timers, sizes, connection counters, etc. But from time to time we may come across complicated issues or even bugs that need more detailed debugging output than even logs can provide.

Displaying errors

As mentioned, HAProxy logs errors to log files, as documented in the section 1.3.1. The Response line. For example, if we send it simple, invalid traffic such as the following:

Then the logs will show the following error:

But to get even more information about the request in question and even see the contents of the request, we can use the Runtime API command “show errors”:

Dumping sessions

Depending on the logging configuration, all connections can be logged to HAProxy’s log files. But, as usual, they are logged only after HAProxy gets a reply from the backend servers or when one of its timers expires. In situations where long timeouts are involved or where sessions are taking long to complete, this might cause the logs to never seem to arrive. In such situations, the Runtime API command “show sess” may be used to dump all current sessions and their related information:

In the above example, we can see two sessions: the first one is an IPv4 session; the second is a session related to our invoking of the Runtime API. From looking into the output, we can, for example, identify that the IPv4 session is in an early stage of processing because no backend (“be=”) was selected yet. Also, to help us further in troubleshooting complex issues, we may use the command “show sess” with a session ID provided as an argument (“show sess ID”) to get even more details about a particular session.

Closing sessions

In addition to displaying active sessions using “show sess”, we can also use the Runtime API to close sessions at will by using “shutdown session”:

Such sessions will appear in the logs containing the flag “K”, indicating they were shut down.

Updating GeoIP databases

GeoIP databases map IP address ranges to geographical locations. Such databases can be used in HAProxy for different purposes. Often times, they are used for performing GeoIP lookups natively within HAProxy and serving the data to backend servers. The backend servers can then depend on the information being available as part of incoming requests, requiring no specific GeoIP code nor causing any slowdowns in the application code itself.

More information about using GeoIP with HAProxy can be found in one of our previous blog posts, Using GeoIP Databases with HAProxy. Another common use case for GeoIP is to include the client country code in the HAProxy logs. This could be achieved by using the following log-format directive:

The above line will use the map_ip converter in order to get the country code from the map file ip-country.lst. Logs based on this format will then look like the following:

Now, in terms of updating the GeoIP databases, let’s assume we have the following new GeoIP entries in a map file:

We could easily add them to the running configuration using the Runtime API command add map. Also, instead of adding entries one by one, we are going to use a little bit of bash shell scripting to automate the data import:

Where to Go From Here

We’ve written several other blog posts that will give you even more examples of using the HAProxy Runtime API. Be sure to read:

• Dynamic Scaling for Microservices with the HAProxy Runtime API

• Introduction to HAProxy Maps

Conclusion

We hope you have enjoyed this blog post providing an introduction to the HAProxy Runtime API and showing some of its most common and practical use cases. The complete HAProxy Runtime API documentation can be found in the HAProxy Management Guide, section 9.3.

If you have a subscription to HAProxy Enterprise, we can provide you with authoritative support, scripts, and modules that will help you make the best use of the Runtime API, including its most advanced features listed among the commands but not specifically elaborated in this blog post.

The Runtime API is evolving along with all other features and improvements that we are adding to HAProxy. One of the planned Runtime API improvements is an HTTP REST interface to complement the existing access methods. Let us know what other features and improvements you would like to see included!

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.