Announcing HAProxy Data Plane API 2.3

HAProxy Data Plane API 2.3 expands its service discovery mechanisms and introduces native support for discovering AWS EC2 instances and auto-scaling groups. It also adds a new configuration file that supports HCL and YAML, an Inotify configuration watcher, and Syslog support.

Register for our webinar to learn more about this release.

HAProxy Data Plane API version 2.3 is now available, and you will find it in the 2.3 version of the Alpine Docker image. It does much to simplify the complexity exposed to users, as well as expand its set of features in areas like service discovery, logging, and validation.

The most obvious change is moving the command-line arguments needed to start the API into a YAML configuration file. This change, which outwardly looks cosmetic, is an indication of the maturity of the API. New features that require their own command-line arguments, such as the new Syslog support, make it clear that this API has grown bigger than its initial role, which was focused on generating an HAProxy configuration. It is fast becoming the interface between HAProxy and the services that surround it. Managing that complexity will be crucial going forward.

This version also supports service discovery of EC2 instances based on AWS instance tags. The API generates server lines in your HAProxy configuration from the instances it discovers. The importance of the cloud can’t be denied or even understated, and gaining service discovery in EC2 for HAProxy will be welcomed news for many organizations.

This latest release of the HAProxy Data Plane API also monitors the HAProxy configuration using Inotify, which means that you can make manual changes to your HAProxy configuration and use the Data Plane API simultaneously without worrying that the methods will conflict. It also adds support for sending its request logs over Syslog and allows you to customize the format using Apache Log Format directives. There are several other exciting features that were introduced, so check out the full release information below!

This release was a community effort and could not have been made possible without all of the hard work from everyone involved, both on GitHub and Slack.

HCL / YAML Configuration File

There are quite a few arguments you need to set when starting the Data Plane API.

This release introduces support for storing those command-line arguments and settings in a YAML configuration file—it also supports the HCL language as a file format. Version 2.3 of the Data Plane API requires this YAML file upon startup. Here’s an example dataplaneapi.yaml file:

Or its HCL equivalent:

Load it with the -f command-line argument. Note how the previous command with all of the arguments is now condensed:

In this file, you can specify the Basic authentication users that have access to the API by setting the user block. In previous versions, you had to configure this as a userlist section in the haproxy.cfg file. You can find complete examples for YAML here and HCL here. A converter for migrating between formats is also supplied through the following command:

This assumes that the root path is the HAProxy Data Plane API base source directory.

AWS EC2 Service Discovery

As organizations continue to move their workloads into the cloud and embrace cloud-native technologies, service discovery has become increasingly important. HAProxy Technologies recognized the importance of this need and introduced support for DNS Service Discovery in HAProxy version 1.8, released in 2017. The previous release of the HAProxy Data Plane API, version 2.2, expanded in this area by adding native support for Consul.

This release adds service discovery support within AWS EC2. Once you configure the HAProxy Data Plane API with the proper credentials, it’s able to poll the AWS API at a defined interval and populate your HAProxy backends with the addresses of your EC2 instances. If a backend does not exist, it will automatically create it. It also harnesses the HAProxy Runtime API to minimize reloads during scaling events.

To get started, first you will need to add the appropriate tags to the instances that you want to be discovered. The API can detect the following AWS tags:

Your AWS tags should look like this:

Next, you will need to configure the HAProxy Data Plane API either by sending a POST request with the configuration details or by editing the HAProxy Data Plane API’s HCL/YAML configuration directly.

Below, we send a POST request using curl to the /service_discovery/aws endpoint to set up service discovery:

Another way to do it is to change the API’s configuration file, adding the service_discovery block:

Your AWS credentials can also be specified through the AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID environment variables, or inherited by the IAM role with AmazonEC2ReadOnlyAccess permissions assigned to the EC2 instance.

Once running, service discovery will populate properly tagged EC2 instances within your HAProxy configuration:

The disabled servers act as empty slots that HAProxy fills in with servers on-the-fly to scale services up and down without requiring a reload.

You can specify services that you want to explicitly allow or deny from discovery by using the Allowlist and Denylist configuration arrays. For example, if you wanted to allow services that have the Environment tag Production only, you would specify this within your configuration.

ACL Runtime API Updates

HAProxy ACLs are expressions that you can define in your HAProxy configuration that allow you to match anything found within the request or response headers and metadata. Then, you can reference them to define routing, blocking, and logging rules. This release introduces support for interacting with defined ACLs via HAProxy’s Runtime API. You can now view, add, and remove ACL entries that are stored in-memory. The following endpoints have been added:

Inotify Configuration Watcher

The HAProxy Data Plane API lets you manage the HAProxy configuration through a RESTful interface. However, in some cases, you may want to make manual changes directly within the configuration. This release introduces support for inotify, which is a Linux kernel API that allows applications to monitor the filesystem for changes. This means that you can modify the HAProxy configuration directly and the HAProxy Data Plane API will automatically detect those changes, load them into its memory, and increment the current version on the HAProxy configuration file. It now also stores a new value at the top of the configuration named _md5hash, which holds the current MD5 hash of the configuration. This new functionality will only re-read the configuration and does not issue any reloads or restarts based on updates. This feature is enabled by default but can be disabled with the command-line argument --disable-inotify.

Syslog Support

Logging is a critical component to observability within any infrastructure. Whether you’re using logs for auditing and accountability purposes or for debugging, they provide deep insight and historical reference of your systems.

This release adds support for sending HAProxy Data Plane API request and debug logs to a Syslog server over a Unix socket or TCP/IP. It adds the following new command-line arguments

This release also introduces support for using the Apache common log format to structure request log entries. The default value is:

%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-agent}i” %{us}T

This can be modified using the --apache-common-log-format option from the command-line.

Customized Configuration Validation

Do you split your HAProxy configuration into multiple .cfg files? By default, the Data Plane API validates only the main configuration file by calling haproxy -f /etc/haproxy/haproxy.cfg -c. You can now specify validate_cmd within the reload section of the API’s YAML configuration file to define an alternate command to use for validation.

After adding validate_cmd, the API sets an environment variable named DATAPLANEAPI_TRANSACTION_FILE that contains the to-be-validated configuration that it generates. You’ll need to define a shell script that validates any HAProxy configuration files you have in addition to the file pointed at by the DATAPLANEAPI_TRANSACTION_FILE variable.

An example validation shell script, which we store at /usr/local/bin/validate-haproxy.sh, is as follows:

Miscellaneous

• New command-line and YAML/HCL configuration options for setting the uid and gid of the HAProxy Data Plane API process. The command-line flags are --uid <uid> and --gid <gid> the HCL/YAML arguments are uid / gid.

• Query string parameters force_reload and skip_reload were added to the Map file and SSL certificate storage endpoints, allowing you to force a reload or skip reloads entirely. The skip_reload parameter has higher precedence.

• The query string parameter acl_name allows for querying an ACL by its name.

• Transactions that fail or are committed after other transactions are made could result in a mismatched version number. These transactions will now be marked as outdated and will return an HTTP 406 Not Acceptable.

• This release supports all available options to the server and default-server directives.

• Support for http-request return has been added, which exposes the native response generator that was introduced in HAProxy 2.2

• The HAProxy Data Plane API can now be configured to write its pid file using the --pid-file command-line argument or pid_file within the dataplaneapi section of the configuration file.

Contributors

We’d like to thank the code contributors who helped make this version possible:

Conclusion

The HAProxy Data Plane API version 2.3 adds service discovery capabilities with a native AWS EC2 integration. It also adds a new configuration file that supports HCL and YAML, an Inotify configuration watcher, and Syslog support. The HAProxy Data Plane API continues to get better thanks to the active community surrounding it. If you’d like to get involved, head over to our GitHub repository to learn more!