How to Log HTTP Headers With HAProxy for Debugging Purposes

HAProxy offers a powerful logging system that allows users to capture information about HTTP transactions. As part of that, logging headers provide insight into what's happening behind the scenes, such as when a web application firewall (WAF) blocks a request because of a problematic header. It can also be helpful when an application has trouble parsing a header, allowing you to review the logged headers to ensure that they conform to a specific format. While logging headers is essential, the process can easily consume storage capacity. In this blog post, we show how to collect HTTP header logs and store them remotely to avoid overwhelming your standard log system.

Warning: Ensure Compliant Handling of HTTP Headers

Before delving into the process of logging HTTP headers with HAProxy, it is crucial to carefully understand the following. Failure to do so may potentially expose you or your company to significant risks and consequences.

HTTP headers can contain sensitive information related to privacy, which falls under the purview of the GDPR (General Data Protection Regulation) or any similar regulation such as LGPDP (Lei Geral de Proteção de Dados Pessoais) in Brazil. It's important to note that certain HTTP headers, like session cookies, are used to provide access to specific sections of a website and may sometimes convey private information. To comply with relevant data protection regulations, you must ensure the privacy and security of user information when handling and logging these headers.

The examples provided below assume your infrastructure meets all necessary privacy and security requirements. This blog post will focus solely on presenting solutions for logging headers, rather than addressing how to secure these logs. It is your responsibility to ensure that the necessary privacy and security measures are in place when implementing these examples.

What Are HTTP Headers?

HTTP headers are additional pieces of information sent between client and server during an HTTP transaction. Headers can be grouped under request headers and response headers. 

• Request headers are sent from the client to the server, bearing information on the client’s preferences and any necessary information about the request (this includes sensitive information). 

• Response headers are sent from the server to the client, instructing the client on how to handle additional information about the response.

The instructions in this blog post focus on gathering information from request headers. Logging these HTTP headers can be useful in understanding requests, identifying threats, and monitoring the flow of data.

Using the Logging System

When it comes to operationalizing your log data, HAProxy provides a wealth of information. If you are just getting started, read our blog post, Introduction to HAProxy Logging, to learn how to set up HAProxy logging, target a Syslog server, understand the log fields, and discover some helpful tools for parsing log files. If you want to go deeper and see HAProxy logs in action, watch our on-demand webinar, Deep Dive Into HAProxy Logging.

HAProxy offers a comprehensive configuration option to generate a log line that includes all headers. The req.hdrs sample fetch is the simplest way to achieve this. It retrieves the request headers block and returns it as a single contiguous string represented as an HTTP/1.1 request, one header per line, including the final empty line. 

To ensure accurate debugging and logging of the original headers sent by the client, call req.hdrs as early as possible, before any header modification takes place. The res.hdrs sample fetch reflects the request headers exactly as they appear when req.hdrs is invoked. By calling it early, you can obtain the unmodified headers for effective debugging purposes.

A log line is generated after the HTTP transaction completes, which means the request buffer is no longer accessible. However, the result of the req.hdrs sample fetch can be stored in a transaction variable and used in a log-format directive. This allows you to include the headers in the log output.

With this example, the following log line is generated. The headers are appended at the end of the standard log line. Any delimiters, such as newline and carriage return, are converted to their ASCII numbers and can be used later for header splitting purposes. Additionally, the empty line that separates the headers from the body is also logged.

Log Length and Storage Capacity

By default, HAProxy truncates log lines after 1024 characters. However, you can adjust this setting by adding a len option (more details below) to the log directive in the global section of the configuration. This allows HAProxy to send a larger amount of data to syslog.

The handling of messages in syslog is defined by RFC 5424. According to section 6.1 of the RFC, the recommended minimum size any log server should be able to process is 480 bytes, with a recommended size of 2048 bytes. There is no maximum value defined in this RFC. Furthermore, its predecessor RFC 3164, which has been the official one for a decade, strictly forbids any client to send packets larger than 1024 bytes (section 4.1), which has set a hard-coded limit in many implementations to this exact value. Therefore, it is crucial that the software implementation of your logging chain (from HAProxy to the syslog daemon and all the processing tools until the storage) supports larger log line sizes.

Additionally, each log line must fit into a UDP packet, which means the maximum length is slightly less than 64k due to protocol overheads.

This setup has limitations and can consume a significant amount of storage capacity. The size of HTTP request headers can rapidly increase, depending on the type of information being exchanged between the client and server. It is not uncommon to encounter requests with 2 or 3 kilobytes of headers. For larger websites, at the time of writing, the accumulated log data for headers could reach several hundred gigabytes or more each day. This factor should be carefully considered when implementing HTTP header logging to ensure appropriate storage resources are available.

To address this, we can send header logs to remote storage. This approach allows us to efficiently manage our header logs without bogging down local systems with the headers generated during HTTP transactions. Offloading header logs to a dedicated remote storage not only alleviates system resources but streamlines header analysis. We outline how to send header logs to remote storage below.

Using Lua to Send Header Logs to Remote Storage

Lua is a versatile language that allows you to extend the functionality of your load balancer (you can refer to our blog post on Lua and its new event framework in HAProxy for more information). In Lua, you can access almost all sample fetches using the f:FETCH_NAME() function. Functions with a dot in their name can be accessed by replacing the dot with an underscore (i.e., req.hdrs becomes req_hdrs).

The provided Lua script serves as an example and performs a few actions. First, it creates a dump_headers function that gathers the headers as a string using txn.f:req_hdrs(), which is equivalent to the req.hdrs() fetch sample. Then, it uses the HTTP client to send the headers to an application on a remote server for offloaded processing. The dump_headers function is registered as an action for http-request rules, enabling its usage in a proxy definition at a later stage.

​​Now, to configure a proxy that invokes the dump_headers action to send the headers to the remote storage application. To log the client's original headers accurately, we recommend that you invoke dump_headers as early as possible in the configuration. This is because dump_headers internally relies on res.hdrs to access the request headers.

On the remote machine that processes the logs, we require a basic web service. While it is possible to use an existing one, for the purpose of this blog post we will create our own minimalistic server using Python due to its simplicity. The Lua script expects the server to receive a POST request. The provided Python script serves as an example, but it is crucial to note that you should not use this example in production environments. If you require such a service, we recommend creating your own implementation with proper resilience and security measures in place, or use an off-the-shelf solution like Logstash, Fluentd, or Loki, although often these tools expect to receive a JSON message.

You can modify this example to log the server's responses as well, but we won’t cover that in these instructions. Keep in mind that Lua has access to only tune.bufsize bytes (default is 16k) for each request and response. Attempting to log the body would not be a wise idea, as no more than tune.bufsize of it are accessible before processing the request, and it may quickly overflow the log server's maximum message size.

Relate HAProxy Logs to Lua with Unique IDs

With the current setup, we have two separate log events in two different files for each request: one from HAProxy and another from the Python application responsible for storing headers. However, there is no direct way to correlate lines from these two files. Thankfully, HAProxy provides a solution: the unique-id-format directive creates a unique identifier for each request. By including this unique ID in both the HAProxy logs and the header dump, we can now effectively correlate the information between the two sources.

Use Asynchronous Operation to Queue Request Headers Efficiently

In the previous example, if the remote HTTP server is unavailable or becomes overwhelmed by the large volume of data to process, the request is blocked until a timeout is triggered. This can result in the client experiencing latency, making the service less performant. 

However, the new Lua queuing framework introduced in HAProxy 2.8 enables you to place the data in a queue to be processed asynchronously by a worker. In this scenario, the Lua action is highly efficient as its primary purpose is to swiftly enqueue the request headers, allowing for near-instantaneous processing of the request.

Each queued item in memory requires approximately 1 kilobyte of RAM for a 1 kilobyte header block. In this example, the queue size is limited to 10,000 items, which corresponds to about 5MB of memory for an average header block size of 3 to 5 kilobytes. This restriction prevents HAProxy from exhausting its allocated memory from tune.lua.maxmem. 

If the remote server experiences a prolonged failure and the queue reaches its maximum capacity, incoming headers will not be added to the queue. Instead, their logging will be silently dropped until space becomes available by removing previous items from the queue.

To use dump_headers_async, we need to modify the existing lua-log-id proxy to invoke this function instead of dump_headers. We can simply create a new proxy that allows us to compare both approaches side-by-side.

Now, even if the external application responsible for storing headers is experiencing issues or is slowed down, the main proxy can still process requests at maximum performance levels.

With the ability to log requests headers asynchronously, toggling this feature still requires commenting or uncommenting the http-request lua.dump_headers_async line. Reloading HAProxy is necessary after making this change. 

Conditional Logging by Client Source IP

So far, the example configuration logs headers for all clients, which may not be very convenient or efficient. To address this, you can change the lua.dump_headers_async call to fire only when an incoming request's IP address matches an entry declared in a map file. If the client's source IP is found in the map, its headers will be logged. Otherwise, no action will be taken for that client. This conditional approach allows for more control over which clients' headers are logged, making the logging process more efficient and customizable.

To implement this feature, the http-request lua.dump_headers_async line needs to be modified to http-request lua.dump_headers_async if { src -m ip -M -f headers.map }. This modification allows the lua-log-async-conditional to conditionally log headers based on the presence of the client's source IP in the headers.map file.

To activate logging in HAProxy, you can use the Runtime API:

To deactivate logging for that IP:

Instead of specifying a single address, you can define a full range of IP addresses by using CIDR notation. For example, use 172.29.1.0/24 instead of 172.29.1.14.

Similar Articles:

• How to Create Powerful, Customized Lua Mailers in HAProxy

Activate logging on demand with a dedicated proxy

You have the option to use a dedicated proxy to manage the logging feature. In the next example, the proxy named debug-status-manager is specifically designed for this purpose.

First, it checks the client's source address to ensure that it originates from a trusted source, which in this example is the range 172.16.29.0/24. Since the source IP can be checked when the connection is established, it is more efficient to reject unauthorized clients using a tcp-request connection rule, especially if the manager operates over HTTPS (although this is not the case in this example). This TCP rule would be executed before any TLS handshake takes place.

Second, if the request path is not /debug-on or /debug-off, or if the clientip parameter is not found, the request will be rejected.

Finally, the content of the headers.map file is modified based on the path used to enable or disable the debug feature. These modifications are done in memory only, which means the actual file on the filesystem remains unchanged. If the proxy is reloaded, the headers.map file will be reset to its original state.

Now, you can use the curl command to control the behavior of the proxy:

Conclusion

HAProxy logs are very powerful and easy to configure to add some debugging information. However, since the initial syslog implementations were limited to 1024 bytes, it is clearly not designed to handle large HTTP request headers.

With Lua and the new HAProxy Lua queuing framework, new possibilities arise for adding custom debugging functionalities. We can harness the power and flexibility of HAProxy to create a feature that enables the logging of debugging information on demand.

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.