How To Identify Requests as Part of an End-To-End Tracing Strategy

Tracing follows requests as they move through an entire network, from the initial client request to the final response. In financial services, end-to-end tracing is essential for maintaining robust security, ensuring comprehensive observability of system operations, and understanding chains of events in case of issues or anomalies. With the right tracing implementation, financial services can ensure secure data transfers, adhere to regulatory requirements, and maintain a detailed record of requests moving through a network.

HAProxy Enterprise can be used to uniquely identify every request as part of a greater tracing strategy. Since HAProxy Enterprise sits before your applications, it’s the logical starting point to create a unique ID that can be traced as the request moves through your network.

To accomplish this, we’ll need a unique identifier (what we will refer to as a correlation ID) added to three places within your system—the request, the load balancer logs, and the response back to the client. We’ll also add a JA3 fingerprint to our logs to identify the requestor more accurately than an IP address.

Let’s get started.

Step one: Adding a correlation ID to the system

First, we will add the correlation ID to the system in the request, log, and response to ensure the unique identifier is carried throughout the entire journey in the backend. This setup makes it easier to trace its path through your network.

Add to the request to your backend so that your backend can process it:

In HAProxy, the unique ID (UUID) is generated using the unique-id-format directive. This directive defines a template for generating unique IDs and can utilize fetch methods to retrieve specific data. The uuid fetch method, in this case, creates a random identifier for each request (if the system is expected to be starved for entropy and there is a higher risk of this producing a duplicate other fields can be added; in the majority of cases a random uuid is also universally unique).

Next, we want to add the same ID to the response back to the client:

Finally, we want to add the ID to load balancer logs using %ID in the log-format:

Step two: Implementing client fingerprinting

Client fingerprints in JA3 format are often useful for managing bots, but in this circumstance, we are logging client fingerprints to correlate requests more accurately.

For example, if multiple requests share the same JA3 fingerprint, it may indicate that they come from the same type of client. This is useful for identifying classes of client software (such as a Python client) and detecting potentially malicious behavior. Capturing the User-Agent string alongside the JA3 fingerprint can provide additional context to further identify bots or suspicious behavior. However, it’s important to note that JA3 fingerprints alone are not reliable for identifying returning clients unless the fingerprint is unique or rare, which could be the case for certain bots.

First, ensure you have the Fingerprint module installed (refer to our documentation). Use the following command:

Next, edit your HAProxy Enterprise configuration file, /etc/hapee-2.9/hapee-lb.cfg. In the global section, add the module-path and module-load directives:

Now set up the logging of the client fingerprint using the JA3 hash. By adding txn the variable becomes available to a single request and response:

If you’re using HAProxy Fusion Control Plane, you can show the fingerprint in the Request Explorer in HAProxy Fusion:

Step three: Logging the correlation ID with the client fingerprint

Now, we will configure HAProxy Enterprise to log both the correlation ID and the client fingerprint together. This setup helps identify the type of client software and allows the backend application to track the processing of individual requests. However, comprehensive request tracking across multiple requests requires additional steps on the backend to associate and relay the correlation ID through all relevant function calls and subsequent service requests.

HAProxy Enterprise:

HAProxy Fusion:

By combining all of these elements, here is what your final configuration should look like to uniquely identify requests as part of your end-to-end tracing strategy:

Here’s how the fingerprint and correlation ID are displayed in HAProxy Fusion’s Request Explorer:

You can see the ID returning in the response header in the browser’s Dev Tools:

Conclusion

Configuring HAProxy Enterprise to log client fingerprints alongside correlation IDs is a first step toward achieving end-to-end tracing. However, setting the correlation ID within HAProxy’s logs does not immediately provide full tracing benefits.

To fully leverage this setup, client applications should use the correlation ID to trace their own function calls and forward them to any service they interact with. Furthermore, integrating tools like OpenTelemetry can further enhance your tracing capabilities.

By taking additional steps, financial services can improve their ability to detect and mitigate fraudulent activities, enhancing their overall security posture.

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.