![How To Identify Requests as Part of an End-To-End Tracing Strategy](https://cdn.haproxy.com/img/containers/partner_integrations/how-financial-services-can-uniquely-identify-requests-as-part-of-an-end-to-end-tracing-strategy-1724175865.png/fcad2b63ecd7e3458148f21a19e964bc/how-financial-services-can-uniquely-identify-requests-as-part-of-an-end-to-end-tracing-strategy-1724175865.webp)
Tracing follows requests as they move through an entire network, from the initial client request to the final response. In financial services, end-to-end tracing is essential for maintaining robust security, ensuring comprehensive observability of system operations, and understanding chains of events in case of issues or anomalies. With the right tracing implementation, financial services can ensure secure data transfers, adhere to regulatory requirements, and maintain a detailed record of requests moving through a network.
HAProxy Enterprise can be used to uniquely identify every request as part of a greater tracing strategy. Since HAProxy Enterprise sits before your applications, it’s the logical starting point to create a unique ID that can be traced as the request moves through your network.
To accomplish this, we’ll need a unique identifier (what we will refer to as a correlation ID) added to three places within your system—the request, the load balancer logs, and the response back to the client. We’ll also add a JA3 fingerprint to our logs to identify the requestor more accurately than an IP address.
Let’s get started.
Step one: Adding a correlation ID to the system
First, we will add the correlation ID to the system in the request, log, and response to ensure the unique identifier is carried throughout the entire journey in the backend. This setup makes it easier to trace its path through your network.
Add to the request to your backend so that your backend can process it:
unique-id-format "%[uuid]" | |
unique-id-header X-Correlation-ID |
In HAProxy, the unique ID (UUID) is generated using the unique-id-format
directive. This directive defines a template for generating unique IDs and can utilize fetch methods to retrieve specific data. The uuid
fetch method, in this case, creates a random identifier for each request (if the system is expected to be starved for entropy and there is a higher risk of this producing a duplicate other fields can be added; in the majority of cases a random uuid is also universally unique).
Next, we want to add the same ID to the response back to the client:
http-after-response set-header X-Correlation-ID %[unique-id] |
Finally, we want to add the ID to load balancer logs using %ID
in the log-format:
log-format "$HAPROXY_HTTP_LOG_FMT %[unique-id]" |
Step two: Implementing client fingerprinting
Client fingerprints in JA3 format are often useful for managing bots, but in this circumstance, we are logging client fingerprints to correlate requests more accurately.
HAProxy Enterprise ships with its own HAProxy Enterprise Bot Management Module?
For example, if multiple requests share the same JA3 fingerprint, it may indicate that they come from the same type of client. This is useful for identifying classes of client software (such as a Python client) and detecting potentially malicious behavior. Capturing the User-Agent
string alongside the JA3 fingerprint can provide additional context to further identify bots or suspicious behavior. However, it’s important to note that JA3 fingerprints alone are not reliable for identifying returning clients unless the fingerprint is unique or rare, which could be the case for certain bots.
First, ensure you have the Fingerprint module installed (refer to our documentation). Use the following command:
sudo apt-get install hapee-2.9r1-lb-fingerprint-ssl |
Next, edit your HAProxy Enterprise configuration file, /etc/hapee-2.9/hapee-lb.cfg
. In the global
section, add the module-path
and module-load
directives:
global | |
module-path /opt/hapee-2.9/modules | |
module-load hapee-lb-fingerprint-ssl.so |
Now set up the logging of the client fingerprint using the JA3 hash. By adding txn
the variable becomes available to a single request and response:
frontend mysite | |
http-request set-var(txn.client_fingerprint) req.fingerprint_ssl.ja3_hash(EL_EXCLUDE:41,EL_SORT) | |
log-format "$HAPROXY_HTTP_LOG_FMT %[var(txn.client_fingerprint)]" |
If you’re using HAProxy Fusion Control Plane, you can show the fingerprint in the Request Explorer in HAProxy Fusion:
log-format-sd "%{+Q,+E}o[request@58750 host=%[var(txn.host)] referer=%[var(txn.referer)] user_agent=%[var(txn.user_agent)]][custom@58750 client_fingerprint=%[var(txn.client_fingerprint)]]" |
Step three: Logging the correlation ID with the client fingerprint
Now, we will configure HAProxy Enterprise to log both the correlation ID and the client fingerprint together. This setup helps identify the type of client software and allows the backend application to track the processing of individual requests. However, comprehensive request tracking across multiple requests requires additional steps on the backend to associate and relay the correlation ID through all relevant function calls and subsequent service requests.
HAProxy Enterprise:
log-format "$HAPROXY_HTTP_LOG_FMT %[unique-id] %[var(txn.client_fingerprint)]" |
HAProxy Fusion:
log-format-sd "%{+Q,+E}o[request@58750 host=%[var(txn.host)] referer=%[var(txn.referer)] user_agent=%[var(txn.user_agent)]][custom@58750 correlation_id=%[unique-id] client_fingerprint=%[var(txn.client_fingerprint)]]" |
By combining all of these elements, here is what your final configuration should look like to uniquely identify requests as part of your end-to-end tracing strategy:
unique-id-format "%[uuid]" | |
unique-id-header X-Correlation-ID | |
http-after-response set-header X-Correlation-ID %[unique-id] | |
log-format-sd "%{+Q,+E}o[request@58750 host=%[var(txn.host)] referer=%[var(txn.referer)] user_agent=%[var(txn.user_agent)]][custom@58750 correlation_id=%[unique-id] client_fingerprint=%[var(txn.client_fingerprint)]]" |
Here’s how the fingerprint and correlation ID are displayed in HAProxy Fusion’s Request Explorer:
![correlation-id-in-request-explorer-1724177505](/assets/posts/correlation-id-in-request-explorer-1724177505.png)
You can see the ID returning in the response header in the browser’s Dev Tools:
![id-returning-in-dev's-tools](/assets/posts/id-returning-in-dev's-tools.png)
Conclusion
Configuring HAProxy Enterprise to log client fingerprints alongside correlation IDs is a first step toward achieving end-to-end tracing. However, setting the correlation ID within HAProxy’s logs does not immediately provide full tracing benefits.
To fully leverage this setup, client applications should use the correlation ID to trace their own function calls and forward them to any service they interact with. Furthermore, integrating tools like OpenTelemetry can further enhance your tracing capabilities.
By taking additional steps, financial services can improve their ability to detect and mitigate fraudulent activities, enhancing their overall security posture.
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.