Decrypt TLS traffic

When diagnosing network issues, you may need to analyze TLS-encrypted traffic to see the underlying application-layer protocol messages. You can use tcpdump to capture packets and save them to a .pcap, or packet capture file. You can then import such a file to Wireshark for analysis, but you must provide additional information to Wireshark so that it can decipher the traffic. You can enable the logging of TLS keys in HAProxy Enterprise, which you can then import into Wireshark. Wireshark will use these secrets to decipher the encrypted packets in your .pcap file. As of version 3.0, you can produce a keylog file for both traffic between clients and the load balancer and traffic between the load balancer and backend servers.

Decrypt traffic between the load balancer and clients Jump to heading #

To analyze TLS traffic between the load balancer and clients:

1. In your load balancer configuration, set tune.ssl.keylog to on in the global section. This activates the retrieval of the TLS keys you will use for decryption in Wireshark.

2. Force the load balancers and clients to use TLS 1.3 by adding the ssl-min-ver argument to your TLS bind line. TLS 1.3 is required for logging the TLS keys and for allowing you to decrypt the traffic in Wireshark:

   haproxy
   frontend  fe_main
     bind *:443 ssl crt /etc/hapee-3.0/certs/cert.pem ssl-min-ver TLSv1.3

   haproxy
   frontend  fe_main
     bind *:443 ssl crt /etc/hapee-3.0/certs/cert.pem ssl-min-ver TLSv1.3

   Tip

   You can also set the ssl-min-ver globally using the option ssl-default-bind-options. For example:

   haproxy
   global
     ssl-default-bind-options ssl-min-ver TLSv1.3

   haproxy
   global
     ssl-default-bind-options ssl-min-ver TLSv1.3

3. Define a custom log format in your frontend that writes TLS session secrets to the access log. The log format uses sample fetches to retrieve the keys. We are using the frontend fetches here, as indicated by fc in the fetch names:

   haproxy
   frontend fe_main 
     log-format "$HAPROXY_HTTP_LOG_FMT CLIENT_EARLY_TRAFFIC_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_client_early_traffic_secret]\nCLIENT_HANDSHAKE_TRAFFIC_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_client_handshake_traffic_secret]\nSERVER_HANDSHAKE_TRAFFIC_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_server_handshake_traffic_secret]\nCLIENT_TRAFFIC_SECRET_0 %[ssl_fc_client_random,hex] %[ssl_fc_client_traffic_secret_0]\nSERVER_TRAFFIC_SECRET_0 %[ssl_fc_client_random,hex] %[ssl_fc_server_traffic_secret_0]\nEXPORTER_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_exporter_secret]\nEARLY_EXPORTER_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_early_exporter_secret]"

   haproxy
   frontend fe_main 
     log-format "$HAPROXY_HTTP_LOG_FMT CLIENT_EARLY_TRAFFIC_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_client_early_traffic_secret]\nCLIENT_HANDSHAKE_TRAFFIC_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_client_handshake_traffic_secret]\nSERVER_HANDSHAKE_TRAFFIC_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_server_handshake_traffic_secret]\nCLIENT_TRAFFIC_SECRET_0 %[ssl_fc_client_random,hex] %[ssl_fc_client_traffic_secret_0]\nSERVER_TRAFFIC_SECRET_0 %[ssl_fc_client_random,hex] %[ssl_fc_server_traffic_secret_0]\nEXPORTER_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_exporter_secret]\nEARLY_EXPORTER_SECRET %[ssl_fc_client_random,hex] %[ssl_fc_early_exporter_secret]"

4. Reload the load balancer to apply the configuration changes:

   nix
   sudo systemctl reload hapee-3.0-lb

   nix
   sudo systemctl reload hapee-3.0-lb

5. Initiate a packet capture between the load balancer and clients using tcpdump to capture the traffic. For example, to save packets to a .pcap file on the load balancer instance named mycap.pcap, you could use the following command. Note that you may need to change the port and network interface (-i) depending on your settings. The port is the port on which clients make TLS connection to your load balancer.

   nix
   sudo tcpdump -s 0 port 443 -i eth0 -w mycap.pcap

   nix
   sudo tcpdump -s 0 port 443 -i eth0 -w mycap.pcap

   Tip

   You can list your network interfaces using a command such as ifconfig -a or ip link show, depending on your OS.

6. While your capture is running, and after a client connects to the load balancer, the access log will contain lines like this that are the keys for the TLS session:

   text
   CLIENT_EARLY_TRAFFIC_SECRET 0007A9877A21DAAA12156C5230F69D219A95DB00F0595F54E7C87C27AE91E1BA
   CLIENT_HANDSHAKE_TRAFFIC_SECRET 0007A9877A21DAAA12156C5230F69D219A95DB00F0595F54E7C87C27AE91E1BA fa3eb968fcb530d416e33cb25e377038ffbf7b4fb943fcf28b4b283e780e02cdc4171a6c7285f972a26828c6747460a3
   SERVER_HANDSHAKE_TRAFFIC_SECRET 0007A9877A21DAAA12156C5230F69D219A95DB00F0595F54E7C87C27AE91E1BA cc6bbfa6e770685b61fbe86b51863678fbbfc3688d55bf4aaff351553bbcb4788460f8e85048cda257d4e6df547fe6d7
   CLIENT_TRAFFIC_SECRET_0 0007A9877A21DAAA12156C5230F69D219A95DB00F0595F54E7C87C27AE91E1BA 3cb0bb3bc836ef9b3b98984bb7f76a1b0d36b5d28acc94b91c8bde7052b17e112afd83078f39edb3eefc8cdcaac06f21
   SERVER_TRAFFIC_SECRET_0 0007A9877A21DAAA12156C5230F69D219A95DB00F0595F54E7C87C27AE91E1BA c11e1a31231ff2561c4c37d0d82132c263a070fb13c897008a4539cf38f1e3ff27a16d9b73efe2d1dc0c9e5df3fed84e
   EXPORTER_SECRET 0007A9877A21DAAA12156C5230F69D219A95DB00F0595F54E7C87C27AE91E1BA 393e485a78edb1c09be95a335d67fa6b82

   text
   CLIENT_EARLY_TRAFFIC_SECRET 0007A9877A21DAAA12156C5230F69D219A95DB00F0595F54E7C87C27AE91E1BA
   CLIENT_HANDSHAKE_TRAFFIC_SECRET 0007A9877A21DAAA12156C5230F69D219A95DB00F0595F54E7C87C27AE91E1BA fa3eb968fcb530d416e33cb25e377038ffbf7b4fb943fcf28b4b283e780e02cdc4171a6c7285f972a26828c6747460a3
   SERVER_HANDSHAKE_TRAFFIC_SECRET 0007A9877A21DAAA12156C5230F69D219A95DB00F0595F54E7C87C27AE91E1BA cc6bbfa6e770685b61fbe86b51863678fbbfc3688d55bf4aaff351553bbcb4788460f8e85048cda257d4e6df547fe6d7
   CLIENT_TRAFFIC_SECRET_0 0007A9877A21DAAA12156C5230F69D219A95DB00F0595F54E7C87C27AE91E1BA 3cb0bb3bc836ef9b3b98984bb7f76a1b0d36b5d28acc94b91c8bde7052b17e112afd83078f39edb3eefc8cdcaac06f21
   SERVER_TRAFFIC_SECRET_0 0007A9877A21DAAA12156C5230F69D219A95DB00F0595F54E7C87C27AE91E1BA c11e1a31231ff2561c4c37d0d82132c263a070fb13c897008a4539cf38f1e3ff27a16d9b73efe2d1dc0c9e5df3fed84e
   EXPORTER_SECRET 0007A9877A21DAAA12156C5230F69D219A95DB00F0595F54E7C87C27AE91E1BA 393e485a78edb1c09be95a335d67fa6b82

   Note that each TLS session (connection) will generate its own keys.

7. Save the lines from the access logs containing the secrets to a text file. Import the file into Wireshark via Preferences > Protocols > TLS > (Pre)-Master-Secret log filename.

8. Open the .pcap file with your captured traffic in Wireshark to see the deciphered traffic.

Once you have finished troubleshooting:

1. In your load balancer configuration, set tune.ssl.keylog to off in the global section, or delete the line entirely. This disables the logging of TLS keys.

2. Remove the log format line that retrieves the TLS keys from your configuration.

3. Reload the load balancer to apply the configuration changes:

   nix
   sudo systemctl reload hapee-3.0-lb

   nix
   sudo systemctl reload hapee-3.0-lb

Decrypt traffic between the load balancer and backend servers Jump to heading #

To analyze TLS traffic between the load balancer and backend servers:

1. In your load balancer configuration, set tune.ssl.keylog to on in the global section. This activates the retrieval of the TLS keys you will use for decryption in Wireshark.

2. Force the load balancer and the backend servers to use TLS 1.3 by adding the ssl-min-ver argument to the servers. TLS 1.3 is required for logging the TLS keys and for allowing you to decrypt the traffic in Wireshark:

   haproxy
   backend servers
     server s1 192.168.56.50:443 ssl verify required ca-file /etc/haproxy/certs/ca.crt ssl-min-ver TLSv1.3

   haproxy
   backend servers
     server s1 192.168.56.50:443 ssl verify required ca-file /etc/haproxy/certs/ca.crt ssl-min-ver TLSv1.3

   Note that here we have also added verify required to our server line and have provided the CA certificate using ca-file. This enforces a check where the load balancer will verify the server certificate. For more information see verify reference.

   Tip

   You can also set the ssl-min-ver globally using the global option ssl-default-bind-options. For example:

   haproxy
   global
     ssl-default-bind-options ssl-min-ver TLSv1.3

   haproxy
   global
     ssl-default-bind-options ssl-min-ver TLSv1.3

3. Define a custom log format in your frontend that writes TLS session secrets to the access log. The log format uses sample fetches to retrieve the keys. We are using the backend fetches here, as indicated by bc in the fetch names:

   haproxy
   frontend fe_main
     log-format "$HAPROXY_HTTP_LOG_FMT CLIENT_EARLY_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_client_early_traffic_secret]\nCLIENT_HANDSHAKE_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_client_handshake_traffic_secret]\nSERVER_HANDSHAKE_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_server_handshake_traffic_secret]\nCLIENT_TRAFFIC_SECRET_0 %[ssl_bc_client_random,hex] %[ssl_bc_client_traffic_secret_0]\nSERVER_TRAFFIC_SECRET_0 %[ssl_bc_client_random,hex] %[ssl_bc_server_traffic_secret_0]\nEXPORTER_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_exporter_secret]\nEARLY_EXPORTER_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_early_exporter_secret]"

   haproxy
   frontend fe_main
     log-format "$HAPROXY_HTTP_LOG_FMT CLIENT_EARLY_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_client_early_traffic_secret]\nCLIENT_HANDSHAKE_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_client_handshake_traffic_secret]\nSERVER_HANDSHAKE_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_server_handshake_traffic_secret]\nCLIENT_TRAFFIC_SECRET_0 %[ssl_bc_client_random,hex] %[ssl_bc_client_traffic_secret_0]\nSERVER_TRAFFIC_SECRET_0 %[ssl_bc_client_random,hex] %[ssl_bc_server_traffic_secret_0]\nEXPORTER_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_exporter_secret]\nEARLY_EXPORTER_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_early_exporter_secret]"

4. Reload the load balancer to apply the configuration changes:

   nix
   sudo systemctl reload hapee-3.0-lb

   nix
   sudo systemctl reload hapee-3.0-lb

5. Initiate a packet capture between the load balancer and the backend servers using tcpdump to capture the traffic. For example, to save packets to a .pcap file on the load balancer instance named mycap.pcap, you could use the following command. Note that you may need to change the port and network interface (-i) depending on your settings. The port is the port on which your load balancer connects to your backend servers.

   nix
   sudo tcpdump -s 0 port 443 -i eth0 -w mycap.pcap

   nix
   sudo tcpdump -s 0 port 443 -i eth0 -w mycap.pcap

   Tip

   You can list your network interfaces using a command such as ifconfig -a or ip link show, depending on your OS.

6. While your capture is running, and after the load balancer connects to a backend server, the access log will contain lines like this that are the keys for the TLS session:

   text
   CLIENT_EARLY_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E260DEAB346F93CE594153D95E33E4BFD5F80 -
   CLIENT_HANDSHAKE_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E260DEAB346F93CE594153D95E33E4BFD5F80 15ab9abf57145fe49c73d9a617eca9b918d5c4dd455c4bb923c04a936475241facbac21f66bca7c459f5179f753f4afa
   SERVER_HANDSHAKE_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E260DEAB346F93CE594153D95E33E4BFD5F80 09bded135c6b85959d0c2eaf09d177cc4fb9e2d9777cbda5a234d0894ef84b64bbd346cc331a16111d4273d639090d5b
   CLIENT_TRAFFIC_SECRET_0 C030AF8EAEE688F1F3A360E5D53E260DEAB346F93CE594153D95E33E4BFD5F80 155b07c8fcef945cbad456f6b11e216fde42f9ac1cdc8c6eff4bed845caf520a2a490ccba3ae06ffe3d9091904674c41
   SERVER_TRAFFIC_SECRET_0 C030AF8EAEE688F1F3A360E5D53E260DEAB346F93CE594153D95E33E4BFD5F80 18ed3dc1188b7ed1085cbdf41b0f0388b80904f6f21b8962f57cdf460d5694f2b2d99f7055ac44f0e6afefc9e790626b
   EXPORTER_SECRET C030AF8EAEE688F1F3A360E5D53E260DEAB346F93CE594153D95E33E4BFD5F80 9479651fd91e38d549b284ecae7c6430743ae56cc4e8fb899eaf0a4016891d3991b01691c1c4c787d95a10c

   text
   CLIENT_EARLY_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E260DEAB346F93CE594153D95E33E4BFD5F80 -
   CLIENT_HANDSHAKE_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E260DEAB346F93CE594153D95E33E4BFD5F80 15ab9abf57145fe49c73d9a617eca9b918d5c4dd455c4bb923c04a936475241facbac21f66bca7c459f5179f753f4afa
   SERVER_HANDSHAKE_TRAFFIC_SECRET C030AF8EAEE688F1F3A360E5D53E260DEAB346F93CE594153D95E33E4BFD5F80 09bded135c6b85959d0c2eaf09d177cc4fb9e2d9777cbda5a234d0894ef84b64bbd346cc331a16111d4273d639090d5b
   CLIENT_TRAFFIC_SECRET_0 C030AF8EAEE688F1F3A360E5D53E260DEAB346F93CE594153D95E33E4BFD5F80 155b07c8fcef945cbad456f6b11e216fde42f9ac1cdc8c6eff4bed845caf520a2a490ccba3ae06ffe3d9091904674c41
   SERVER_TRAFFIC_SECRET_0 C030AF8EAEE688F1F3A360E5D53E260DEAB346F93CE594153D95E33E4BFD5F80 18ed3dc1188b7ed1085cbdf41b0f0388b80904f6f21b8962f57cdf460d5694f2b2d99f7055ac44f0e6afefc9e790626b
   EXPORTER_SECRET C030AF8EAEE688F1F3A360E5D53E260DEAB346F93CE594153D95E33E4BFD5F80 9479651fd91e38d549b284ecae7c6430743ae56cc4e8fb899eaf0a4016891d3991b01691c1c4c787d95a10c

   Note that each TLS session (connection) will generate its own keys.

7. Save the lines from the access logs containing the secrets to a text file. Import the file into Wireshark via Preferences > Protocols > TLS > (Pre)-Master-Secret log filename.

8. Open the .pcap file with your captured traffic in Wireshark to see the deciphered traffic.

Once you have finished troubleshooting:

1. In your load balancer configuration, set tune.ssl.keylog to off in the global section, or delete the line entirely. This disables the logging of TLS keys.

2. Remove the log format line that retrieves the TLS keys from your configuration.

3. Reload the load balancer to apply the configuration changes:

   nix
   sudo systemctl reload hapee-3.0-lb

   nix
   sudo systemctl reload hapee-3.0-lb

See also Jump to heading #

• tune.ssl.keylog reference
• ssl-default-bind-options reference
• verify reference
• ca-file reference