Debugging

This section describes how to use debugging features.

Trace a source in real time Jump to heading #

Use a trace to monitor load balancer events in real time. A trace captures events related to a particular source such as HTTP, FastCGI, or the Peers protocol.

The trace command is already supported in the Runtime API. Version 2.7 adds experimental support for an analogous trace directive in the load balancer configuration file.

To define a trace in the configuration file, add the expose-experimental-directives directive followed by the trace directive in the global section.

haproxy
global
  expose-experimental-directives
  trace h1 event +any

haproxy
global
  expose-experimental-directives
  trace h1 event +any

To review the events, use journalctl.

nix
sudo journalctl --follow --unit hapee-3.0-lb | head -3

nix
sudo journalctl --follow --unit hapee-3.0-lb | head -3

output
text
-- Logs begin at Thu 2023-02-23 18:51:08 UTC. --
Feb 27 22:05:37 node1 hapee-lb[22321]: [00|h1|1|mux_h1.c:2126] sending request headers : 
  [B] [MSG_RQBEFORE, MSG_RPBEFORE] - req=(.fl=0x00001400 .curr_len=0 .body_len=0)  
  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "GET / HTTP/1.1" - h1c=0x55ec8c41cca0(0x80002200) 
  conn=0x55ec8c2640e0(0x00000300) h1s=0x55ec8c26e090(0x00100010) sd=0x55ec8c3dfff0(0x00000001) 
  sc=0x55ec8c2af5f0(0x00000011)
    
Feb 27 22:05:37 node1 hapee-lb[22321]: [00|h1|1|mux_h1.c:2529] H1 request fully xferred : 
  [B] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=0)  
  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - h1c=0x55ec8c41cca0(0x80002200) 
  conn=0x55ec8c2640e0(0x00000300) h1s=0x55ec8c26e090(0x00004010) sd=0x55ec8c3dfff0(0x00000001) 
  sc=0x55ec8c2af5f0(0x00000011)

output
text
-- Logs begin at Thu 2023-02-23 18:51:08 UTC. --
Feb 27 22:05:37 node1 hapee-lb[22321]: [00|h1|1|mux_h1.c:2126] sending request headers : 
  [B] [MSG_RQBEFORE, MSG_RPBEFORE] - req=(.fl=0x00001400 .curr_len=0 .body_len=0)  
  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "GET / HTTP/1.1" - h1c=0x55ec8c41cca0(0x80002200) 
  conn=0x55ec8c2640e0(0x00000300) h1s=0x55ec8c26e090(0x00100010) sd=0x55ec8c3dfff0(0x00000001) 
  sc=0x55ec8c2af5f0(0x00000011)
    
Feb 27 22:05:37 node1 hapee-lb[22321]: [00|h1|1|mux_h1.c:2529] H1 request fully xferred : 
  [B] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=0)  
  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - h1c=0x55ec8c41cca0(0x80002200) 
  conn=0x55ec8c2640e0(0x00000300) h1s=0x55ec8c26e090(0x00004010) sd=0x55ec8c3dfff0(0x00000001) 
  sc=0x55ec8c2af5f0(0x00000011)

You can also use the show events Runtime API command to display the trace events.

nix
echo "show events buf0" | sudo socat stdio /var/run/hapee-3.0/hapee-lb.sock | head -3

nix
echo "show events buf0" | sudo socat stdio /var/run/hapee-3.0/hapee-lb.sock | head -3

output
text
<0>2021-02-22T16:44:21.014274+00:00 [00|h1|0|mux_h1.c:1459] rcvd H1 request headers : 
  [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x5617388cd310(0x00000000) 
  h1s=0x5617388cd620(0x00000010)
      
<0>2021-02-22T16:44:21.017431+00:00 [00|h1|0|mux_h1.c:1492] H1 request fully rcvd : 
  [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x5617388cd310(0x00000000) 
  h1s=0x5617388cd620(0x00000410)

output
text
<0>2021-02-22T16:44:21.014274+00:00 [00|h1|0|mux_h1.c:1459] rcvd H1 request headers : 
  [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x5617388cd310(0x00000000) 
  h1s=0x5617388cd620(0x00000010)
      
<0>2021-02-22T16:44:21.017431+00:00 [00|h1|0|mux_h1.c:1492] H1 request fully rcvd : 
  [F] [MSG_DONE, MSG_RPBEFORE] - "GET / HTTP/1.1" - h1c=0x5617388cd310(0x00000000) 
  h1s=0x5617388cd620(0x00000410)

To display the trace definition, use the show trace Runtime API command.

nix
echo "show trace" | sudo socat stdio /var/run/hapee-3.0/hapee-lb.sock

nix
echo "show trace" | sudo socat stdio /var/run/hapee-3.0/hapee-lb.sock

output
text
Supported trace sources and states (.=stopped, w=waiting, R=running) :
[.] h2         -> none [drp 0]  [HTTP/2 multiplexer]
[.] fcgi       -> none [drp 0]  [FastCGI multiplexer]
[R] h1         -> stdout [drp 0]  [HTTP/1 multiplexer]
[.] stream     -> none [drp 0]  [Applicative stream]
[.] peers      -> none [drp 0]  [Peers protocol]
[.] check      -> none [drp 0]  [Health-check]
[.] pt         -> none [drp 0]  [Passthrough multiplexer]

output
text
Supported trace sources and states (.=stopped, w=waiting, R=running) :
[.] h2         -> none [drp 0]  [HTTP/2 multiplexer]
[.] fcgi       -> none [drp 0]  [FastCGI multiplexer]
[R] h1         -> stdout [drp 0]  [HTTP/1 multiplexer]
[.] stream     -> none [drp 0]  [Applicative stream]
[.] peers      -> none [drp 0]  [Peers protocol]
[.] check      -> none [drp 0]  [Health-check]
[.] pt         -> none [drp 0]  [Passthrough multiplexer]

Anonymize debug output Jump to heading #

You can anonymize the output of some Runtime API commands that might otherwise reveal sensitive or confidential data such as IP addresses, names of sections, and hostnames.

In anonymization, sensitive fields are replaced with 24-bit hashes that preserve sufficient consistency between elements to help troubleshooters identify relationships between elements.

To anonymize a Runtime API command, precede it with set anon on;.

nix
echo "set anon on; show sess" | sudo socat stdio /var/run/hapee-3.0/hapee-lb.sock

nix
echo "set anon on; show sess" | sudo socat stdio /var/run/hapee-3.0/hapee-lb.sock

The hash key is a 32-bit number between 0 and 4294967295, where 0 disables the anonymizing function. The following methods are supported for setting the hash key:

• Specify a key at each invocation: set anon on <key>;

• Specify a global hash key for all invocations: set anon global-key <key> CLI command or anonkey global configuration directive

• Omit a key and allow HAProxy Enterprise to generate a random key

  nix
  echo "set anon global-key 123456789; set anon on; show sess" | sudo socat stdio /var/run/hapee-3.0/hapee-lb.sock

  nix
  echo "set anon global-key 123456789; set anon on; show sess" | sudo socat stdio /var/run/hapee-3.0/hapee-lb.sock

Anonymize configuration file Jump to heading #

To dump an anonymized copy of the HAProxy Enterprise configuration file with IP addresses, section names, and hostnames masked, use the hapee-lb command with the -dC<key> flag, specifying an arbitrary number as the hash key.

nix
/opt/hapee-3.0/sbin/hapee-lb -dC123456789 -f /etc/hapee-3.0/hapee-lb.cfg

nix
/opt/hapee-3.0/sbin/hapee-lb -dC123456789 -f /etc/hapee-3.0/hapee-lb.cfg

The key generates hashed values in the same way as the set anon CLI command, allowing you to correlate output generated by the two methods when troubleshooting.

See also Jump to heading #

• set anon
• set anon global-key
• show events
• show trace
• trace