Traces

Use a trace to monitor load balancer events in real time. A trace captures events related to a particular source such as HTTP/1, HTTP/2, HTTP/3 over QUIC, FastCGI, or the Peers protocol. The syntax for defining a trace in the configuration file was added in HAProxy version 2.7, but changed in version 3.1.

As an alternative, you can also use the Runtime API trace command.

Enable tracing Jump to heading #

To define a trace in the configuration file, add a traces section. In that section, add one or more trace directives that configure the traces to start. Below, we start a single h1 trace:

haproxy
traces
  trace h1  sink stdout  level user  event +any  verbosity clean  start now

haproxy
traces
  trace h1  sink stdout  level user  event +any  verbosity clean  start now

Viewing the output of the trace differs depending on the sink value. In this example, it’s sent to stdout.

output
text
Feb 27 22:05:37 node1 hapee-lb[22321]: [00|h1|1|mux_h1.c:2126] sending request headers : 
  [B] [MSG_RQBEFORE, MSG_RPBEFORE] - req=(.fl=0x00001400 .curr_len=0 .body_len=0)  
  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "GET / HTTP/1.1" - h1c=0x55ec8c41cca0(0x80002200) 
  conn=0x55ec8c2640e0(0x00000300) h1s=0x55ec8c26e090(0x00100010) sd=0x55ec8c3dfff0(0x00000001) 
  sc=0x55ec8c2af5f0(0x00000011)
    
Feb 27 22:05:37 node1 hapee-lb[22321]: [00|h1|1|mux_h1.c:2529] H1 request fully xferred : 
  [B] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=0)  
  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - h1c=0x55ec8c41cca0(0x80002200) 
  conn=0x55ec8c2640e0(0x00000300) h1s=0x55ec8c26e090(0x00004010) sd=0x55ec8c3dfff0(0x00000001) 
  sc=0x55ec8c2af5f0(0x00000011)

output
text
Feb 27 22:05:37 node1 hapee-lb[22321]: [00|h1|1|mux_h1.c:2126] sending request headers : 
  [B] [MSG_RQBEFORE, MSG_RPBEFORE] - req=(.fl=0x00001400 .curr_len=0 .body_len=0)  
  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - "GET / HTTP/1.1" - h1c=0x55ec8c41cca0(0x80002200) 
  conn=0x55ec8c2640e0(0x00000300) h1s=0x55ec8c26e090(0x00100010) sd=0x55ec8c3dfff0(0x00000001) 
  sc=0x55ec8c2af5f0(0x00000011)
    
Feb 27 22:05:37 node1 hapee-lb[22321]: [00|h1|1|mux_h1.c:2529] H1 request fully xferred : 
  [B] [MSG_DONE, MSG_RPBEFORE] - req=(.fl=0x00001511 .curr_len=0 .body_len=0)  
  res=(.fl=0x00001404 .curr_len=0 .body_len=0) - h1c=0x55ec8c41cca0(0x80002200) 
  conn=0x55ec8c2640e0(0x00000300) h1s=0x55ec8c26e090(0x00004010) sd=0x55ec8c3dfff0(0x00000001) 
  sc=0x55ec8c2af5f0(0x00000011)

Trace syntax Jump to heading #

Define traces inside a traces section. Within that section, add one or more trace directives. The syntax for a trace directive is:

text
trace <source> [<args...>]
  optional args: 
    * event [+|-|!]<name>|none|any 
    * level <level>
    * verbosity <level>
    * sink <sink>
    * lock <criterion>
    * follow <other_source>
    * { pause | start | stop } [ [+|-|!]event|now]

text
trace <source> [<args...>]
  optional args: 
    * event [+|-|!]<name>|none|any 
    * level <level>
    * verbosity <level>
    * sink <sink>
    * lock <criterion>
    * follow <other_source>
    * { pause | start | stop } [ [+|-|!]event|now]

Defining a trace involves setting several arguments that can have different values depending on the source. The trace directive expects:

• the source: the category of events to capture.
• the events: the events within the category to capture.
• the level: the depth of detail.
• the verbosity: the granularity of details reported.
• the sink: where to send the trace output.
• the lock: criteria to lock onto and follow, such a specific HTTP stream.
• the follow: passively follow another source’s locked pointer, such as session.

To start a trace, add a start argument. The simplest form is start now, which immediately starts the trace. But you can also set an event that once encountered will start the trace. Also, you can set events that will trigger pausing or stopping the trace.

Example:

haproxy
traces
  trace h1  sink stdout  level user  event +any  verbosity clean  start now

haproxy
traces
  trace h1  sink stdout  level user  event +any  verbosity clean  start now

Trace arguments Jump to heading #

In this section, you’ll see which values are available for each argument.

Sources Jump to heading #

A source defines the category of events to capture. The trace directive supports the following sources:

Events Jump to heading #

Use the event argument to define source-specific event reporting.

haproxy
trace event [+|-|!]<name>|none|any  ...

haproxy
trace event [+|-|!]<name>|none|any  ...

Set event any to capture all events, event none to capture none, or enable specific events by prefixing the event name with + or disable an event by prefixing it with - or !. Each source supports a different set of events, described in the following sections.

Levels Jump to heading #

Use the level argument to specify the depth of detail to capture.

haproxy
level <level>

haproxy
level <level>

The level can be one of the following values:

Lock criteria Jump to heading #

Use the lock argument to lock onto one of the listed criteria. This means that the trace will follow a single connection or a single stream, for example, after the initial event triggers.

haproxy
lock criterion

haproxy
lock criterion

Sinks Jump to heading #

Use the sink argument to specify where to send the trace data:

haproxy
sink <sink>

haproxy
sink <sink>

All sources support these sinks:

Examples:

• Specify stdout as the sink for the h1 source:

  haproxy
  traces
    trace h1 sink stdout  start now

  haproxy
  traces
    trace h1 sink stdout  start now

• Use a ring buffer to send output to file /tmp/h1.traces:

  haproxy
  ring buf1
    size 10485760 # 10MB
    format timed
    backing-file /tmp/h1.traces
    
  traces
    trace h1  sink buf1 level user  event +any  verbosity complete start now

  haproxy
  ring buf1
    size 10485760 # 10MB
    format timed
    backing-file /tmp/h1.traces
    
  traces
    trace h1  sink buf1 level user  event +any  verbosity complete start now

• Use a ring buffer to send the output to a UDP syslog server address:

  haproxy
  ring buf1
    format rfc3164
    maxlen 1200
    size 32764
    timeout connect 5s
    timeout server 10s
    server traffic 127.0.0.5:514 # sends over TCP to log-forward
  log-forward traffic
    bind 127.0.0.5:514      # TCP listener
    log 10.0.0.4:514 local0 # to remote server over UDP
    
  traces
    trace h1  sink buf1 level user  event +any  verbosity complete start now

  haproxy
  ring buf1
    format rfc3164
    maxlen 1200
    size 32764
    timeout connect 5s
    timeout server 10s
    server traffic 127.0.0.5:514 # sends over TCP to log-forward
  log-forward traffic
    bind 127.0.0.5:514      # TCP listener
    log 10.0.0.4:514 local0 # to remote server over UDP
    
  traces
    trace h1  sink buf1 level user  event +any  verbosity complete start now

Verbosity Jump to heading #

Use the verbosity argument to set the granularity of detail reported for the trace source:

haproxy
verbosity <level>

haproxy
verbosity <level>

The verbosity available depends on the source you have selected.

See also Jump to heading #

• show events
• show trace
• trace