Load balancer statistics

The Statistics dashboard provides a near real-time feed of information that you can use to troubleshoot proxied services, get insights about your traffic, monitor the load on servers, and even perform administrative actions such as placing a server in maintenance mode or enabling health checks.

Enable the dashboard Jump to heading #

To enable the Statistics dashboard, add the following frontend to the load balancer configuration:

haproxy
frontend stats
  mode http
  bind *:8404
  stats enable
  stats uri /stats
  stats refresh 10s
  stats admin if LOCALHOST

haproxy
frontend stats
  mode http
  bind *:8404
  stats enable
  stats uri /stats
  stats refresh 10s
  stats admin if LOCALHOST

In this frontend:

• The bind line sets which address and port you’ll use to access the dashboard.
• The stats enable line is required to define this frontend as an active statistics dashboard.
• The optional stats uri line changes the URL’s path.
• The stats refresh line configures how often the dashboard will automatically refresh within your browser.
• The optional stats admin line allows users from LOCALHOST to perform administrative actions.

To add columns for SSL, HTTP/1, HTTP/2, HTTP/3, and QUIC modules, add the stats show-modules directive.

For more information, see stats enable.

Access the dashboard Jump to heading #

Access the dashboard at http://<IP ADDRESS>:8404/stats. Example for HAProxy ALOHA:

The dashboard is the same for HAProxy Enterprise except for the uppermost line providing details for the executable name, version, and release date.

Administrative actions Jump to heading #

To enable administrative actions, add the stats admin line. You can control who gets administrative privileges:

• Use an ACL statement to restrict who can access administrative actions.
• Specify stats admin if TRUE to allow all users to perform administrative actions.
• Specify stats auth <username>:<password> to enforce Basic authentication.

To perform an action, select servers or an entire backend, then select the action from the drop-down menu, and then click Apply. Example for HAProxy Enterprise:

General process information Jump to heading #

The General process information section provides details about the load balancer process itself. Note that it includes the global maxconn value, which is a key performance parameter because it controls how much traffic is sent to the servers. When the number of connections exceeds maxconn, the load balancer queues the requests. Keep in mind that maxconn can be changed for each frontend.

The color legend shows what colors are used to indicate server state:

The Display options allow you to customize the display.

The External resources section provides links to HAProxy Technologies, the product update site, and documentation.

Metrics Jump to heading #

The metrics cover the health of your servers, current request rates, response times, and more. These metrics give you granular data on a per-frontend, backend, and server basis.

A summary of the metrics follows:

Frontend statistics Jump to heading #

A frontend is what a client connects to. As requests enter the load balancer, and as responses are returned to the client, they pass through the frontend. So, it has access to end-to-end timings, message sizes, and health indicators that encompass the whole request/response lifecycle. Below we explore the statistics related to frontends.

Queue Jump to heading #

The Queue section is not used for frontends.

Session rate Jump to heading #

The Session rate section, in the context of a frontend, describes the rate at which clients are connecting to the load balancer.

• The Cur column shows the current rate at which client sessions, or fully established connections between client and server, are being created. If you hover over this field, the page displays the following drilldown metrics:

  Metric	What it means
  Current connection rate per second	Rate at which clients are connecting to the load balancer (having not yet created full sessions).
  Current session rate per second	Rate at which sessions, which are entities that hold the state of an end-to-end connection (client to load balancer and load balancer to backend server), are being created.
  Current request rate per second	Rate at which HTTP requests are being received over established connections.

• The Max column shows the most sessions that have ever been in use simultaneously. If you hover over this field, the page displays the following drilldown metrics:

  Metric	What it means
  Max connection rate per second	Highest rate at which clients have connected to the load balancer (having not yet created full sessions).
  Max session rate per second	Highest rate at which clients have established sessions, which are entities that hold the state of an end-to-end connection (client to load balancer and load balancer to backend server).
  Max request rate per second	Highest rate at which HTTP requests have been received over established connections.

• The Limit column shows the maximum number of sessions per second that the frontend will accept, as set by the rate-limit sessions setting. If this limit is exceeded, additional connections are kept pending in the socket’s backlog (in system buffers).

Sessions Jump to heading #

In the context of a frontend, the Sessions section counts the number of sessions, or full client-to-server connections, in use on the load balancer.

• The Cur column shows the current number of established sessions. The Max column shows the most sessions that have ever been simultaneously established.

• The Limit column shows the most simultaneous sessions that are allowed, as defined by the maxconn setting in the frontend. That particular frontend will stop accepting new connections when this limit is reached. If maxconn is not set, then Limit is the same as the maxconn value in the global section of your configuration. If that isn’t set, the value is based on your system.

• When you hover over the Total column, the page displays the following drilldown metrics:

  Metric	What it means
  Cum. connections	Cumulative number of connections established since the load balancer was last reloaded.
  Cum. sessions	Cumulative number of sessions (end-to-end connections) established since the last reload.
  Cum. HTTP requests	Cumulative number of HTTP requests since the last reload.
  HTTP 1xx responses	Total number of HTTP requests that received a 1xx response.
  HTTP 2xx responses	Total number of HTTP requests that received a 2xx response.
  Compressed 2xx	Total number of 2xx responses that were compressed, if compression has been enabled. It also shows the percentage of requests that were compressed.
  HTTP 3xx responses	Total number of HTTP requests that received a 3xx response.
  HTTP 4xx responses	Total number of HTTP requests that received a 4xx response.
  HTTP 5xx responses	Total number of HTTP requests that received a 5xx response.
  Other responses	Total number of HTTP requests that received a response not covered by the other metrics.
  Intercepted requests	Total number of requests intercepted and redirected to the Stats page.
  Cache lookups	Total number of times the cache was checked for a resource.
  Cache hits	Total number of times a cached resource was returned.
  Failed hdr rewrites	Total number of times that an HTTP header could not be added or set because there was not enough space left in the buffer. Try increasing tune.maxrewrite or tune.bufsize.

• The LbTot and Last columns are not used in the Sessions section.

Bytes Jump to heading #

In the context of a frontend, the Bytes section displays the cumulative amount of data sent and received between the load balancer and downstream clients.

• The In column shows the total number of bytes received.
• The Out column shows the total number of bytes sent.

Denied Jump to heading #

The Denied section shows the number of requests and responses that were rejected because of security concerns.

• The Req column shows the number of requests that were denied due to any of the following configuration directives placed in the frontend or listen section:

  • http-request deny
  • http-request reject
  • http-request silent-drop
  • http-request tarpit
  • http-response silent-drop
  • tcp-request connection silent-drop
  • tcp-request content reject
  • tcp-request content silent-drop

• The Resp column shows the number of responses that were denied by an http-response deny directive that was placed in the frontend or listen.

Errors Jump to heading #

In the context of a frontend, only the Req column is used in the Errors section. It shows the number of requests that encountered an error.

Possible causes include:

• early termination from the client
• a read error from the client
• the client timed out
• the client closed the connection
• the client sent a malformed request
• the request was tar pitted

Warnings Jump to heading #

The Warnings section is not used for frontends.

Server Jump to heading #

The only field from the Server section that applies to a frontend is the Status field.

When Status is OPEN, the frontend is operating normally and ready to receive traffic. You can disable the frontend, which will change its status to STOP, by executing the disable frontend Runtime API command. Here’s an example of calling that command:

nix
echo "disable frontend www" | \
  sudo socat stdio tcp4-connect:127.0.0.1:9999

nix
echo "disable frontend www" | \
  sudo socat stdio tcp4-connect:127.0.0.1:9999

Use the enable frontend command, which has the same syntax, to change the frontend’s status back to OPEN.

Backend Statistics Jump to heading #

A backend is a pool of load-balanced servers. The load balancer sends requests to a backend and then receives a response from one of the active servers. Statistics here include information about the health of each server, timings related to queuing, connecting, receiving responses, and request rates.

Queue Jump to heading #

The Queue section applies only to backends and shows how long clients are waiting for a server to become available. The load balancer is able to queue connections when you use the maxconn setting. This reduces strain on your backend servers.

• The Cur column shows the number of client connections that are currently queued and not yet assigned to a server.
• The Max column shows the most connections that have ever been queued at once.
• The Limit column shows the maximum number of connections that are allowed to be queued, as defined by the maxqueue setting on each server line. Setting a maximum limit on how many connections can be queued is one way to ensure that clients don’t stay queued for too long. You can also set timeout queue to set a time limit for how long a client can stay queued before the load balancer returns an error response.

Session rate Jump to heading #

In the context of a backend server, a session means the same as a connection. These statistics show the rate at which connections are being made to a server.

• The Cur column shows the current rate, per second, at which connections are being established to the server.
• The Max column shows the highest rate at which connections have ever been established to the given server.
• The Limit column is not used.

Sessions Jump to heading #

In the context of a backend, the Sessions section shows the number of current connections to any of the active servers.

• The Cur column lists the number of active connections to a server.

• The Max column shows the most connections that have ever been simultaneously established to the given server.

• The Limit column shows the maximum number of connections allowed for a server, as set by the maxconn parameter on a server line.

  The backend row shows the value of fullconn for Limit or, if not set, it uses the following formula: sum of the Sessions Limit values for frontends that route to this backend, divided by 10. So, if you have two frontends that each have a Sessions Limit of 1000, their sum would be 2000. Divide that by 10 to get the value 200 for the limit for this backend.

• The Total column shows the cumulative number of connections that have used the given server. When you hover over this field, a dropdown displays the following drilldown metrics:

  Metric	What it means
  Cum. sessions	Cumulative number of connections established to this server.
  New connections	Number of times a new connection was established instead of reusing an existing one.
  Reused connections	Number of times an existing connection was reused.
  Cum. HTTP responses	Cumulative number of HTTP responses received from this server.
  HTTP 1xx responses	Total number of HTTP 1xx responses from this server.
  HTTP 2xx responses	Total number of HTTP 2xx responses from this server.
  HTTP 3xx responses	Total number of HTTP 3xx responses from this server.
  HTTP 4xx responses	Total number of HTTP 4xx responses from this server.
  HTTP 5xx responses	Total number of HTTP 5xx responses from this server.
  Other responses	Total number of HTTP responses not covered by the other metrics.
  Failed hdr rewrites	Total number of times that an HTTP header could not be added or set because there was not enough space left in the buffer. Try increasing tune.maxrewrite or tune.bufsize.
  Queue time	The amount of time in milliseconds that a connection stayed queued while waiting for a connection slot to the server, averaged over the last 1024 successful connections.
  Connect time	The amount of time in milliseconds it took to successfully connect to the server, averaged over the last 1024 successful connections.
  Response time	The server response time in milliseconds averaged over the last 1024 successful connections.
  Total time	The total session time in milliseconds averaged over the last 1024 successful connections.

• The LbTot column shows the total number of times the given server was selected to serve a request. This can be due to normal load balancing or because of a redispatch from a failed server.

• The Last column shows the time since the last connection was received. At the bottom of this section, in gray, the metrics are summed and reported for the backend overall.

Bytes Jump to heading #

In the context of a backend, the Bytes section displays the amount of data sent and received between the load balancer and the upstream server.

• The In column shows the number of bytes sent to the server.
• The Out column shows the number bytes received back. The bottom row shows a total for the backend as a whole across all servers.

Denied Jump to heading #

The Denied section shows the number of requests and responses that were rejected because of security concerns.

• The Req column only applies to the backend as a whole. It shows the number of requests that were rejected by any of the following configuration directives present in a backend:

  • http-request deny
  • http-request reject
  • http-request silent-drop
  • http-request tarpit
  • http-response silent-drop
  • tcp-request content reject
  • tcp-request content silent-drop
  • tcp-response content silent-drop

• The Resp column shows the number of responses that were rejected for any given server due to any of the following configuration directives present in a backend:

  • http-response deny
  • tcp-response content reject

The bottom row shows the sum of these individual server metrics.

Errors Jump to heading #

The Errors section shows the number of errors related to communicating with a backend server.

• The Req column is not used.
• The Conn column shows the number of requests that encountered an error while trying to connect to the server. For example, if you shut down a server while the load balancer had already initiated connection to it, then you would see this number rise. If enabled, the load balancer will use health checks to eventually remove the server from the rotation.
• The Resp column shows errors encountered while getting the response. For example, closing the connection with tcp-response content close will increase this count.

Warnings Jump to heading #

The Warnings section in a backend displays the number of retries and redispatches.

• The Retr column shows the total number of times that a connection was retried. This requires that you’ve added a retries directive to your backend.
• The Redis column shows the number of times that the load balancer failed to establish a connection to a server and redispatched it to another server. This requires that you’ve added an option redispatch directive.

Server Jump to heading #

In the context of a backend, the Server section shows various details about the status, health, and weight of each server.

• The Status column displays whether the server is currently up and for how long. It can display any of the following statuses:

  Status	What it means
  UP	The server is reporting as healthy.
  DOWN	The server is reporting as unhealthy and unable to receive requests.
  NOLB	You’ve added http-check disable-on-404 to the backend, and the configured URL for health checks has returned an HTTP 404 response.
  MAINT	The server has been disabled or put into maintenance mode.
  DRAIN	The server has been put into drain mode.
  no check	Health checks are not enabled for this server.

• The LastChk columns show a value like L7OK/200 in 1ms. That particular value means that a Layer 7 health check was performed; it returned an HTTP 200 OK response and it did so within 1 millisecond. Or, if you see L4OK in 0ms, that means that the load balancer was able to make a Layer 4 connection to the server.

• The Wght column shows the proportion of traffic it will accept, as set by the weight parameter on the server line.

• The Act column shows whether the server is active (marked with a Y) or a backup (marked with a -).

• The Bck column shows whether the server is a backup (marked with a Y) or active (marked with a -).

• The Chk column shows the number of failed health checks.

• The Dwn column shows the number of transitions from UP to DOWN.

• The Dwntme column shows how long the server has been down.

• The Thrtle column shows the current throttle percentage. If you’ve added a slowstart parameter to a server line, then when you disable and later enable that server, the throttle value indicates the percentage of traffic the server will accept. The percentage will gradually rise to 100% over the period you’ve set.

See also Jump to heading #

• Backend core concepts
• disable frontend
• enable frontend
• Frontend core concepts
• Health checks
• maxconn reference
• maxqueue reference
• option redispatch reference
• rate-limit sessions reference
• slowstart reference
• stats admin reference
• stats auth reference
• stats enable reference
• stats show-modules reference
• stats uri reference
• timeout queue reference