Alerts and monitoring

Statistics dashboard

The HAProxy Statistics dashboard provides a near real-time feed of information that you can use to troubleshoot proxied services, get insights about your traffic, monitor the load on servers, and even perform administrative actions such as placing a server in maintenance mode or enabling health checks.

Enable the dashboard Jump to heading

HAProxy Fusion

In HAProxy Fusion, the Statistics dashboard is already enabled. Access it via Overview > Stats.

HAProxy ALOHA

In HAProxy ALOHA, statistics also appear on the Stats tab.

To enable the Statistics dashboard, add the following frontend to the load balancer configuration. This will serve the dashboard at http://<IP ADDRESS>:8404/stats.

haproxy
frontend stats
mode http
bind :8404
stats enable
stats refresh 10s
stats uri /stats
stats show-modules
haproxy
frontend stats
mode http
bind :8404
stats enable
stats refresh 10s
stats uri /stats
stats show-modules

In this frontend:

  • bind sets which address and port you’ll use to access the dashboard.
  • stats enable is required to define this frontend as an active statistics dashboard.
  • stats refresh configures how often the dashboard will automatically refresh within your browser.
  • stats uri (optional) changes the URL’s path.
  • stats show-modules Available since HAProxy 2.3 / HAProxy Enterprise 2.3r1 / HAProxy ALOHA 13.0 (optional) adds statistics for SSL, HTTP/1, HTTP/2, HTTP/3, and QUIC modules.

For more information, see stats enable.

Enable administrative actions Jump to heading

From the dashboard, you can perform some administrative actions. When administrative actions are enabled, a dropdown menu appears at the bottom of the page with actions you can apply. To perform an action, select rows, then select the action from the drop-down menu, and then click Apply.

To enable administrative actions, add the stats admin line to the stats frontend:

haproxy
frontend stats
mode http
bind :8404
stats enable
stats refresh 10s
stats uri /stats
stats show-modules
stats admin if TRUE
haproxy
frontend stats
mode http
bind :8404
stats enable
stats refresh 10s
stats uri /stats
stats show-modules
stats admin if TRUE

where:

  • stats admin allows users to perform administrative actions.

You can control who gets administrative privileges:

  • Use an ACL statement to restrict who can access administrative actions. For example, the statement if TRUE gives all users access, but you could also create a condition that checks the client’s source IP address.
  • Specify stats auth <username>:<password> to enforce Basic authentication.

Actions include:

Administrative Action Description
Set state to READY The selected server will receive traffic. The final status of the server will be determined based on its configured health check.
Set state to DRAIN The selected server won’t accept any new connections other than those accepted through session persistence.
Set state to MAINT The selected server won’t accept any new connections at all and health checks will be stopped.
Health: disable checks Disable a server’s health checks.
Health: enable checks Enable a server’s health checks.
Health: force UP Force a server’s health check status immediately to UP.
Health: force NOLB Force a server’s health check status to NOLB. This stops the server from accepting new, non-persistent connections.
Health: force DOWN Force a server’s health check status immediately to DOWN.
Agent: disable checks Disable a server’s agent checks.
Agent: enable checks Enable a server’s agent checks.

What the dashboard displays Jump to heading

The dashboard shows information for the load balancer process, frontends, and backends. At the top of the screen you’ll find general process information, a color legend, display options, and helpful links to external resources.

General process information Jump to heading

The General process information section provides details about the load balancer process itself, along with settings from the global section of the configuration such as global maxconn.

Statistics dashboard process information

Color legend Jump to heading

The color legend shows what colors are used to indicate server state:

Statistics dashboard legend

Display options Jump to heading

The Display options allow you to customize the display.

Statistics display options

where:

Option Function
Scope Display only frontend, backend, and listen sections whose names contain the specified text.
Hide ‘DOWN’ servers Show only UP servers.
Disable refresh Stop refreshing of the page.
Refresh now Trigger a page refresh.
CSV export Export statistics to file as comma-separated-values.
JSON export (schema) Export statistics to file as JSON schema.

External resources Jump to heading

The External resources section provides helpful links.

Statistics external resources

Frontend statistics Jump to heading

A frontend is what a client connects to. As requests enter the load balancer, and as responses are returned to the client, they pass through the frontend. So, it has access to end-to-end timings, message sizes, and health indicators that encompass the whole request/response lifecycle. Below we explore the statistics related to frontends.

Statistic Description
Queue The Queue section is not used for frontends.
Session rate

The Session rate section, in the context of a frontend, describes the rate at which clients are connecting to the load balancer.

The Cur column shows the current rate at which client sessions, or fully established connections between client and server, are being created. If you hover over this field, the page displays the following drilldown metrics:

  • Current connection rate per second: Rate at which clients are connecting to the load balancer (having not yet created full sessions).
  • Current session rate per second: Rate at which sessions, which are entities that hold the state of an end-to-end connection (client to load balancer and load balancer to backend server), are being created.
  • Current request rate per second: Rate at which HTTP requests are being received over established connections.

The Max column shows the most sessions that have ever been in use simultaneously. If you hover over this field, the page displays the following drilldown metrics:

  • Max connection rate per second: Highest rate at which clients have connected to the load balancer (having not yet created full sessions).
  • Max session rate per second: Highest rate at which clients have established sessions, which are entities that hold the state of an end-to-end connection (client to load balancer and load balancer to backend server).
  • Max request rate per second: Highest rate at which HTTP requests have been received over established connections.

The Limit column shows the maximum number of sessions per second that the frontend will accept, as set by the rate-limit sessions setting. If this limit is exceeded, additional connections are kept pending in the socket’s backlog (in system buffers).

Sessions

In the context of a frontend, the Sessions section counts the number of sessions, or full client-to-server connections, in use on the load balancer.

The Cur column shows the current number of established sessions.

The Max column shows the most sessions that have ever been simultaneously established.

The Limit column shows the most simultaneous sessions that are allowed, as defined by the maxconn setting in the frontend. That particular frontend will stop accepting new connections when this limit is reached. If maxconn is not set, then Limit is the same as the maxconn value in the global section of your configuration. If that isn’t set, the value is based on your system.

When you hover over the Total column, the page displays the following drilldown metrics:

  • Cum. connections: Cumulative number of connections established since the load balancer was last reloaded.
  • Cum. sessions: Cumulative number of sessions (end-to-end connections) established since the last reload.
  • Cum. HTTP requests: Cumulative number of HTTP requests since the last reload.
  • HTTP 1xx responses: Total number of HTTP requests that received a 1xx response.
  • HTTP 2xx responses: Total number of HTTP requests that received a 2xx response.
  • Compressed 2xx: Total number of 2xx responses that were compressed, if compression has been enabled. It also shows the percentage of requests that were compressed.
  • HTTP 3xx responses: Total number of HTTP requests that received a 3xx response.
  • HTTP 4xx responses: Total number of HTTP requests that received a 4xx response.
  • HTTP 5xx responses: Total number of HTTP requests that received a 5xx response.
  • Other responses: Total number of HTTP requests that received a response not covered by the other metrics.
  • Intercepted requests: Total number of requests intercepted and redirected to the Stats page.
  • Cache lookups: Total number of times the cache was checked for a resource.
  • Cache hits: Total number of times a cached resource was returned.
  • Failed hdr rewrites: Total number of times that an HTTP header could not be added or set because there was not enough space left in the buffer. Try increasing tune.maxrewrite or tune.bufsize.

The LbTot and Last columns are not used in the Sessions section.

Bytes

In the context of a frontend, the Bytes section displays the cumulative amount of data sent and received between the load balancer and downstream clients.

  • In: Total number of bytes received.
  • Out: Total number of bytes sent.
Denied

The Denied section shows the number of requests and responses that were rejected because of security concerns.

The Req column shows the number of requests that were denied due to any of the following configuration directives placed in the frontend or listen section:

  • http-request deny
  • http-request reject
  • http-request silent-drop
  • http-request tarpit
  • http-response silent-drop
  • tcp-request connection silent-drop
  • tcp-request content reject
  • tcp-request content silent-drop

The Resp column shows the number of responses that were denied by an http-response deny directive that was placed in the frontend or listen.

Errors

In the context of a frontend, only the Req column is used in the Errors section. It shows the number of requests that encountered an error. Possible causes include:

  • early termination from the client
  • a read error from the client
  • the client timed out
  • the client closed the connection
  • the client sent a malformed request
  • the request was tarpitted
Warnings The Warnings section is not used for frontends.
Server

The only field from the Server section that applies to a frontend is the Status field. When Status is OPEN, the frontend is operating normally and ready to receive traffic. You can disable the frontend, which will change its status to STOP, by executing the disable frontend Runtime API command. Use the enable frontend command to change the frontend’s status back to OPEN.

Backend statistics Jump to heading

A backend is a pool of load-balanced servers. The load balancer sends requests to a backend and then receives a response from one of the active servers. Statistics here include information about the health of each server, timings related to queuing, connecting, receiving responses, and request rates.

Statistic Description
Queue

The Queue section applies only to backends and shows how long clients are waiting for a server to become available. The load balancer is able to queue connections when you use the maxconn setting. This reduces strain on your backend servers.

  • Cur: The number of client connections that are currently queued and not yet assigned to a server.
  • Max: The most connections that have ever been queued at once.
  • Limit: The maximum number of connections that are allowed to be queued, as defined by the maxqueue setting on each server line. Setting a maximum limit on how many connections can be queued is one way to ensure that clients don’t stay queued for too long. You can also set timeout queue to set a time limit for how long a client can stay queued before the load balancer returns an error response.
Session rate

In the context of a backend server, a session means the same as a connection. These statistics show the rate at which connections are being made to a server.

  • Cur: The current rate, per second, at which connections are being established to the server.
  • Max: The highest rate at which connections have ever been established to the given server.
  • Limit: Not used.
Sessions

In the context of a backend, the Sessions section shows the number of current connections to any of the active servers.

  • Cur: The number of active connections to a server.
  • Max: The most connections that have ever been simultaneously established to the given server.
  • Limit: The maximum number of connections allowed for a server, as set by the maxconn parameter on a server line. The backend row shows the value of fullconn for Limit or, if not set, it uses the following formula: sum of the Sessions Limit values for frontends that route to this backend, divided by 10. So, if you have two frontends that each have a Sessions Limit of 1000, their sum would be 2000. Divide that by 10 to get the value 200 for the limit for this backend.
  • Total: The cumulative number of connections that have used the given server. When you hover over the Total field, a dropdown displays the following drilldown metrics:
    • Cum. sessions: Cumulative number of connections established to this server.
    • New connections: Number of times a new connection was established instead of reusing an existing one.
    • Reused connections: Number of times an existing connection was reused.
    • Cum. HTTP responses: Cumulative number of HTTP responses received from this server.
    • HTTP 1xx responses: Total number of HTTP 1xx responses from this server.
    • HTTP 2xx responses: Total number of HTTP 2xx responses from this server.
    • HTTP 3xx responses: Total number of HTTP 3xx responses from this server.
    • HTTP 4xx responses: Total number of HTTP 4xx responses from this server.
    • HTTP 5xx responses: Total number of HTTP 5xx responses from this server.
    • Other responses: Total number of HTTP responses not covered by the other metrics.
    • Failed hdr rewrites: Total number of times that an HTTP header could not be added or set because there was not enough space left in the buffer. Try increasing tune.maxrewrite or tune.bufsize.
    • Queue time: The amount of time in milliseconds that a connection stayed queued while waiting for a connection slot to the server, averaged over the last 1024 successful connections.
    • Connect time: The amount of time in milliseconds it took to successfully connect to the server, averaged over the last 1024 successful connections.
    • Response time: The server response time in milliseconds averaged over the last 1024 successful connections.
    • Total time: The total session time in milliseconds averaged over the last 1024 successful connections.
  • LbTot: The total number of times the given server was selected to serve a request. This can be due to normal load balancing or because of a redispatch from a failed server.
  • Last: The time since the last connection was received. At the bottom of this section, in gray, the metrics are summed and reported for the backend overall.
Bytes

In the context of a backend, the Bytes section displays the amount of data sent and received between the load balancer and the upstream server.

  • In: The number of bytes sent to the server.
  • Out: The number bytes received back. The bottom row shows a total for the backend as a whole across all servers.
Denied

The Denied section shows the number of requests and responses that were rejected because of security concerns.

  • Req: Applies only to the backend as a whole. It shows the number of requests that were rejected by any of the following configuration directives present in a backend:
    • http-request deny
    • http-request reject
    • http-request silent-drop
    • http-request tarpit
    • http-response silent-drop
    • tcp-request content reject
    • tcp-request content silent-drop
    • tcp-response content silent-drop
  • Resp: the number of responses that were rejected for any given server due to any of the following configuration directives present in a backend:
    • http-response deny
    • tcp-response content reject

The bottom row shows the sum of these individual server metrics.

Errors

The Errors section shows the number of errors related to communicating with a backend server.

  • Req: Not used.
  • Conn: The number of requests that encountered an error while trying to connect to the server. For example, if you shut down a server while the load balancer had already initiated connection to it, then you would see this number rise. If enabled, the load balancer will use health checks to eventually remove the server from the rotation.
  • Resp: Errors encountered while getting the response. For example, closing the connection with tcp-response content close will increase this count.
Warnings

The Warnings section in a backend displays the number of retries and redispatches.

  • Retr: Total number of times that a connection was retried. This requires that you’ve added a retries directive to your backend.
  • Redis: The number of times that the load balancer failed to establish a connection to a server and redispatched it to another server. This requires that you’ve added an option redispatch directive.
Server

In the context of a backend, the Server section shows various details about the status, health, and weight of each server.

  • Status: Displays whether the server is currently up and for how long. It can display any of the following statuses:
    • UP: The server is reporting as healthy.
    • DOWN: The server is reporting as unhealthy and unable to receive requests.
    • NOLB: You’ve added http-check disable-on-404 to the backend, and the configured URL for health checks has returned an HTTP 404 response.
    • MAINT: The server has been disabled or put into maintenance mode.
    • DRAIN: The server has been put into drain mode.
    • no check: Health checks are not enabled for this server.
  • LastChk: Show a value like L7OK/200 in 1ms. That particular value means that a Layer 7 health check was performed; it returned an HTTP 200 OK response and it did so within 1 millisecond. Or, if you see L4OK in 0ms, that means that the load balancer was able to make a Layer 4 connection to the server.
  • Wght: The proportion of traffic it will accept, as set by the weight parameter on the server line.
  • Act: Shows whether the server is active (marked with a Y) or a backup (marked with a -).
  • Bck: Shows whether the server is a backup (marked with a Y) or active (marked with a -).
  • Chk: The number of failed health checks.
  • Dwn: The number of transitions from UP to DOWN.
  • Dwntme: How long the server has been down.
  • Thrtle: The current throttle percentage. If you’ve added a slowstart parameter to a server line, then when you disable and later enable that server, the throttle value indicates the percentage of traffic the server will accept. The percentage will gradually rise to 100% over the period you’ve set.

See also Jump to heading

Do you have any suggestions on how we can improve the content of this page?