Global Profiling Engine
Configure real-time aggregation of stick table data
The Global Profiling Engine collects stick table data from all HAProxy Enterprise nodes in the cluster in real time. It then aggregates that data and pushes it back to all of the nodes. For example, if LoadBalancer1 receives two requests and LoadBalancer2 receives three requests, the Global Profiling Engine will sum those numbers to get a total of five, then push that to both LoadBalancer1 and LoadBalancer2. This is helpful for an active/active load balancer configuration wherein the nodes need to share client request information to form an accurate picture of activity across the cluster.
The aggregated data does not overwrite the data on the load balancer nodes. Instead, it is pushed to secondary stick tables that have, for example, a suffix of .aggregate
. You would use a fetch method to retrieve the aggregated data and perform an action, such as rate limiting.
Stick table data is transferred between the HAProxy Enterprise servers and the Global Profiling Engine server by using the peers protocol, a protocol created specifically for this purpose. You must configure which servers should participate, both on the Global Profiling Engine server and on each HAProxy Enterprise node.
Configure HAProxy Enterprise nodes Jump to heading
An HAProxy Enterprise node must be configured to share its stick table data with the Global Profiling Engine server. Once aggregated, the profiling engine sends the data back to each node where it is stored in a new stick table.
Follow these steps on each load balancer:
-
Edit the file
/etc/hapee-3.0/hapee-lb.cfg
.Add a
peers
section.hapee-lb.cfghaproxyglobal[...]# By setting this, you are directing HAProxy Enterprise to use the server line# that specifies this name as the local node.localpeer enterprise1[...]peers mypeers# This is the address and port that the load balancer will receive aggregated data from the GPE serverbind 0.0.0.0:10000# The local HAProxy Enterprise node hostname defined by one of the following:# 1) the value provided when the load balancer process is started with the -L argument# 2) the localpeer name from the global section of the load balancer configuration (suggested method)# 3) the hostname as returned by the system hostname command (default)server enterprise1# The Global Profiling Engine# If you run GPE on the same server, use a different port hereserver gpe 192.168.50.40:10000# stick tables definitionstable request_rates type ip size 100k expire 30s store http_req_rate(10s)table request_rates.aggregate type ip size 100k expire 30s store http_req_rate(10s)hapee-lb.cfghaproxyglobal[...]# By setting this, you are directing HAProxy Enterprise to use the server line# that specifies this name as the local node.localpeer enterprise1[...]peers mypeers# This is the address and port that the load balancer will receive aggregated data from the GPE serverbind 0.0.0.0:10000# The local HAProxy Enterprise node hostname defined by one of the following:# 1) the value provided when the load balancer process is started with the -L argument# 2) the localpeer name from the global section of the load balancer configuration (suggested method)# 3) the hostname as returned by the system hostname command (default)server enterprise1# The Global Profiling Engine# If you run GPE on the same server, use a different port hereserver gpe 192.168.50.40:10000# stick tables definitionstable request_rates type ip size 100k expire 30s store http_req_rate(10s)table request_rates.aggregate type ip size 100k expire 30s store http_req_rate(10s)Inside it:
-
Define a
bind
line to set the IP address and port at which this node should receive data back from the Global Profiling Engine server. In this example, thebind
directive listens on all IP addresses at port 10000 and receives aggregated data. -
Define a
server
line for the current load balancer server. The server name value is important because it must match the name you set in the Global Profiling Engine server’s configuration for the correspondingpeer
line. The hostname may be one of the following, in order of precedence:- the value provided with the
-L
argument specified on the command line used to start the load balancer process - the
localpeer
name specified in theglobal
section of the load balancer configuration (this method is used in this example) - the host name returned by the system
hostname
command. This is the default, but we recommend using one of the other two methods
In this example, the local HAProxy Enterprise node is listed with only its hostname,
enterprise1
. It is not necessary to specify its IP address and port. - the value provided with the
-
Define a
server
line for the Global Profiling Engine server. Set its IP address and port. The name you set here is also important. It must match the correspondingpeer
line in the Global Profiling Engine server’s configuration. -
Define stick tables. For each one, add a duplicate line where the table name has the suffix
.aggregate
. In this example, the non-aggregated stick tablerequest_rates
will store current HTTP request rates. The stick tables record the rate at which clients make requests over 10 seconds. We clear out stale records after 30 seconds by setting theexpire
parameter on the stick table. Thetype
parameter sets the key for the table, which in this case is an IP address. The stick tablerequest_rates.aggregate
receives its data from the Global Profiling Engine. Its suffix,.aggregate
, will match the the profiling engine’s configuration.
localpeer definition
If you receive an error for your load balancer configuration that looks like the following after specifying the name for your load balancer on the
server
line:text[WARNING] (6125) : config : Removing incomplete section 'peers mypeers' (no peer named 'enterprise1')text[WARNING] (6125) : config : Removing incomplete section 'peers mypeers' (no peer named 'enterprise1')Specify your hostname value for
localpeer
in yourglobal
section:hapee-lb.cfghaproxygloballocalpeer enterprise1hapee-lb.cfghaproxygloballocalpeer enterprise1This global setting is required in cases where your hostname (retrieved using the system
hostname
command) is different from your desired peer name. Be sure to update your GPE configuration to use the name you specify as thelocalpeer
name, as well as update your load balancer configuration to reference that name on theserver
line for your load balancer in yourpeers
section). -
-
Add directives to your frontend, backend or listen sections that populate the non-aggregated stick tables with data.
Below, the
http-request track-sc0
line adds request rate information for each client that connects to the load balancer, using the client’s source IP address (src
) as the key in the stick table.hapee-lb.cfghaproxyfrontend fe_mainbind :80default_backend webservers# add records to the stick table using the client's# source IP address as the table keyhttp-request track-sc0 src table mypeers/request_rateshapee-lb.cfghaproxyfrontend fe_mainbind :80default_backend webservers# add records to the stick table using the client's# source IP address as the table keyhttp-request track-sc0 src table mypeers/request_rates -
Add directives that read the aggregated data returned from the Global Profiling Engine server. That data is stored in the table with the suffix
.aggregate
.Below, the
http-request deny
line rejects clients that have a request rate greater than 1000. The client’s request rate is an aggregate amount calculated from all active load balancers. Note that this line reads data from therequest_rates.aggregate
table.hapee-lb.cfghaproxy# perform actions like rate limitinghttp-request deny deny_status 429 if { sc_http_req_rate(0,mypeers/request_rates.aggregate) gt 1000 }hapee-lb.cfghaproxy# perform actions like rate limitinghttp-request deny deny_status 429 if { sc_http_req_rate(0,mypeers/request_rates.aggregate) gt 1000 } -
Restart HAProxy Enterprise.
nixsudo systemctl restart hapee-3.0-lbnixsudo systemctl restart hapee-3.0-lb
Configure the Global Profiling Engine Jump to heading
The Global Profiling Engine server collects stick table data from HAProxy Enterprise load balancers in your cluster, but you must set which load balancers will be allowed to participate by listing them in the configuration file.
Use dynamic configuration Jump to heading
Available since
- HAProxy Enterprise - GPE version 1.0 (
hapee-extras-gpe10
package or newer)
Load balancers can connect to the GPE server without you adding them explicitly to the GPE configuration file. Include the dynamic-peers
directive in either:
- the
aggregations
section to enable it for only that section. - the
global
section to enable it for multipleaggregations
sections.
For example, to set dynamic-peers
in an aggregations
section:
-
On the Global Profiling Engine server, edit the file
/etc/hapee-extras/hapee-gpe-stktagg.cfg
. Add anaagregations
section that includesdynamic-peers
:hapee-gpe-stktagg.cfghaproxyglobal# Enables the Global Profiling Engine APIstats socket /var/run/hapee-extras/gpe-api.sockaggregations data# set how to map non-aggregated to aggregated stick tablesfrom any to .aggregate# the profiling engine listens at this addresspeer gpe 0.0.0.0:10000 local# register load balancer on the flydynamic-peershapee-gpe-stktagg.cfghaproxyglobal# Enables the Global Profiling Engine APIstats socket /var/run/hapee-extras/gpe-api.sockaggregations data# set how to map non-aggregated to aggregated stick tablesfrom any to .aggregate# the profiling engine listens at this addresspeer gpe 0.0.0.0:10000 local# register load balancer on the flydynamic-peers -
Optional: If you have multiple
aggregations
sections, which is useful for serving multiple clusters of load balancers, then you can simplify your setup by setting abind
directive in theglobal
section instead of setting apeer
line with thelocal
keyword in eachaggregations
section. This sets the address at which to listen for incoming stick table data.hapee-gpe-stktagg.cfghaproxyglobal# Enables the Global Profiling Engine APIstats socket /var/run/hapee-extras/gpe-api.sockbind 0.0.0.0:10000aggregations data# set how to map non-aggregated to aggregated stick tablesfrom any to .aggregate# register load balancer on the flydynamic-peershapee-gpe-stktagg.cfghaproxyglobal# Enables the Global Profiling Engine APIstats socket /var/run/hapee-extras/gpe-api.sockbind 0.0.0.0:10000aggregations data# set how to map non-aggregated to aggregated stick tablesfrom any to .aggregate# register load balancer on the flydynamic-peersIf you do this, then on the load balancers the
peer
line for the GPE server must use the same name as theaggregations
section. Here, the name isdata
.hapee-lb.cfghaproxypeers mypeerspeer data 192.168.56.26:10000peer enterprise1 192.168.50.41:10000peer enterprise2 192.168.50.42:10000...hapee-lb.cfghaproxypeers mypeerspeer data 192.168.56.26:10000peer enterprise1 192.168.50.41:10000peer enterprise2 192.168.50.42:10000...
Use static configuration Jump to heading
You can specify the IP address of each load balancer that is allowed to connect:
-
On the Global Profiling Engine server, edit the file
/etc/hapee-extras/hapee-gpe-stktagg.cfg
.In the
aggregations
section, add apeer
line for the Global Profiling Engine itself and for each HAProxy Enterprise node. Each peer’s name (e.g.enterprise1
) should match the name you set in the HAProxy Enterprise configuration, since that is how the profiling engine validates the peer.Peer names
Be sure that the peer names you specify in the GPE server’s configuration match exactly the names you specified in your load balancer configuration. For example, the following load balancer configuration sets the load balancer’s
localpeer
name toenterprise1
and we reference this name again in thepeers
section:hapee-lb.cfghaproxygloballocalpeer enterprise1...peers mypeersbind 0.0.0.0:10000server enterprise1...hapee-lb.cfghaproxygloballocalpeer enterprise1...peers mypeersbind 0.0.0.0:10000server enterprise1...As such, it must appear in the GPE server’s configuration as
enterprise1
in order for GPE to make connection to the load balancer.hapee-gpe-stktagg.cfghaproxyglobal# Enables the Global Profiling Engine APIstats socket /var/run/hapee-extras/gpe-api.sockaggregations data# set how to map non-aggregated to aggregated stick tablesfrom any to .aggregate# the profiling engine listens at this addresspeer gpe 0.0.0.0:10000 local# the load balancers listen at these addressespeer enterprise1 192.168.50.41:10000peer enterprise2 192.168.50.42:10000hapee-gpe-stktagg.cfghaproxyglobal# Enables the Global Profiling Engine APIstats socket /var/run/hapee-extras/gpe-api.sockaggregations data# set how to map non-aggregated to aggregated stick tablesfrom any to .aggregate# the profiling engine listens at this addresspeer gpe 0.0.0.0:10000 local# the load balancers listen at these addressespeer enterprise1 192.168.50.41:10000peer enterprise2 192.168.50.42:10000In this example:
-
The Global Profiling Engine API provides a programmable API, which listens at the socket
/var/run/hapee-extras/gpe-api.sock
. Thestats socket
directive enables a CLI that lets you view data that the aggregator has stored. -
In the
aggregations
section, thefrom
line defines how non-aggregated stick tables map to aggregated stick tables, and what the suffix for the aggregated stick tables should be. The keywordany
means that any stick table found will be aggregated. Aggregated data is pushed to tables with the same name, but ending with the suffix.aggregate
. In the example, the engine expects stick tables to be named likerequest_rates
and it will push aggregated data torequest_rates.aggregate
.You can also use a more specific mapping. In the example below, the engine expects stick tables to be named like
request_rates.nonaggregate
and it will push aggregated data torequest_rates.aggregate
. Stick tables without the.nonaggregate
suffix will be ignored.haproxyfrom .nonaggregate to .aggregatehaproxyfrom .nonaggregate to .aggregate -
The
peer
line with thelocal
argument indicates the local GPE server. -
HAProxy Enterprise
peer
lines must use the same name you set on theserver
line in the HAProxy Enterprise configuration (e.g.enterprise1
), and they must specify the IP addresses and ports where the load balancers are receiving aggregated data.
-
-
Restart the Global Profiling Engine service:
haproxysudo systemctl restart hapee-extras-gpehaproxysudo systemctl restart hapee-extras-gpe
Verify your setup Jump to heading
Check that the Global Profiling Engine and load balancers are setup correctly by utilizing their APIs.
-
On the load balancer, call the Runtime API function
show peers
to check that the Global Profiling Engine is listed and that itslast_status
isESTA
(established):Below, the
show peers
command lists connected peers:nixecho "show peers" | sudo socat stdio unix-connect:/var/run/hapee-3.0/hapee-lb.sock | head -2nixecho "show peers" | sudo socat stdio unix-connect:/var/run/hapee-3.0/hapee-lb.sock | head -2outputtext0x5651d4a03010: [07/Jul/2021:17:02:30] id=mypeers disabled=0 flags=0x2213 resync_timeout=<PAST> task_calls=920x5651d4a06540: id=gpe(remote,active) addr=192.168.50.40:10000 last_status=ESTA last_hdshk=2m17soutputtext0x5651d4a03010: [07/Jul/2021:17:02:30] id=mypeers disabled=0 flags=0x2213 resync_timeout=<PAST> task_calls=920x5651d4a06540: id=gpe(remote,active) addr=192.168.50.40:10000 last_status=ESTA last_hdshk=2m17s -
Call the Runtime API function
show table
to see data in non-aggregated and aggregated stick tables.Below, we view data in the stick table named
request_rates.aggregate
:nixecho "show table mypeers/request_rates.aggregate" | sudo socat stdio unix-connect:/var/run/hapee-3.0/hapee-lb.socknixecho "show table mypeers/request_rates.aggregate" | sudo socat stdio unix-connect:/var/run/hapee-3.0/hapee-lb.sockoutputtext# table: mypeers/request_rates.aggregate, type: ip, size:102400, used:10x7fc0e401fb80: key=192.168.50.1 use=0 exp=28056 http_req_rate(10000)=5outputtext# table: mypeers/request_rates.aggregate, type: ip, size:102400, used:10x7fc0e401fb80: key=192.168.50.1 use=0 exp=28056 http_req_rate(10000)=5 -
On the Global Profiling Engine server, call the
show aggrs
function to see load balancers that are registered as peers. Astate
of0x7
means a successful connection. If you see a state of0xffffffff
, that means that a connection was not successful. Often, this is caused by the peer names not matching between the Global Profiling Engine’s configuration and the HAProxy Enterprise configuration.Below, the
show aggrs
command shows that the peer namedenterprise1
has connected:nixecho "show aggrs" | sudo socat stdio /var/run/hapee-extras/gpe-api.socknixecho "show aggrs" | sudo socat stdio /var/run/hapee-extras/gpe-api.sockoutputtextaggregations datapeer 'enterprise1'(0) sync_ok: 1 accept: 1(last: 6080) connect: 1(last: 16086) state: 0x7 sync_state: 0x3sync_req_cnt: 0 sync_fin_cnt: 0 sync_cfm_cnt: 0outputtextaggregations datapeer 'enterprise1'(0) sync_ok: 1 accept: 1(last: 6080) connect: 1(last: 16086) state: 0x7 sync_state: 0x3sync_req_cnt: 0 sync_fin_cnt: 0 sync_cfm_cnt: 0
Optional: Bind outgoing connections to an interface Jump to heading
If the server where you are running the Global Profiling Engine has multiple network interfaces, you can configure the engine to bind to a specific one for outgoing data sent to HAProxy Enterprise servers.
To bind outgoing connections to a specific address, use the source
directive in the global
section.
IPv4 examples
hapee-gpe-stktagg.cfghaproxy
globalsource 126.123.10.12:12345
hapee-gpe-stktagg.cfghaproxy
globalsource 126.123.10.12:12345
The port is optional. It defaults to 0 for random ports.
hapee-gpe-stktagg.cfghaproxy
globalsource 126.123.10.12
hapee-gpe-stktagg.cfghaproxy
globalsource 126.123.10.12
IPv6 examples
hapee-gpe-stktagg.cfghaproxy
globalsource [2607:f8b0:400e:c00::ef]:12345
hapee-gpe-stktagg.cfghaproxy
globalsource [2607:f8b0:400e:c00::ef]:12345
The port is optional. It defaults to 0 for random ports.
hapee-gpe-stktagg.cfghaproxy
globalsource [2607:f8b0:400e:c00::ef]
hapee-gpe-stktagg.cfghaproxy
globalsource [2607:f8b0:400e:c00::ef]
GPE with session persistence Jump to heading
Available since
- HAProxy Enterprise 2.9r1
A special situation arises when you want to use the Global Profiling Engine to sync session persistence data across load balancers. Session persistence uses a stick table to track which server a client was routed to initially and from then on continues to route that client to the same server.
-
For example, consider the
backend
below that enables session persistence, but without GPE:haproxybackend serversstick-table type ip size 1m expire 30mstick on srcserver s1 192.168.0.10:80 checkserver s2 192.168.0.11:80 checkhaproxybackend serversstick-table type ip size 1m expire 30mstick on srcserver s1 192.168.0.10:80 checkserver s2 192.168.0.11:80 check -
We need to make the following changes to the
backend
:- Remove the
stick-table
line. - Make the
stick on
directive reference thesessions
table in thepeers
section namedmypeers
. - By default, each load balancer can arrange the servers differently. However, we need to ensure consistent server IDs across all load balancers, so we use the
id
argument to set the IDs explicitly.
haproxybackend serversstick on src table mypeers/sessionsserver s1 192.168.0.10:80 check id 1server s2 192.168.0.11:80 check id 2haproxybackend serversstick on src table mypeers/sessionsserver s1 192.168.0.10:80 check id 1server s2 192.168.0.11:80 check id 2 - Remove the
-
Move the
stick-table
definition to thepeers
section:haproxypeers mypeersbind 0.0.0.0:10000server enterprise1server gpe 192.168.50.40:10000table sessions type ip size 1m expire 30m store server_id,server_keytable sessions.aggregate type ip size 1m expire 30m store server_id,server_key write-to mypeers/sessionshaproxypeers mypeersbind 0.0.0.0:10000server enterprise1server gpe 192.168.50.40:10000table sessions type ip size 1m expire 30m store server_id,server_keytable sessions.aggregate type ip size 1m expire 30m store server_id,server_key write-to mypeers/sessionsIn this example:
- We have moved the stick table to the
peers
section and named itsessions
. You must set itsstore
argument toserver_id,server_key
. - A table named
sessions.aggregate
syncs session persistence data to GPE, which then syncs it to all load balancers. The aggregate table must set thewrite-to
argument so that the data is written back to thesessions
table. Thewrite-to
parameter allows remote load balancers to update the localsessions
table with session persistence data.
- We have moved the stick table to the
-
Make this same change on the other load balancer.
Multi-level setup Jump to heading
You can aggregate stick tables from other Global Profiling Engines, which allows you to aggregate stick tables across different data centers, for example.
We will consider the following setup:
The top-level aggr3 Global Profiling Engine will sum the counters from the intermediate aggr1 and aggr2 aggregate stick tables. It will then send the top-level aggregate stick table to all HAProxy Enterprise nodes.
You can also host multiple top-level servers for high availability. In that case, intermediate servers simply push their data to both. See below for details.
Configure the top-level Global Profiling Engine Jump to heading
Follow these steps on the server you wish to be the top-level Global Profiling Engine.
-
Edit the file
/etc/hapee-extras/hapee-gpe-stktagg.cfg
.hapee-gpe-stktagg.cfghaproxyglobalstats socket /var/run/hapee-extras/gpe-api.sockaggregations toplevelfrom .intermediate to .aggregatepeer top-gpe 0.0.0.0:10000 localpeer intermediate-gpe1 192.168.56.111:10000 downpeer intermediate-gpe2 192.168.56.112:10000 downhapee-gpe-stktagg.cfghaproxyglobalstats socket /var/run/hapee-extras/gpe-api.sockaggregations toplevelfrom .intermediate to .aggregatepeer top-gpe 0.0.0.0:10000 localpeer intermediate-gpe1 192.168.56.111:10000 downpeer intermediate-gpe2 192.168.56.112:10000 down-
The current server has the
local
keyword set on itspeer
line. -
In this example, two other Global Profiling Engine servers, intermediate-gpe1 and intermediate-gpe2, are listed with the
down
keyword, which means that they are one level down from the top. -
The top-level Global Profiling Engine will aggregate stick table data from the intermediate servers. Their stick tables should have the
.intermediate
suffix. -
The top-level Global Profiling Engine will push aggregated data back to the intermediate servers. The globally aggregated stick tables should have the
.aggregate
suffix.
-
Configure the intermediate Global Profiling Engines Jump to heading
Follow these steps on the servers you wish to be the intermediate-level Global Profiling Engines.
-
Edit the file
/etc/hapee-extras/hapee-gpe-stktagg.cfg
.intermediate-gpe1
hapee-gpe-stktagg.cfghaproxyglobalstats socket /var/run/hapee-extras/gpe-api.sockaggregations myaggrfrom any to .intermediateforward .aggregatepeer intermediate-gpe1 0.0.0.0:10000 localpeer top-gpe 192.168.56.113:10000 uppeer enterprise1 192.168.50.41:10000peer enterprise2 192.168.50.42:10000hapee-gpe-stktagg.cfghaproxyglobalstats socket /var/run/hapee-extras/gpe-api.sockaggregations myaggrfrom any to .intermediateforward .aggregatepeer intermediate-gpe1 0.0.0.0:10000 localpeer top-gpe 192.168.56.113:10000 uppeer enterprise1 192.168.50.41:10000peer enterprise2 192.168.50.42:10000intermediate-gpe2
hapee-gpe-stktagg.cfghaproxyglobalstats socket /var/run/hapee-extras/gpe-api.sockaggregations myaggrfrom any to .intermediateforward .aggregatepeer intermediate-gpe2 0.0.0.0:10000 localpeer top-gpe 192.168.56.113:10000 uppeer enterprise3 192.168.50.51:10000peer enterprise4 192.168.50.52:10000hapee-gpe-stktagg.cfghaproxyglobalstats socket /var/run/hapee-extras/gpe-api.sockaggregations myaggrfrom any to .intermediateforward .aggregatepeer intermediate-gpe2 0.0.0.0:10000 localpeer top-gpe 192.168.56.113:10000 uppeer enterprise3 192.168.50.51:10000peer enterprise4 192.168.50.52:10000-
The current server has the
local
keyword set on itspeer
line. -
The upper-level Global Profiling Engine peer is denoted by the
up
keyword. -
Each intermediate Global Profiling Engine is aware of only the HAProxy Enterprise nodes it manages and of the top-level Global Profiling Engine.
-
The intermediate-level Global Profiling Engines will aggregate stick table data from the HAProxy Enterprise servers.
-
The
forward
line relays the top-level server’s.aggregate
stick tables to the HAProxy Enterprise servers. -
The intermediate-level Global Profiling Engines will push aggregated data back to the HAProxy Enterprise servers. The aggregated stick tables should have the
.aggregate
suffix.
-
Configure for high availability Jump to heading
To create a highly available setup, you can have multiple top-level servers. Add the group
parameter to them so that the intermediate servers recognize that there are two top-level servers.
intermediate-gpe1
hapee-gpe-stktagg.cfghaproxy
globalstats socket /var/run/hapee-extras/gpe-api.sockaggregations myaggrfrom any to .intermediateforward .aggregatepeer intermediate-gpe1 0.0.0.0:10000 localpeer top-gpe1 192.168.56.113:10000 up group 1peer top-gpe2 192.168.56.114:10000 up group 1peer enterprise1 192.168.50.41:10000peer enterprise2 192.168.50.42:10000
hapee-gpe-stktagg.cfghaproxy
globalstats socket /var/run/hapee-extras/gpe-api.sockaggregations myaggrfrom any to .intermediateforward .aggregatepeer intermediate-gpe1 0.0.0.0:10000 localpeer top-gpe1 192.168.56.113:10000 up group 1peer top-gpe2 192.168.56.114:10000 up group 1peer enterprise1 192.168.50.41:10000peer enterprise2 192.168.50.42:10000
See also Jump to heading
Do you have any suggestions on how we can improve the content of this page?