Troubleshoot the Global Profiling Engine

This section describes troubleshooting steps for the Global Profiling Engine.

Troubleshooting: peer connections Jump to heading #

If calling the /aggs URL with curl on the aggregation server does not produce any data even after adequate time has passed for the first bucket to populate, it may be the case that the Global Profiling Engine does not have connection to its configured HAProxy Enterprise peers.

To verify peer connections:

1. Use the show peers Runtime API command on the HAProxy Enterprise server:

   nix
   echo "show peers" | sudo socat stdio unix-connect:/var/run/hapee-3.0/hapee-lb.sock

   nix
   echo "show peers" | sudo socat stdio unix-connect:/var/run/hapee-3.0/hapee-lb.sock

   output
   text
   0xaaaad88dbc50: [06/Sep/2023:10:07:50] id=mypeers disabled=0 flags=0x6213 resync_timeout=<PAST> task_calls=166688
     0xaaaad88e2750: id=gpe(remote,active) addr=192.168.64.50:10000 last_status=ESTA last_hdshk=0s
           reconnect=4s heartbeat=2s confirm=0 tx_hbt=10896 rx_hbt=0 no_hbt=10882 new_conn=10894 proto_err=0 coll=0
           flags=0x0 appctx:0xaaaad8d064b0 st0=7 st1=0 task_calls=4 state=EST
           shared tables:
             0xaaaad88fdd40 local_id=2 remote_id=2049 flags=0x0 remote_data=0x2
                 last_acked=0 last_pushed=24 last_get=0 teaching_origin=24 update=24
                 table:0xaaaad88e3860 id=mypeers/st_responsetime update=24 localupdate=24 commitupdate=24 refcnt=1
           Dictionary cache not dumped (use "show peers dict")
             0xaaaad88fdaf0 local_id=1 remote_id=2050 flags=0x0 remote_data=0x2
                 last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                 table:0xaaaad88e3bc0 id=mypeers/st_responsetime.aggregate update=17 localupdate=0 commitupdate=0 refcnt=1
           Dictionary cache not dumped (use "show peers dict")
     0xaaaad88e16d0: id=hapee(local,inactive) addr=0.0.0.0:10000 last_status=NONE last_hdshk=<NEVER>
           reconnect=<NEVER> heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=0 proto_err=0 coll=0
           flags=0x0
           shared tables:
             0xaaaad88fde10 local_id=2 remote_id=0 flags=0x0 remote_data=0x0
                 last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                 table:0xaaaad88e3860 id=mypeers/st_responsetime update=24 localupdate=24 commitupdate=24 refcnt=1
           Dictionary cache not dumped (use "show peers dict")
             0xaaaad88fdbc0 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
                 last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                 table:0xaaaad88e3bc0 id=mypeers/st_responsetime.aggregate update=17 localupdate=0 commitupdate=0 refcnt=1
           Dictionary cache not dumped (use "show peers dict")

   output
   text
   0xaaaad88dbc50: [06/Sep/2023:10:07:50] id=mypeers disabled=0 flags=0x6213 resync_timeout=<PAST> task_calls=166688
     0xaaaad88e2750: id=gpe(remote,active) addr=192.168.64.50:10000 last_status=ESTA last_hdshk=0s
           reconnect=4s heartbeat=2s confirm=0 tx_hbt=10896 rx_hbt=0 no_hbt=10882 new_conn=10894 proto_err=0 coll=0
           flags=0x0 appctx:0xaaaad8d064b0 st0=7 st1=0 task_calls=4 state=EST
           shared tables:
             0xaaaad88fdd40 local_id=2 remote_id=2049 flags=0x0 remote_data=0x2
                 last_acked=0 last_pushed=24 last_get=0 teaching_origin=24 update=24
                 table:0xaaaad88e3860 id=mypeers/st_responsetime update=24 localupdate=24 commitupdate=24 refcnt=1
           Dictionary cache not dumped (use "show peers dict")
             0xaaaad88fdaf0 local_id=1 remote_id=2050 flags=0x0 remote_data=0x2
                 last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                 table:0xaaaad88e3bc0 id=mypeers/st_responsetime.aggregate update=17 localupdate=0 commitupdate=0 refcnt=1
           Dictionary cache not dumped (use "show peers dict")
     0xaaaad88e16d0: id=hapee(local,inactive) addr=0.0.0.0:10000 last_status=NONE last_hdshk=<NEVER>
           reconnect=<NEVER> heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=0 proto_err=0 coll=0
           flags=0x0
           shared tables:
             0xaaaad88fde10 local_id=2 remote_id=0 flags=0x0 remote_data=0x0
                 last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                 table:0xaaaad88e3860 id=mypeers/st_responsetime update=24 localupdate=24 commitupdate=24 refcnt=1
           Dictionary cache not dumped (use "show peers dict")
             0xaaaad88fdbc0 local_id=1 remote_id=0 flags=0x0 remote_data=0x0
                 last_acked=0 last_pushed=0 last_get=0 teaching_origin=0 update=0
                 table:0xaaaad88e3bc0 id=mypeers/st_responsetime.aggregate update=17 localupdate=0 commitupdate=0 refcnt=1
           Dictionary cache not dumped (use "show peers dict")

   There should be an entry for the Global Profiling engine, as well as all connected HAProxy Enterprise peers. If no data is returned or if peers are missing, there may be an issue with the configuration.

2. If entries are missing, check both the peers section of /etc/hapee-3.0/hapee-lb.cfg and the peers entries in the Global Profiling Engine configuration file /etc/hapee-extras/hapee-gpe-stktagg.cfg. Check the following:

   • The hostname for the Global Profiling Engine server should be its configured hostname. Use the hostname command to retrieve its name.

     nix
     hostname

     nix
     hostname

     output
     text
     gpe

     output
     text
     gpe

   • In each of the configuration files, /etc/hapee-3.0/hapee-lb.cfg, for the load balancer configuration, and /etc/hapee-extras/hapee-gpe-stktagg.cfg, for the Global Profiling Engine configuration, verify that the hostnames for your load balancer servers are identical everywhere they are referenced.

     The hostname value you provide for the load balancers may be one of three things:

     • the value provided with the -L argument specified on the command line used to start the load balancer process
     • the localpeer name specified in the global section of the load balancer configuration
     • the host name returned by the system hostname command.

     Be sure that this value is the one you specify for as the hostname of the load balancer on the server line in the peers section of your load balancer configuration and on the corresponding peer entry in the aggregations section of the GPE configuration.