Troubleshooting

Network troubleshooting

ARP diagnostics Jump to heading

This troubleshooting method uses Address Resolution Protocol (ARP) to check MAC / IP address associations.

From the Web user interface (web UI) Jump to heading

  1. Open the DiagTools tab.

  2. From the Select list, choose arp who-has (interface, source*, destination).

  3. Enter the following parameters:

    • interface: the interface where to send the ARP packet

    • source*: (optional) source IP

    • destination: the destination IP from which to get the MAC address

  4. Click on Run.

From the command line interface (CLI) Jump to heading

  1. Get root rights by typing root.

  2. Run arping -c 5 -I <interface> [-s <source>] <destination IP>.

    output
    text
    ARPING 10.0.32.10 from 10.0.32.11 eth0
    Unicast reply from 10.0.32.10 [00:0D:C5:36:C6:72] 0.636ms
    Unicast reply from 10.0.32.10 [00:0D:C5:36:C6:72] 0.608ms
    Unicast reply from 10.0.32.10 [00:0D:C5:36:C6:72] 0.650ms
    Unicast reply from 10.0.32.10 [00:0D:C5:36:C6:72] 0.666ms
    Unicast reply from 10.0.32.10 [00:0D:C5:36:C6:72] 0.612ms
    output
    text
    ARPING 10.0.32.10 from 10.0.32.11 eth0
    Unicast reply from 10.0.32.10 [00:0D:C5:36:C6:72] 0.636ms
    Unicast reply from 10.0.32.10 [00:0D:C5:36:C6:72] 0.608ms
    Unicast reply from 10.0.32.10 [00:0D:C5:36:C6:72] 0.650ms
    Unicast reply from 10.0.32.10 [00:0D:C5:36:C6:72] 0.666ms
    Unicast reply from 10.0.32.10 [00:0D:C5:36:C6:72] 0.612ms

Analyze output Jump to heading

In the example above, the MAC address is associated to the IP 10.0.32.10.

Tips Jump to heading

Duplicated IPs Jump to heading

ARP WHO-HAS packets are useful to fix duplicated IPs issues.

When you can reach a server or a VIP on the network, you can use ARP WHO-HAS to check if the MAC address associated with the IP address is the one expected or not:

  • If expected, the problem could be on a upper layer.

  • If not expected, then the MAC address of the server has the duplicated IP.

Packet capture Jump to heading

The diagnostic tool capture captures network traffic to retrieve the following information:

  • Source MAC and IP address

  • VRID

  • VRRP priority

From the Web user interface (web UI) Jump to heading

  1. Open the DiagTools tab.

  2. From Select list, choose capture (interface*, filter*, ignored).

  3. Enter the following parameters:

    • interface: Capture on a single interface. Default capture on all interfaces

    • filter: (optional) which packets to capture. By default, all packets are captured.

      For example:

      • A protocol name: vrrp, icmp

      • A specific IP address: host <ip>

      • A specific TCP or UDP port: port <port>

  4. Click Run.

From the command line interface (CLI) Jump to heading

  1. Get root rights by typing root.

  2. Run tcpdump -vvvenns0 -c 5 [-i <interface>] <filter>. Note that the output below is purposely truncated.

    output
    text
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    04:59:41.088388 00:15:5d:75:2e:1b > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 106: (tos 0x0, ttl 255, id 879, offset
    0 flags [none], [...]
    04:59:41.157687 00:15:5d:75:2e:45 > 00:15:5d:75:2e:1b, ethertype ARP (0x0806), length 42: arp who-has 10.0.0.190
    (00:15:5d:75:2e:1b) tell 10.0.0.187
    04:59:41.157723 00:15:5d:75:2e:1b > 00:15:5d:75:2e:45, ethertype ARP (0x0806), length 42: arp reply 10.0.0.190 is-at
    00:15:5d:75:2e:1b
    04:59:41.306124 00:15:5d:75:2e:1b > 00:15:5d:75:2e:45, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 48464,
    offset 0, flags [DF], [...]
    04:59:41.307063 00:15:5d:75:2e:45 > 00:15:5d:75:2e:1b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 6419,
    offset 0, flags [DF], [...]
    5 packets captured
    0 packets received by filter
    0 packets dropped by kernel
    output
    text
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    04:59:41.088388 00:15:5d:75:2e:1b > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 106: (tos 0x0, ttl 255, id 879, offset
    0 flags [none], [...]
    04:59:41.157687 00:15:5d:75:2e:45 > 00:15:5d:75:2e:1b, ethertype ARP (0x0806), length 42: arp who-has 10.0.0.190
    (00:15:5d:75:2e:1b) tell 10.0.0.187
    04:59:41.157723 00:15:5d:75:2e:1b > 00:15:5d:75:2e:45, ethertype ARP (0x0806), length 42: arp reply 10.0.0.190 is-at
    00:15:5d:75:2e:1b
    04:59:41.306124 00:15:5d:75:2e:1b > 00:15:5d:75:2e:45, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 48464,
    offset 0, flags [DF], [...]
    04:59:41.307063 00:15:5d:75:2e:45 > 00:15:5d:75:2e:1b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 6419,
    offset 0, flags [DF], [...]
    5 packets captured
    0 packets received by filter
    0 packets dropped by kernel

Tips Jump to heading

Capture VRRP traffic Jump to heading

To capture VRRP traffic, use vrrp as <filter> and choose an interface

output
text
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:12:27.936839 00:0c:49:65:47:0e > ... , proto VRRP (112), 10.9.104.253 > 224.0.0.18, ... , vrid 30, prio 35, ... , addrs: ...
14:12:28.250381 00:0c:59:20:c7:f6 > ... , proto VRRP (112), 10.9.104.241 > 224.0.0.18, ... , vrid 166, prio 133, ... , addrs: ...
14:12:28.460930 00:0d:b9:18:b9:74 > ... , proto VRRP (112), 10.9.104.253 > 224.0.0.18, ... , vrid 55, prio 70, ... , addrs: ...
14:12:28.939831 00:0c:49:65:47:0e > ... , proto VRRP (112), 10.9.104.253 > 224.0.0.18, ... , vrid 30, prio 35, ... , addrs: ...
14:12:28.250381 00:0c:59:20:c7:f6 > ... , proto VRRP (112), 10.9.104.241 > 224.0.0.18, ... , vrid 166, prio 133, ... , addrs: ...
5 packets captured
0 packets received by filter
0 packets dropped by kernel
output
text
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:12:27.936839 00:0c:49:65:47:0e > ... , proto VRRP (112), 10.9.104.253 > 224.0.0.18, ... , vrid 30, prio 35, ... , addrs: ...
14:12:28.250381 00:0c:59:20:c7:f6 > ... , proto VRRP (112), 10.9.104.241 > 224.0.0.18, ... , vrid 166, prio 133, ... , addrs: ...
14:12:28.460930 00:0d:b9:18:b9:74 > ... , proto VRRP (112), 10.9.104.253 > 224.0.0.18, ... , vrid 55, prio 70, ... , addrs: ...
14:12:28.939831 00:0c:49:65:47:0e > ... , proto VRRP (112), 10.9.104.253 > 224.0.0.18, ... , vrid 30, prio 35, ... , addrs: ...
14:12:28.250381 00:0c:59:20:c7:f6 > ... , proto VRRP (112), 10.9.104.241 > 224.0.0.18, ... , vrid 166, prio 133, ... , addrs: ...
5 packets captured
0 packets received by filter
0 packets dropped by kernel

Ping Jump to heading

Pinging ensures that an IP address is reachable through the network.

From the Web user interface (web UI) Jump to heading

  1. Open the DiagTools tab.

  2. From the Select list, choose ping (source*, destination, ignored).

  3. Enter the following parameters:

    • source*: (optional) source IP to use
    • destination: the destination IP to reach
  4. Click Run.

From the command line interface (CLI) Jump to heading

  1. Get root rights by typing root.
  2. Run ping -n -c 5 [-s <source>] <destination>.

Successful Jump to heading

output
text
PING 10.0.33.101 (10.0.33.101) 56(84) bytes of data.
64 bytes from 10.0.33.101: icmp_seq=1 ttl=64 time=0.051 ms
64 bytes from 10.0.33.101: icmp_seq=2 ttl=64 time=0.030 ms
64 bytes from 10.0.33.101: icmp_seq=3 ttl=64 time=0.024 ms
64 bytes from 10.0.33.101: icmp_seq=4 ttl=64 time=0.030 ms
64 bytes from 10.0.33.101: icmp_seq=5 ttl=64 time=0.036 ms
--- 10.0.33.101 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4008ms
rtt min/avg/max/mdev = 0.024/0.034/0.051/0.009 ms
output
text
PING 10.0.33.101 (10.0.33.101) 56(84) bytes of data.
64 bytes from 10.0.33.101: icmp_seq=1 ttl=64 time=0.051 ms
64 bytes from 10.0.33.101: icmp_seq=2 ttl=64 time=0.030 ms
64 bytes from 10.0.33.101: icmp_seq=3 ttl=64 time=0.024 ms
64 bytes from 10.0.33.101: icmp_seq=4 ttl=64 time=0.030 ms
64 bytes from 10.0.33.101: icmp_seq=5 ttl=64 time=0.036 ms
--- 10.0.33.101 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4008ms
rtt min/avg/max/mdev = 0.024/0.034/0.051/0.009 ms

Unsuccessful Jump to heading

output
text
PING 10.0.33.101 (10.0.33.101) 56(84) bytes of data.
--- 10.0.33.101 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms
output
text
PING 10.0.33.101 (10.0.33.101) 56(84) bytes of data.
--- 10.0.33.101 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms

Routing diagnostics Jump to heading

This diagnostic tool prints HAProxy ALOHA routing table.

From the Web user interface (web UI) Jump to heading

  1. Open the DiagTools tab.

  2. From the Select list, choose advanced routes (ignored, ignored, ignored).

  3. Optionally, enter the parameter:

    • interface: (optional) limit output to routes associated to this interface
  4. Click on Run.

From the command line interface (CLI) Jump to heading

  1. Get root rights by typing root.

  2. Run ip route list.

Single default gateway Jump to heading

output
text
default via 10.0.0.161 dev eth0
10.0.0.160/27 dev eth0 proto kernel scope link src 10.0.0.190
127.255.255.1 dev sync proto kernel scope link src 127.255.255.2
192.168.10.0/24 dev eth2 proto kernel scope link src 192.168.10.3
192.168.13.0/24 dev eth3 proto kernel scope link src 192.168.13.3
192.168.14.0/24 dev eth1 proto kernel scope link src 192.168.14.3
192.168.15.0/24 via 192.168.10.1 dev eth2
output
text
default via 10.0.0.161 dev eth0
10.0.0.160/27 dev eth0 proto kernel scope link src 10.0.0.190
127.255.255.1 dev sync proto kernel scope link src 127.255.255.2
192.168.10.0/24 dev eth2 proto kernel scope link src 192.168.10.3
192.168.13.0/24 dev eth3 proto kernel scope link src 192.168.13.3
192.168.14.0/24 dev eth1 proto kernel scope link src 192.168.14.3
192.168.15.0/24 via 192.168.10.1 dev eth2

Multiple default gateways Jump to heading

output
text
default via 10.0.0.161 dev eth0
default via 192.168.14.1 dev eth1 metric 1
default via 192.168.10.1 dev eth2 metric 2
default via 192.168.13.1 dev eth3 metric 3
10.0.0.160/27 dev eth0 proto kernel scope link src 10.0.0.190
127.255.255.1 dev sync proto kernel scope link src 127.255.255.2
192.168.10.0/24 dev eth2 proto kernel scope link src 192.168.10.3
192.168.13.0/24 dev eth3 proto kernel scope link src 192.168.13.3
192.168.14.0/24 dev eth1 proto kernel scope link src 192.168.14.3
192.168.15.0/24 via 192.168.10.1 dev eth2
output
text
default via 10.0.0.161 dev eth0
default via 192.168.14.1 dev eth1 metric 1
default via 192.168.10.1 dev eth2 metric 2
default via 192.168.13.1 dev eth3 metric 3
10.0.0.160/27 dev eth0 proto kernel scope link src 10.0.0.190
127.255.255.1 dev sync proto kernel scope link src 127.255.255.2
192.168.10.0/24 dev eth2 proto kernel scope link src 192.168.10.3
192.168.13.0/24 dev eth3 proto kernel scope link src 192.168.13.3
192.168.14.0/24 dev eth1 proto kernel scope link src 192.168.14.3
192.168.15.0/24 via 192.168.10.1 dev eth2

Do you have any suggestions on how we can improve the content of this page?