Introduction:
The nsxcli includes the capture command, a useful debugging tool to capture specified network traffic at various points and stages in the network processing pipeline. It is a valuable and powerful tool to troubleshoot a variety of network issues. In this article, we will look at using nsxcli capture in a data path troubleshooting approach.
Preferred over OS-specific syntax for capture utilities such as tcpdump and pktcap-uw, the nsxcli capture provides a consistent user command-line interface across ESXi, KVM, Edge, and Manager. For example, the capture filter expression syntax is the same across all enabled NSX-T devices.
The data path troubleshooting strategy will be to:
– identify capture locations at critical points along the data path based
– use Traceflow to validate the path and to determine the Port for the corresponding capture location
– use nsxcli capture to collect traffic at the capture location
Network Topology:
In this example, we will use nsxcli captures to follow Guest VM ping traffic from virtual to physical. Capture locations shown are the critical points chosen along the end-to-end data path.
Traceflow
Advanced Networking & Security Traceflow is an excellent tool to illustrate the data path. Here the Source is VM1, and the destination is an external Internet IP beyond the physical network.
The Capture locations identified in the Network Topology have been transferred over to the resulting Traceflow.
Notice that for capture location 7 that there are a few points in the detailed Traceflow that represent the interconnectivity between tier-0 and tier-1. Depending on the troubleshooting scenario, it might be necessary to add additional capture locations. It is important to note that Traceflow hops have further details, including Port, which we will use in nsxcli captures.
At capture location 7, the Port is lab-tier-0-lab-tier-1-t1_lrp.
nsxcli Capture Command Options:
Capture Location 1:
Analysis will begin close to the Guest VM, on the connected N-VDS port, specified as the Guest VM NIC name or the N-VDS port number.
- run esxtop on the ESXi host where the Guest VM resides to identify the Guest's port number and NIC name: [root@esxcna01-s1:~] esxtop, n 4:06:24pm up 3 days 19:44, 756 worlds, 2 VMs, 2 vCPUs; CPU load average: 0.01, 0.01, 0.01 PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PSZTX PKTRX/s MbRX/s PSZRX %DRPTX %DRPRX 33554433 Management n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50331649 Management n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50331650 vmnic0 - DvsPortset-0 0.00 0.00 0.00 274.66 0.68 325.00 0.00 0.00 50331651 Shadow of vmnic0 n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50331652 vmnic1 - DvsPortset-0 116.98 0.24 269.00 157.67 0.44 366.00 0.00 0.00 50331653 Shadow of vmnic1 n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50331654 vmk0 vmnic1 DvsPortset-0 50.86 0.14 367.00 20.35 0.01 74.00 0.00 0.00 50331655 vmk1 vmnic1 DvsPortset-0 66.12 0.10 194.00 66.12 0.09 169.00 0.00 0.00 50331656 71259:VM2.eth0 vmnic0 DvsPortset-0 0.00 0.00 0.00 5.09 0.00 60.00 0.00 0.00 67108865 Management n/a DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108866 vmnic2 - DvsPortset-1 10.17 0.01 116.00 264.49 0.67 333.00 0.00 0.00 67108867 Shadow of vmnic2 n/a DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108868 vmk10 vmnic2 DvsPortset-1 10.17 0.01 66.00 5.09 0.00 60.00 0.00 0.00 67108869 vmk50 void DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108870 vdr-vdrPort vmnic2 DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108871 69749:VM1.eth0 vmnic2 DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 - The N-VDS port identifier is 67108871, and the Guest VM interface name is VM1.eth0 - Alternatively, this can also be determined from the N-VDS confuguration: [root@esxcna01-s1:~] nsxdp-cli vswitch instance list DvsPortset-1 (NSXToverlay) c0 8d c9 87 a2 d6 40 c4-9f 34 2c b7 9c bf 3d 76 Total Ports:1536 Available:1516 Client PortID DVPortID MAC Uplink Management 67108865 00:00:00:00:00:00 n/a vmnic2 67108866 uplink1 00:00:00:00:00:00 Shadow of vmnic2 67108867 00:50:56:58:f5:37 n/a vmk10 67108868 10 00:50:56:6a:99:31 vmnic2 vmk50 67108869 b7f52c2a-11ed-43c4-8e8e-c9a156e65e64 00:50:56:66:0f:29 void vdr-vdrPort 67108870 vdrPort 02:50:56:56:44:52 vmnic2 VM1.eth0 67108871 74908cb5-3374-4f6a-89b2-d207aa8c2d87 00:50:56:96:98:58 vmnic2 <--- - Here are two possible nsxcli captures: root@esxcna01-s1:~] nsxcli -c start capture interface VM1.eth0 expression ipproto 0x01 11:42:16.945357 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 1, length 64 11:42:17.947101 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 2, length 64 11:42:18.949292 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 3, length 64 [root@esxcna01-s1:~] nsxcli -c start capture interface 67108871 expression ipproto 0x01 11:43:19.048945 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 63, length 64 11:43:20.051061 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 64, length 64 11:43:21.053129 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 65, length 64
Capture Location 2:
Analysis will continue before the Distributed Firewall (DFW) filter is applied, known as “stage pre”:
- run summarize-dvfilter on the ESXi host where the Guest VM resides to identify the Guest's DFW filter name: [root@esxcna01-s1:~] summarize-dvfilter | grep -A 2 VM1 world 69749 vmm0:VM1 vcUuid:'50 16 ec 3c e5 9b 78 15-35 af 0c 18 d3 ea a9 19' port 67108871 VM1.eth0 vNic slot 2 name: nic-69749-eth0-vmware-sfw.2 <--- DFW filter name starts with "nic-" and ends with "-vmware-sfw.2" esxcna01-s1.core.hypervizor.com> start capture dvfilter nic-69749-eth0-vmware-sfw.2 stage pre expression ipproto 0x01 11:50:18.752232 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 482, length 64 11:50:18.787494 02:50:56:56:44:52 > 00:50:56:96:98:58, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.70.100: ICMP echo reply, id 2341, seq 482, length 64 11:50:19.754051 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 483, length 64 11:50:19.788627 02:50:56:56:44:52 > 00:50:56:96:98:58, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.70.100: ICMP echo reply, id 2341, seq 483, length 64
Capture Location 3:
Capture after the Distributed Firewall (DFW) filter is applied, known as “stage post”:
esxcna01-s1.core.hypervizor.com> start capture dvfilter nic-69749-eth0-vmware-sfw.2 stage post expression ipproto 0x01 11:50:45.784836 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 509, length 64 11:50:45.820445 02:50:56:56:44:52 > 00:50:56:96:98:58, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.70.100: ICMP echo reply, id 2341, seq 509, length 64 11:50:46.785223 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 510, length 64 11:50:46.822170 02:50:56:56:44:52 > 00:50:56:96:98:58, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.70.100: ICMP echo reply, id 2341, seq 510, length 64
Capture Location 4:
Capture as traffic arrives on the ESXi hosts’s virtual distributed router interface:
esxcna01-s1.core.hypervizor.com> start capture interface vdrPort direction output expression ipproto 0x01 11:56:30.286711 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 853, length 64 11:56:31.287938 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 854, length 64 11:56:32.289692 00:50:56:96:98:58 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 855, length 64
Capture Location 5:
Capture traffic on the ESXi Compute hosts’ Tunnel Endpoint (TEP) interface. Since traffic is GENEVE encapsulated, expression ipproto 0x01” is no longer appropriate. “expression ip 192.168.110.184” will match on TEP interface traffic.
[root@esxcna01-s1:~] esxcfg-vmknic -l Interface Port Group/DVPort/Opaque Network IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type NetStack vmk0 623 IPv4 192.168.110.81 255.255.255.0 192.168.110.255 00:50:56:01:44:05 1500 65535 true STATIC defaultTcpipStack vmk0 623 IPv6 fe80::250:56ff:fe01:4405 64 00:50:56:01:44:05 1500 65535 true STATIC, PREFERRED defaultTcpipStack vmk1 624 IPv4 10.10.20.81 255.255.255.0 10.10.20.255 00:50:56:6a:5e:cf 1500 65535 true STATIC defaultTcpipStack vmk1 624 IPv6 fe80::250:56ff:fe6a:5ecf 64 00:50:56:6a:5e:cf 1500 65535 true STATIC, PREFERRED defaultTcpipStack vmk10 10 IPv4 192.168.110.184 255.255.255.0 192.168.110.255 00:50:56:6a:99:31 1600 65535 true STATIC vxlan vmk10 10 IPv6 fe80::250:56ff:fe6a:9931 64 00:50:56:6a:99:31 1600 65535 true STATIC, PREFERRED vxlan vmk50 b7f52c2a-11ed-43c4-8e8e-c9a156e65e64 IPv4 169.254.1.1 255.255.0.0 169.254.255.255 00:50:56:66:0f:29 1500 65535 true STATIC hyperbus vmk50 b7f52c2a-11ed-43c4-8e8e-c9a156e65e64 IPv6 fe80::250:56ff:fe66:f29 64 00:50:56:66:0f:29 1500 65535 true STATIC, PREFERRED hyperbus esxcna01-s1.core.hypervizor.com> start capture interface vmk10 expression ip 192.168.110.184 16:27:48.147456 00:50:56:6a:99:31 > 00:50:56:96:e9:12, ethertype IPv4 (0x0800), length 66: 192.168.110.184.49154 > 192.168.110.181.3784: BFDv1, Control, State Down, Flags: [none], length: 24 16:27:48.347435 00:50:56:6a:99:31 > 00:50:56:96:f6:d8, ethertype IPv4 (0x0800), length 66: 192.168.110.184.49152 > 192.168.110.180.3784: BFDv1, Control, State Up, Flags: [none], length: 24 16:27:49.047468 00:50:56:6a:99:31 > 00:50:56:96:e9:12, ethertype IPv4 (0x0800), length 66: 192.168.110.184.49154 > 192.168.110.181.3784: BFDv1, Control, State Down, Flags: [none], length: 24
This completes Computer Cluster ESXi host captures.
Capture Location 6:
Capture traffic on the Edge VM Tunnel Endpoint (TEP) interface. Since traffic is GENEVE encapsulated, “expression ip 192.168.110.180” will match on TEP interface traffic.
- VRF 0 has the Edge VM TEP interface: nsxtedge01> get logical-router Logical Router UUID VRF LR-ID Name Type Ports 736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL 3 cbf4e534-3ad4-4cfb-83b2-79e03f7c80c0 1 12 DR-lab-tier-1 DISTRIBUTED_ROUTER_TIER1 5 019feeec-649c-449d-998f-01a2f5fed8c9 2 2054 SR-lab-tier-0 SERVICE_ROUTER_TIER0 6 0b06252f-a1f6-4d15-b106-d8128d3f0691 3 3073 SR-lab-tier-1 SERVICE_ROUTER_TIER1 5 e9322040-ebe6-426c-914a-72858fd86322 4 11 DR-lab-tier-0 DISTRIBUTED_ROUTER_TIER0 4 nsxtedge01> vrf 0 nsxtedge01(vrf)> get int Logical Router UUID VRF LR-ID Name Type 736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL Interfaces (IPv6 DAD Status A-Assigned, D-Duplicate, T-Tentative) Interface : 9fd3c667-32db-5921-aaad-7a88c80b5e9f Ifuid : 258 Mode : blackhole Interface : f322c6ca-4298-568b-81c7-a006ba6e6c88 Ifuid : 257 Mode : cpu Interface : 72dd0b68-e71e-5b53-b801-f7c246f3fdc9 Ifuid : 379 Name : Mode : lif IP/Mask : 192.168.110.180/24 MAC : 00:50:56:96:f6:d8 LS port : d0955bdb-7b4d-5a88-ba92-ac4eb212ff00 Urpf-mode : PORT_CHECK DAD-mode : LOOSE RA-mode : RA_INVALID Admin : up Op_state : up MTU : 1600 - 72dd0b68-e71e-5b53-b801-f7c246f3fdc9 is the Edge VM TEP interface: nsxtedge01> start capture interface 72dd0b68-e71e-5b53-b801-f7c246f3fdc9 direction dual expression ip 192.168.110.180 19:31:21.152657 00:50:56:96:f6:d8 > 00:50:56:6a:99:31, ethertype IPv4 (0x0800), length 116: 192.168.110.180.39521 > 192.168.110.184.6081: Geneve, Flags [O], vni 0x0, proto TEB (0x6558): 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 66: 192.168.110.180.55516 > 192.168.110.184.3784: BFDv1, Control, State Up, Flags: [none], length: 24 <base64>AFBWapkxAFBWlvbYCABFwABmAABAAEAR2wnAqG60wKhuuJphF8EAUl8hAIBlWAAAAAAAAAAAAAAAAAAAAAAIAEXAADQGBAAA/xFWN8CobrTAqG642NwOyAAgnn0gwAMYkTKoISZUzlwAAYagAA9CQAAAAAA=</base64> 19:31:21.169028 00:50:56:6a:99:31 > 00:50:56:96:f6:d8, ethertype IPv4 (0x0800), length 116: 192.168.110.184.56313 > 192.168.110.180.6081: Geneve, Flags [O], vni 0x0, proto TEB (0x6558): 00:50:56:6a:99:31 > 00:50:56:96:f6:d8, ethertype IPv4 (0x0800), length 66: 192.168.110.184.49152 > 192.168.110.180.3784: BFDv1, Control, State Up, Flags: [none], length: 24 <base64>AFBWlvbYAFBWapkxCABFAABmAABAAEAR28nAqG64wKhutNv5F8EAUheXAIBlWAAAAAAAUFaW9tgAUFZqmTEIAEUAADQAAAAA/xFc+8CobrjAqG60wAAOyAAgAAAgwAMYJlTOXJEyqCEAAYagAA9CQAAAAAA=</base64>
Capture Location 7:
Identify Edge VM interface lab-tier-0-lab-tier-1-t0_lrp and lab-tier-0-lab-tier-1-t1_lrp, and ensure traffic is over nsxtedge01 by placing nsxtedge02 in maintenance mode.
nsxtedge02> set maintenance-mode enabled Maintenance Mode: enabled nsxtedge01> get logical-router interface stats | find name|IP/|MAC|interface|-Packets interface : 3fc00f41-9f1b-49fa-8e14-3e19cea010f3 name : bp-sr0-port IP/Mask : 169.254.0.2/25;fe80::50:56ff:fe56:5300/64(NA) MAC : 02:50:56:56:53:00 RX-Packets : 20 TX-Packets : 30 interface : eb3e4379-4b7e-41d4-83ee-95a3152caff3 name : sr0-internal-routing-port IP/Mask : 169.254.0.130/25;fe80::50:56ff:fe56:5200/64(NA) MAC : 02:50:56:56:52:00 RX-Packets : 664833 TX-Packets : 662167 interface : 233e568d-9f70-43f9-9e55-8c2a926a2655 name : external-uplink1 IP/Mask : 192.168.100.102/24 MAC : 00:50:56:96:25:72 RX-Packets : 102597 TX-Packets : 172178 interface : 3766169e-3dd6-4672-b016-9c6f9db0987c name : bp-dr-port IP/Mask : 169.254.0.1/25;fe80::50:56ff:fe56:4452/64(NA) MAC : 02:50:56:56:44:52 RX-Packets : 47 TX-Packets : 0 interface : e01935a4-3720-4bb1-92e5-cb6d55050304 name : lab-tier-0-lab-tier-1-t0_lrp IP/Mask : 100.64.160.0/31;fc7a:1e3f:83d2:6800::1/64(NA);fe80::50:56ff:fe56:4452/64(NA) MAC : 02:50:56:56:44:52 RX-Packets : 154570 TX-Packets : 897 interface : 6889225f-2206-4337-88c6-58fc5adb78af name : lab-tier-0-lab-tier-1-t1_lrp IP/Mask : 100.64.160.1/31;fc7a:1e3f:83d2:6800::2/64(NA);fe80::50:56ff:fe56:4455/64(NA) MAC : 02:50:56:56:44:55 RX-Packets : 143228 TX-Packets : 154691 interface : c08409a9-bd3f-4a62-835a-3d278cec0ffd name : bp-sr0-port IP/Mask : 169.254.0.2/28;fe80::50:56ff:fe56:5300/64(NA) MAC : 02:50:56:56:53:00 RX-Packets : 154989 TX-Packets : 493 interface : 6988c30f-6e40-4195-9a0b-f1a967c71a04 name : infra-seg1-dlrp IP/Mask : 192.168.70.1/24 MAC : 02:50:56:56:44:52 RX-Packets : 267 TX-Packets : 143340 interface : 6620db9a-123a-48cd-93e4-728ed0fd4815 name : bp-dr-port IP/Mask : 169.254.0.1/28;fe80::50:56ff:fe56:4452/64(NA) MAC : 02:50:56:56:44:52 RX-Packets : 99 TX-Packets : 0 interface : 0842709d-5019-48f3-bb1b-228fc45ed058 name : lab-tier-1-dhcp-dlrp IP/Mask : 192.168.60.2/24 MAC : 02:50:56:56:44:52 RX-Packets : 23 TX-Packets : 0 interface : 72dd0b68-e71e-5b53-b801-f7c246f3fdc9 name : IP/Mask : 192.168.110.180/24 MAC : 00:50:56:96:f6:d8 RX-Packets : 3423780 TX-Packets : 1675321 lab-tier-0-lab-tier-1-t1_lrp: nsxtedge01> start capture interface 6889225f-2206-4337-88c6-58fc5adb78af expression ipproto 0x01 18:28:58.811837 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 24365, length 64 <base64>AlBWVkRSAlBWVkRVCABFAABU/ihAAD8BJmTAqEZkCAgICAgAi7sJJV8t6cz3XQAAAABV9A4AAAAAABAREhMUFRYXGBkaGxwdHh8gISIjJCUmJygpKissLS4vMDEyMzQ1Njc=</base64> 18:28:58.844249 02:50:56:56:44:52 > 02:50:56:56:44:55, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.70.100: ICMP echo reply, id 2341, seq 24365, length 64 <base64>AlBWVkRVAlBWVkRSCABFAABUAAAAACgBe40ICAgIwKhGZAAAk7sJJV8t6cz3XQAAAABV9A4AAAAAABAREhMUFRYXGBkaGxwdHh8gISIjJCUmJygpKissLS4vMDEyMzQ1Njc=</base64> lab-tier-0-lab-tier-1-t0_lrp: nsxtedge01> start capture interface e01935a4-3720-4bb1-92e5-cb6d55050304 expression ipproto 0x01 18:28:26.775462 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 24333, length 64 <base64>AlBWVkRSAlBWVkRVCABFAABU69NAAD8BOLnAqEZkCAgICAgA3GcJJV8Nycz3XQAAAAAlaA4AAAAAABAREhMUFRYXGBkaGxwdHh8gISIjJCUmJygpKissLS4vMDEyMzQ1Njc=</base64> 18:28:26.807430 02:50:56:56:44:52 > 02:50:56:56:44:55, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.70.100: ICMP echo reply, id 2341, seq 24333, length 64 <base64>AlBWVkRVAlBWVkRSCABFAABUAAAAACgBe40ICAgIwKhGZAAA5GcJJV8Nycz3XQAAAAAlaA4AAAAAABAREhMUFRYXGBkaGxwdHh8gISIjJCUmJygpKissLS4vMDEyMzQ1Njc=</base64>
Capture Location 8:
Identify Edge VM external uplink interface external-uplink1:
nsxtedge01> get logical-router interface stats | find name|IP/|MAC|interface|-Packets interface : 3fc00f41-9f1b-49fa-8e14-3e19cea010f3 name : bp-sr0-port IP/Mask : 169.254.0.2/25;fe80::50:56ff:fe56:5300/64(NA) MAC : 02:50:56:56:53:00 RX-Packets : 20 TX-Packets : 30 interface : eb3e4379-4b7e-41d4-83ee-95a3152caff3 name : sr0-internal-routing-port IP/Mask : 169.254.0.130/25;fe80::50:56ff:fe56:5200/64(NA) MAC : 02:50:56:56:52:00 RX-Packets : 664833 TX-Packets : 662167 interface : 233e568d-9f70-43f9-9e55-8c2a926a2655 name : external-uplink1 IP/Mask : 192.168.100.102/24 MAC : 00:50:56:96:25:72 RX-Packets : 102597 TX-Packets : 172178 interface : 3766169e-3dd6-4672-b016-9c6f9db0987c name : bp-dr-port IP/Mask : 169.254.0.1/25;fe80::50:56ff:fe56:4452/64(NA) MAC : 02:50:56:56:44:52 RX-Packets : 47 TX-Packets : 0 interface : e01935a4-3720-4bb1-92e5-cb6d55050304 name : lab-tier-0-lab-tier-1-t0_lrp IP/Mask : 100.64.160.0/31;fc7a:1e3f:83d2:6800::1/64(NA);fe80::50:56ff:fe56:4452/64(NA) MAC : 02:50:56:56:44:52 RX-Packets : 154570 TX-Packets : 897 interface : 6889225f-2206-4337-88c6-58fc5adb78af name : lab-tier-0-lab-tier-1-t1_lrp IP/Mask : 100.64.160.1/31;fc7a:1e3f:83d2:6800::2/64(NA);fe80::50:56ff:fe56:4455/64(NA) MAC : 02:50:56:56:44:55 RX-Packets : 143228 TX-Packets : 154691 interface : c08409a9-bd3f-4a62-835a-3d278cec0ffd name : bp-sr0-port IP/Mask : 169.254.0.2/28;fe80::50:56ff:fe56:5300/64(NA) MAC : 02:50:56:56:53:00 RX-Packets : 154989 TX-Packets : 493 interface : 6988c30f-6e40-4195-9a0b-f1a967c71a04 name : infra-seg1-dlrp IP/Mask : 192.168.70.1/24 MAC : 02:50:56:56:44:52 RX-Packets : 267 TX-Packets : 143340 interface : 6620db9a-123a-48cd-93e4-728ed0fd4815 name : bp-dr-port IP/Mask : 169.254.0.1/28;fe80::50:56ff:fe56:4452/64(NA) MAC : 02:50:56:56:44:52 RX-Packets : 99 TX-Packets : 0 interface : 0842709d-5019-48f3-bb1b-228fc45ed058 name : lab-tier-1-dhcp-dlrp IP/Mask : 192.168.60.2/24 MAC : 02:50:56:56:44:52 RX-Packets : 23 TX-Packets : 0 interface : 72dd0b68-e71e-5b53-b801-f7c246f3fdc9 name : IP/Mask : 192.168.110.180/24 MAC : 00:50:56:96:f6:d8 RX-Packets : 3423780 TX-Packets : 1675321 nsxtedge01> start capture interface 233e568d-9f70-43f9-9e55-8c2a926a2655 direction dual expression ipproto 0x01 19:40:09.559601 00:50:56:96:25:72 > 00:50:56:01:3c:bc, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 49359, length 64 <base64>AFBWATy8AFBWliVyCABFAABUgpVAAD4BovfAqEZkCAgICAgAepAJJcDPGC/5XQAAAADYGgsAAAAAABAREhMUFRYXGBkaGxwdHh8gISIjJCUmJygpKissLS4vMDEyMzQ1Njc=</base64> 19:40:09.596320 00:50:56:01:3c:bc > 00:50:56:96:25:72, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.70.100: ICMP echo reply, id 2341, seq 49359, length 64 <base64>AFBWliVyAFBWATy8CABFAABUAAAAACkBeo0ICAgIwKhGZAAAgpAJJcDPGC/5XQAAAADYGgsAAAAAABAREhMUFRYXGBkaGxwdHh8gISIjJCUmJygpKissLS4vMDEyMzQ1Njc=</base64>
Capture Location 9:
Capture Edge VM uplink traffic as it exits the ESXi host. Since this Edge Cluster ESXi host is not NSX-T prepared, nsxcli capture is not available. We will resort to an ESXi hosts pktcap-uw OS level traffic capture. Here we have determined that Edge traffic is out ESXi host interface vmnic3.
[root@esx03-s1:~] nsxcli, n 7:46:58pm up 4 days 23:25, 617 worlds, 1 VMs, 4 vCPUs; CPU load average: 0.17, 0.17, 0.17 PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PSZTX PKTRX/s MbRX/s PSZRX %DRPTX %DRPRX 33554433 Management n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Start capture 50331649 Management n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50331650 vmnic0 - DvsPortset-0 0.00 0.00 0.00 2746.58 14.84 708.00 0.00 0.00 50331651 Shadow of vmnic0 n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50331652 vmnic1 - DvsPortset-0 452.68 0.71 206.00 2293.90 14.13 807.00 0.00 0.00 50331653 Shadow of vmnic1 n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50331654 vmk0 vmnic1 DvsPortset-0 152.59 0.29 252.00 25.43 0.01 60.00 0.00 0.00 50331655 vmk1 vmnic1 DvsPortset-0 264.49 0.38 190.00 254.31 0.34 174.00 0.00 0.00 50331656 vmk2 vmnic1 DvsPortset-0 0.00 0.00 0.00 10.17 0.00 60.00 0.00 0.00 50331657 vdr-vdrPort vmnic1 DvsPortset-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50331658 78429:NSX-T-Edge01-2.5.0.0.0-1 vmnic1 DvsPortset-0 10.17 0.01 137.00 20.35 0.02 115.00 0.00 0.00 50331659 78429:NSX-T-Edge01-2.5.0.0.0-1 vmnic1 DvsPortset-0 25.43 0.02 118.00 35.60 0.03 118.00 0.00 0.00 67108865 Management n/a DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108866 vmnic2 - DvsPortset-1 0.00 0.00 0.00 15.26 0.01 108.00 0.00 0.00 67108867 Shadow of vmnic2 n/a DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108868 vmnic3 - DvsPortset-1 5.09 0.01 166.00 10.17 0.01 79.00 0.00 0.00 67108869 Shadow of vmnic3 n/a DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108870 78429:NSX-T-Edge01-2.5.0.0.0-1 vmnic3 DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108871 78429:NSX-T-Edge01-2.5.0.0.0-1 vmnic3 DvsPortset-1 5.09 0.01 166.00 10.17 0.01 79.00 0.00 0.00 [root@esx03-s1:~] pktcap-uw --uplink vmnic3 --dir 1 --proto 0x01 -o - | tcpdump-uw -r - -nn reading from file -, link-type EN10MB (Ethernet) 19:52:42.604135 IP 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 50110, lengt 19:52:43.605763 IP 192.168.70.100 > 8.8.8.8: ICMP echo request, id 2341, seq 50111, length 64 tcpdump-uw: pcap_loop: error reading dump file: Interrupted system call [root@esx03-s1:~] pktcap-uw --uplink vmnic3 --dir 0 --proto 0x01 -o - | tcpdump-uw -r - -nn reading from file -, link-type EN10MB (Ethernet) 19:52:55.655378 IP 8.8.8.8 > 192.168.70.100: ICMP echo reply, id 2341, seq 50123, length 64 19:52:56.661685 IP 8.8.8.8 > 192.168.70.100: ICMP echo reply, id 2341, seq 50124, length 64
Capture Location 10:
Here is an additional capture performed on an NSX-T Manager appliance. This illustrates the syntax is the same across all enabled NSX-T devices.
nsxtmgr01> start capture interface eth0 expression ipproto 0x01 19:58:29.995778 00:50:56:01:3c:b9 > 00:50:56:96:5d:ac, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 192.168.110.17: ICMP echo request, id 3031, seq 1, length 64 19:58:29.995932 00:50:56:96:5d:ac > 00:50:56:01:3c:b9, ethertype IPv4 (0x0800), length 98: 192.168.110.17 > 192.168.70.100: ICMP echo reply, id 3031, seq 1, length 64 19:58:31.002133 00:50:56:01:3c:b9 > 00:50:56:96:5d:ac, ethertype IPv4 (0x0800), length 98: 192.168.70.100 > 192.168.110.17: ICMP echo request, id 3031, seq 2, length 64 19:58:31.002178 00:50:56:96:5d:ac > 00:50:56:01:3c:b9, ethertype IPv4 (0x0800), length 98: 192.168.110.17 > 192.168.70.100: ICMP echo reply, id 3031, seq 2, length 64
The capture filter expression syntax is the same across all enabled NSX-T devices!
