Troubleshooting DHCP in an NSX-T environment can be challenging. This blog post provides a structured approach to following a client DHCP request along the communication path to help determine the root cause as to why IP address assignment may be failing.
The Lab Topology
In this case DHCP Relay is configured on an NSX-T 2.4.1 environment, using the Simplified UI, where the DHCP Server is outside of the NSX-T environment.
Background Notes: Configuring NSX-T DHCP Relay
There are three configuration settings to pay close attention to when setting up DHCP Relay using the Simplified UI in NSX-T:
- 1: Define the IP Address Management DHCP configuration:
- 2: Assign IP Address Management configuration to the Tier-1 Gateway:
- 3: Specify a DHCP Range IP on the DHCP Client Segment:
Although unexpected, this configuration is actually required to realize DHCP relay functionality on the DHCP Client Segment. This requirement is expected to be improved upon in a subsequent NSX-T release.
Using Traffic Captures to follow a DHCP Client request
The follow diagram illustrates locations along the path that traffic captures can be performed to identify delivery of client DHCP request packets.
1. Traffic Capture at the DHCP vDS port
To set up for this capture, begin by determining the ESXi host that is hosting the Guest VM, and identify the vDS port ID that connects to the Guest VM:
- here Guest VM photon-dhcp-client is hosted by ESXi host esxcna-s1, with MAC Address 00:50:56:96:0e:25:
- Use net-stats to determine the vDS port ID that connects to the Guest VM:
[root@esxcna03-s1:~] net-stats -l PortNum Type SubType SwitchName MACAddress ClientName 50331650 4 0 DvsPortset-0 00:50:56:01:10:de vmnic0 50331652 4 0 DvsPortset-0 00:50:56:01:10:e2 vmnic1 50331654 3 0 DvsPortset-0 00:50:56:01:10:de vmk0 50331655 3 0 DvsPortset-0 00:50:56:6d:6d:ba vmk1 67108866 4 0 DvsPortset-1 00:50:56:01:10:e1 vmnic2 67108868 3 0 DvsPortset-1 00:50:56:67:0f:0f vmk10 67108869 3 0 DvsPortset-1 00:50:56:6a:3b:45 vmk50 67108878 5 0 DvsPortset-1 00:50:56:96:0e:25 photon-dhcp-client.eth0 [ Note that Guest VM photon-dhcp-client is connected to vDS PortNum 67108878:
Perform traffic capture with dir 0 to monitor packets received on the vDS :
[root@esxcna03-s1:~] pktcap-uw --switchport 67108878 --dir 0 -o - | tcpdump-uw -r - -nn reading from file -, link-type EN10MB (Ethernet) 09:50:35.528093 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:50:56:96:0e:25, length 295 09:51:39.651635 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:50:56:96:0e:25, length 295 09:52:43.018149 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:50:56:96:0e:25, length 295 09:53:30.490694 IP6 fe80::250:56ff:fe96:e25.546 > ff02::1:2.547: dhcp6 inf-req (The Guest VM with MAC address 00:50:56:96:0e:25 is sending a broadcast request, initiating the DHCP request process.) Perform traffic capture, with dir 1, to monitor packets received on the vDS: [root@esxcna03-s1:~] pktcap-uw --switchport 67108871 --dir 1 -o - | tcpdump-uw -r - -nn reading from file -, link-type EN10MB (Ethernet) 10:08:32.469028 IP 0.0.0.0 > 224.0.0.1: igmp query v3 [max resp time 1.0s] 10:10:37.469419 IP 0.0.0.0 > 224.0.0.1: igmp query v3 [max resp time 1.0s] (There is no DHCP reply.)
From this we have been able to determine that the DHCP request leaves the Guest VM, but that no response is received.
2. Traffic Capture at the Downlink
The Guest VM’s default gateway is a Tier-1 gateway logical interface. With the Guest VM on the Compute Cluster and the Edge on the Edge Cluster, the DHCP request needs to exit ESXi host esxcna-s1 on a physical network interface. To set up for this capture we will need to identify the ESXi host physical NIC, and perform the capture before GEVEVE Tunnel encapsulation, so that it’s readable.
From the ESXi host, use esxtop: [root@esxcna03-s1:~] esxtop - then type n, to look at network details
Traffic will be exiting the host on physical NIC vmnic2.
pktcap-uw --uplink vmnic2 --dir 1 -o - | tcpdump-uw -r - -nn 10:39:45.836434 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:50:56:96:0e:25, length 295 10:39:45.836678 IP 192.168.70.1.67 > 192.168.100.2.67: BOOTP/DHCP, Request from 00:50:56:96:0e:25, length 295
Now things are getting interesting!
- with DHCP relay configured on the Tier-1 Gateway, the DHCP request is sourced from the Tier-1 gateway logical interface, 192.168.70.1.
- the destination is UDP port 67, to DHCP Relay target IP 192.168.100.2.
- the ESXi host Tier-1 DR picks up 0.0.0.0.68 > 255.255.255.255.67, and translates it to 192.168.70.1.67 > 192.168.100.2.67
If you are not seeing this, then there is something incorrectly configured. Review the NSX-T Configuration notes for DHCP Relay shown above.
3. Traffic Capture at the RouterLink
So far all our traffic captures have been ESXi hosts based, but now we’ll be moving over to the Edge, to verify the DHCP request is making it over the Tier-1 to Tier-0 RouterLink.
Find the vrf for SERVICE_ROUTER_TIER1
nsxtedge01> get logical-router
Logical Router
UUID VRF LR-ID Name Type Ports
736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL 3
7be9fece-e558-4949-b1a2-eaffa26fe0c5 21 8194 SR-lab-tier-0 SERVICE_ROUTER_TIER0 6
51a2da6f-3a98-43c1-9ed0-5c94bccd47ab 22 6149 SR-lab-tier-1 SERVICE_ROUTER_TIER1 5
3ef116ea-7adc-48bb-bc89-89fd16502087 23 6146 DR-lab-tier-0 DISTRIBUTED_ROUTER_TIER0 4
f6826f40-7fe1-4882-92c7-f15b5ffd1a24 25 6148 DR-lab-tier-1 DISTRIBUTED_ROUTER_TIER1 4
The Tier-1 Service Router is vrf 22, which is Active/Active on nsxtedge01 and nsxtedge02, so it doesn't matter on which Edge this step is performed.
nsxtedge01> vrf 22
nsxtedge01(tier1_sr)> get high-availability status
Service Router
UUID : 51a2da6f-3a98-43c1-9ed0-5c94bccd47ab
state : Active
type : TIER1
mode : A/S
failover mode : Non-preemptive
rank : 1
service count : 0
service score : n/a
HA ports state
UUID : 7f6dcbe0-36d4-4928-992d-b33847136a2d
op_state : Up
addresses : f
nsxtedge01 is Active.
The interface that interconnects Tier-1 and Tier-0, in our case is lab-tier-0-lab-tier-1-t1_lrp, by default with an IP 100.64.0.1. We will need it's UUID:
nsxtedge01(tier1_sr)> get int | find Name|IP|MAC|VNI|Interface
UUID VRF LR-ID Name Type
Interfaces
Interface : 456aff90-7b17-4599-8a83-324266e6f368
Name : bp-dr-port
IP/Mask : 169.254.0.1/28;fe80::50:56ff:fe56:4452/64
MAC : 02:50:56:56:44:52
VNI : 71689
Interface : fae133d1-f989-4432-bbc4-fa94788aa7dd
Name : infra-dhcp-client-subnet-dlrp
IP/Mask : 192.168.70.1/24
MAC : 02:50:56:56:44:52
VNI : 71688
Interface : 0f2e1f4d-945a-5d16-9703-6865d24ccda6
Interface : 9420646c-a6e8-5330-be5f-26f65a9b14b5
UUID VRF LR-ID Name Type
Interfaces
Interface : 7f6dcbe0-36d4-4928-992d-b33847136a2d
Name : lab-tier-0-lab-tier-1-t1_lrp
IP/Mask : 100.64.0.1/31;fe80::50:56ff:fe56:4455/64;fc17:b79d:a22e:6800::2/64
MAC : 02:50:56:56:44:55
VNI : 71681
Interface : 7065626d-f831-4c4e-a070-e5465b25e01a
Name : bp-sr0-port
IP/Mask : 169.254.0.2/28;fe80::50:56ff:fe56:5300/64
MAC : 02:50:56:56:53:00
VNI : 71689
Interface : 49d99ff6-ee9f-4e33-9488-cbe7ed5232e1
IP/Mask : 127.0.0.1/8;::1/128
Interface : 714366db-18b8-50fd-ba24-893ea87daacf
Interface : 0719f236-02dc-5a87-9395-e4d2fcea2beb
nsxtedge01(tier1_sr)> exit
lab-tier-0-lab-tier-1-t1_lrp is interface 7f6dcbe0-36d4-4928-992d-b33847136a2d.
Now we are ready for the traffic capture:
nsxtedge01> start capture interface 7f6dcbe0-36d4-4928-992d-b33847136a2d direction dual expression ip 192.168.70.1
11:40:32.394409 02:50:56:56:44:55 > 02:50:56:56:44:52, ethertype IPv4 (0x0800), length 337: 192.168.70.1.67 > 192.168.100.2.67: BOOTP/DHCP, Request from 00:50:56:96:0e:25, length 295
AlBWVkRSAlBWVkRVCABFwAFDAAAAAAoRg5bAqEYBwKhkAgBDAEMBLwAAAQEGAAnjWWYaDQAAAAAAAAAAAAAAAAAAwKhGAQBQVpYOJQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABjglNjNQEBPRP/tiIP6wACAACrEXZBt7fc75RFNwgBAwwPBiF5KjkCAkAMDnBob3Rvbi1tYWNoaW5l/w==
11:40:32.396563 02:50:56:56:44:52 > 02:50:56:56:44:55, ethertype IPv4 (0x0800), length 70: 192.168.100.2 > 192.168.70.1: ICMP 192.168.100.2 udp port 67 unreachable, length 36
AlBWVkRVAlBWVkRSCABFwAA4LZAAAP4BYyDAqGQCwKhGAQMD+0cAAAAARcABQwAAAAAIEYWWwKhGAcCoZAIAQwBDAS8AAA==
4. Traffic Capture at the Uplink
We will continue with an Edge capture, but this time on the Tier-0 uplink to physical, known in this setup as interface external-uplink1.
Find the vrf for SERVICE_ROUTER_TIER0
nsxtedge01> get logical-router
Logical Router
UUID VRF LR-ID Name Type Ports
736a80e3-23f6-5a2d-81d6-bbefb2786666 0 0 TUNNEL 3
7be9fece-e558-4949-b1a2-eaffa26fe0c5 21 8194 SR-lab-tier-0 SERVICE_ROUTER_TIER0 6
51a2da6f-3a98-43c1-9ed0-5c94bccd47ab 22 6149 SR-lab-tier-1 SERVICE_ROUTER_TIER1 5
3ef116ea-7adc-48bb-bc89-89fd16502087 23 6146 DR-lab-tier-0 DISTRIBUTED_ROUTER_TIER0 4
f6826f40-7fe1-4882-92c7-f15b5ffd1a24 25 6148 DR-lab-tier-1 DISTRIBUTED_ROUTER_TIER1 4
The Tier-0 Service Router is vrf 21.
nsxtedge01> vrf 21
nsxtedge01(tier0_sr)> get int | find Name|IP|MAC|VNI|Interface
UUID VRF LR-ID Name Type
Interfaces
Interface : 412553f5-8fc2-4d69-adf8-20679c116cc5
Name : lab-tier-0-lab-tier-1-t0_lrp
IP/Mask : 100.64.0.0/31;fc17:b79d:a22e:6800::1/64;fe80::50:56ff:fe56:4452/64
MAC : 02:50:56:56:44:52
VNI : 71681
Interface : a61773ef-855d-4532-843b-21135ab43d28
Name : bp-dr-port
IP/Mask : 169.254.0.1/25;fe80::50:56ff:fe56:4452/64
MAC : 02:50:56:56:44:52
VNI : 71683
Interface : 766d09cb-84fa-52f1-ab94-2ee237bf8078
Interface : 434b3511-b209-524d-8150-6ea91189b169
UUID VRF LR-ID Name Type
Interfaces
Interface : 7cc827ff-bde3-4f45-afc5-751d18d16c14
Name : external-uplink1
IP/Mask : 192.168.100.102/24
MAC : 00:50:56:96:6f:58
Interface : 296743c2-af2d-5879-bbd6-c9a636748171
Interface : df10c015-3abf-4966-9063-be69e2d006ec
Name : sr1-internal-routing-port
IP/Mask : 169.254.0.131/25;fe80::50:56ff:fe56:5201/64
MAC : 02:50:56:56:52:01
VNI : 71684
Interface : 754701a9-6ee5-4e86-9f51-e39b3989732a
Name : bp-sr1-port
IP/Mask : 169.254.0.3/25;fe80::50:56ff:fe56:5301/64
MAC : 02:50:56:56:53:01
VNI : 71683
Interface : 16d4537c-5242-5ae5-a729-6d12284033f9
Interface : a4634c58-2130-455e-b0a6-8ef1ae1d3b38
IP/Mask : 127.0.0.1/8;::1/128
external-uplink1 is interface 7cc827ff-bde3-4f45-afc5-751d18d16c14.
nsxtedge01(tier0_sr)> exit
nsxtedge01> start capture interface 7cc827ff-bde3-4f45-afc5-751d18d16c14 direction dual expression ip 192.168.70.1
12:04:05.384971 00:50:56:96:6f:58 > 00:50:56:01:3c:bc, ethertype IPv4 (0x0800), length 337: 192.168.70.1.67 > 192.168.100.2.67: BOOTP/DHCP, Request from 00:50:56:96:0e:25, length 295
AFBWATy8AFBWlm9YCABFwAFDAAAAAAkRhJbAqEYBwKhkAgBDAEMBLwAAAQEGAAnjWWYfkgAAAAAAAAAAAAAAAAAAwKhGAQBQVpYOJQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABjglNjNQEBPRP/tiIP6wACAACrEXZBt7fc75RFNwgBAwwPBiF5KjkCAkAMDnBob3Rvbi1tYWNoaW5l/w==
From this we have been able to determine that the DHCP request exits Tier-0 to physical, but that no response is being received. In this case we've identified that the issue is outside the NSX-T environment!
The Solution:
In this scenario, we find that the Cisco IOS based DHCP service has been administratively shut down:
vRouter_DC01#sh running-config version 15.4 service timestamps debug datetime msec service timestamps log datetime msec no service dhcp <-------------------------- this is an issue! no platform punt-keepalive disable-kernel-core platform console virtual ! hostname vRouter_DC01 ! ... ip domain lookup source-interface GigabitEthernet3 ip domain name core.hypervizor.com ip name-server 192.168.110.10 ip dhcp excluded-address 192.168.70.1 192.168.70.49 ! ip dhcp pool dhcp-client-pool import all network 192.168.70.0 255.255.255.0 default-router 192.168.70.1 dns-server 192.168.110.10 domain-name core.hypervizor.com Let's restart the Cisco IOS DHCP server vRouter_DC01#conf terminal Enter configuration commands, one per line. End with CNTL/Z. vRouter_DC01(config)#service dhcp At the same time, let's looks for a DHCP reply back on the ESXi hosts vDS port: [root@esxcna03-s1:~] pktcap-uw --switchport 67108878 --dir 0 -o - | tcpdump-uw -r - -nn reading from file -, link-type EN10MB (Ethernet) tcpdump-uw: pcap_loop: error reading dump file: Interrupted system call [root@esxcna03-s1:~] pktcap-uw --switchport 67108871 --dir 1 -o - | tcpdump-uw -r - -nn reading from file -, link-type EN10MB (Ethernet) 12:21:10.145263 ARP, Request who-has 192.168.70.50 tell 192.168.70.1, length 46 12:21:10.543664 ARP, Request who-has 192.168.70.50 tell 192.168.70.1, length 46 12:21:11.145513 ARP, Request who-has 192.168.70.50 tell 192.168.70.1, length 46 12:21:11.543481 ARP, Request who-has 192.168.70.50 tell 192.168.70.1, length 46 12:21:12.143247 IP 192.168.70.1.67 > 192.168.70.50.68: BOOTP/DHCP, Reply, length 328 12:21:12.145332 ARP, Request who-has 192.168.70.50 tell 192.168.70.1, length 46 12:21:12.148482 IP 192.168.70.1.67 > 192.168.70.50.68: BOOTP/DHCP, Reply, length 328 12:21:17.161778 ARP, Reply 192.168.70.1 is-at 02:50:56:56:44:52, length 46 12:21:17.170259 IP 192.168.110.10.53 > 192.168.70.50.60359: 39210 1/0/1 A 216.239.35.0 (61) 12:21:17.170459 IP 192.168.110.10.53 > 192.168.70.50.37973: 17776 1/0/1 AAAA 2001:4860:4806:: (73) 12:21:17.246284 IP 216.239.35.0.123 > 192.168.70.50.43897: NTPv4, Server, length 48 12:21:49.554039 IP 216.239.35.0.123 > 192.168.70.50.43540: NTPv4, Server, length 48
192.168.70.1.67 > 192.168.70.50.68: BOOTP/DHCP is the response we’ve been looking for. The Guest VM has received an IP address of 192.168.70.50.
The goal of this post has been to demonstrate that with a combination of ESXi host and NSX-T Edge traffic captures, it’s possible to methodically follow IP traffic over the communication path, to help isolate possible issues.
An important caveat to be aware of:
Once an IP Address Management type, say DHCP Server or DHCP Relay has been applied to a Tier-1 Gateway, it can’t be changed. This means you need to plan the DHCP setup carefully in advance before deployment, or you’re going to need to delete and then redeploy the Tier-1 Gateway.
Verify the setup through Advanced Networking a Security:
With the configuration effort having been performed through the Simplified UI, you can check your work in Advanced Networking and Security. You will not be able to make changes here since “These fields are defined by another object in NSX and can’t be edited.” This means that since these are Policy UI created objects you can’t edit then via the Management Plane UI, but as a sanity check, you can look here to verify that they have been realized correctly.
Additional details on DHCP Server Configuration in NSX-T can be found here: https://spillthensxt.com/dhcp-server-configuration-in-nsx-t/
