Background
NSX-T 2.4.2 includes a new ESXi host Python script named nsxcfg-vswitch which can be used to recover network connectivity to an N-VDS based kernel management interface. I learned about this gem from one of my peers, NSX-T Escalation Engineer Micheal (Rui) Liu.
Included as part of VIB installation, think of nsxcfg-vswitch as an emergency tool that can be used to restore management network connectivity to a host by migrating the management kernel interface from an NSX-T vSphere Distributed Switch (N-VDS) to a vSphere Standard Switch (VSS), without impacting any NSX-T features.
This same operation can be performed with some additional effort using a more traditional method, with esxcfg-vswitch, as documented here: https://kb.vmware.com/s/article/71080.
Keep in mind that nsxcfg-vswitch was not available in NSX-T 2.4.1:
[root@esxcna01-s1:~] esxcli software vib list | grep nsx-opsagent nsx-opsagent 2.4.1.0.0-6.5.13716717 <- NSX-T 2.4.1 [root@esxcna01-s1:~] find . -name nsxcfg-vswitch <- no matches
nsxcfg-vswitch is available in NSX-T 2.4.2 on ESXi hosts prepared with NSX-T VIBs:
[root@esxcna01-s1:~] esxcli software vib list | grep nsx-opsagent nsx-opsagent 2.4.2.0.0-6.5.14269648 <- NSX-T 2.4.2 [root@esxcna01-s1:~] find . -name nsxcfg-vswitch ./usr/lib64/vmware/nsx-opsagent/bin/nsxcfg-vswitch <- location of Python script
Sample Procedure, moving Management kernel interface vmk0 from an N-VDS to a VSS:
1) With loss of connectivity to vmk0, unless reachable via another kernel interface, connect to the host console using the Direct Console User Interface (DCUI):
F2 to enter the DCUI and verify ESXi Shell is enabled
Alt-F1 to switch to console, for command-line access
2) Use esxcfg-vswitch to verify that virtual Standard Switch vSwitch0 exists and has the unused PortGroup named “Management Network”:
[root@esxcna01-s1:~] esxcfg-vswitch -l Switch Name Num Ports Used Ports Configured Ports MTU Uplinks vSwitch0 1536 1 128 1500 <--- vSwitch0 exists PortGroup Name VLAN ID Used Ports Uplinks Management Network 0 0 <--- Management Network exists
3) If not, use esxcfg-vswitch to create vSwitch0 or add a new portgroup to vSwitch0, as detailed here: https://kb.vmware.com/s/article/1000258
https://kb.vmware.com/s/article/1000258
[root@esxcna01-s1:~] esxcfg-vswitch –a vSwitch0 [root@esxcna01-s1:~] esxcfg-vswitch –A "Management Network" vSwitch0 [root@esxcna01-s1:~] esxcfg-vswitch -l
4) Review the existing N-VDS Setup:
[root@esxcna01-s1:/usr/lib64/vmware/nsx-opsagent/bin] nsxdp-cli vswitch instance list ... DvsPortset-2 (NSXTvlan) 3f b0 1c 1c 9e b2 42 75-96 5c 10 7b 00 f9 9e 55 Total Ports:1536 Available:1515 Client PortID DVPortID MAC Uplink Management 83886081 00:00:00:00:00:00 n/a vmnic3 83886082 uplink1 00:00:00:00:00:00 Shadow of vmnic3 83886083 00:50:56:5e:eb:56 n/a vdr-vdrPort 83886084 vdrPort 02:50:56:56:44:52 vmnic3 vmk0 83886085 e4f20136-e2d5-4ec6-a126-c6e387777f30 00:50:56:6a:f5:24 vmnic3 - notice that Management kernel interface vmk0 is on the N-VDS named NSXTvlan, over physical NIC vmnic3
5) Sop the nsx-opsagent process:
[root@esxcna01-s1:~] /etc/init.d/nsx-opsagent status opsAgent is running [root@esxcna01-s1:~] /etc/init.d/nsx-opsagent stop nsx-opsagent stop watchdog-opsAgent: Terminating watchdog process with PID 68127 nsx-opsagent service is stopped
6) Move kernel interface vmk0 and physical interface vmnic3 from the N-VDS to VSS vSwitch0:
[root@esxcna01-s1:~] cd /usr/lib64/vmware/nsx-opsagent/bin
[root@esxcna01-s1:/usr/lib64/vmware/nsx-opsagent/bin] ./nsxcfg-vswitch
Usage: nsxcfg-vswitch [options]
Options:
-h, --help show this help message and exit
-l, --list List all N-VDS
-L, --list-details List details of N-VDS, network or dvport
-d DVSWITCH, --dvswitch=DVSWITCH
N-VDS name for --list-details
-n NETWORK, --network=NETWORK
OpaqueNetwork or LogicalSwitch name for --list-details
-D DVP, --dvp=DVP Port or VIF ID for --list-details
-m, --migrate-into-vss
Migrate VMkernel interface and/or Physical NIC into
VSS from N-VDS
-I VMK, --vmk=VMK VMkernel interface to migrate for --migrate-into-vss
-p PG, --pg=PG Destination VSS portgroup for --migrate-into-vss
-P PNIC, --pnic=PNIC Physical NIC to migrate for --migrate-into-vss
[root@esxcna03-s1:/usr/lib64/vmware/nsx-opsagent/bin]
run "./nsxcfg-vswitch -m -p <portgroup> -I <management-vmk-name> -P <pnic>" to migrate the management vmk and the pnic into the VSS that owns the portgroup. For instance, by default, management vmk is vmk0 connected to portgroup "Management Network" inside vSwitch0 and it shares MAC with vmnic0, and they were both in vSwitch0 before they were migrated into NVDS, to restore them, run
esxcna03# ./nsxcfg-vswitch -m -p "Management Network" -I vmk0 -P vmnic3
The CLI can also be used to migrate only the vmk or pnic, like this:
./nsxcfg-vswitch -m -p <portgroup> -I <management-vmk-name>
./nsxcfg-vswitch -m -p <portgroup> -P <pnic>
[root@esxcna01-s1:/usr/lib64/vmware/nsx-opsagent/bin] ./nsxcfg-vswitch -m -p "Management Network" -I vmk0 -P vmnic3
Migrating PNIC vmnic3 out of NVDS [3f b0 1c 1c 9e b2 42 75-96 5c 10 7b 00 f9 9e 55]…
Migrated PNIC vmnic3 out of NVDS [3f b0 1c 1c 9e b2 42 75-96 5c 10 7b 00 f9 9e 55]
Migrating vmk0 vmnic3 into VSS vSwitch0
Migration is done
7) Use esxtop, to verify that vmk0 is now over vmnic3:
root@esxcna01-s1:~] esxtop then type "n" for the network view: 10:06:22pm up 2 days 12:40, 623 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.01, 0.01, 0.01 PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PSZTX PKTRX/s MbRX/s PSZRX %DRPTX %DRPRX 33554433 Management n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 33554435 vmnic3 - vSwitch0 0.00 0.00 0.00 2678.01 12.95 633.00 0.00 0.00 33554436 Shadow of vmnic3 n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50331649 Management n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50331650 vmnic0 - DvsPortset-0 0.00 0.00 0.00 2678.01 12.95 633.00 0.00 0.00 50331651 Shadow of vmnic0 n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50331652 vmnic1 - DvsPortset-0 12.36 0.02 221.00 2665.65 12.93 635.00 0.00 0.00 50331653 Shadow of vmnic1 n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50331654 vmk0 vmnic3 DvsPortset-0 6.37 0.01 265.00 8.11 0.01 121.00 0.00 0.00 67108865 Management n/a DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108868 vmk10 void DvsPortset-1 1.16 0.00 60.00 0.00 0.00 0.00 0.00 0.00 67108869 vmk50 void DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 67108870 vdr-vdrPort void DvsPortset-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 83886081 Management n/a DvsPortset-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 83886084 vdr-vdrPort void DvsPortset-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
8) Go to MP UI to fix the transport node config so that it won’t migrate the management vmk and the pnic into the N-VDS again, either by edit the transport node if it is standalone, or the transport node profile if the transport node is part of a cluster using the profile.
9) Start the nsx-opsagent process:
root@esxcna01-s1:~] /etc/init.d/nsx-opsagent start
10) check MP UI to make sure the transport node’s config status is success.
Keep in mind that NSX-T 2.4.2 includes nsxcfg-vswitch, which may help with ESXi host recovery. If you liked this article, you may be interested in configuring NSX-T Edge connectivity to physical.
