In my last blog, we saw how to re-register a vCenter server with NSX-T after the NSX-T is redeployed. In this blog post we will see how to cleanup an Orphaned ESXi host which is NSX-T configured.
If we have the NSX-T manager available, we can easily get it done by going to System > Fabric > Nodes > Select the ESXi host and click on REMOVE NSX. This will cleanup the ESXi host and remove all the components added while it got prepared.
However, in a scenario where the NSX-T manager is unavailable or lost permanently, we can follow the below mentioned steps to cleanup the ESXi hosts manually.
Review the ESXi :
I have a ESXi host which is prepared for NSX-T. Let’s see the vmkernel interfaces currently added to the host by running
esxcfg-vmknic -l command.
As in the below screenshot, we have vmk10 and vmk50 added to the ESXi host.
esxcfg-vswitch -l to see the virtual switches added. As you see in the below screenshot, Overlay-TZ-NVDS is added to the host when it got prepared.
Now, let’s run
nsxdp-cli vswitch instance list to figure out what vmkernel interfaces are connected to the N-VDS. You can also see which vmnics are connected as well.
Cleanup The ESXi :
Before starting, make sure there is no virtual machines running on the ESXi that you are planning to cleanup. It is recommended to put the ESXi into maintenance mode.
Once ready, login to the ESXi host using SSH and run
vsipioctl clearallfilters command to remove filters.
vsipioctl clearallfiltersERROR: Command clearallfilters is dangerous and can cause unintended consequence! ERROR: Please supply first option -Override in order to override the safety guard and actually run the command. [root@esxi-3:~]
Note: You will have to append ‘-Override’ to run it successfully.
vsipioctl clearallfilters -OverrideRemoving all vmware-sfw filters… Cleared dvfilter include table. Updated all VMs to remove filters. Destroyed all filters (please ignore `Function not implemented' error if there is).
From the ESXi host, run
/etc/init.d/netcpad stop command to stop netcpa.
/etc/init.d/netcpad stopnetCP agent service monitor is not running watchdog-netcpa: Terminating watchdog process with PID 2103498 Clear HyperBus ARPs watchdog-dfwpktlogs: Terminating watchdog process with PID 2103475 Memory reservation released for netcpa netCP agent service is stopped
From the ESXi host, get into nsxcli command mode and run
del nsx command to manually uninstall the NSX-T Data Center configuration and modules. This command should clean NSX-T configuration from the ESXi host. It is supposed to delete the opaque switch and all the vibs installed on the host.
Once completed, you will see the the below output.
[root@esxi-3:~] nsxcli esxi-3.gsslabs.org>
As you see in the below screenshot, I do not see the vmk10 and vmk50 vmkernel interfaces on the ESXi host.
If the vmkernel Interfaces are not removed, you can run the below command to delete the same manually.
esxcli network ip interface remove --interface-name=vmk50 esxcli network ip interface remove --interface-name=vmk10
The Overlay-TZ-NVDS virtual switch is also removed from the ESXi host.
Now, run the below command and confirm all the NSX-T vibs are also removed.
esxcli software vib list | grep -i nsx
In some cases, I have seen the NSX vibs are left behind even after running del nsx command. Try rebooting the ESXi host and see if it cleans up. If not you can manually remove the vibs by running the below command.
NSX-T 2.4 version :
esxcli software vib remove -n nsx-aggservice -n nsx-cli-libs -n nsx-common-libs -n nsx-context-mux -n nsx-esx-datapath -n nsx-exporter -n nsx-host -n nsx-metrics-libs -n nsx-mpa -n nsx-nestdb-libs -n nsx-nestdb -n nsx-netcpa -n nsx-opsagent -n nsx-platform-client -n nsx-profiling-libs -n nsx-proxy -n nsx-python-gevent -n nsx-python-greenlet -n nsx-python-logging -n nsx-python-protobuf -n nsx-rpc-libs -n nsx-sfhc -n nsx-shared-libs -n nsx-upm-libs -n nsx-vdpi -n nsxcli --no-live-install --force
NSX-T 2.3 version :
esxcli software vib remove -n nsx-aggservice -n nsx-cli-libs -n nsx-common-libs -n nsx-da -n nsx-esx-datapath -n nsx-exporter -n nsx-host -n nsx-metrics-libs -n nsx-mpa -n nsx-nestdb-libs -n nsx-nestdb -n nsx-netcpa -n nsx-opsagent -n nsx-platform-client -n nsx-profiling-libs -n nsx-proxy -n nsx-python-gevent -n nsx-python-greenlet -n nsx-python-logging -n nsx-python-protobuf -n nsx-rpc-libs -n nsx-sfhc -n nsx-shared-libs -n nsxcli -n epsec-mux --no-live-install --force
Once done, run
nsxcli from the ESXi host and confirm vib is removed. The command should fail now.
Finally, you may also delete the NSX-T related log files from /var/log on the ESXi host.
[root@esxi-3:/var/log] ls -l | grep nsx lrwxrwxrwx 1 root root 16 Sep 3 05:24 nsx -> /scratch/log/nsx lrwxrwxrwx 1 root root 27 Sep 3 06:12 nsx-nestdb.log -> /scratch/log/nsx-nestdb.log lrwxrwxrwx 1 root root 29 Sep 3 06:12 nsx-opsagent.log -> /scratch/log/nsx-opsagent.log lrwxrwxrwx 1 root root 26 Sep 3 06:12 nsx-proxy.log -> /scratch/log/nsx-proxy.log -rw-r--r-- 1 root root 0 Sep 3 05:36 nsxaVim.err lrwxrwxrwx 1 root root 24 Sep 3 06:12 nsxaVim.log -> /scratch/log/nsxaVim.log -rw-r--r-- 1 root root 64 Sep 3 05:36 nsxaVim.ps lrwxrwxrwx 1 root root 23 Sep 3 05:24 nsxcli.log -> /scratch/log/nsxcli.log
The steps mentioned in this blog will help to cleanup just the NSX-T configuration from ESXi host. We can also use the
Reset System Configuration option that is available from the DCUI of the ESXi host to clean up. However this will reset all the configuration of the ESXi and we will have to start from scratch.
You can also refer VMware document 2.3 and 2.4 for details.