How to cleanup an orphaned ESXi host that is NSX-T prepared

In my last blog, we saw how to re-register a vCenter server with NSX-T after the NSX-T is redeployed. In this blog post we will see how to cleanup an Orphaned ESXi host which is NSX-T configured.

If we have the NSX-T manager available, we can easily get it done by going to System > Fabric > Nodes > Select the ESXi host and click on REMOVE NSX. This will cleanup the ESXi host and remove all the components added while it got prepared.

However, in a scenario where the NSX-T manager is unavailable or lost permanently, we can follow the below mentioned steps to cleanup the ESXi hosts manually.

Review the ESXi :

I have a ESXi host which is prepared for NSX-T. Let’s see the vmkernel interfaces currently added to the host by running esxcfg-vmknic -l command.

As in the below screenshot, we have vmk10 and vmk50 added to the ESXi host.

Run esxcfg-vswitch -l to see the virtual switches added. As you see in the below screenshot, Overlay-TZ-NVDS is added to the host when it got prepared.

Now, let’s run nsxdp-cli vswitch instance list to figure out what vmkernel interfaces are connected to the N-VDS. You can also see which vmnics are connected as well.

Cleanup The ESXi :

Before starting, make sure there is no virtual machines running on the ESXi that you are planning to cleanup. It is recommended to put the ESXi into maintenance mode.

Once ready, login to the ESXi host using SSH and run vsipioctl clearallfilters command to remove filters.

[root@esxi-3:~] vsipioctl clearallfilters
 ERROR: Command clearallfilters is dangerous and can cause unintended consequence!
 ERROR: Please supply first option -Override in order to override the safety guard and actually run the command.
 [root@esxi-3:~]

Note: You will have to append ‘-Override’ to run it successfully.

[root@esxi-3:~] vsipioctl clearallfilters -Override
 Removing all vmware-sfw filters…
 Cleared dvfilter include table.
 Updated all VMs to remove filters.
 Destroyed all filters (please ignore `Function not implemented' error if there is).

From the ESXi host, run /etc/init.d/netcpad stop command to stop netcpa.

[root@esxi-3:~] /etc/init.d/netcpad stop
 netCP agent service monitor is not running
 watchdog-netcpa: Terminating watchdog process with PID 2103498
 Clear HyperBus ARPs
 watchdog-dfwpktlogs: Terminating watchdog process with PID 2103475
 Memory reservation released for netcpa
 netCP agent service is stopped

From the ESXi host, get into nsxcli command mode and run del nsx command to manually uninstall the NSX-T Data Center configuration and modules. This command should clean NSX-T configuration from the ESXi host. It is supposed to delete the opaque switch and all the vibs installed on the host.

Once completed, you will see the the below output.

[root@esxi-3:~] nsxcli
 esxi-3.gsslabs.org> del nsx
 Terminated

As you see in the below screenshot, I do not see the vmk10 and vmk50 vmkernel interfaces on the ESXi host.

If the vmkernel Interfaces are not removed, you can run the below command to delete the same manually.

esxcli network ip interface remove --interface-name=vmk50
esxcli network ip interface remove --interface-name=vmk10

The Overlay-TZ-NVDS virtual switch is also removed from the ESXi host.

Now, run the below command and confirm all the NSX-T vibs are also removed.

esxcli software vib list | grep -i nsx

In some cases, I have seen the NSX vibs are left behind even after running del nsx command. Try rebooting the ESXi host and see if it cleans up. If not you can manually remove the vibs by running the below command.

NSX-T 2.4 version :

esxcli software vib remove -n nsx-aggservice -n nsx-cli-libs -n nsx-common-libs -n nsx-context-mux -n nsx-esx-datapath -n nsx-exporter -n nsx-host -n nsx-metrics-libs -n nsx-mpa -n nsx-nestdb-libs -n nsx-nestdb -n nsx-netcpa -n nsx-opsagent -n nsx-platform-client -n nsx-profiling-libs -n nsx-proxy -n nsx-python-gevent -n nsx-python-greenlet -n nsx-python-logging -n nsx-python-protobuf -n nsx-rpc-libs -n nsx-sfhc -n nsx-shared-libs -n nsx-upm-libs -n nsx-vdpi -n nsxcli --no-live-install --force

NSX-T 2.3 version :

esxcli software vib remove -n nsx-aggservice -n nsx-cli-libs -n nsx-common-libs -n nsx-da -n nsx-esx-datapath -n nsx-exporter -n nsx-host -n nsx-metrics-libs -n nsx-mpa -n nsx-nestdb-libs -n nsx-nestdb -n nsx-netcpa -n nsx-opsagent -n nsx-platform-client -n nsx-profiling-libs -n nsx-proxy -n nsx-python-gevent -n nsx-python-greenlet -n nsx-python-logging -n nsx-python-protobuf -n nsx-rpc-libs -n nsx-sfhc -n nsx-shared-libs -n nsxcli -n epsec-mux --no-live-install --force

Once done, run nsxcli from the ESXi host and confirm vib is removed. The command should fail now.

Finally, you may also delete the NSX-T related log files from /var/log on the ESXi host.

[root@esxi-3:/var/log] ls -l | grep nsx
 lrwxrwxrwx    1 root     root            16 Sep  3 05:24 nsx -> /scratch/log/nsx
 lrwxrwxrwx    1 root     root            27 Sep  3 06:12 nsx-nestdb.log -> /scratch/log/nsx-nestdb.log
 lrwxrwxrwx    1 root     root            29 Sep  3 06:12 nsx-opsagent.log -> /scratch/log/nsx-opsagent.log
 lrwxrwxrwx    1 root     root            26 Sep  3 06:12 nsx-proxy.log -> /scratch/log/nsx-proxy.log
 -rw-r--r--    1 root     root             0 Sep  3 05:36 nsxaVim.err
 lrwxrwxrwx    1 root     root            24 Sep  3 06:12 nsxaVim.log -> /scratch/log/nsxaVim.log
 -rw-r--r--    1 root     root            64 Sep  3 05:36 nsxaVim.ps
 lrwxrwxrwx    1 root     root            23 Sep  3 05:24 nsxcli.log -> /scratch/log/nsxcli.log

The steps mentioned in this blog will help to cleanup just the NSX-T configuration from ESXi host. We can also use the Reset System Configuration option that is available from the DCUI of the ESXi host to clean up. However this will reset all the configuration of the ESXi and we will have to start from scratch.

You can also refer VMware document 2.3 and 2.4 for details.

---