How to disable Bosh Resurrector in TKGi

Bosh resurrection is one of the amazing features that are available in the Tanzu Kubernetes Grid Integrated environment. Recently, I was working on an issue on the Tanzu Kubernetes Grid Integrated environment and learned something new. I thought it might help someone else and hence writing it down here.

What is BOSH Resurrector ?

The Bosh Resurrector is a plugin that is responsible for automatically monitoring and recreating VMs that become inaccessible. It Reacts to server hardware failure and network disruptions by recreating virtual machines on active, stable ESXi hosts. Resurrector detects operating system failures of bosh deployed and managed VMS by continuously monitoring virtual machines and recreating them as required.

How to enable Bosh Resurrector?

To enable bosh resurrector, login to the Operations Manager web interface and select the bosh director tile in the product dashboard. Now, go to director config and select Enable VM Resurrector Plugin. Once done, save the settings and apply the changes.

Check Bosh Resurrection status

To check the status of the bosh resurrection, you can run the below curl command. As in the below image, you can see the resurrection is true for an environment that is enabled.

bosh curl /resurrection

bosh -d service-instance_9d9a8b55-1fef-4c51-846e-687eaa5e260a curl /resurrection

Disable Bosh Resurrection

To disable the bosh resurrection, you can run the below bosh command. This command will disable resurrection on all the deployments in the current bosh environment.

bosh update-resurrection off

You may also run the below bosh command to disable the resurrection. However, the point to note is that even though the command is pointing to one TKGi cluster, it will still disable bosh resurrection on all the bosh deployments in the environment. You may review the below image for reference. I have tested this in TKGi 1.10 version and I believe this is the same in the latest versions as well.

bosh -d service-instance_9d9a8b55-1fef-4c51-846e-687eaa5e260a update-resurrection off

Disable Bosh Resurrection in a single deployment

Now the question is how to disable bosh resurrection on a single bosh deployment for any maintenance activities in one cluster. It’s clear that bosh -d <Deployment_ID> update-resurrection off command is not helpful to disable the resurrection in a single cluster.

To disable bosh resurrection on an individual cluster, follow the below steps. 

1. Create a resurrection.yml file with the below content

rules:
- enabled: false
  include:
    deployments:
    - service-instance_9d9a8b55-1fef-4c51-846e-687eaa5e260a

• 2. Run the below command to disable bosh resurrection for the cluster service-instance_9d9a8b55-1fef-4c51-846e-687eaa5e260a

bosh update-config –type resurrection –name <disable-resurrection-cluster-1> resurrection.yml 

When you run bosh configs command, you will see a new bosh config added for disabling the resurrection

Note: Bosh curl command that we ran earlier will still report the bosh resurrection is enabled for the cluster. However, I have confirmed that bosh is not recreating the VM that goes down from this TKGi cluster.

Once the maintenance activity is completed, to turn the resurrection back on in the deployment above, you can delete the bosh config with the help of the below bosh command. The config ID in this example is 17 as in the above image.

bosh delete-config <Config_ID>

Disable resurrection for all deployments except for one

To disable resurrection on all the deployments except for service-instance_9d9a8b55-1fef-4c51-846e-687eaa5e260a, use the below resurrection.yml file

rules:
- enabled: false
  exclude:
    deployments:
    - service-instance_9d9a8b55-1fef-4c51-846e-687eaa5e260a

Thank you for reading this blog post. If it has helped you, feel free to share it with others. Happy Learning!