Updating the Harbor certificate in Enterprise PKS fails

In this blog post, I will discuss about an issue faced while updating the harbor custom certificate from Enterprise PKS Management Console(EPMC). Review this Pivotal document for details on how to update the custom certificate on harbor vm in Enterprise PKS.

Symptoms:

• Error: “failed to apply Harbor config”
• This issue occurs if this is the task is performed from the EPMC or Opsmanager
• Harbor is using NFS-backed storage for harbor-registry
• Once failed, the Harbor deployment VM is in a stopped state
• Monit summary shows no process running in the Harbor vm
• bosh task give the below error

Task 1854 | 22:04:51 | Updating instance harbor-app: harbor-app/2becda9c-b215-48f4-b932-7734f4c6a4b2 (0) (canary) (00:01:37) L Error: Action Failed get_task: Task e0ce7d47-7741-4812-6499-319e7231ffca result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront. Task 1854 | 22:06:28 | Error: Action Failed get_task: Task e0ce7d47-7741-4812-6499-319e7231ffca result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.

You see messages similar to the following in the /var/vcap/sys/log/harbor/pre-start.stderr.log

Removing network harbor-app_harbor-clair Removing network harbor-app_harbor-notary Removing network harbor-app_harbor-chartmuseum Removing network harbor-app_notary-sig Warning: Stopping rpcbind.service, but it can still be activated by: rpcbind.socket mount.nfs: /harbor-registry is busy or already mounted Warning: Stopping rpcbind.service, but it can still be activated by: rpcbind.socket mount.nfs: /harbor-registry is busy or already mounted (END)

Manually starting the harbor vm fails with the below error.

Task 1871 | 22:24:18 | Updating instance harbor-app: harbor-app/2becda9c-b215-48f4-b932-7734f4c6a4b2 (0) (canary) (00:00:55) L Error: Action Failed get_task: Task 041411e1-88ed-4890-6772-fe24064dc7f3 result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront. Task 1871 | 22:25:13 | Error: Action Failed get_task: Task 041411e1-88ed-4890-6772-fe24064dc7f3 result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront.

Cause:

This issue can occur if the NFS share is already mounted and a re-mount attempt is made to the same folder. You can run the below command to confirm that the NFS share is already mounted. 

cat /proc/mounts | grep nfs
10.20.x.x:/vmware_harbor_repo /harbor-registry nfs4 rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.31.0.5,local_lock=none,addr=10.20.x.x 0 0

Resolution:

This is a known issue affecting Harbor and there is currently no resolution. However you can workaround this issue by manually un-mounting the registry from the harbor vm by running the below command from inside the harbor node.

sudo -i
umount /harbor-registry

Once this is done, apply the changes again from EPMC or Opsmanager.

---