VMware Enterprise PKS upgrade failed with error

Recently while working on a VMware Enterprise PKS upgrade, we noticed a issue which halts the upgrade halfway. The error message for the failure was “2 of 5 post-start scripts failed. Failed Jobs: telemetry-agent-image”

Symptoms

Bosh task reports errors:

Task 13565 | 23:39:22 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 13565 | 23:39:25 | Updating instance worker: worker/4427fda7-d962-4ec5-a594-b3526c05394a (3) (canary) (00:02:34)31m
L Error: Action Failed get_task: Task 16e218ba-5984-4fd8-6633-52ea7a790bf7 result: 2 of 5 post-start scripts failed. Failed Jobs: telemetry-agent-image, kubelet. Successful Jobs: bosh-dns, sink-resources-images, wavefront-proxy-images.[0m
Task 13565 | 23:41:59 | [31mError: Action Failed get_task: Task 16e218ba-5984-4fd8-6633-52ea7a790bf7 result: 2 of 5 post-start scripts failed. Failed Jobs: telemetry-agent-image, kubelet. Successful Jobs: bosh-dns, sink-resources-images, wavefront-proxy-images.[0m

post-start.stderr.log: ( Worker > /var/vcap/sys/log/telemetry-agent-image/ )

Error processing tar file(exit status 1): open /usr/share/man/man8/systemd-suspend.service.8.gz: no space left on device
Error processing tar file(exit status 1): open /usr/share/man/man1/watch.1.gz: no space left on device
Error processing tar file(exit status 1): symlink tset.1.gz /usr/share/man/man1/reset.1.gz: no space left on device
post-start.stderr.log (END)

You see an error similar to the following when running the docker-load command:

docker load -i /var/vcap/packages/telemetry-agent-image/pkstelemetrybot_telemetry-agent:fda6005.tar

8241afc74c6f: Loading layer [==================================================>] 120.8MB/120.8MB
Error processing tar file(exit status 1): open /usr/share/man/man8/systemd-remount-fs.service.8.gz: no space left on device

Cause

The local docker registry doesn’t have enough free space
Docker image registry is filled with dangling images

Resolution

To resolve the issue, identify the dangling images in the registry and remove them.

For more information review : how-to-identify-dangling-registry-images-in-docker-registry

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: