The /var/log partition grows exponentially on VMware NSX-T Edge Node

Recently, I worked on an issue with NSX-T Edge node, where the /var/log partition grows to 100%. On further investigation, we identified that the issue is seen in NSX-T version 2.3.0 and 2.3.1. I am writing it down here if anyone faces the same issue and needed a workaround.

Symptoms :

  • /var/log partition is 100%
  • /var/log/lb/ directory takes up all the space
  • lbconf_gen.log files does not get rotated and grows exponentially
  • Issue is seen in NSX-T Edge nodes in version 2.3.0 and 2.3.1 

Cause :

The issue occurs because the lbconf_gen.log under /var/log/lb/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/ grows exponentially without getting rotated and fills up the space.

Resolutions :

To workaround this issue, append the below entries to /etc/logrotate.d/nsx-edge-lb file on the Edge node.

/var/log/lb/*/lbconf_gen.log
{
daily
missingok
rotate 3
maxsize 3K
compress
delaycompress
notifempty
create 0640 lb lb
postrotate
uuid=echo $1 | cut -d '/' -f5
chown root:adm /var/log/lb/$uuid/lbconf_gen.log.2.gz
endscript
}

Make sure to change the rotate and maxsize parameters to reasonable values


NOTE : If your NSX-T environment is running in production, I suggest you to take assistance from the VMware NSX-T technical support team to validate and execute the workaround without making any mistakes.


2 thoughts on “The /var/log partition grows exponentially on VMware NSX-T Edge Node

Add yours

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: