Recently, I worked on an issue with NSX-T Edge node, where the /var/log partition grows to 100%. On further investigation, we identified that the issue is seen in NSX-T version 2.3.0 and 2.3.1. I am writing it down here if anyone faces the same issue and needed a workaround.
- /var/log partition is 100%
- /var/log/lb/ directory takes up all the space
- lbconf_gen.log files does not get rotated and grows exponentially
- Issue is seen in NSX-T Edge nodes in version 2.3.0 and 2.3.1
The issue occurs because the lbconf_gen.log under /var/log/lb/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/ grows exponentially without getting rotated and fills up the space.
To workaround this issue, append the below entries to /etc/logrotate.d/nsx-edge-lb file on the Edge node.
create 0640 lb lb
uuid=echo $1 | cut -d '/' -f5
chown root:adm /var/log/lb/$uuid/lbconf_gen.log.2.gz
Make sure to change the rotate and maxsize parameters to reasonable values
NOTE : If your NSX-T environment is running in production, I suggest you to take assistance from the VMware NSX-T technical support team to validate and execute the workaround without making any mistakes.