The /var/log partition grows exponentially on VMware NSX-T Edge Node

Recently, I worked on an issue with NSX-T Edge node, where the /var/log partition grows to 100%. On further investigation, we identified that the issue is seen in NSX-T version 2.3.0 and 2.3.1. I am writing it down here if anyone faces the same issue and needed a workaround.

Symptoms :

• /var/log partition is 100%
• /var/log/lb/ directory takes up all the space
• lbconf_gen.log files does not get rotated and grows exponentially
• Issue is seen in NSX-T Edge nodes in version 2.3.0 and 2.3.1 

Cause :

The issue occurs because the lbconf_gen.log under /var/log/lb/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/ grows exponentially without getting rotated and fills up the space.

Resolutions :

To workaround this issue, append the below entries to /etc/logrotate.d/nsx-edge-lb file on the Edge node.

/var/log/lb/*/lbconf_gen.log
{
        daily
        missingok
        rotate 3
        maxsize 3K
        compress
        delaycompress
        notifempty
        create 0640 lb lb
        postrotate
                uuid=echo $1 | cut -d '/' -f5
                chown root:adm /var/log/lb/$uuid/lbconf_gen.log.2.gz
        endscript
}

Make sure to change the rotate and maxsize parameters to reasonable values

---

NOTE : If your NSX-T environment is running in production, I suggest you to take assistance from the VMware NSX-T technical support team to validate and execute the workaround without making any mistakes.

---