Logging into the NSX-T Manager gives error “Some of appliance components is not functioning”

Recently, while I was working in my lab environment, I faced an issue with my NSX-T deployment it was not allowing me to log in to it. The error message did not help me much to understand the root cause of the issue. I am documenting it here in this blog just in case if anyone hits the same issue.

Symptoms :

• Single node NSX-T Manager cluster
• Going to https://IP-Address-of-NSX-T-Mgr/ loads the login page

• Once you click on login, the below error appears

• cbm.log @ /var/log/cbm on the NSX-T manager gives the below error

2019-09-20T09:05:38.498Z INFO CorfuRuntime-0 CorfuRuntime - fetchLayout: Invoking the systemDownHandler.
2019-09-20T09:05:38.543Z INFO netty-16 NettyClientRouter - connectAsync[tcp://192.168.0.15:9000/]: Channel connection failed, reconnecting...
2019-09-20T09:05:39.545Z INFO netty-17 NettyClientRouter - connectAsync[tcp://192.168.0.15:9000/]: Channel connection failed, reconnecting...
2019-09-20T09:05:39.999Z WARN CorfuRuntime-0 CorfuRuntime - Tried to get layout from 192.168.0.15:9000 but failed by timeout
2019-09-20T09:05:39.999Z WARN CorfuRuntime-0 CorfuRuntime - Couldn't connect to any up-to-date layout servers, retrying in PT1S, Retried 7312 times, systemDownHandlerTriggerLimit = 90
2019-09-20T09:05:39.999Z INFO CorfuRuntime-0 CorfuRuntime - fetchLayout: Invoking the systemDownHandler.
(END)

• corfu.9000.log @ /var/log/corfu on the NSX-T Manager gives the below error

2019-09-20T06:03:43.097Z | ESC[34mINFO ESC[0;39m | CorfuServer | o.c.i.ServerContext | getClientGroup: Type NioEventLoopGroup with 24 threads
2019-09-20T06:03:43.152Z | ESC[34mINFO ESC[0;39m | CorfuServer | o.c.i.ServerContext | getWorkerGroup: Type NioEventLoopGroup with 24 threads
2019-09-20T06:03:43.154Z | ESC[34mINFO ESC[0;39m | CorfuServer | o.c.i.ServerContext | getBossGroup: Type NioEventLoopGroup
2019-09-20T06:03:43.167Z | ESC[34mINFO ESC[0;39m | CorfuServer | o.e.sizeof.impl.AgentLoader | Unavailable or unrecognised attach API : java.lang.ClassNotFoundException: com.sun.tools.attach.VirtualMachine
2019-09-20T06:03:43.180Z | ESC[34mINFO ESC[0;39m | CorfuServer | o.e.sizeof.impl.JvmInformation | Detected JVM data model settings of: 64-Bit HotSpot JVM with Compressed OOPs
2019-09-20T06:03:43.185Z | ESC[34mINFO ESC[0;39m | CorfuServer | org.corfudb.util.MetricsUtils | Corfu metrics collection and all reporting types are disabled
2019-09-20T06:03:43.836Z | ESC[39mDEBUGESC[0;39m | CorfuServer | i.n.util.ResourceLeakDetector | -Dio.netty.leakDetection.level: simple
2019-09-20T06:03:43.837Z | ESC[39mDEBUGESC[0;39m | CorfuServer | i.n.util.ResourceLeakDetector | -Dio.netty.leakDetection.targetRecords: 4
2019-09-20T06:03:43.845Z | ESC[39mDEBUGESC[0;39m | CorfuServer | i.netty.buffer.AbstractByteBuf | -Dio.netty.buffer.bytebuf.checkAccessible: true
2019-09-20T06:03:43.846Z | ESC[39mDEBUGESC[0;39m | CorfuServer | .u.ResourceLeakDetectorFactory | Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@5efe6e77
2019-09-20T06:03:47.434Z | ESC[1;31mERRORESC[0;39m | CorfuServer | o.c.i.log.StreamLogFiles | Checksum mismatch detected while trying to read file sun.nio.ch.FileChannelImpl@82dd757
2019-09-20T06:03:47.671Z | ESC[1;31mERRORESC[0;39m | CorfuServer | o.c.infrastructure.CorfuServer | CorfuServer: Server exiting due to unrecoverable error:
org.corfudb.runtime.exceptions.DataCorruptionException: null
at org.corfudb.infrastructure.log.StreamLogFiles.parseEntry(StreamLogFiles.java:525)
at org.corfudb.infrastructure.log.StreamLogFiles.readAddressSpace(StreamLogFiles.java:562)
at org.corfudb.infrastructure.log.StreamLogFiles.lambda$getSegmentHandleForAddress$1(StreamLogFiles.java:677)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1688)
at org.corfudb.infrastructure.log.StreamLogFiles.getSegmentHandleForAddress(StreamLogFiles.java:660)
at org.corfudb.infrastructure.log.StreamLogFiles.initializeLogMetadata(StreamLogFiles.java:136)
at org.corfudb.infrastructure.log.StreamLogFiles.<init>(StreamLogFiles.java:112)
at org.corfudb.infrastructure.LogUnitServer.<init>(LogUnitServer.java:114)
at org.corfudb.infrastructure.CorfuServer.startServer(CorfuServer.java:304)
at org.corfudb.infrastructure.CorfuServer.lambda$main$0(CorfuServer.java:270)
at java.lang.Thread.run(Thread.java:748)

Cause :

From the above logs, it’s clear that the Corfu database is not available due to data corruption.

Resolution :

Unfortunately, the only way to recover from this state is to restore from a backup or deploy a new NSX-T manager.

Note : If anyone of you faces this issue, I highly recommend you reach out to the VMware NSX-T technical support team for assistance before making any further changes.

---

When I saw the error message at first, it did not give me much of an idea about what is causing the issue or whether the issue is recoverable or not. After reviewing the logs, I am clear about what the next steps are. I hope this blog post helps if you face this issue in your environment.

---