Grey ring in OpsCenter UI and DSR errors in agent logs
Summary
The ring tab of a DC appears greyed out
Errors related to the /var/lib/datastax-agent/tmp/persistent/dsr/repair-service.db appear in the agent log
Applies to
Problem only occurs with tarball agents: issue seen on 6.5.3 and 6.1.11 (from investigation, the issue was introduced in 6.1.7)
DataStax OpsCenter 6.5
DataStax OpsCenter 6.1
Symptoms
The content of the ring tab in OpsCenter shows the nodes greyed out with an unknown status
Datastax-agent output some error as per the excerpt below
WARN [C3P0PooledConnectionPoolManager[identityToken->xyz]-HelperThread-#0]:07:02,397 com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7c47eac7 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: java.sql.SQLException: path to '/var/lib/datastax-agent/tmp/persistent/dsr/repair-service.db': '/var/lib/datastax-agent' does not exist
Cause
This error was potentially introduced with OpsCenter 6.1.7 and the Distributed Subrange Repair (DSR) feature. The tarball agents appear to be looking for a directory that only exists by default with the packaged installers /var/lib/datastax-agent/tmp. The problem was only noted after OpsCenter is restarted but agents kept running.
Solution
This issue was escalated to development under OPSC-15533. The solution is as follow:
Go to the cluster configuration file in OpsCenter. ie: /etc/opscenter/clusters/<clustername>.conf
Add the following section redirecting the agents to a valid temp directory. In the example below, make sure the user running the agent can read/write/execute to the folder /path/to/agent/tmp:
Summary
/var/lib/datastax-agent/tmp/persistent/dsr/repair-service.db
appear in the agent logApplies to
Symptoms
Cause
This error was potentially introduced with OpsCenter 6.1.7 and the Distributed Subrange Repair (DSR) feature. The tarball agents appear to be looking for a directory that only exists by default with the packaged installers
/var/lib/datastax-agent/tmp
.The problem was only noted after OpsCenter is restarted but agents kept running.
Solution
This issue was escalated to development under OPSC-15533. The solution is as follow:
/etc/opscenter/clusters/<clustername>.conf
/path/to/agent/tmp
: