Summary

• The ring tab of a DC appears greyed out
• Errors related to the /var/lib/datastax-agent/tmp/persistent/dsr/repair-service.db appear in the agent log

Applies to

• Problem only occurs with tarball agents: issue seen on 6.5.3 and 6.1.11 (from investigation, the issue was introduced in 6.1.7)
• DataStax OpsCenter 6.5
• DataStax OpsCenter 6.1

Symptoms

• The content of the ring tab in OpsCenter shows the nodes greyed out with an unknown status

• Datastax-agent output some error as per the excerpt below

WARN [C3P0PooledConnectionPoolManager[identityToken->xyz]-HelperThread-#0]:07:02,397 com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7c47eac7 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:   
 java.sql.SQLException: path to '/var/lib/datastax-agent/tmp/persistent/dsr/repair-service.db': '/var/lib/datastax-agent' does not exist

Cause

This error was potentially introduced with OpsCenter 6.1.7 and the Distributed Subrange Repair (DSR) feature. The tarball agents appear to be looking for a directory that only exists by default with the packaged installers /var/lib/datastax-agent/tmp.
The problem was only noted after OpsCenter is restarted but agents kept running.

Solution

This issue was escalated to development under OPSC-15533. The solution is as follow:

• Go to the cluster configuration file in OpsCenter. ie: /etc/opscenter/clusters/<clustername>.conf
• Add the following section redirecting the agents to a valid temp directory. In the example below, make sure the user running the agent can read/write/execute to the folder /path/to/agent/tmp:

[agent_config] 
tmp_dir = /path/to/agent/tmp