Summary

• The ring tab of a DC appears greyed out
• Errors related to the /var/lib/datastax-agent/tmp/persistent/dsr/repair-service.db appear in the agent log

Applies to

• Problem only occurs with tarball agents: issue seen on 6.5.3 and potentially from 6.1.7 - under investigation
• OpsCenter 6.5.3

Symptoms

• The content of the ring tab in OpsCenter shows the nodes greyed out with an unknown status

• Datastax-agent output some error as per the excerpt below

WARN [C3P0PooledConnectionPoolManager[identityToken->xyz]-HelperThread-#0]:07:02,397 com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7c47eac7 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:   
 java.sql.SQLException: path to '/var/lib/datastax-agent/tmp/persistent/dsr/repair-service.db': '/var/lib/datastax-agent' does not exist

Cause

This error was potentially introduced with OpsCenter 6.1.7 and the Distributed Subrange Repair (DSR) feature. The tarball agents appear to be looking for a directory that only exists by default with the packaged installers /var/lib/datastax-agent/tmp.
The problem was only noted after OpsCenter is restarted but agents kept running.

Workaround

• Go to the cluster configuration file in opscenter. ie: /etc/opscenter/clusters/<clustername>.conf
• Add the following section redirecting the agents to a valid temp directory (in the example below, make sure the user running the agent can read/write/execute to the folder /path/to/agent/tmp):

[agent_config] 
tmp_dir = /path/to/agent/tmp 

Solution

This issue is escalated to development under OPSC-15533 for further investigation.
Click follow at the top of this KBA to get updates for new information.
Alternatively, the release notes for newer versions would contain the fix reference when implemented. ie, for OpsCenter 6.5.x:
OpsCenter Release Notes