Symptoms

The following error repeats constantly in the agent.log

ERROR [jmx-cf-metrics] 2020-05-13 11:53:27,726 JMX InstanceNotFoundException: org.apache.cassandra.metrics:type=Table,keyspace=KEYSPACE_NAME,scope=TABLE_NAME,name=NodeSyncIncrementalValidationsCoveredAgesSeconds:values
 javax.management.InstanceNotFoundException: org.apache.cassandra.metrics:type=Table,keyspace=KEYSPACE_NAME,scope=TABLE_NAME,name=NodeSyncIncrementalValidationsCoveredAgesSeconds

Applies to

• DataStax Enterprise 6.0.10 and higher patch release (6.0.11, 6.0.12...)
• OpsCenter 6.7 and above

Cause

Due to a versioning issue, DSE 6.0.10 and higher 6.0 are incorrectly detected by the agent which pick up the wrong definitions files.
The metric NodeSyncIncrementalValidationsCoveredAgesSeconds was added in DSE 6.8.0 and OpsCenter tries to query this metric on the versions mentioned.

Workaround

1- disable auto update of definitions in opscenterd.conf by adding the following section at the bottom of the opscenterd.conf file

[definitions] 
auto_update = false

For further information, the parameter is documented here:
https://docs.datastax.com/en/opscenter/6.5/opsc/configure/opscConfigProps_r.html#opscConfigProps__definitions-auto_update

2- delete the file cassandra-metrics-4.0.0.680.json on the /var/lib/opscenter/definitions/ directory on the OpsCenter server

3- identify the keyspace containing the data for the 6.0 cluster being monitored (“OpsCenter” by default if no storage cluster is in use). ie: /etc/opscenter/clusters/dse6011.conf

[storage_cassandra]
keyspace = OpsCenter_dse6011

4- confirm in cqlsh the keys associated to the 4.0.0.680 definitions file are present in the repository and delete them:

cqlsh> select key, blobAsText(key), blobAsText(column1) from "OpsCenter_dse6011".settings where key=0x636f6e6669672d63617373616e6472612d6d6574726963732d342e302e302e363830;

 key                                                                    | system.blobastext(key)             | system.blobastext(column1)
------------------------------------------------------------------------+------------------------------------+----------------------------
 0x636f6e6669672d63617373616e6472612d6d6574726963732d342e302e302e363830 | config-cassandra-metrics-4.0.0.680 |                       data
 0x636f6e6669672d63617373616e6472612d6d6574726963732d342e302e302e363830 | config-cassandra-metrics-4.0.0.680 |                        md5
 
cqlsh> delete from "OpsCenter_dse6011".settings where key=0x636f6e6669672d63617373616e6472612d6d6574726963732d342e302e302e363830; 
cqlsh> select key, blobAsText(key), blobAsText(column1) from "OpsCenter_dse6011".settings where key=0x636f6e6669672d63617373616e6472612d6d6574726963732d342e302e302e363830;

5- repeat the operation for the config-index entry of cassandra-metrics-4.0.0.680:

cqlsh> select key, blobAsText(key), column1, blobAsText(column1) from "OpsCenter_dse6011".settings where key=0x636f6e6669672d696e646578 and column1 = 0x63617373616e6472612d6d6574726963732d342e302e302e363830;

 key                        | system.blobastext(key) | column1                                                  | system.blobastext(column1)
----------------------------+------------------------+----------------------------------------------------------+-----------------------------
 0x636f6e6669672d696e646578 |           config-index | 0x63617373616e6472612d6d6574726963732d342e302e302e363830 | cassandra-metrics-4.0.0.680

cqlsh> delete from "OpsCenter_dse6011".settings where key=0x636f6e6669672d696e646578 and column1 = 0x63617373616e6472612d6d6574726963732d342e302e302e363830;

6- restart OpsCenter + the agents on the nodes impacted when above is done. ie:
sudo service opscenterd restart
sudo service datastax-agent restart

Solution

This issue is being investigated under internal reference OPSC-16608. Please register to this technote to get new updates on the topic or contact support for further information