Symptoms
The following error repeats constantly in the agent.log
ERROR [jmx-cf-metrics] 2020-05-13 11:53:27,726 JMX InstanceNotFoundException: org.apache.cassandra.metrics:type=Table,keyspace=KEYSPACE_NAME,scope=TABLE_NAME,name=NodeSyncIncrementalValidationsCoveredAgesSeconds:values javax.management.InstanceNotFoundException: org.apache.cassandra.metrics:type=Table,keyspace=KEYSPACE_NAME,scope=TABLE_NAME,name=NodeSyncIncrementalValidationsCoveredAgesSeconds
Applies to
- DataStax Enterprise 6.0.10 and higher patch release (6.0.11, 6.0.12...)
- OpsCenter 6.7 and above
Cause
Due to a versioning issue, DSE 6.0.10 and higher 6.0 are incorrectly detected by the agent which pick up the wrong definitions files.
The metric NodeSyncIncrementalValidationsCoveredAgesSeconds
was added in DSE 6.8.0 and OpsCenter tries to query this metric on the versions mentioned.
Workaround
1- disable auto update of definitions in opscenterd.conf by adding the following section at the bottom of the opscenterd.conf file
[definitions] auto_update = false
For further information, the parameter is documented here:
https://docs.datastax.com/en/opscenter/6.5/opsc/configure/opscConfigProps_r.html#opscConfigProps__definitions-auto_update
2- delete the file cassandra-metrics-4.0.0.680.json
on the /var/lib/opscenter/definitions/ directory on the OpsCenter server
3- identify the keyspace containing the data for the 6.0 cluster being monitored (“OpsCenter” by default if no storage cluster is in use). ie: /etc/opscenter/clusters/dse6011.conf
[storage_cassandra] keyspace = OpsCenter_dse6011
4- confirm in cqlsh the keys associated to the 4.0.0.680 definitions file are present in the repository and delete them:
cqlsh> select key, blobAsText(key), blobAsText(column1) from "OpsCenter_dse6011".settings where key=0x636f6e6669672d63617373616e6472612d6d6574726963732d342e302e302e363830; key | system.blobastext(key) | system.blobastext(column1) ------------------------------------------------------------------------+------------------------------------+---------------------------- 0x636f6e6669672d63617373616e6472612d6d6574726963732d342e302e302e363830 | config-cassandra-metrics-4.0.0.680 | data 0x636f6e6669672d63617373616e6472612d6d6574726963732d342e302e302e363830 | config-cassandra-metrics-4.0.0.680 | md5 cqlsh> delete from "OpsCenter_dse6011".settings where key=0x636f6e6669672d63617373616e6472612d6d6574726963732d342e302e302e363830; cqlsh> select key, blobAsText(key), blobAsText(column1) from "OpsCenter_dse6011".settings where key=0x636f6e6669672d63617373616e6472612d6d6574726963732d342e302e302e363830;
5- repeat the operation for the config-index
entry of cassandra-metrics-4.0.0.680
:
cqlsh> select key, blobAsText(key), column1, blobAsText(column1) from "OpsCenter_dse6011".settings where key=0x636f6e6669672d696e646578 and column1 = 0x63617373616e6472612d6d6574726963732d342e302e302e363830; key | system.blobastext(key) | column1 | system.blobastext(column1) ----------------------------+------------------------+----------------------------------------------------------+----------------------------- 0x636f6e6669672d696e646578 | config-index | 0x63617373616e6472612d6d6574726963732d342e302e302e363830 | cassandra-metrics-4.0.0.680 cqlsh> delete from "OpsCenter_dse6011".settings where key=0x636f6e6669672d696e646578 and column1 = 0x63617373616e6472612d6d6574726963732d342e302e302e363830;
6- restart OpsCenter + the agents on the nodes impacted when above is done. ie:sudo service opscenterd restart
sudo service datastax-agent restart
Solution
This issue is being investigated under internal reference OPSC-16608. Please register to this technote to get new updates on the topic or contact support for further information