Missing data and gaps appear in Grafana
DataStax Enterprise (DSE) 6.0
DataStax Enterprise 6.7
DSE Metrics collector
- Gaps of data appear in graphs on Grafana
- Checking the graphs from Prometheus show the same behaviour
- CPU usage of collectd process on DSE nodes, shown as ld-2.23.so in top, indicate constant usage of a full CPU. ie:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19234 cassand+ 20 0 1424648 1.032g 2588 S 100.0 13.2 9:14.54 ld-2.23.so
/var/log/cassandracan appear flooded with following errors:
[2019-07-11 21:24:30] plugin_dispatch_values: Low water mark reached. Dropping 0% of metrics. [2019-07-11 21:24:32] plugin_dispatch_values: Low water mark reached. Dropping 0% of metrics.
The issue is related to the number of metrics retrieved by collectd and based on the number of tables present in the schema of DSE.
/etc/dse/collectd.conf.tmpl, add a timeout + increase the ReadThreads and WriteThreads from 5 to 8:
Timeout 6 ReadThreads 8 WriteThreads 8
A set of should also be added to the
jvm.options file for DSE
Restart DSE so the change is taken into account
Note it may be necessary to wait several hours after the restart to see the CPU dropping and the gaps issue may still occur immediately after the change. If after half a day, the issue still occur, consider the change mentioned below on top of the previous one.
Tweaking filters may be necessary, depending on the amount of tables the system hosts to reduce the number of metrics collected:
dsetool insights_filters --add --global --deny "org\\.apache\\.cassandra\\.metrics\\.table.cas_.*" dsetool insights_filters --add --global --deny "org\\.apache\\.cassandra\\.metrics\\.index_table.cas_.*" dsetool insights_filters --add --global --deny "org\\.apache\\.cassandra\\.metrics\\.keyspace\\..*" dsetool insights_filters --show_filters
This issue was escalated to Development under reference DSP-19559 and the changes above have been integrated in DSE 6.7.9 and 6.8.2.