NodeSync Status tab in OpsCenter reports false urgencies

Summary

This article relates to an issue where the NodeSync Status tab in OpsCenter falsely reports that "[N]% of the data was not synced before the deadline".

Applies to

• DataStax Enterprise 6.7
• DataStax Enterprise 6.0
• DSE OpsCenter 6.5
• DSE OpsCenter 6.7

Symptoms

The NodeSync Status tab shows a portion of the progress bar as red. When you hover over the red portion, the displayed text indicates that some percent of data was not able to be synced by the deadline:

However, NodeSync has been running successfully previously and/or per node Nodesync rate is correctly set to an appropriate value. 

Cause

In this case, you may be hitting OPSC-15823: "NodeSync UI - false urgencies due to expired entries in nodesync_status table".  This issue can arise when there has been a topology change to the cluster (adding or removing nodes, etc).

Workaround

Caution: Verify the alert is false before applying the workaround.

1. Ensure the rate configured is sufficient for the amount of data to be synced

# run to get recommended rate
nodetool nodesyncservice ratesimulator recommended -v

# compare recommended rate to actual rate.  Raise rate as needed. This command must be run on each node.
nodetool nodesyncservice getrate

# raise rate as needed
nodetool nodesyncservice setrate [new rate in kb]

To persist the rate limit, use the rate_in_kb setting in cassandra.yaml.

If the rate is sufficient, NodeSync has been running as long as the highest deadline_target_sec and you have performed a cluster topology change, then OPSC-15823 is likely being hit and the workaround should be used.

2. Disable the NodeSynce Service on each node with the following command:

$nodetool nodesyncservice disable

3. Truncate the system_distributed.nodesync_status table:

cqlsh> truncate table system_distributed.nodesync_status;

4. Re-enable NodeSync Service on each node with the following command:

$nodetool nodesyncservice enable

5. Verify NodeSync is completing successfully.

Let NodeSync run for the length of the longest deadline on a table.  The red error on the progress bar should no longer be displayed.  If this workaround doesn't resolve the issue, contact DataStax Support.

Solution

As always, DataStax recommends reading the DSE release notes and upgrading to the latest version. You can watch for a resolution for the associated Jira OPSC-15823.