Hints file with unknown CFID can cause nodes to fail

Hints files that contain an unknown column family id may cause nodes to stop and also fail to restart

DSE5.0.0 to DSE 5.0.11, DSE 5.1.0 to DSE5.1.4 inclusive

Symptoms

The following exception may be seen when the node tried to replay hints from a corrupted hint file:

ERROR [HintsDispatcher:59] 2018-03-07 15:00:01  HintsDispatchExecutor.java:243 - Failed to dispatch hints file 81ce70a6-91c8-4956-b2c3-26db27cb37fc-1520427216096-1.hints: file is corrupted ({})
org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch exception

Cause

This issue is caused by the following Cassandra bug:
https://issues.apache.org/jira/browse/CASSANDRA-13696

A hints file that contains an unknown column family on the node will cause the hints file to fail to replay. This in turn can cause a i/o exception and either cause the node to stop and / or fail to restart.

Workaround

Deleting the offending hint file should mitigate the situation and allow the node to be restarted. Note the logs may not always specify the actual file but searching for all or part of the UUID in the message above on the node's filesystem will help to locate the hints file.

Although you should always run repair in your cluster as part of normal operations, when deleting hints files, it is advisable to run repair afterwards as soon as practically possible.

Solution

Upgrade to DSE5.0.12, DSE5.1.4 or later