Node schema out of sync caused by excessive tombstones in system.hints

Summary

The customer had a problem where their schema was out of sync with the cluster which was caused by excessive tombstones in their system.hints table

Symptoms

The customer noticed the following output from nodetool describeschema:

Cluster Information: 
Name: main_cluster 
Snitch: com.datastax.bdp.snitch.DseDelegateSnitch 
Partitioner: org.apache.cassandra.dht.RandomPartitioner 
Schema versions: 
UNREACHABLE: [10.120.1.1]

91fa204f-31b2-3675-aa02-b3402bdcdace: [10.140.4.1]

3322829b-c49f-3b05-bab7-bdc1dfa21fef: [10.140.4.2, 10.120.4.3, 10.120.4.1]

They had already tried the following command with no effect:

nodetool resetlocalschema 

Cause

The system.hints table had so many tombstones, that when the node tried to replay the hints it failed with the following error:

ERROR [HintedHandoff:1] 2014-12-12 05:12:57,513 SliceQueryFilter.java (line 200) Scanned over 100000 tombstones in system.hints; query aborted (see tombstone_failure_threshold) 
ERROR [HintedHandoff:1] 2014-12-12 05:12:57,514 CassandraDaemon.java (line 199) Exception in thread Thread[HintedHandoff:1,1,main] 
org.apache.cassandra.db.filter.TombstoneOverwhelmingException 

Workaround

The system hints table can be truncated but repair must be run afterwards

Solution

The node can be re-boot strapped into the cluster, it will then stream the data and sync with the correct schema version. There are also other methods of adding a new node or syncing nodes into the cluster but these are not covered in this note. For more information see the documentation: http://datastax.com/documentation/cassandra/2.0/cassandra/gettingStartedCassandraIntro.html