Summary
The customer had a problem where their schema was out of sync with the cluster which was caused by excessive tombstones in their system.hints table
Symptoms
The customer noticed the following output from nodetool describeschema:
Cluster Information: Name: main_cluster Snitch: com.datastax.bdp.snitch.DseDelegateSnitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema versions: UNREACHABLE: [10.120.1.1] 91fa204f-31b2-3675-aa02-b3402bdcdace: [10.140.4.1] 3322829b-c49f-3b05-bab7-bdc1dfa21fef: [10.140.4.2, 10.120.4.3, 10.120.4.1]
They had already tried the following command with no effect:
nodetool resetlocalschema
Cause
The system.hints table had so many tombstones, that when the node tried to replay the hints it failed with the following error:
ERROR [HintedHandoff:1] 2014-12-12 05:12:57,513 SliceQueryFilter.java (line 200) Scanned over 100000 tombstones in system.hints; query aborted (see tombstone_failure_threshold)
ERROR [HintedHandoff:1] 2014-12-12 05:12:57,514 CassandraDaemon.java (line 199) Exception in thread Thread[HintedHandoff:1,1,main] org.apache.cassandra.db.filter.TombstoneOverwhelmingException
Workaround
The system hints table can be truncated but repair must be run afterwards
Solution
The node can be re-boot strapped into the cluster, it will then stream the data and sync with the correct schema version. There are also other methods of adding a new node or syncing nodes into the cluster but these are not covered in this note. For more information see the documentation: http://datastax.com/documentation/cassandra/2.0/cassandra/gettingStartedCassandraIntro.html