DataStax Help Center

Node schema out of sync caused by excessive tombstones in system.hints


The customer had a problem where their schema was out of sync with the cluster which was caused by excessive tombstones in their system.hints table


The customer noticed the following output from nodetool describeschema:

Cluster Information: 
Name: main_cluster 
Snitch: com.datastax.bdp.snitch.DseDelegateSnitch 
Partitioner: org.apache.cassandra.dht.RandomPartitioner 
Schema versions: 

91fa204f-31b2-3675-aa02-b3402bdcdace: []

3322829b-c49f-3b05-bab7-bdc1dfa21fef: [,,]

They had already tried the following command with no effect:

nodetool resetlocalschema 


The system.hints table had so many tombstones, that when the node tried to replay the hints it failed with the following error:

ERROR [HintedHandoff:1] 2014-12-12 05:12:57,513 (line 200) Scanned over 100000 tombstones in system.hints; query aborted (see tombstone_failure_threshold) 
ERROR [HintedHandoff:1] 2014-12-12 05:12:57,514 (line 199) Exception in thread Thread[HintedHandoff:1,1,main]  org.apache.cassandra.db.filter.TombstoneOverwhelmingException 


The system hints table can be truncated but repair must be run afterwards


The node can be re-boot strapped into the cluster, it will then stream the data and sync with the correct schema version. There are also other methods of adding a new node or syncing nodes into the cluster but these are not covered in this note. For more information see the documentation:

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request


Powered by Zendesk