Summary
When performing streaming activities such as repair or bootstrap, an error specifying the node is unable to find sufficient resources for a given token is seen in the system.log
Symptoms
The error message observed in the system.log
may look similar to the following example
java.lang.IllegalStateException: unable to find sufficient sources for streaming range (-3685074324747697686,-3680615207285604279] in keyspace <some_keyspace>
Cause
The node reporting this error thinks that all the nodes that own the specified token range in this example are either offline or otherwise unavailable.
Solution
Either make sure all nodes are up, or increase the RF of the keyspace in question.
For example, the keyspace in question might be the system_traces
keyspace which by default uses aSimpleStrategy
replication strategy. If you’re using multiple DCs, alter the keyspace to use NetworkTopologyStrategy
, making sure that there is at least 1 replica in each DC
cqlsh> alter keyspace system_traces WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1':1,'DC2':1,'DC3':1};
In the case of the system_traces
keyspace, if no query tracing is being performed, and there aren’t even any traces in the system_traces tables. it is only required to alter the keyspace; no repair is necessary because there is nothing to repair.
Otherwise, if there is data involved, it is recommended to run a repair of the keyspace in question, after altering the keyspace.