Users have reported read and login failures after a keyspace's replication definition has been changed.
In a situation where users are unable to login to Cassandra on some nodes after the replication factor for the
system_auth keyspace has been changed (e.g. increased to 3 from default 1), attempts to login result in the following errors:
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host node5/10.12.34.56:9042: Username and/or password are incorrect
com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host node75/10.98.76.54:9042: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
In the case of reads, similar errors relating to missing data or not enough replicas are returned.
The most common cause of this issue is the data not being present in replicas. This is because a repair has not been run after the replication has been changed, i.e. the new replica(s) does not have the data.
A variation of the issue is where replication for a keyspace has been configured with the incorrect data center name and so queries against the non-existent DC fails.
Follow these steps to resolve this issue:
Step 1 - Review the keyspace definition to ensure that the data center names are defined correctly. Fix errors as appropriate.
cqlsh> DESCRIBE KEYSPACE <keyspace> ;
Ensure the DC names are in the correct case, this is a common mistake
Step 2 - Pick a node in the cluster and run a primary-range repair:
$ nodetool repair -pr -- <keyspace>
-pr option will only repair one node's primary range which is more efficient as the work is not duplicated.
Step 3 - Wait for the repair to complete on this node before proceeding to the next step.
Step 4 - Repeat steps 2 and 3 on the next node until all nodes in the cluster have been repaired.
Mark Curtis' article on Solr queries which return "Unavailable shards for ranges" error.
For additional information on replication of DSE Security keyspaces, see Configuring system_auth and dse_security keyspace replication.
Jeremiah Jordan's blog post on Repair in Cassandra for a good explanation of repair with the