Summary
When querying the cluster, requests could not be completed with nodes unavailable to service the query.
Symptoms
Data Manipulation Language (DML) queries (e.g. SELECT
, INSERT
, UPDATE
) against the database fail despite all nodes being operational.
Nodes can be confirmed to be operational since Data Definition Language (DDL) queries (e.g. DESCRIBE
, ALTER
) return the expected results.
Below is a sample output from a cqlsh query:
cqlsh> SELECT * FROM music.albums ; Unable to complete request: one or more nodes were unavailable.
Cause
In this instance, the issue is a result of incorrect replication settings on the keyspace.
The sample nodetool status
output shows that there is a single-datacentre named Cassandra
:
Datacenter: Cassandra
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.0.1.2 1.67 GB 256 36.8% 109dc707-1df2-4f50-9ec5-d611f1772f5a RAC1
UN 10.0.1.3 1.66 GB 256 30.0% a88a9570-9ead-4cdd-a81b-daf68013f8ab RAC1 UN 10.0.1.4 1.66 GB 256 33.2% 239d8975-1783-4935-addb-691173214045 RAC1
However, the music keyspace in the example above shows that the data is replicated to a non-existent datacentre named DC1
:
cqlsh> DESCRIBE KEYSPACE music ;
CREATE KEYSPACE music WITH replication = {
'class': 'NetworkTopologyStrategy',
'DC1': '3'
};
...
Solution
Update the replication settings on the keyspace with the correct datacentre name.
Step 1 - Alter the keyspace definition:
cqlsh> ALTER KEYSPACE music WITH replication = {
'class': 'NetworkTopologyStrategy',
'Cassandra': '3'
};
Step 2 - (OPTIONAL) It may also be necessary to run a rolling repair around the ring:
$ nodetool repair -pr -- music
See also
KB article - Read and login failures after updating keyspace replication
Comments
0 comments
Please sign in to leave a comment.