DataStax Help Center

Node with vnodes added incorrectly to a single-token cluster

Summary

This article provides steps on how to recover a node with vnodes (virtual nodes) inadvertently added to a single-token cluster.

Symptoms

While expanding a cluster, an administrator notices that the new node has been configured with vnodes enabled. A sample nodetool status output shows that node 10.1.2.3 has 256 tokens compared to the other nodes which only have a single token:

Datacenter: DC
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.1.2.3 12.98 GB 256 100.0% 4da62e08-b408-443a-98f3-efd2d0bc30c9 RAC1
UN 10.1.4.5 13.31 GB 1 100.0% 84a55b07-7fea-4a6b-a9e3-245ae4522a40 RAC1
UN 10.1.6.7 13.35 GB 1 100.0% 1f9e855f-e466-4948-84fa-094df511a2ab RAC1

Cause

The node was configured incorrectly as a result of human error.

Solution

Follow these steps to resolve this issue:

Step 1 - Decommission the node. This assigns the ranges owned by the node to the other nodes in the ring and streams its data out to its neighbours.

$ nodetool decommission

Monitor the progress with nodetool netstats.

Step 2 - Once the decommission has completed, reconfigure the following properties in the cassandra.yaml:

  • comment out num_tokens
  • set the token assignment initial_token

Step 3 - Delete the following directories. This will ensure that all their contents are cleaned out completely.

  • data_file_directories
  • commitlog
  • saved_caches

Step 4 - Recreate the directories above and make sure Cassandra has full permissions.

Step 5 - Bootstrap the node again by starting DSE.

See also

For more information on decommissioning nodes, see Removing a node.

If you have not done so already, recalculate the token assignments. For more information, see Generating tokens.

Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk