This article provides steps on how to recover a node with vnodes (virtual nodes) inadvertently added to a single-token cluster.
While expanding a cluster, an administrator notices that the new node has been configured with vnodes enabled. A sample
nodetool status output shows that node
10.1.2.3 has 256 tokens compared to the other nodes which only have a single token:
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.1.2.3 12.98 GB 256 100.0% 4da62e08-b408-443a-98f3-efd2d0bc30c9 RAC1
UN 10.1.4.5 13.31 GB 1 100.0% 84a55b07-7fea-4a6b-a9e3-245ae4522a40 RAC1
UN 10.1.6.7 13.35 GB 1 100.0% 1f9e855f-e466-4948-84fa-094df511a2ab RAC1
The node was configured incorrectly as a result of human error.
Follow these steps to resolve this issue:
Step 1 - Decommission the node. This assigns the ranges owned by the node to the other nodes in the ring and streams its data out to its neighbours.
$ nodetool decommission
Monitor the progress with
Step 2 - Once the decommission has completed, reconfigure the following properties in the
- comment out
- set the token assignment
Step 3 - Delete the following directories. This will ensure that all their contents are cleaned out completely.
Step 4 - Recreate the directories above and make sure Cassandra has full permissions.
Step 5 - Bootstrap the node again by starting DSE.
For more information on decommissioning nodes, see Removing a node.
If you have not done so already, recalculate the token assignments. For more information, see Generating tokens.