Summary
Several hours after decommissioning a node, the node still shows as UL or LEAVING state despite having very little data.
Symptoms
In the context of this article, a stuck or hung decommission is characterised by:
- DataStax Enterprise (DSE) or Cassandra process is still running
- no activity in the
system.log nodetool statusshows the node in UL or LEAVING statenodetool netstatsshows the node in LEAVING mode and not sending streamsnodetool compactionstatsshows 0 pending tasks- very low or close to zero CPU utilisation
Cause
A common cause of this problem is administrators incorrectly running nodetool drain prior to decommissioning a node.
When a node is "drained", data is flushed to disk and Cassandra stops listening for connections from clients and other nodes in the cluster. Despite the Cassandra process and JVM still running, for all intents and purposes Cassandra is no longer operational.
The nodetool drain command is used to prepare a node for a Cassandra or DSE upgrade. Do not run the command as part of a decommission process.
Workaround
Use these steps to get the node to decommission.
Step 1 - Restart DSE on the node.
Step 2 - Run nodetool decommission again and the node should be removed from the cluster as expected.
See also
DataStax doc - nodetool drain command
Comments
1 comment
nodetool decommission is still having issues. We try to decommission Spark DC from cluster. followed steps recommended in decom DC doc. 1. Stopped keyspace replication to spark DC 2. ran nodetool decommission which said node was leaving (UL) 3 ) Opscenter said its done 100% but nodetool netstats was still saying node was streaming to and other nodes in DC said it was streaming from
4) node was not removed from cluster and remained in UL state. then I had to try nodetool removenode $nodeid command to remove node in DC one at a time and it worked. to use nodetool remove node , we had to stop DSE on node to be remove and then ran nodetool remove from its peer. Then repeated same process on other 2 nodes in Spark DC sequentially. Please can you update documentation to use right command for decommission nodes or all nodes in DC. nodetool decommission never worked recently in a cleaner fashion.
Please sign in to leave a comment.