DataStax Help Center

Node stuck in LEAVING state after being decommissioned

Summary

Several hours after decommissioning a node, the node still shows as UL or LEAVING state despite having very little data.

Symptoms

In the context of this article, a stuck or hung decommission is characterised by:

  • DataStax Enterprise (DSE) or Cassandra process is still running
  • no activity in the system.log
  • nodetool status shows the node in UL or LEAVING state
  • nodetool netstats shows the node in LEAVING mode and not sending streams
  • nodetool compactionstats shows 0 pending tasks
  • very low or close to zero CPU utilisation

Cause

A common cause of this problem is administrators incorrectly running nodetool drain prior to decommissioning a node.

When a node is "drained", data is flushed to disk and Cassandra stops listening for connections from clients and other nodes in the cluster. Despite the Cassandra process and JVM still running, for all intents and purposes Cassandra is no longer operational.

The nodetool drain command is used to prepare a node for a Cassandra or DSE upgrade. Do not run the command as part of a decommission process.

Workaround

Use these steps to get the node to decommission.

Step 1 - Restart DSE on the node.

Step 2 - Run nodetool decommission again and the node should be removed from the cluster as expected.

See also

DataStax doc - nodetool drain command

Erick Ramirez -
Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

Comments

  • Avatar
    Athinanthny Senthil

    nodetool decommission is still having issues. We try to decommission Spark DC from cluster. followed steps recommended in decom DC doc. 1. Stopped keyspace replication to spark DC 2. ran nodetool decommission which said node was leaving (UL) 3 ) Opscenter said its done 100% but nodetool netstats was still saying node was streaming to and other nodes in DC said it was streaming from
    4) node was not removed from cluster and remained in UL state. then I had to try nodetool removenode $nodeid command to remove node in DC one at a time and it worked. to use nodetool remove node , we had to stop DSE on node to be remove and then ran nodetool remove from its peer. Then repeated same process on other 2 nodes in Spark DC sequentially. Please can you update documentation to use right command for decommission nodes or all nodes in DC. nodetool decommission never worked recently in a cleaner fashion.

Powered by Zendesk