DataStax Help Center

Node bootup hung, cannot get past "Initializing system.local"

Summary

A node which has been re-provisioned or re-purposed, i.e. previously decommissioned from a running cluster, is unable to startup with the new cluster settings.

Some customers decommission some nodes in a cluster and decide to either use them to build another cluster or add them to an existing cluster. The nodes are reconfigured but hang when DSE is started.

Symptoms

DSE appears to go through the normal startup process but eventually hangs when it gets to the line Initializing system.local as in the log extract below:

INFO [main] 2014-11-03 18:21:35,351 ColumnFamilyStore.java (line 249) Initializing system.hints 
INFO [main] 2014-11-03 18:21:35,355 ColumnFamilyStore.java (line 249) Initializing system.schema_keyspaces
INFO [main] 2014-11-03 18:21:35,359 ColumnFamilyStore.java (line 249) Initializing system.range_xfers
INFO [main] 2014-11-03 18:21:35,364 ColumnFamilyStore.java (line 249) Initializing system.schema_columnfamilies
INFO [main] 2014-11-03 18:21:35,368 ColumnFamilyStore.java (line 249) Initializing system.NodeIdInfo
INFO [main] 2014-11-03 18:21:35,372 ColumnFamilyStore.java (line 249) Initializing system.paxos
INFO [main] 2014-11-03 18:21:35,379 ColumnFamilyStore.java (line 249) Initializing system.schema_columns
INFO [main] 2014-11-03 18:21:35,383 ColumnFamilyStore.java (line 249) Initializing system.IndexInfo
INFO [main] 2014-11-03 18:21:35,671 ColumnFamilyStore.java (line 249) Initializing system.peers
INFO [main] 2014-11-03 18:21:35,674 ColumnFamilyStore.java (line 249) Initializing system.local

Cause

This issue has occurred because the node has not been prepared correctly for the new cluster after it has been decommissioned.

Some of the known causes are:

  • data directory has not been cleared of its old contents
  • similarly, commitlog and saved_caches directories have not been emptied
  • one of the directories, e.g. commitlog, has been recreated with incorrect permissions and Cassandra is unable to write to it

Solution

Follow these steps to resolve the issue:

Step 1 - Shutdown DSE.

Step 2 - Delete the following directories. This will ensure that all their contents are deleted completely.

  • data
  • commitlog
  • saved_caches

Step 3 - Recreate the directories above and make sure Cassandra has full permissions.

Step 4 - Start DSE.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk