Node startup fails with error "ConfigurationException: Cannot change the number of tokens"

Summary

A DataStax Enterprise node is unable to startup and join the cluster with a ConfigurationException relating to the token assignments.

Symptoms

Node startup fails reporting an error while loading the configuration file cassandra.yaml. Below is a sample exception stack trace from DSE 4.5.8:

 INFO 17:01:33,587 Loading settings from file:/home/automaton/dse-4.5.8/resources/cassandra/conf/cassandra.yaml
ERROR 17:01:33,678 Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot change the number of tokens from 256 to 1
 at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:836)
 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:621)
 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:510)
 at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:437)
 at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:430)
 at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
 at com.datastax.bdp.server.DseDaemon.main(DseDaemon.java:671)
Cannot change the number of tokens from 256 to 1
Fatal configuration error; unable to start server. See log for stacktrace.

Cause

The node was configured incorrectly as a result of human error.

The node was previously configured with vnodes (256 tokens in the example above) but was changed back to single-token and started up with auto_bootstrap: true in cassandra.yaml.

The bootstrap failed when it compared the number of previously saved tokens in the system.local table against the num_tokens: property in cassandra.yaml.

Solution

Follow these steps to resolve this issue:

Step 1 - Calculate the token for the node as described in the Generating tokens document.

Step 2 - Reconfigure the token properties in cassandra.yaml:

• comment out num_tokens
• set the token assignment initial_token
• leave auto_bootstrap: true

Step 3 - Delete the following directories. This will ensure that all their contents are cleaned out completely.

• <data_file_directories>
• commitlog
• saved_caches

Step 4 - Recreate the directories above and make sure Cassandra has full permissions.

Step 5 - Bootstrap the node again by starting DSE.

See also

If you have not done so already, recalculate the token assignments. For more information, see Generating tokens.