Summary
A DataStax Enterprise node is unable to startup and join the cluster with a ConfigurationException relating to the token assignments.
Symptoms
Node startup fails reporting an error while loading the configuration file cassandra.yaml. Below is a sample exception stack trace from DSE 4.5.8:
INFO 17:01:33,587 Loading settings from file:/home/automaton/dse-4.5.8/resources/cassandra/conf/cassandra.yaml
ERROR 17:01:33,678 Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot change the number of tokens from 256 to 1
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:836)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:621)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:510)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:437)
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:430)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
at com.datastax.bdp.server.DseDaemon.main(DseDaemon.java:671)
Cannot change the number of tokens from 256 to 1
Fatal configuration error; unable to start server. See log for stacktrace.
Cause
The node was configured incorrectly as a result of human error.
The node was previously configured with vnodes (256 tokens in the example above) but was changed back to single-token and started up with auto_bootstrap: true in cassandra.yaml.
The bootstrap failed when it compared the number of previously saved tokens in the system.local table against the num_tokens: property in cassandra.yaml.
Solution
Follow these steps to resolve this issue:
Step 1 - Calculate the token for the node as described in the Generating tokens document.
Step 2 - Reconfigure the token properties in cassandra.yaml:
- comment out
num_tokens - set the token assignment
initial_token - leave auto_bootstrap: true
Step 3 - Delete the following directories. This will ensure that all their contents are cleaned out completely.
<data_file_directories>commitlogsaved_caches
Step 4 - Recreate the directories above and make sure Cassandra has full permissions.
Step 5 - Bootstrap the node again by starting DSE.
See also
If you have not done so already, recalculate the token assignments. For more information, see Generating tokens.
Comments