Full ERROR message example
ERROR [main] 2020-06-16 09:14:42,218 Ec2Snitch.java:210 - This ec2-enabled snitch appears to be using the legacy naming scheme for regions, but existing nodes in cluster are using the opposite: region(s) = [us-west-2], availability zone(s) = [2a]. Please check the ec2_naming_scheme property in the cassandra-rackdc.properties configuration file for more details. ERROR [main] 2020-06-16 09:14:42,219 CassandraDaemon.java:789 - Exception encountered during startup java.lang.IllegalStateException: null at org.apache.cassandra.service.StorageService.validateEndpointSnitch(StorageService.java:573) at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:530) at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:800) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:659) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:610) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:373) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:650) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:767)
What does this ERROR mean?
This error means that the endpoint snitch validation process in Cassandra versions 4.0+ found a mismatch in the naming conventions for regions and availability zones between this node and the rest of the cluster.
Why does this ERROR occur?
In late 2017, AWS changed the naming conventions for region names to end with a digit. Cassandra allows custom datacenter suffixes, and an operator could conceivably make the suffix "a", and thus create a region name that looks just like one of the region's availability zones. (for example, "us-east-1a"). A mismatch in naming schemes would essentially place the nodes in different clusters and potentially make nodes unreachable, so Cassandra throws an error to make it evident as to what has happened.
How do you fix this ERROR?
As the error states, you should check the ec2_naming_scheme property in the cassandra-rackdc.properties configuration file for more details. The ec2_naming_scheme should be in the file for Cassandra 4.0+ and commented out if no prior change was made. The default value is to use the standard naming scheme.
# Datacenter and rack naming convention used by the Ec2Snitch and Ec2MultiRegionSnitch.
# Options are:
# legacy : datacenter name is the part of the availability zone name preceding the last "-"
# when the zone ends in -1 and includes the number if not -1. Rack is the portion of
# the availability zone name following the last "-".
# Examples: us-west-1a => dc: us-west, rack: 1a; us-west-2b => dc: us-west-2, rack: 2b;
# YOU MUST USE THIS VALUE IF YOU ARE UPGRADING A PRE-4.0 CLUSTER
# standard : Default value. datacenter name is the standard AWS region name, including the number.
# rack name is the region plus the availability zone letter.
# Examples: us-west-1a => dc: us-west-1, rack: us-west-1a; us-west-2b => dc: us-west-2, rack: us-west-2b;
Make sure that the naming convention is the same across all nodes. If you are upgrading a pre-4.0 cluster, you must use the legacy naming convention.