DSE "EverywhereStrategy" is not understood by Apache Cassandra (non-DSE) nodes and can cause restart failures

Summary

The recommended procedure to upgrade from Open Source Apache Cassandra (COSS) to DSE is to create a new cluster and stream the data across. However, if a DSE node is added into a COSS cluster there may be issues restarting any of the COSS nodes.

Symptoms

The cluster will appear to function ok but when a COSS node restart is attempted the following error may be observed

ERROR [main] 2016-02-26 12:08:26,197 CassandraDaemon.java:579 - Exception encountered during startup 
java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Unable to find replication strategy class 'org.apache.cassandra.locator.EverywhereStrategy' 

Cause

DSE uses a default replication strategy of "EverywhereStrategy" for the dse_system keyspace. This is propagated into the cluster's schema when the DSE node is added and makes its way into the system.schema* tables. Upon restarting the COSS node will try to read its sstables and error out when it tries to understand this unknown replication strategy

Workaround

The best approach is to keep the DSE node in the cluster and then alter the replication strategy for the dse_system keyspace to one that is understood by all nodes, for example:

ALTER KEYSPACE dse_system WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1':1 ,'DC2':1};

Afterwards the dse_system keyspace could be removed if so wished

Solution

At the time of writing this article there is no permanent solution available. An internal jira has been created to track the issue:

DSP-8794 - EverywhereStrategy "poisons" the schema preventing COSS nodes restarting