Node upgraded from DSE 4.8 to 5.0 fails to start with NPE for LegacySchemaMigrator

Summary

This article discusses a rare situation where nodes upgraded to DSE 5.0 fails to start.

Applies to

• DataStax Enterprise 5.0.13 and earlier

Symptoms

In very rare and specific scenarios, a node previously running with DSE 4.8 which has been upgraded to earlier versions of DSE 5.0 fails to start immediately after the software binary upgrade. During startup, the LegacySchemaMigrator responsible for converting the old schema fails with a null pointer exception. Here is an example stack trace from a node upgraded to DSE 5.0.12:

ERROR [main] 2018-05-15 02:01:50,152  CassandraDaemon.java:678 - Exception encountered during startup
java.lang.NullPointerException: null
        at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:156) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.serializers.AbstractTextSerializer.deserialize(AbstractTextSerializer.java:41) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.serializers.AbstractTextSerializer.deserialize(AbstractTextSerializer.java:28) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:113) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.cql3.UntypedResultSet$Row.getString(UntypedResultSet.java:267) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.schema.LegacySchemaMigrator.createColumnFromColumnRow(LegacySchemaMigrator.java:743) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.schema.LegacySchemaMigrator.createColumnsFromColumnRows(LegacySchemaMigrator.java:698) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.schema.LegacySchemaMigrator.decodeTableMetadata(LegacySchemaMigrator.java:334) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.schema.LegacySchemaMigrator.readTableMetadata(LegacySchemaMigrator.java:275) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.schema.LegacySchemaMigrator.readTable(LegacySchemaMigrator.java:246) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readTables$7(LegacySchemaMigrator.java:239) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_131]
        at org.apache.cassandra.schema.LegacySchemaMigrator.readTables(LegacySchemaMigrator.java:239) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.schema.LegacySchemaMigrator.readKeyspace(LegacySchemaMigrator.java:188) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.schema.LegacySchemaMigrator.lambda$readSchema$4(LegacySchemaMigrator.java:179) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_131]
        at org.apache.cassandra.schema.LegacySchemaMigrator.readSchema(LegacySchemaMigrator.java:179) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySchemaMigrator.java:79) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:235) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:469) ~[dse-core-5.0.12.jar:5.0.12]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582) ~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
        at com.datastax.bdp.DseModule.main(DseModule.java:91) [dse-core-5.0.12.jar:5.0.12]

Cause

Range tombstones in Apache Cassandra 2.1 SSTables are handled incorrectly in Cassandra 3.0 where two CQL rows get created incorrectly when only one should exist (CASSANDRA-14008).

In rare conditions where a range tombstone exists in one of the legacy CQL schema SSTables, the redundant row which gets created contains null values leading to the NullPointerException for the LegacySchemaMigrator which runs on startup after a node is upgraded to DSE 5.0.

The fix was not included in earlier versions of DSE 5.0 until DSE 5.0.14.

Solution

If the cluster is affected by the symptoms above, upgrade to DSE 5.0.14 or later to obtain the fix which allows upgraded nodes to start successfully.