Summary

When customers upgrade to 5.1 and start the node, it fails with "java.lang.RuntimeException: com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin". 

Symptoms

The node fails to start with the following error in system.log and debug.log:

ERROR [main] 2017-05-02 18:40:06,067  CassandraDaemon.java:705 - Exception encountered during startup
java.lang.RuntimeException: com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin com.datastax.bdp.reporting.snapshots.histograms.HistogramInfoPlugin
        at com.datastax.bdp.server.DseDaemon.start(DseDaemon.java:497) ~[dse-core-5.1.0.jar:5.1.0]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:614) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
        at com.datastax.bdp.DseModule.main(DseModule.java:93) [dse-core-5.1.0.jar:5.1.0]
Caused by: com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin com.datastax.bdp.reporting.snapshots.histograms.HistogramInfoPlugin
        at com.datastax.bdp.plugin.PluginManager.activateDirect(PluginManager.java:422) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.plugin.PluginManager.activate(PluginManager.java:208) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.plugin.PluginManager.preStart(PluginManager.java:104) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.server.DseDaemon.preStart(DseDaemon.java:517) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.server.DseDaemon.start(DseDaemon.java:487) ~[dse-core-5.1.0.jar:5.1.0]
        ... 2 common frames omitted
Caused by: java.lang.RuntimeException: Error preparing CQL statement
        at com.datastax.bdp.cassandra.cql3.StatementUtils.prepareStatement(StatementUtils.java:143) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.cassandra.cql3.StatementUtils.prepareStatement(StatementUtils.java:149) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.reporting.CqlWriter.createTable(CqlWriter.java:54) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.reporting.snapshots.histograms.AbstractHistogramWriter.lambda$createTable$18(AbstractHistogramWriter.java:93) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.gms.DseVersionNotifier.runAtVersion(DseVersionNotifier.java:145) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.snitch.EndpointStateTracker.runAtVersion(EndpointStateTracker.java:97) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.reporting.snapshots.histograms.AbstractHistogramWriter.createTable(AbstractHistogramWriter.java:89) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.reporting.snapshots.histograms.HistogramInfoPlugin.lambda$setupSchema$283(HistogramInfoPlugin.java:151) ~[dse-core-5.1.0.jar:5.1.0]
        at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_121]
        at com.datastax.bdp.reporting.snapshots.histograms.HistogramInfoPlugin.setupSchema(HistogramInfoPlugin.java:151) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.plugin.PluginManager.initialize(PluginManager.java:451) ~[dse-core-5.1.0.jar:5.1.0]
        at com.datastax.bdp.plugin.PluginManager.activateDirect(PluginManager.java:401) ~[dse-core-5.1.0.jar:5.1.0]
        ... 6 common frames omitted
Caused by: org.apache.cassandra.exceptions.InvalidRequestException: Undefined column name dropped_mutations
        at org.apache.cassandra.config.ColumnDefinition$Raw$Literal.find(ColumnDefinition.java:587) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
        at org.apache.cassandra.config.ColumnDefinition$Raw$Literal.find(ColumnDefinition.java:580) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
        at org.apache.cassandra.config.ColumnDefinition$Raw$Literal.prepare(ColumnDefinition.java:561) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
        at org.apache.cassandra.cql3.statements.ModificationStatement$Parsed.getColumnDefinition(ModificationStatement.java:906) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
        at org.apache.cassandra.cql3.statements.UpdateStatement$ParsedInsert.prepareInternal(UpdateStatement.java:160) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
        at org.apache.cassandra.cql3.statements.ModificationStatement$Parsed.prepare(ModificationStatement.java:807) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
        at org.apache.cassandra.cql3.statements.ModificationStatement$Parsed.prepare(ModificationStatement.java:793) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
        at org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:558) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
        at org.apache.cassandra.cql3.QueryProcessor.prepare(QueryProcessor.java:413) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
        at org.apache.cassandra.cql3.QueryProcessor.prepare(QueryProcessor.java:404) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
        at com.datastax.bdp.cassandra.cql3.StatementUtils.prepareStatement(StatementUtils.java:136) ~[dse-core-5.1.0.jar:5.1.0]
        ... 17 common frames omitted
INFO  [StorageServiceShutdownHook] 2017-05-02 18:40:06,069  DseDaemon.java:876 - DSE shutting down...

Cause

It is related with the schema change on column dropped_mutations in several CQL tables of DSE 5.1 Performance Object. When the related options of performance object, such as histogram_data_options and cluster_summary_stats_options, are enabled, the user will see the error after upgrade to 5.1 and can't start the node.

  

Workaround

1. Disable all PO options in dse.yaml, start the node.
2. Repeat step 1 and upgrade all the nodes to 5.1.
3. Back up the schema of keyspace dse_perf 
4. Drop all tables under keyspace dse_perf; Or for simple operation, drop keyspace dse_perf, then recreate an empty keyspace with "CREATE KEYSPACE dse_perf ..." statement.
5. Enable the desirable options in performance objects.
6. Rolling restart the nodes.

The corresponding tables will be recreated after the rolling restart.

Solution

DSP-13301 is filed and currently under development review.