Summary
When customers upgrade to 5.1 and start the node, it fails with "java.lang.RuntimeException: com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin".
Symptoms
The node fails to start with the following error in system.log and debug.log:
ERROR [main] 2017-05-02 18:40:06,067 CassandraDaemon.java:705 - Exception encountered during startup
java.lang.RuntimeException: com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin com.datastax.bdp.reporting.snapshots.histograms.HistogramInfoPlugin
at com.datastax.bdp.server.DseDaemon.start(DseDaemon.java:497) ~[dse-core-5.1.0.jar:5.1.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:614) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at com.datastax.bdp.DseModule.main(DseModule.java:93) [dse-core-5.1.0.jar:5.1.0]
Caused by: com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin com.datastax.bdp.reporting.snapshots.histograms.HistogramInfoPlugin
at com.datastax.bdp.plugin.PluginManager.activateDirect(PluginManager.java:422) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.plugin.PluginManager.activate(PluginManager.java:208) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.plugin.PluginManager.preStart(PluginManager.java:104) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.server.DseDaemon.preStart(DseDaemon.java:517) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.server.DseDaemon.start(DseDaemon.java:487) ~[dse-core-5.1.0.jar:5.1.0]
... 2 common frames omitted
Caused by: java.lang.RuntimeException: Error preparing CQL statement
at com.datastax.bdp.cassandra.cql3.StatementUtils.prepareStatement(StatementUtils.java:143) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.cassandra.cql3.StatementUtils.prepareStatement(StatementUtils.java:149) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.reporting.CqlWriter.createTable(CqlWriter.java:54) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.reporting.snapshots.histograms.AbstractHistogramWriter.lambda$createTable$18(AbstractHistogramWriter.java:93) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.gms.DseVersionNotifier.runAtVersion(DseVersionNotifier.java:145) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.snitch.EndpointStateTracker.runAtVersion(EndpointStateTracker.java:97) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.reporting.snapshots.histograms.AbstractHistogramWriter.createTable(AbstractHistogramWriter.java:89) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.reporting.snapshots.histograms.HistogramInfoPlugin.lambda$setupSchema$283(HistogramInfoPlugin.java:151) ~[dse-core-5.1.0.jar:5.1.0]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[na:1.8.0_121]
at com.datastax.bdp.reporting.snapshots.histograms.HistogramInfoPlugin.setupSchema(HistogramInfoPlugin.java:151) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.plugin.PluginManager.initialize(PluginManager.java:451) ~[dse-core-5.1.0.jar:5.1.0]
at com.datastax.bdp.plugin.PluginManager.activateDirect(PluginManager.java:401) ~[dse-core-5.1.0.jar:5.1.0]
... 6 common frames omitted
Caused by: org.apache.cassandra.exceptions.InvalidRequestException: Undefined column name dropped_mutations
at org.apache.cassandra.config.ColumnDefinition$Raw$Literal.find(ColumnDefinition.java:587) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.config.ColumnDefinition$Raw$Literal.find(ColumnDefinition.java:580) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.config.ColumnDefinition$Raw$Literal.prepare(ColumnDefinition.java:561) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.cql3.statements.ModificationStatement$Parsed.getColumnDefinition(ModificationStatement.java:906) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.cql3.statements.UpdateStatement$ParsedInsert.prepareInternal(UpdateStatement.java:160) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.cql3.statements.ModificationStatement$Parsed.prepare(ModificationStatement.java:807) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.cql3.statements.ModificationStatement$Parsed.prepare(ModificationStatement.java:793) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:558) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.cql3.QueryProcessor.prepare(QueryProcessor.java:413) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.cql3.QueryProcessor.prepare(QueryProcessor.java:404) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at com.datastax.bdp.cassandra.cql3.StatementUtils.prepareStatement(StatementUtils.java:136) ~[dse-core-5.1.0.jar:5.1.0]
... 17 common frames omitted
INFO [StorageServiceShutdownHook] 2017-05-02 18:40:06,069 DseDaemon.java:876 - DSE shutting down...
Cause
It is related with the schema change on column dropped_mutations
in several CQL tables of DSE 5.1 Performance Object. When the related options of performance object, such as histogram_data_options and cluster_summary_stats_options
, are enabled, the user will see the error after upgrade to 5.1 and can't start the node.
Workaround
- Disable all PO options in dse.yaml, start the node.
- Repeat step 1 and upgrade all the nodes to 5.1.
- Back up the schema of keyspace dse_perf
- Drop all tables under keyspace dse_perf; Or for simple operation, drop keyspace dse_perf, then recreate an empty keyspace with "CREATE KEYSPACE dse_perf ..." statement.
- Enable the desirable options in performance objects.
- Rolling restart the nodes.
Solution
DSP-13301 is filed and currently under development review.