Summary
This article discusses the root cause for a DataStax Enterprise node failing to startup in non-Analytics mode.
Symptoms
When attempting to startup a DSE node in real-time Cassandra mode only, DSE fails to start with a runtime exception. Below are sample entries from a DSE 4.8.6 system.log
:
ERROR [main] 2016-04-28 20:14:23,324 DseDaemon.java:359 - Unable to start DSE server. java.lang.RuntimeException: com.datastax.bdp.plugin.PluginManager$PluginActivationException: Plugin activation failed at com.datastax.bdp.plugin.PluginManager.activate(PluginManager.java:170) ~[dse-core-4.8.6.jar:4.8.6] at com.datastax.bdp.plugin.PluginManager.preStart(PluginManager.java:88) ~[dse-core-4.8.6.jar:4.8.6] at com.datastax.bdp.server.DseDaemon.preStart(DseDaemon.java:383) ~[dse-core-4.8.6.jar:4.8.6] at com.datastax.bdp.server.DseDaemon.start(DseDaemon.java:352) ~[dse-core-4.8.6.jar:4.8.6] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:577) [cassandra-all-2.1.13.1218.jar:2.1.13.1218] at com.datastax.bdp.DseModule.main(DseModule.java:74) [dse-core-4.8.6.jar:4.8.6] Caused by: com.datastax.bdp.plugin.PluginManager$PluginActivationException: Plugin activation failed at com.datastax.bdp.plugin.PluginManager.activateDirect(PluginManager.java:310) ~[dse-core-4.8.6.jar:4.8.6] at com.datastax.bdp.plugin.PluginManager.activate(PluginManager.java:162) ~[dse-core-4.8.6.jar:4.8.6] ... 5 common frames omitted Caused by: java.lang.RuntimeException: SparkApplicationInfoRollupPlugin cannot be used in non-Spark node. at com.datastax.bdp.spark.reporting.SparkApplicationInfoRollupPlugin.init(SparkApplicationInfoRollupPlugin.java:34) ~[dse-spark-4.8.6.jar:4.8.6] at com.datastax.bdp.reporting.snapshots.AbstractSnapshotInfoPlugin.setupSchema(AbstractSnapshotInfoPlugin.java:57) ~[dse-core-4.8.6.jar:4.8.6] at com.datastax.bdp.plugin.PluginManager.initialize(PluginManager.java:359) ~[dse-core-4.8.6.jar:4.8.6] at com.datastax.bdp.plugin.PluginManager.activateDirect(PluginManager.java:289) ~[dse-core-4.8.6.jar:4.8.6] ... 6 common frames omitted
Cause
The issue occurs because some Analytics options have been enabled in dse.yaml
despite the node not being started up in Analytics mode, for example:
# Spark cluster summary stats options spark_cluster_info_options: enabled: true ...
# Spark application stats options spark_application_info_options: enabled: true ...
As a result, the Spark plugins are getting activated at startup:
INFO [main] 2016-04-28 20:13:57,261 Workload.java:85 - Setting my workload to Cassandra ... INFO [main] 2016-04-28 20:14:22,759 PluginBean.java:54 - SparkClusterInfo plugin is now enabled INFO [main] 2016-04-28 20:14:22,759 AsyncSnapshotInfoBean.java:52 - SparkClusterInfo plugin using 1 async writers INFO [main] 2016-04-28 20:14:22,760 SnapshotInfoBean.java:58 - SparkClusterInfo refresh rate set to 10000 (was 0) INFO [main] 2016-04-28 20:14:22,777 HiveMetaStorePlugin.java:50 - Hive metastore version: 1 INFO [main] 2016-04-28 20:14:22,782 NodeObjectLatencyPlugin.java:56 - Initializing data object io tracker plugin INFO [main] 2016-04-28 20:14:22,788 PluginBean.java:54 - SparkApplicationInfo plugin is now enabled INFO [main] 2016-04-28 20:14:22,789 SnapshotInfoBean.java:58 - SparkApplicationInfo refresh rate set to 10000 (was 0)
Solution
Since the Spark options are only valid when the node is started in Analytics mode, update dse.yaml
and disable them as follows:
# Spark cluster summary stats options spark_cluster_info_options: enabled: false ...
# Spark application stats options spark_application_info_options: enabled: false ...
This should allow DSE to startup as a Cassandra real-time mode only.
See also
DataStax doc - Monitoring Spark with Spark Performance Objects