Summary
In some cases, inadvertently altering one of the two DSE insights keyspaces, a known issue with the DSE insights plugin can prevent a node from starting up and causes issues with trying to add or replace nodes in a cluster.
Applies to
- DSE 5.1.14 to 5.1.16
- DSE 6.0.9
- DSE 6.7.4
Symptoms
The node will fail to start with a large error stack. The important part of this error stack is here in the system.log
:
1) Error in custom provider, java.lang.RuntimeException: java.lang.RuntimeException: Error preparing "SELECT JSON config from dse_insights_local.insights_config where key = 1" statement at com.datastax.bdp.insights.InsightsModule.insightsClient(InsightsModule.java:120) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.insights.InsightsModule) at com.datastax.bdp.insights.InsightsModule.insightsClient(InsightsModule.java:120) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.insights.InsightsModule) while locating com.datastax.insights.client.InsightsClient for parameter 0 at com.datastax.bdp.insights.InsightsClientRuntimeManager.<init>(InsightsClientRuntimeManager.java:37) at com.datastax.bdp.insights.InsightsModule.configure(Unknown Source) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.insights.InsightsModule) while locating com.datastax.bdp.insights.InsightsClientRuntimeManager for parameter 0 at com.datastax.bdp.insights.InsightsPlugin.<init>(InsightsPlugin.java:66) at com.datastax.bdp.insights.InsightsModule.configure(Unknown Source) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.insights.InsightsModule) while locating com.datastax.bdp.insights.InsightsPlugin Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Error preparing "SELECT JSON config from dse_insights_local.insights_config where key = 1" statement
Subsequently the nodes will show as present in the cluster, but with DN
in nodetool status.
Cause
When the node starts up, the DSE insights plugin checks for some user-defined types (UDTs) and tables that are declared in one of the two DSE insights keyspaces dse_insights
and dse_insights_local
. If these are missing (for example, you may have dropped them by mistake) or the RF has been changed (for example, using NetworkTopologyStrategy but not replicated to the DC where the nodes are being added), then the node will fail to start.
Workaround
Until the cluster is upgraded to DSE 5.1.17 or later, the only way to resolve this issue is to fix the keyspace replication for the dse_insights
and dse_insights_local
keyspaces.
Run the following commands in cqlsh on a node that is up and functional in the cluster.
Note: You must use the UPDATE
command to change the dse_insights_local
keyspace because cqlsh
restricts assignment of LocalStrategy
to keyspaces.
alter KEYSPACE dse_insights WITH replication = {'class': 'org.apache.cassandra.locator.EverywhereStrategy'}; UPDATE system_schema.keyspaces SET replication = {'class': 'org.apache.cassandra.locator.LocalStrategy'} WHERE keyspace_name = 'dse_insights_local'
Solution
Upgrade to DSE 5.1.17 or later for the fix to prevent the DSE insights plugin from stopping the node startup. Upgrading prevents the issue from happening, but you must perform the workaround steps to resolve any existing missing or incorrect keyspace replication problems.
This problem is tracked by internal Jira DSP-19521.