"Error in custom provider" prevents DSE node startup

Summary

In some cases, inadvertently altering one of the two DSE insights keyspaces, a known issue with the DSE insights plugin can prevent a node from starting up and causes issues with trying to add or replace nodes in a cluster.

Applies to

• DSE 5.1.14 to 5.1.16
• DSE 6.0.9
• DSE 6.7.4

Symptoms

The node will fail to start with a large error stack. The important part of this error stack is here in the system.log:

1) Error in custom provider, java.lang.RuntimeException: java.lang.RuntimeException: Error preparing "SELECT JSON config from dse_insights_local.insights_config where key = 1" statement
  at com.datastax.bdp.insights.InsightsModule.insightsClient(InsightsModule.java:120) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.insights.InsightsModule)
  at com.datastax.bdp.insights.InsightsModule.insightsClient(InsightsModule.java:120) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.insights.InsightsModule)
  while locating com.datastax.insights.client.InsightsClient
    for parameter 0 at com.datastax.bdp.insights.InsightsClientRuntimeManager.<init>(InsightsClientRuntimeManager.java:37)
  at com.datastax.bdp.insights.InsightsModule.configure(Unknown Source) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.insights.InsightsModule)
  while locating com.datastax.bdp.insights.InsightsClientRuntimeManager
    for parameter 0 at com.datastax.bdp.insights.InsightsPlugin.<init>(InsightsPlugin.java:66)
  at com.datastax.bdp.insights.InsightsModule.configure(Unknown Source) (via modules: com.datastax.bdp.DseModule -> com.datastax.bdp.insights.InsightsModule)
  while locating com.datastax.bdp.insights.InsightsPlugin
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Error preparing "SELECT JSON config from dse_insights_local.insights_config where key = 1" statement  

Subsequently the nodes will show as present in the cluster, but with DN in nodetool status.

Cause

When the node starts up, the DSE insights plugin checks for some user-defined types (UDTs) and tables that are declared in one of the two DSE insights keyspaces dse_insights and dse_insights_local. If these are missing (for example, you may have dropped them by mistake) or the RF has been changed (for example, using NetworkTopologyStrategy but not replicated to the DC where the nodes are being added), then the node will fail to start.

Workaround

Until the cluster is upgraded to DSE 5.1.17 or later, the only way to resolve this issue is to fix the keyspace replication for the dse_insights and dse_insights_local keyspaces.

Run the following commands in cqlsh on a node that is up and functional in the cluster.

Note: You must use the UPDATE command to change the dse_insights_local keyspace because cqlsh restricts assignment of LocalStrategy to keyspaces.

alter KEYSPACE dse_insights WITH replication = {'class': 'org.apache.cassandra.locator.EverywhereStrategy'};
UPDATE system_schema.keyspaces SET replication = {'class': 'org.apache.cassandra.locator.LocalStrategy'}  WHERE keyspace_name = 'dse_insights_local'

Solution

Upgrade to DSE 5.1.17 or later for the fix to prevent the DSE insights plugin from stopping the node startup. Upgrading prevents the issue from happening, but you must perform the workaround steps to resolve any existing missing or incorrect keyspace replication problems. 

This problem is tracked by internal Jira DSP-19521.