DataStax Help Center

Search nodes with multiple entries in logs for AbstractSolrSecondaryIndex.java NullPointerException

Summary

Solr nodes are overwhelmed with warnings on re-index of expired columns.

Symptoms

Multiple entries in the logs of the same warning of the form:

WARN [SolrSecondaryIndex <keyspace.table> TTL scheduler.] 2015-03-04 01:56:39,505 AbstractSolrSecondaryIndex.java (line 1410)
java.lang.NullPointerException

These warnings are repeated constantly and eventually fill up the logs in some instances.

The affected DSE versions are 4.0.3, 4.5.5 and 4.6.0.

Cause

Issue was identified to be a bug in the Indexer Rebuild task. The internal bug ID is DSP-4869. The Index Rebuild thread attempts to remove documents from the index but fails when it can not find rows in the database.

In one particular instance, this bug was triggered as a result of an infrastructure outage that affected hosts running Solr nodes. When the cluster was restored, the DBAs failed to perform a repair on the cluster and data was missing from several nodes.

The Index Rebuild process runs regularly and keeps iterating through the expired documents and so keeps generating the same warnings in the logs.

Workaround

It is important to note that the bug only gets triggered when data is missing from tables and the Index Rebuild thread attempts to remove Solr documents which have expired. If cluster data is consistent and cluster repairs are up to date, this issue would not manifest itself.

Otherwise, recover from this issue as follows:

Step 1 - Restart DSE on affected nodes.

Step 2 - Rebuild the Solr indexes.

Solution

A fix for DSP-4869 has been made available in DSE versions 4.0.6, 4.5.7, 4.6.2 and 4.7.0.

Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

Comments

  • Avatar
    Eric Wasserman

    We hit this problem with a single node that failed and was restored. We have the repair service running but didn't initiate a manual repair. Doing a Full Rebuild of the index on the core that was complaining did the trick.

  • Avatar
    Erick Ramirez

    Thanks for the feedback, Eric. We are glad it worked for you. Cheers!

Powered by Zendesk