Solr nodes are overwhelmed with warnings on re-index of expired columns.
Multiple entries in the logs of the same warning of the form:
WARN [SolrSecondaryIndex <keyspace.table> TTL scheduler.] 2015-03-04 01:56:39,505 AbstractSolrSecondaryIndex.java (line 1410) java.lang.NullPointerException
These warnings are repeated constantly and eventually fill up the logs in some instances.
The affected DSE versions are 4.0.3, 4.5.5 and 4.6.0.
Issue was identified to be a bug in the Indexer Rebuild task. The internal bug ID is DSP-4869. The Index Rebuild thread attempts to remove documents from the index but fails when it can not find rows in the database.
In one particular instance, this bug was triggered as a result of an infrastructure outage that affected hosts running Solr nodes. When the cluster was restored, the DBAs failed to perform a
repair on the cluster and data was missing from several nodes.
The Index Rebuild process runs regularly and keeps iterating through the expired documents and so keeps generating the same warnings in the logs.
It is important to note that the bug only gets triggered when data is missing from tables and the Index Rebuild thread attempts to remove Solr documents which have expired. If cluster data is consistent and cluster repairs are up to date, this issue would not manifest itself.
Otherwise, recover from this issue as follows:
Step 1 - Restart DSE on affected nodes.
Step 2 - Rebuild the Solr indexes.
A fix for DSP-4869 has been made available in DSE versions 4.0.6, 4.5.7, 4.6.2 and 4.7.0.