Prior to DSE versions of 5.1.12, 6.0.5 and 6.7.1, spark queries may fail with Solr optimization enabled in the following scenarios:
- Search had been enabled on a cluster.
- A search index was created.
- The cluster is restarted without Search enabled but Analytics is enabled
OR
- Search is enabled in one Data Center.
- A search index is created.
- Search is disabled in another Data Center which has Analytics enabled.
An improvement has been implemented in DSE versions 5.1.12+, 6.0.5+ ,and 6.7.1+ which will allow SolrPredicateRules only to be applied if the local DC has Search enabled.
If you are using Analytics on a cluster in either of the scenarios above, and there is a predicate query on a non-partition key, then the query will generate the following exception:
Error: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.InvalidQueryException: Cannot execute this query as it might involve data filtering and thus may have unpredictable performance. If you want to execute this query despite the performance unpredictability, use ALLOW FILTERING (state=,code=0)
The workaround is to disable Search optimization, e.g set the following parameter in spark-defaults.conf file:
spark.sql.dse.search.enableOptimization false
or set in the application code:
("spark.sql.dse.solr.enable_optimization", "off")