FAQ - Identifying Lucene flushes in log files for DSE 5.1x and 5.0.x

Summary

This article provides information on the events that cause Lucene flushes in DSE Search and how to identify them in the logs.

Applies to

• DataStax Enterprise 5.1.x
• DataStax Enterprise 5.0.x

Background

An important part of DSE Search tuning is understanding how often Lucene segments are flushed to disk. Understanding how to review the logs to identify this frequency can help gain insight when investigating performance problems.

What log events cause Lucene segments to flush to disk?

The following log events cause Lucene segments to be flushed to disk:

• MemtableFlushWriter - Cassandra memtable flush.
• Solr TTL scheduler - DSE search does a periodic sweep for deletions so this scheduler can be propagated into the Solr index.
• SolrSecondaryIndex <keyspace_name.table_name> index reloader - Solr core re-index
• commitScheduler-<number>-thread - DSE Search periodic soft commit
• <keyspace_name.table_name> Index WorkPool work thread - DSE search periodic soft commit using Steal of index worker (DSE 5.1 and earlier)

Lucene segments contain the documents that have been flushed to on-disk index. These segments are later merged using the selected merge policy that is configured in solrconfig.xml. The default merge policy is sufficient for most workloads.

What log file can I review?

These events are typically logged in the debug.log on every DSE node.

How do I know what time periods to review?

When you experienced a problem, it is useful to compare your normal merge events with the problem period to see if the frequency and type of these flush events have changed. For example, even though you typically see the TTL schedule, you might notice that the TTL scheduler is finding an increased number of documents to delete that could have coincided with a bulk load that was performed on the cluster.

What if the debug logs and system.log are updated in different time periods?

If you use OpsCenter, the diagnostics report will include the latest file as it does for other log files. However, the debug logs often rotate faster than the system.log. If you are engaging support to look at a given incident, make sure your log window covers the specific time window of the problem period and if not, send those additional log files.