Compaction and memtable flush failing with null AssertionError from IncrementalTrieWriterPageAware.java

Summary

This article discusses an issue experienced when nodes are writing data to SSTables.

Applies to

• DataStax Enterprise 6.0.3

Symptoms

During compaction or when flushing memtables to disk, a null assertion error gets thrown by compaction or flush writer threads. Here is an example stack trace in the system.log generated while replaying commit logs on startup:

INFO  [main] 2018-10-09 12:34:57,461  CommitLog.java:187 - Replaying \
  /var/lib/cassandra/commitlog/CommitLog-600-1538030955810.log, \
  /var/lib/cassandra/commitlog/CommitLog-600-1538030955811.log
ERROR [PerDiskMemtableFlushWriter_0:2] 2018-10-09 12:34:00,414  CassandraDaemon.java:119 - Exception in thread Thread[PerDiskMemtableFlushWriter_0:2,5,main]
java.lang.AssertionError: null
        at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware$WriteRecursion.complete(IncrementalTrieWriterPageAware.java:401)
        at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware$Recursion.process(IncrementalTrieWriterPageAware.java:288)
        at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware.writeRecursive(IncrementalTrieWriterPageAware.java:410)
        at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware.performCompletion(IncrementalTrieWriterPageAware.java:127)
        at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware.performCompletion(IncrementalTrieWriterPageAware.java:67)
        at org.apache.cassandra.io.tries.IncrementalTrieWriterBase.complete(IncrementalTrieWriterBase.java:92)
        at org.apache.cassandra.io.sstable.format.trieindex.RowIndexWriter.complete(RowIndexWriter.java:77)
        at org.apache.cassandra.io.sstable.format.trieindex.PartitionWriter.finish(PartitionWriter.java:184)
        at org.apache.cassandra.io.sstable.format.trieindex.PartitionWriter.writePartition(PartitionWriter.java:106)
        at org.apache.cassandra.io.sstable.format.trieindex.TrieIndexSSTableWriter.append(TrieIndexSSTableWriter.java:186)
        at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:47)
        at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:825)
        at org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:903)
        at org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:679)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
        at java.lang.Thread.run(Thread.java:748)

Here is a similar stack trace resulting from a compaction thread:

ERROR [CompactionExecutor:3] 2018-09-14 12:16:31,132  CassandraDaemon.java:119 - Exception in thread Thread[CompactionExecutor:3,5,main]
java.lang.AssertionError: null
    at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware$WriteRecursion.complete(IncrementalTrieWriterPageAware.java:401)
    at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware$Recursion.process(IncrementalTrieWriterPageAware.java:288)
    at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware.writeRecursive(IncrementalTrieWriterPageAware.java:410)
    at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware.performCompletion(IncrementalTrieWriterPageAware.java:127)
    at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware.performCompletion(IncrementalTrieWriterPageAware.java:67)
    at org.apache.cassandra.io.tries.IncrementalTrieWriterBase.complete(IncrementalTrieWriterBase.java:92)
    at org.apache.cassandra.io.sstable.format.trieindex.RowIndexWriter.complete(RowIndexWriter.java:77)
    at org.apache.cassandra.io.sstable.format.trieindex.PartitionWriter.finish(PartitionWriter.java:184)
    at org.apache.cassandra.io.sstable.format.trieindex.PartitionWriter.writePartition(PartitionWriter.java:106)
    at org.apache.cassandra.io.sstable.format.trieindex.TrieIndexSSTableWriter.append(TrieIndexSSTableWriter.java:186)
    at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:139)
    at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
    at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:145)
    at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:210)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:92)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:101)
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:310)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
    at java.lang.Thread.run(Thread.java:748)

Cause

Cassandra operations such as compactions, reads and writes rely heavily on being able to do comparisons to order and/or merge results. DSE 6.0 introduced the use of trie-based indices to significantly improve the performance of DSE's version of Cassandra.

An issue which causes a stack overflow error during compaction was fixed in DSE 6.0.3 (DB-2364). The fix however inadvertently broke a bit of code which builds the trie data structures, using the wrong offset which results in a null value when calculating the trie size (DB-2477).

Solution

DB-2477 has been fixed in DSE 6.0.4. Upgrade to the latest version of DSE 6.0 to get the latest fixes and improvements.