Summary
This article discusses an issue experienced when nodes are writing data to SSTables.
Applies to
- DataStax Enterprise 6.0.3
Symptoms
During compaction or when flushing memtables to disk, a null assertion error gets thrown by compaction or flush writer threads. Here is an example stack trace in the system.log generated while replaying commit logs on startup:
INFO [main] 2018-10-09 12:34:57,461 CommitLog.java:187 - Replaying \
/var/lib/cassandra/commitlog/CommitLog-600-1538030955810.log, \
/var/lib/cassandra/commitlog/CommitLog-600-1538030955811.log ERROR [PerDiskMemtableFlushWriter_0:2] 2018-10-09 12:34:00,414 CassandraDaemon.java:119 - Exception in thread Thread[PerDiskMemtableFlushWriter_0:2,5,main] java.lang.AssertionError: null at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware$WriteRecursion.complete(IncrementalTrieWriterPageAware.java:401) at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware$Recursion.process(IncrementalTrieWriterPageAware.java:288) at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware.writeRecursive(IncrementalTrieWriterPageAware.java:410) at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware.performCompletion(IncrementalTrieWriterPageAware.java:127) at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware.performCompletion(IncrementalTrieWriterPageAware.java:67) at org.apache.cassandra.io.tries.IncrementalTrieWriterBase.complete(IncrementalTrieWriterBase.java:92) at org.apache.cassandra.io.sstable.format.trieindex.RowIndexWriter.complete(RowIndexWriter.java:77) at org.apache.cassandra.io.sstable.format.trieindex.PartitionWriter.finish(PartitionWriter.java:184) at org.apache.cassandra.io.sstable.format.trieindex.PartitionWriter.writePartition(PartitionWriter.java:106) at org.apache.cassandra.io.sstable.format.trieindex.TrieIndexSSTableWriter.append(TrieIndexSSTableWriter.java:186) at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:47) at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:825) at org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:903) at org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:679) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) at java.lang.Thread.run(Thread.java:748)
Here is a similar stack trace resulting from a compaction thread:
ERROR [CompactionExecutor:3] 2018-09-14 12:16:31,132 CassandraDaemon.java:119 - Exception in thread Thread[CompactionExecutor:3,5,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware$WriteRecursion.complete(IncrementalTrieWriterPageAware.java:401)
at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware$Recursion.process(IncrementalTrieWriterPageAware.java:288)
at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware.writeRecursive(IncrementalTrieWriterPageAware.java:410)
at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware.performCompletion(IncrementalTrieWriterPageAware.java:127)
at org.apache.cassandra.io.tries.IncrementalTrieWriterPageAware.performCompletion(IncrementalTrieWriterPageAware.java:67)
at org.apache.cassandra.io.tries.IncrementalTrieWriterBase.complete(IncrementalTrieWriterBase.java:92)
at org.apache.cassandra.io.sstable.format.trieindex.RowIndexWriter.complete(RowIndexWriter.java:77)
at org.apache.cassandra.io.sstable.format.trieindex.PartitionWriter.finish(PartitionWriter.java:184)
at org.apache.cassandra.io.sstable.format.trieindex.PartitionWriter.writePartition(PartitionWriter.java:106)
at org.apache.cassandra.io.sstable.format.trieindex.TrieIndexSSTableWriter.append(TrieIndexSSTableWriter.java:186)
at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:139)
at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:145)
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:210)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:92)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:101)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:310)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
at java.lang.Thread.run(Thread.java:748)
Cause
Cassandra operations such as compactions, reads and writes rely heavily on being able to do comparisons to order and/or merge results. DSE 6.0 introduced the use of trie-based indices to significantly improve the performance of DSE's version of Cassandra.
An issue which causes a stack overflow error during compaction was fixed in DSE 6.0.3 (DB-2364). The fix however inadvertently broke a bit of code which builds the trie data structures, using the wrong offset which results in a null value when calculating the trie size (DB-2477).
Solution
DB-2477 has been fixed in DSE 6.0.4. Upgrade to the latest version of DSE 6.0 to get the latest fixes and improvements.