Full ERROR Message Example
ERROR [AntiCompactionExecutor:2] 2020-08-28 04:10:25,762 CompactionManager.java:1597 - Error anticompacting [BigTableReader(path='/var/lib/cassandra/data/keyspace/table-f934d8b0faba11e6936d59819ac49009/mc-14269510-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/keyspace/table-f934d8b0faba11e6936d59819ac49009/mc-14334269-big-Data.db')]
TYPICALLY PRODUCES A STACK TRACE SIMILAR TO THE FOLLOWING:
ERROR [CompactionExecutor:44736] 2020-08-22 10:09:50,205 CompactionManager.java:1320 - Error anticompacting [BigTableReader(path='/var/lib/cassandra/data/keyspace/table-09cef1213ba111e78303758e6998286b/mc-35-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/keyspace/table-09cef1213ba111e78303758e6998286b/mc-33-big-Data.db')]
java.lang.AssertionError: null
at org.apache.cassandra.db.rows.ComplexColumnData$Builder.addCell(ComplexColumnData.java:254) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:636) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:562) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:220) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:159) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.db.rows.Row$Merger.merge(Row.java:539) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:482) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:446) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:220) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:159) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:428) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:288) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
What does this ERROR message mean?
Anti-compaction is completed as part of the incremental repair process and involves separating repaired and un-repaired ranges into separate sstables, an ERROR during anti-compaction means an exception occurred while segregating the sstables, this can happen during the read or write phase. The stack trace will provide the most detail as to why the anti-compaction failed.
Why does this ERROR occur?
The error typically occurs because an sstable could not be read, or because there is not enough storage space to write a new sstable to disk.
How do you fix this ERROR?
When this ERROR is returned, the full stack trace should be reviewed in the log and the underlying problem should be addressed before attempting to repair the table again.
The two most common scenarios that cause this problem are documented below.
SSTable corruption
If an sstable is corrupted and cannot be read during anti-compaction, look for a message similar to the following in the log:
ERROR [AntiCompactionExecutor:2] 2020-08-29 04:10:25,762 CompactionManager.java:1597 - Error anticompacting [BigTableReader(path='/var/lib/cassandra/data/keyspace/table-f934d8b0faba11e6936d59819ac49119/mc-14269509-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/keyspace/table-f934d8b0faba11e6936d59819ac49119/mc-14334268-big-Data.db')] org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/lib/cassandra/data/keyspace/table-f934d8b0faba11e6936d59819ac49119/mc-14334268-big-Data.db
If this happens, first try repairing the table using "nodetool scrub keyspace table" to see if it can repaired online.
If this fails, bring the database offline and attempt to repair the table using sstablescrub, for example:
$ sudo sstablescrub keyspace table
If the corruption is significant enough it cannot be repaired, either restore the table from a backup, or if the Replication Factor is such that another copy of the data exists in the cluster, delete the corrupted group of sstables and repair the table to repopulate the node's data by streaming from a healthy node, for example:
$ nodetool repair keyspace table
No space left on device
If you run out of disk space during anti-compaction, look for a message similar to the following in the log:
ERROR [CompactionExecutor:2014] 2020-09-05 00:53:10,310 CompactionManager.java:1118 - Error anticompacting SSTableReader(path='/var/lib/cassandra/data/keyspace/table-bdc977d0c68711e59262d3d06e12d2b6/sstable-name-65730-Data.db') org.apache.cassandra.io.FSWriteError: java.io.IOException: No space left on device
If this happens, procure additional storage space, ensure it is properly configured in the data_file_directories section of the cassandra.yaml and restart Cassandra before attempting to repair the table again.