Support Policy      Security Notices
  1. DataStax Support
  2. ERROR Messages
  3. Errors

ERROR: Error anticompacting

Follow

Full ERROR Message Example

ERROR [AntiCompactionExecutor:2] 2020-08-28 04:10:25,762  CompactionManager.java:1597 - Error anticompacting [BigTableReader(path='/var/lib/cassandra/data/keyspace/table-f934d8b0faba11e6936d59819ac49009/mc-14269510-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/keyspace/table-f934d8b0faba11e6936d59819ac49009/mc-14334269-big-Data.db')]

TYPICALLY PRODUCES A STACK TRACE SIMILAR TO THE FOLLOWING:

ERROR [CompactionExecutor:44736] 2020-08-22 10:09:50,205  CompactionManager.java:1320 - Error anticompacting [BigTableReader(path='/var/lib/cassandra/data/keyspace/table-09cef1213ba111e78303758e6998286b/mc-35-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/keyspace/table-09cef1213ba111e78303758e6998286b/mc-33-big-Data.db')]
java.lang.AssertionError: null
    at org.apache.cassandra.db.rows.ComplexColumnData$Builder.addCell(ComplexColumnData.java:254) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:636) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:562) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:220) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:159) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.db.rows.Row$Merger.merge(Row.java:539) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:482) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:446) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:220) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:159) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:428) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:288) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]
    at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159]

What does this ERROR message mean?

Anti-compaction is completed as part of the incremental repair process and involves separating repaired and un-repaired ranges into separate sstables, an ERROR during anti-compaction means an exception occurred while segregating the sstables, this can happen during the read or write phase. The stack trace will provide the most detail as to why the anti-compaction failed.

Why does this ERROR occur?

The error typically occurs because an sstable could not be read, or because there is not enough storage space to write a new sstable to disk.

How do you fix this ERROR?

When this ERROR is returned, the full stack trace should be reviewed in the log and the underlying problem should be addressed before attempting to repair the table again.

The two most common scenarios that cause this problem are documented below.

SSTable corruption

If an sstable is corrupted and cannot be read during anti-compaction, look for a message similar to the following in the log:

ERROR [AntiCompactionExecutor:2] 2020-08-29 04:10:25,762  CompactionManager.java:1597 - Error anticompacting [BigTableReader(path='/var/lib/cassandra/data/keyspace/table-f934d8b0faba11e6936d59819ac49119/mc-14269509-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/keyspace/table-f934d8b0faba11e6936d59819ac49119/mc-14334268-big-Data.db')]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/lib/cassandra/data/keyspace/table-f934d8b0faba11e6936d59819ac49119/mc-14334268-big-Data.db

If this happens, first try repairing the table using "nodetool scrub keyspace table" to see if it can repaired online.
If this fails, bring the database offline and attempt to repair the table using sstablescrub, for example:

$ sudo sstablescrub keyspace table

If the corruption is significant enough it cannot be repaired, either restore the table from a backup, or if the Replication Factor is such that another copy of the data exists in the cluster, delete the corrupted group of sstables and repair the table to repopulate the node's data by streaming from a healthy node, for example:

$ nodetool repair keyspace table

No space left on device

If you run out of disk space during anti-compaction, look for a message similar to the following in the log:

ERROR [CompactionExecutor:2014] 2020-09-05 00:53:10,310  CompactionManager.java:1118 - Error anticompacting SSTableReader(path='/var/lib/cassandra/data/keyspace/table-bdc977d0c68711e59262d3d06e12d2b6/sstable-name-65730-Data.db')
org.apache.cassandra.io.FSWriteError: java.io.IOException: No space left on device

If this happens, procure additional storage space, ensure it is properly configured in the data_file_directories section of the cassandra.yaml and restart Cassandra before attempting to repair the table again.

Created: 2020-09-24 20:38:59 UTC
Last Update: 2020-09-24 20:38:59 UTC

Related articles

Comments

0 comments

Please sign in to leave a comment.