DataStax Help Center

Incremental repair fails with "received out of order wrt DecoratedKey" error


Under some circumstances an assertion error might be seen when running incremental repairs, especially with, but not limited to tables using Leveled compaction strategy. 


The following is typical of the type or error seen in the /var/log/cassandra/system.log file:

ERROR [ValidationExecutor:158] 2016-03-22 10:26:26,747 - Failed creating a merkle tree for [repair #25c18291-f010-11e5-947f-598e34c38298 on mykeyspace/table1, (7613954818482387724,7617142735463175341]], / (see log for details)
ERROR [ValidationExecutor:158] 2016-03-22 10:26:26,747 - Exception in thread Thread[ValidationExecutor:158,1,main]
java.lang.AssertionError: row DecoratedKey(7614440285461857870, 0345e912) received out of order wrt DecoratedKey(7616204422921296916, 0226cb69)
        at ~[cassandra-all-]
        at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction( ~[cassandra-all-]
        at org.apache.cassandra.db.compaction.CompactionManager.access$600( ~[cassandra-all-]
        at org.apache.cassandra.db.compaction.CompactionManager$ ~[cassandra-all-]
        at ~[na:1.8.0_74]
        at java.util.concurrent.ThreadPoolExecutor.runWorker( ~[na:1.8.0_74]
        at java.util.concurrent.ThreadPoolExecutor$ [na:1.8.0_74]
        at [na:1.8.0_74]



This is due to anti-compaction in incremental repairs failing to remove old sstables. As a result, an AssertionError gets thrown when the old sstables are repaired again in a subsequent run because the rows in the old sstables are now considered out-of-order. The problem is predominantly caused by CASSANDRA-11548. There is also input from CASSANDRA-9935. The following internal jira tracks these:

DSP-9327 - Need fix for CASSANDRA-11548


This problem is addressed with an upgrade to DSE versions 4.7.9, 4.8.7 or later.



Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request


Powered by Zendesk