DataStax Help Center

Incremental repair fails with "received out of order wrt DecoratedKey" error

Summary

Under some circumstances an assertion error might be seen when running incremental repairs, especially with, but not limited to tables using Leveled compaction strategy. 

Symptoms

The following is typical of the type or error seen in the /var/log/cassandra/system.log file:

ERROR [ValidationExecutor:158] 2016-03-22 10:26:26,747  Validator.java:245 - Failed creating a merkle tree for [repair #25c18291-f010-11e5-947f-598e34c38298 on mykeyspace/table1, (7613954818482387724,7617142735463175341]], /10.1.2.3 (see log for details)
ERROR [ValidationExecutor:158] 2016-03-22 10:26:26,747  CassandraDaemon.java:229 - Exception in thread Thread[ValidationExecutor:158,1,main]
java.lang.AssertionError: row DecoratedKey(7614440285461857870, 0345e912) received out of order wrt DecoratedKey(7616204422921296916, 0226cb69)
        at org.apache.cassandra.repair.Validator.add(Validator.java:127) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
        at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1051) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
        at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:89) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
        at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:662) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_74]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_74]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_74]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]

 

Cause

This is due to anti-compaction in incremental repairs failing to remove old sstables. As a result, an AssertionError gets thrown when the old sstables are repaired again in a subsequent run because the rows in the old sstables are now considered out-of-order. The problem is predominantly caused by CASSANDRA-11548. There is also input from CASSANDRA-9935. The following internal jira tracks these:

DSP-9327 - Need fix for CASSANDRA-11548

Solution

This problem is addressed with an upgrade to DSE versions 4.7.9, 4.8.7 or later.

 

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk