Full ERROR Message Example
ERROR [ValidationExecutor:2607] 2014-09-29 08:05:43,906 Validator.java (line 242) Failed creating a merkle tree for [repair #df973db0-47ae-11e4-8593-097f88b99f2b on <keyspace>/<table>, (5328258759137893466,5332261958806667240]], /<node IP Address> (see log for details)
What does this ERROR message mean?
This error will occur when there is an issue during the validation phase of the anti-entropy repair process. The error message includes the repair session ID, keyspace, table, token ranges and initiating node IP of that specific repair session.
Why does this ERROR occur?
Generally this error is caused by an underlying error related another part of the system. Look for additional error messages from the ValidationExecutor thread. Here are some examples of other underlying issues which can cause this error.
Multiple Repair Sessions
This occurrence was caused by inadvertently executing multiple repair sessions on the same token ranges concurrently.
2017-06-13 06:08:16,187 [ValidationExecutor:593] ERROR o.a.c.r.Validator Failed creating a merkle tree for [repair #b09d5d41-4ffe-11e7-8e95-5bb7b32c67d5 on OpsCenter/settings, [(8889172701211476431,8906446960321752952], (8643931488678411911,8856710018453695505], (-1740258122363769154,-1656184310864004355], (8944056438784578624,-9137192055313332485], (8856710018453695505,8889172701211476431], (8906446960321752952,8944056438784578624], (8412927907909524792,8643931488678411911], (-4099117174112970346,-4022271442049336905]]], /10.54.5.63 (see log for details) 2017-06-13 06:08:16,187 [ValidationExecutor:594] ERROR o.a.c.s.CassandraDaemon Exception in thread Thread[ValidationExecutor:594,1,main] java.lang.RuntimeException: Cannot start multiple repair sessions over the same sstables at org.apache.cassandra.service.ActiveRepairService$ParentRepairSession.markSSTablesRepairing(ActiveRepairService.java:526) ~[cassandra-all-3.0.8.1293.jar:3.0.8.1293] at org.apache.cassandra.db.compaction.CompactionManager.getSSTablesToValidate(CompactionManager.java:1187) ~[cassandra-all-3.0.8.1293.jar:3.0.8.1293] at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1091) ~[cassandra-all-3.0.8.1293.jar:3.0.8.1293] at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:80) ~[cassandra-all-3.0.8.1293.jar:3.0.8.1293] at org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:714) ~[cassandra-all-3.0.8.1293.jar:3.0.8.1293] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_05] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_05] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_05] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
Mmap Failure
Here the validation error was caused by insufficient mmap limits for the Cassandra process.
ERROR [ValidationExecutor:2607] 2014-09-29 08:05:43,906 Validator.java (line 242) Failed creating a merkle tree for [repair #df973db0-47ae-11e4-8593-097f88b99f2b on OpsCenter/settings, (5328258759137893466,5332261958806667240]], /10.20.21.16 (see log for details)
ERROR [ValidationExecutor:2607] 2014-09-29 08:05:43,907 CassandraDaemon.java (line 199) Exception in thread Thread[ValidationExecutor:2607,1,main]
FSReadError in /var/lib/cassandra/data/OpsCenter/settings/snapshots/df973db0-47ae-11e4-8593-097f88b99f2b/OpsCenter-settings-jb-26979-Index.db
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:200)
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:168)
at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:457)
at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:422)
at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:203)
at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:184)
at org.apache.cassandra.db.ColumnFamilyStore.getSnapshotSSTableReader(ColumnFamilyStore.java:1843)
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:766)
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:398)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192)
... 13 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:885)
... 14 more
SStable Corruption
A corruption on the underlying sstables caused this validation error.
ERROR [ValidationExecutor:51] 2016-03-31 12:58:49,610 Validator.java:245 - Failed creating a merkle tree for [repair #d5d62de0-f761-11e5-bf9d-6b0c2e45941b on OpsCenter/rollups300, (-7738447546856083245,-7695742441447957635]], /10.135.132.160 (see log for details) ERROR [ValidationExecutor:51] 2016-03-31 12:58:49,610 CassandraDaemon.java:227 - Exception in thread Thread[ValidationExecutor:51,1,main] org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /data/OpsCenter/rollups300-16563691d78111e5b8c36b0c2e45941b/OpsCenter-rollups300-ka-29670-Data.db
How do you fix this ERROR?
The solution to the validation error depends entirely on the underlying cause. To fix the error, identify the root cause by looking for additional errors on the ValidationExecutor thread.
Comments
0 comments
Please sign in to leave a comment.