Summary
A "validation failed" was seen when running a repair
Symptoms
When the user attempted to run a repair, the following exception was seen in the /var/log/cassandra/system.log
INFO [AntiEntropyStage:1] 2015-06-05 07:13:34,981 RepairJob.java (line 161) [repair #610799a0-0b52-11e5-b97e-b7a7c52693de] requesting merkle trees for mytable (to [/10.50.3.144, /10.50.3.143, ip-10-
50-4-241.internal/10.50.4.241])
...
ERROR [ValidationExecutor:48] 2015-06-05 07:13:34,990 Validator.java (line 242) Failed creating a merkle tree for [repair #610799a0-0b52-11e5-b97e-b7a7c52693de on myks/mytable, (-1282988750579997629,-1185701225543990938]], /10.50.4.241 (see log for details)
ERROR [ValidationExecutor:48] 2015-06-05 07:13:34,990 CassandraDaemon.java (line 199) Exception in thread Thread[ValidationExecutor:48,1,main]
java.lang.RuntimeException: Cannot get comparator 1 in org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type). This might due to a mismatch between the schema and the data read
at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:133)
at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:140)
at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:59)
at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:36)
at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:112)
at org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:116)
at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:150)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:186)
at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:98)
at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:85)
at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:196)
at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:74)
at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:55)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:115)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:800)
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:306)
at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:285)
at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45)
at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:124)
... 23 more
Cause
It appears that some corruption had occurred to an sstable. It was originally reported in the following Cassandra jira:
https://issues.apache.org/jira/browse/CASSANDRA-8120
Solution
There are two tools: "nodetool scrub" and "sstablescrub", it is best to run the former first. If that is not successful then run the latter.
Run a nodetool scrub on the on the table:
http://docs.datastax.com/en/cassandra/2.0/cassandra/tools/toolsScrub.html
If nodetool scrub does not finish successfully then use the offline sstablescrub:
http://docs.datastax.com/en/cassandra/2.0/cassandra/tools/toolsSSTableScrub_t.html
Note: sstablescrub should be run with DSE offline (i.e. the DSE service is not running on the node where you are running sstable scrub)