"Validation failed" when running a nodetool repair

Summary

A "validation failed" was seen when running a repair

Symptoms

When the user attempted to run a repair, the following exception was seen in the /var/log/cassandra/system.log

 INFO [AntiEntropyStage:1] 2015-06-05 07:13:34,981 RepairJob.java (line 161) [repair #610799a0-0b52-11e5-b97e-b7a7c52693de] requesting merkle trees for mytable (to [/10.50.3.144, /10.50.3.143, ip-10-
50-4-241.internal/10.50.4.241])
...
ERROR [ValidationExecutor:48] 2015-06-05 07:13:34,990 Validator.java (line 242) Failed creating a merkle tree for [repair #610799a0-0b52-11e5-b97e-b7a7c52693de on myks/mytable, (-1282988750579997629,-1185701225543990938]], /10.50.4.241 (see log for details)
ERROR [ValidationExecutor:48] 2015-06-05 07:13:34,990 CassandraDaemon.java (line 199) Exception in thread Thread[ValidationExecutor:48,1,main]
java.lang.RuntimeException: Cannot get comparator 1 in org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type). This might due to a mismatch between the schema and the data read
        at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:133)
        at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:140)
        at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:59)
        at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:36)
        at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:112)
        at org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:116)
        at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:150)
        at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:186)
        at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:98)
        at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:85)
        at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:196)
        at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:74)
        at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:55)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:115)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:800)
        at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
        at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
        at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:306)
        at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:285)
        at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45)
        at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:124)
        ... 23 more

Cause

It appears that some corruption had occurred to an sstable. It was originally reported in the following Cassandra jira:

https://issues.apache.org/jira/browse/CASSANDRA-8120

Solution

There are two tools: "nodetool scrub" and "sstablescrub", it is best to run the former first. If that is not successful then run the latter.

Run a nodetool scrub on the on the table:

http://docs.datastax.com/en/cassandra/2.0/cassandra/tools/toolsScrub.html

If nodetool scrub does not finish successfully then use the offline sstablescrub:

http://docs.datastax.com/en/cassandra/2.0/cassandra/tools/toolsSSTableScrub_t.html

Note: sstablescrub should be run with DSE offline (i.e. the DSE service is not running on the node where you are running sstable scrub)