A schema update results in compaction failures with assertion errors.
Compaction threads fail with the following sample exception stack from DataStax Enterprise 4.6.5 (Cassandra 2.0.14):
ERROR [CompactionExecutor:6488] 2015-07-14 19:33:14,247 CassandraDaemon.java (line 258) Exception in thread Thread[CompactionExecutor:6488,1,main] java.lang.AssertionError: Added column does not sort as the last column at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:116) at org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:121) at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:155) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:186) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:98) at org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:85) ...
This issue is caused by changing the clustering order on a table where data already exists. In the example above, the original schema defined the clustering column in the default ascending order. At some point after data was inserted into the table, the table schema was altered such that the clustering column has been modified to be in descending order.
NOTE - The reverse also applies, i.e. clustering column sorted in descending order but later altered to ascending order.
Since there is already data in the table, the compaction fails since the row is sorted in the wrong order and cannot be added. For the same reason, existing data cannot be deserialised.
There is no real solution to this problem. Depending on the size of the data which needs to be corrected, it may be possible to extract the contents of the affected sstables (e.g. with
sstable2json), then re-insert the data into Cassandra.
DataStax doc -
ALTER TABLE command
DataStax doc -