Summary
After an upgrade, UDT errors occurred on multiple tables.
Applies to
- Upgrades to DSE 5.1, including DSE 5.1.14
Error
WARN [ReadStage-1] 2019-04-24 11:40:18,795 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-1,5,main]: Found 2 incompatible versions of column inappropriate_feedbacks in foo.bar: one of type org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.UserType(foo,696e617070726f7072696174655f666565646261636a,617574686f725f66696e6765727072696e75:org.apache.cassandra.db.marshal.UTF8Type,7375626d697373696f6e5f7473:org.apache.cassandra.db.marshal.TimestampType,726561736f6e5f74657874:org.apache.cassandra.db.marshal.UTF8Type))) and one of type org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UserType(foo,696e617070726f7072696174655f666565646261636a,617574686f725f66696e6765727072696e75:org.apache.cassandra.db.marshal.UTF8Type,7375626d697373696f6e5f7473:org.apache.cassandra.db.marshal.TimestampType,726561736f6e5f74657874:org.apache.cassandra.db.marshal.UTF8Type)) (but both types are incompatible)
Solution
To resolve the errors:
1. Stop repairs.
2. Run offline sstablescrub against all tables flagged as having UDT errors in system logs:
sstablescrub -e fix-only [keyspace] [table]
3. Start repairs
If repairs fail:
1. Note where they failed, which keyspace and table.
2. Run sstablescrub against that table:
sstablescrub [keyspace] [table]
It should fail with header error, so you will need to run:
sstablescrub -e fix-only [keyspace] [table]
Start your repairs and keep repeating the "If repairs fail" step if the repairs fail again.