Full ERROR Message Example

INFO  [main] 2020-07-28 07:26:46,093  ColumnFamilyStore.java:551 - Removing SSTables from failed streaming session. Found 3 files to cleanup.
ERROR [main] 2020-07-28 07:51:06,737  SSTable.java:261 - Missing component: /var/lib/cassandra/data/keyspace/table-2f9f7620998111e7b91a43fd28aa150e/keyspace-table-ka-63717-Summary.db
...

What does this ERROR message mean?

One of the files associated to an sstable is missing on the file system

Why does this ERROR occur?

This issue can be resulting from a disk corruption or an instable JVM that didnt allow for the files associated to a flushing sstable to be written in time.

How do you fix this ERROR?

• The data associated to the missing file cannot be rebuilt and this content should be isolated.
• Identify from the /var/log/cassandra/system.log or /var/log/cassandra/output.log the problematic files associated to an sstable. Based on ther example error message above, under /var/lib/cassandra/data/keyspace/table-2f9f7620998111e7b91a43fd28aa150e/, the files would be named in a similar patter to the examples below (exact naming pattern will vary):
  for C* 2.0 and DSE 4.8: keyspace-table-ka-63717-*
  for C* 3.x and DSE 5.x: mc-63717-*

• Stop the node where the problem was noted
• Move the various files associated to the problematic sstable outside the /var/lib/cassandra/data. ie:
  for C* 2.0 and DSE 4.8:

  mv /var/lib/cassandra/data/keyspace/table-2f9f7620998111e7b91a43fd28aa150e/keyspace-table-ka-63717-* /tmp

  for C* 3.x and DSE 5.x:

  mv /var/lib/cassandra/data/keyspace/table-2f9f7620998111e7b91a43fd28aa150e/mc-63717-* /tmp

• Start the node
• Run a repair for the problematic keyspace/table. ie:

  nodetool repair -full \-\- keyspace table