Can I delete sstables from a node and will it cause any problems?

Summary

Is it ok to delete old sstables if you don't have any more disc space or any room for compaction?

Question

If we are running low on disc space on a number of our nodes, don't have enough space for compaction and don't have the facility in the short term to add more nodes or add more disc space. Could we delete some old sstables? Will this cause any problems?

Answer

Assuming you have exhausted all options of:

• Deleting data and running compaction
• Deleting any old snapshot data
• Adding more disc space
• Adding more nodes into the cluster

Then the last resort would be to remove older sstables. 

If you have decided to remove these files then the most obvious side effect is missing data; subsequent reads coming into the nodes will not find any sstables and thus potentially data loss could occur. However it is assumed that you would have gotten to this point by exhausting all other options as above. The following procedure is used for removing sstable files. Note this will incur irreversible data loss:

On each node in the cluster:

1. modify read repair chance on the concerned keyspaces to 0

2. Cancel any pending repair operations or wait until completed.

3. Ensure the repair service in OpsCenter (if used) is turned off.

4. Stop the node

5. Move the selected sstable file off the node

6. Start the node

7. Check logs for errors and make sure the node has joined the cluster ok, then move to the next node

Always consult support first. This procedure is a last resort action and you are entirely responsible for any actions carried out against your cluster.