Is it ok to delete old sstables if you don't have any more disc space or any room for compaction?
If we are running low on disc space on a number of our nodes, don't have enough space for compaction and don't have the facility in the short term to add more nodes or add more disc space. Could we delete some old sstables? Will this cause any problems?
Assuming you have exhausted all options of:
- Deleting data and running compaction
- Deleting any old snapshot data
- Adding more disc space
- Adding more nodes into the cluster
Then the last resort would be to remove older sstables.
If you have decided to remove these files then the most obvious side effect is missing data; subsequent reads coming into the nodes will not find any sstables and thus potentially data loss could occur. However it is assumed that you would have gotten to this point by exhausting all other options as above. The following procedure is used for removing sstable files. Note this will incur irreversible data loss:
On each node in the cluster:
1. modify read repair chance on the concerned keyspaces to 0
2. Cancel any pending repair operations or wait until completed.
3. Ensure the repair service in OpsCenter (if used) is turned off.
4. Stop the node
5. Move the selected sstable file off the node
6. Start the node
7. Check logs for errors and make sure the node has joined the cluster ok, then move to the next node
Always consult support first. This procedure is a last resort action and you are entirely responsible for any actions carried out against your cluster.