Customers using JBOD with SizeTieredCompaction may find that certain disks in the pool of disks fill up quicker than others. This causes two problems.
- The disk usage is unbalanced.
- Compactions fail due to a lack of diskspace.
Measuring disk consumption, one can see that 1 or 2 disks will fill to 90% or above while other disks are underutilized.
Running a df on the disks can demonstrate this.
The logic behind multiple data directories specified in the Cassandra.yaml. currently picks the disks first by number of current tasks, then by free space. This can lead to an imbalance amongst the disks. There are two Cassandra Jiras regarding this problem.
- https://issues.apache.org/jira/browse/CASSANDRA-8329 (Fixed; 2.0.12): "LeveledCompactionStrategy should split large files across data directories when compacting"
- https://issues.apache.org/jira/browse/CASSANDRA-7386 (Fixed; 2.1.3): "JBOD threshold to prevent unbalanced disk utilization"
There are a few suggestions for working with this.
1) If you are a candidate for using leveled compaction strategy, moving to LCS could provide you with the a solution. WARNING: Moving to LCS just to remedy this situation is not recommended. Read HERE for specifics about LCS strategy and when to use it.
2) Manually move the sstable files.
- Identify the sstable sets. Remember you will want to move <keyspace>-<table>-<generation>-<SEQUENCE>-<type>.db. Its the sstable set in a sequence that you want to move.
- Identify the size of the files. Doing something like 'ls -lR' for anything ending in 'Data.db'
- Sort that by size.
- At this point you might want to base the inquiry on something larger than a specific size. For instance anything larger than 1GB.
- Copy this information along with the drive from where it came into a spreadsheet.
Put the sizes of all files from each drive into a different column for each drive and then total the columns.
- From there you can move the the sizes back and forth from different columns until the totals are roughly equal.
- Note the original location and new location of the files and create a list of mv commands.
- Shut down the node and move the files to the new disks.
Note: Try to settle on a configuration that would allow minimal number of moves.
The jiras for this have been addressed in Cassandra 2.0.12 and 2.1.3.