DataStax Help Center

System logs full of "Spinning trying to capture all readers" with compaction backing up

Summary

In releases of DSE from 4.7.x and 4.8.x there is a potential of a problem where sstables are not unmarked compacting once they are done with when using cleanup/scrub/upgradesstables. This can cause backed up compactions on a node and lead to performance problems.

Symptoms

The following messages is typical of what might be seen in the logs:

INFO  [Thrift:5] 2016-03-15 14:39:03,277  ColumnFamilyStore.java:1924 - Spinning trying to capture all readers...


It may also be observed that that nodetool tpstats shows an increasing number in ‘pending compactions’ on the node, which might also be accompanied by a growth in the number of sstables in the data directory.

Cause

This is a known issue covered in CASSANDRA-10829 

Workaround

It might be possible to restart the node to delay the problem. The logs above can be monitored to check for the presence of this error. The sooner is it caught and restarted the less backlog of compactions there will be.

If the node seems to never be able to catch up with compactions then it might be necessary to re-bootstrap the node but this should be a last resort.

Solution

 

At the time of writing this article an Upgrade to DSE4.7.8 or DSE4.8.5 is required.

 

 

Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk