Summary
Large GC pauses were observed in the Apache Cassandra log file, which created a lot of unnecessary "noise" in the log files.
Symptoms
The /var/log/cassandra/system.log
was showing very high pauses like so:
INFO [Service Thread] 2016-04-20 22:53:00,273 GCInspector.java:258 - ParNew GC in 24898ms. CMS Old Gen: 1233265384 -> 1233265632; Par Eden Space: 671088640 -> 0; Par Survivor Space: 6228992 -> 27236208
However when checking the JVM GC logs the corresponding time for pause is nowhere near the same. The node is also responding normally, and not showing as down in gossip or in any output from nodetool status
Cause
The running JVM version at the time was 1.7.0.25
which was quite outdated.
Solution
Upgrading to the latest version of java available for the currently installed branch solved the issue. It is normally recommended to keep java patched to the latest available release for updates and security fixes.
For more information on java downloads and updates, please refer to the following link:
http://www.java.com/en/download/help/