Nodes appear unresponsive due to a Linux futex_wait() kernel bug

Summary

Some customers have reported nodes randomly freeze and become unresponsive for an unknown reason.

Symptoms

Unresponsive nodes have the following characteristics:

no garbage collection activity in the logs
no compactions in progress
unable to run nodetool commands
no response on native transport, Thrift or JMX ports
low or close to zero CPU utilisation
high CPU utilisation which eventually leads to node being unresponsive

In some instances where a customer is able to generate a thread dump, the jstack output is dominated by BLOCKED threads, for example:

Thread 104823: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1068 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Cause

The problem is due to a Linux futex_wait() bug which causes user processes to deadlock and hang. A futex_wait() call (and any processes making a call) can stay blocked forever. JVM synchronization method calls such as lock(), park() and unpark() all make futex_wait() calls at some point and fall victim to the bug.

The bug exists in RHEL 6.6, CentOS 6.6 and related Linux distributions. At the time of writing, RHEL 7.x/CentOS 7.x were also affected.

Earlier kernels in RHEL 6.5, CentOS 6.5 (and related distributions) are not affected by the bug.

Solution

Upgrade to Linux kernels containing the get_futex_key_refs() fix such as RHEL 6.6.z and CentOS 6.6.z.

Here is an example to check on a RHEL server for the installed patches:

$ sudo rpm -q --changelog kernel-`uname -r` | grep futex | grep ref
- [kernel] futex: Mention key referencing differences between shared and private futexes (Larry Woodman) [1167405]
- [kernel] futex: Ensure get_futex_key_refs() always implies a barrier (Larry Woodman) [1167405]

For further information on distributions which contain the fix, please consult the relevant vendor or distributor of the operating system.

Summary

Symptoms

Cause

Solution

See also

Comments