Summary

A direct buffer out of memory condition can potentially occur under certain circumstances due to the way the JVM uses temporary buffer cache of the java nio package

Applies to

• DSE 5.0
• DSE 5.1.0-5.1.9
• DSE 6.0

Symptoms

The DSE process will unexpectedly stop with the following error in the /var/log/cassandra/system.log

ERROR [CompactionExecutor:2] 2018-04-30 09:31:36,251  CassandraDaemon.java:207 - Exception in thread Thread[CompactionExecutor:2,1,main]
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_171]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.8.0_171]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_171]
        at org.apache.cassandra.utils.memory.BufferPool.allocate(BufferPool.java:108) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.utils.memory.BufferPool.access$1000(BufferPool.java:45) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.utils.memory.BufferPool$LocalPool.allocate(BufferPool.java:387) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.utils.memory.BufferPool$LocalPool.access$000(BufferPool.java:314) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:120) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:92) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.io.util.RandomAccessReader.allocateBuffer(RandomAccessReader.java:87) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.access$100(CompressedRandomAccessReader.java:38) ~[cassandra-all-3.0.13.1735.jar:5.0.9]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader$Builder.createBuffer(CompressedRandomAccessReader.java:275) ~[cassandra-all-3.0.13.1735.jar:5.0.9]
        at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:74) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:59) ~[cassandra-all-3.0.13.1735.jar:5.0.9]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader$Builder.build(CompressedRandomAccessReader.java:283) ~[cassandra-all-3.0.13.1735.jar:5.0.9]
        at org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:145) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.io.util.SegmentedFile.createReader(SegmentedFile.java:133) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1739) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(AbstractSSTableIterator.java:94) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:51) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:41) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:69) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:584) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:497) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:359) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:395) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:93) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:153) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.compaction.CompactionManager$13.run(CompactionManager.java:1478) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_171]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]

Cause

The java nio package uses a temporary cache for off heap byte buffers. In Java versions earlier than 1.8.102 and 1.9.104, there was no limit on this cache. TheJDK-8147468 bug outlines the issue for Java 9 (see backport bugs for earlier releases).

Workaround

Upgrade Java to 1.8.102 or later, and then apply the following setting into your cassandra-env.sh or your jvm.options file:

-Djdk.nio.maxCachedBufferSize=1048576

Internal DataStax Jira DB-2028 covers this issue.

Solution

Upgrade to DSE 5.1.10, DSE 6.0.1, or later where this max cached direct buffer limit on NIO is  correctly set to 1 MB by default. See Installing supporting software.