Summary

A direct buffer out of memory condition can potentially occur under certain circumstances due to the way the JVM may use the temporary buffer cache of the java nio package

Applies to

DSE 5.0, DSE 5.1.0-5.1.9, DSE6.0.0

Symptoms

The DSE process will unexpectedly stop with the following error in the /var/log/cassandra/system.log

ERROR [CompactionExecutor:2] 2018-04-30 09:31:36,251  CassandraDaemon.java:207 - Exception in thread Thread[CompactionExecutor:2,1,main]
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_171]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.8.0_171]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_171]
        at org.apache.cassandra.utils.memory.BufferPool.allocate(BufferPool.java:108) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.utils.memory.BufferPool.access$1000(BufferPool.java:45) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.utils.memory.BufferPool$LocalPool.allocate(BufferPool.java:387) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.utils.memory.BufferPool$LocalPool.access$000(BufferPool.java:314) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:120) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:92) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.io.util.RandomAccessReader.allocateBuffer(RandomAccessReader.java:87) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.access$100(CompressedRandomAccessReader.java:38) ~[cassandra-all-3.0.13.1735.jar:5.0.9]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader$Builder.createBuffer(CompressedRandomAccessReader.java:275) ~[cassandra-all-3.0.13.1735.jar:5.0.9]
        at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:74) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:59) ~[cassandra-all-3.0.13.1735.jar:5.0.9]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader$Builder.build(CompressedRandomAccessReader.java:283) ~[cassandra-all-3.0.13.1735.jar:5.0.9]
        at org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:145) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.io.util.SegmentedFile.createReader(SegmentedFile.java:133) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1739) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(AbstractSSTableIterator.java:94) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:51) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:41) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:69) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:584) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:497) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:359) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:395) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:93) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:153) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at org.apache.cassandra.db.compaction.CompactionManager$13.run(CompactionManager.java:1478) ~[cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_171]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [cassandra-all-3.0.13.1735.jar:3.0.13.1735]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]

Cause

The java nio package uses a temporary cache for off heap byte buffers. In java versions prior to 1.8.102 and 1.9.104 there was no limit on this cache. The following bug outlines the issue for java 9 (see backport bugs for earlier releases)
https://bugs.openjdk.java.net/browse/JDK-8147468

Workaround

Upgrade java to 1.8.102 or later and then apply the following setting into your cassandra-env.sh or your jvm.optionsfile

-Djdk.nio.maxCachedBufferSize=1048576

Internal jira DB-2028 covers this issue

Solution

Upgrade to DSE 5.1.10, 6.0.1 or later where this setting will be default