Nodes become unresponsive during high app traffic periods when mmap is enabled on DSE 6

Summary

This article discusses an issue where incorrectly configured DSE 6 nodes cause performance issues during high traffic periods.

Applies to

• DataStax Enterprise 6.0 or later

Symptoms

Nodes operate as normal but as the application traffic builds up, the read performance degrades exponentially and nodes either:

• run out of memory
• become unresponsive
• or both

In some situations, the Linux oom-killer terminates the DSE process when the server has run out of resources.

Cause

Apache Cassandra uses memory-mapped file I/O through the Unix system call mmap() (or mmap for short). This system call allows Cassandra to use the operating system virtual memory to hold copies of data files to speed up reading SSTables. A hidden cassandra.yaml property called disk_access_mode determines how data files are accessed. The valid options for disk_access_mode are:

• auto (default) - both SSTable data and index files are mapped on 64-bit systems; only index files are mapped for 32-bit systems
• mmap - both data and index files are mapped to memory
• mmap_index_only - only index files are mapped to memory
• standard - standard IO; no files are mapped to memory

There was a known issue with the default auto setting when clusters were upgraded to DSE 5.0 or 5.1. The recommendation was to set [disk_access_mode: mmap_index_only] on DSE 5.0 and DSE 5.1 clusters.

The thread-per-core (TPC) design and asynchronous architecture in DSE 6 with mmap enabled does not perform well during high traffic periods particularly when the "hot data" (set of partitions being heavily read) is larger than the available [off-heap] memory.

When large amounts of SSTables constantly get mapped to memory, the operating system frequently raises page faults which in turn cause TPC threads to get blocked. The nodes will eventually reach a tipping point where enough page faults will block a sufficient proportion of TPC threads that the nodes appear unresponsive to the application.

Solution

DataStax does not recommend setting disk_access_mode to mmap or mmap_index_only. There are very limited cases where mapping files to memory in DSE 6 is valid, namely:

• all application tables use MemoryOnlyStrategy compaction strategy (DSE In-memory)
• servers use non-volatile memory (NVM) storage

If mmap is enabled, the following entries are logged on DSE startup:

INFO [main] 2019-05-13 13:18:02,418 DatabaseDescriptor.java:425 - \
  DiskAccessMode is mmap, indexAccessMode is mmap, commitlogAccessMode is mmap
INFO [main] 2019-05-13 13:18:02,418 DatabaseDescriptor.java:428 - Memory-mapped access mode is selected. \
  Because page faults block and thus cause thread-per-core architectures to be inefficient, \
  this should only be used if all data is either locked to memory or \
  if the underlying storage is non-volatile memory.

See also

KB article - Increased memory utilisation on nodes after upgrading to DSE 5.0 or 5.1