Summary
This article discusses an issue where incorrectly configured DSE 6 nodes cause performance issues during high traffic periods.
Applies to
- DataStax Enterprise 6.0 or later
Symptoms
Nodes operate as normal but as the application traffic builds up, the read performance degrades exponentially and nodes either:
- run out of memory
- become unresponsive
- or both
In some situations, the Linux oom-killer
terminates the DSE process when the server has run out of resources.
Cause
Apache Cassandra uses memory-mapped file I/O through the Unix system call mmap()
(or mmap
for short). This system call allows Cassandra to use the operating system virtual memory to hold copies of data files to speed up reading SSTables. A hidden cassandra.yaml
property called disk_access_mode
determines how data files are accessed. The valid options for disk_access_mode
are:
auto
(default) - both SSTable data and index files are mapped on 64-bit systems; only index files are mapped for 32-bit systemsmmap
- both data and index files are mapped to memorymmap_index_only
- only index files are mapped to memorystandard
- standard IO; no files are mapped to memory
There was a known issue with the default auto
setting when clusters were upgraded to DSE 5.0 or 5.1. The recommendation was to set [disk_access_mode: mmap_index_only
] on DSE 5.0 and DSE 5.1 clusters.
The thread-per-core (TPC) design and asynchronous architecture in DSE 6 with mmap
enabled does not perform well during high traffic periods particularly when the "hot data" (set of partitions being heavily read) is larger than the available [off-heap] memory.
When large amounts of SSTables constantly get mapped to memory, the operating system frequently raises page faults which in turn cause TPC threads to get blocked. The nodes will eventually reach a tipping point where enough page faults will block a sufficient proportion of TPC threads that the nodes appear unresponsive to the application.
Solution
DataStax does not recommend setting disk_access_mode
to mmap
or mmap_index_only
. There are very limited cases where mapping files to memory in DSE 6 is valid, namely:
- all application tables use
MemoryOnlyStrategy
compaction strategy (DSE In-memory) - servers use non-volatile memory (NVM) storage
If mmap
is enabled, the following entries are logged on DSE startup:
INFO [main] 2019-05-13 13:18:02,418 DatabaseDescriptor.java:425 - \ DiskAccessMode is mmap, indexAccessMode is mmap, commitlogAccessMode is mmap INFO [main] 2019-05-13 13:18:02,418 DatabaseDescriptor.java:428 - Memory-mapped access mode is selected. \ Because page faults block and thus cause thread-per-core architectures to be inefficient, \ this should only be used if all data is either locked to memory or \ if the underlying storage is non-volatile memory.
See also
KB article - Increased memory utilisation on nodes after upgrading to DSE 5.0 or 5.1