This article discusses an issue where incorrectly configured DSE 6 nodes cause performance issues during high traffic periods.
- DataStax Enterprise 6.0 or later
Nodes operate as normal but as the application traffic builds up, the read performance degrades exponentially and nodes either:
- run out of memory
- become unresponsive
- or both
In some situations, the Linux
oom-killer terminates the DSE process when the server has run out of resources.
Apache Cassandra uses memory-mapped file I/O through the Unix system call
mmap for short). This system call allows Cassandra to use the operating system virtual memory to hold copies of data files to speed up reading SSTables. A hidden
cassandra.yaml property called
disk_access_mode determines how data files are accessed. The valid options for
auto(default) - both SSTable data and index files are mapped on 64-bit systems; only index files are mapped for 32-bit systems
mmap- both data and index files are mapped to memory
mmap_index_only- only index files are mapped to memory
standard- standard IO; no files are mapped to memory
There was a known issue with the default
auto setting when clusters were upgraded to DSE 5.0 or 5.1. The recommendation was to set [
disk_access_mode: mmap_index_only] on DSE 5.0 and DSE 5.1 clusters.
The thread-per-core (TPC) design and asynchronous architecture in DSE 6 with
mmap enabled does not perform well during high traffic periods particularly when the "hot data" (set of partitions being heavily read) is larger than the available [off-heap] memory.
When large amounts of SSTables constantly get mapped to memory, the operating system frequently raises page faults which in turn cause TPC threads to get blocked. The nodes will eventually reach a tipping point where enough page faults will block a sufficient proportion of TPC threads that the nodes appear unresponsive to the application.
DataStax does not recommend setting
mmap_index_only. There are very limited cases where mapping files to memory in DSE 6 is valid, namely:
- all application tables use
MemoryOnlyStrategycompaction strategy (DSE In-memory)
- servers use non-volatile memory (NVM) storage
mmap is enabled, the following entries are logged on DSE startup:
INFO [main] 2019-05-13 13:18:02,418 DatabaseDescriptor.java:425 - \ DiskAccessMode is mmap, indexAccessMode is mmap, commitlogAccessMode is mmap INFO [main] 2019-05-13 13:18:02,418 DatabaseDescriptor.java:428 - Memory-mapped access mode is selected. \ Because page faults block and thus cause thread-per-core architectures to be inefficient, \ this should only be used if all data is either locked to memory or \ if the underlying storage is non-volatile memory.