This article discusses an issue where nodes experience unusual performance issues after upgrading a cluster from DSE 4.8 to DSE 5.0 or DSE 5.1.
- DataStax Enterprise 5.1
- DataStax Enterprise 5.0
Customers have reported performance issues not observed when clusters were running with DSE 4.8. Shortly after upgrading to DSE 5.0 or 5.1, nodes experience issues despite factors such as application load, cluster traffic, access patterns, time of day/week/month, and hardware/network resources being equal.
- Significant increase in off-heap memory usage (for example, when monitored using Linux utilities such as
- Increased P99 and/or maximum read latencies (like from
- Intermittent read request timeouts in the extreme.
In a rare case, DSE being started on a node fails to finish the startup sequence because most of the memory gets consumed very quickly and is eventually exhausted so the Linux
oom-killer terminates the DSE process.
Apache Cassandra uses memory-mapped file I/O through the Unix system call
mmap for short). This mmap system call allows Cassandra to use the operating system's virtual memory to hold copies of data files so reading SSTables is fast. A hidden
cassandra.yaml property called
disk_access_mode determines how data files are accessed. The valid options are:
auto(default) - both SSTable data and index files are mapped on 64-bit systems; only index files are mapped for 32-bit systems
mmap- both data and index files are mapped to memory
mmap_index_only- only index files are mapped to memory
standard- Cassandra uses standard IO and no files are mapped to memory
In versions of Cassandra 2.1 or earlier (DataStax 4.8 or earlier), reading compressed SSTables involved the data being copied on-heap then sent to an on-heap buffer to be decompressed and behaved as if disk access mode was set to
mmap_index_only despite the default mode being
With the added support for direct buffer decompression in Cassandra 2.2 (CASSANDRA-8464), the behaviour for disk access mode changed to the way it was designed, i.e. default
auto mode on 64-bit systems now
mmap() both SSTable data and index files. Note that DSE 5.0 includes Cassandra 3.0 and DSE 5.1 includes Cassandra 3.11, see DataStax Enterprise, Apache Cassandra, CQL, and SSTable compatibility.
In cases where there are lots of random reads, and the set of SSTables being heavily read is larger than the available memory, the affected nodes will have a high number of page faults. In some cases, the affected servers run out of memory and the Linux
oom-killer terminates DSE.
Since CASSANDRA-8464 allows mapping compressed data directly, it is more efficient to map only index files.
With the default
disk_access_mode: auto during startup, DSE logs an entry similar to below:
INFO [main] 2019-05-02 12:33:21,572 DatabaseDescriptor.java:350 - \ DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
After restarting DSE, an entry in the logs will be similar to:
INFO [main] 2019-05-02 17:53:50,437 DatabaseDescriptor.java:356 - \ DiskAccessMode is standard, indexAccessMode is mmap
This log entry indicates that SSTable data files are used with standard disk IO but index files will be mapped to memory.
WARNING - Enabling
mmap in DSE 6 has a detrimental effect to a cluster's performance. See the article below for details.
- KB article - Nodes become unresponsive during high app traffic periods when mmap is enabled on DSE 6
- DataStax Enterprise documentation for Cassandra.yaml
- Branimir Lambov
- Jake Luciani
- Mark Curtis
- Michael Keeney
- Stefania Alborghetti
- Thanh Tranh
- Wei Deng