When to Take Heap Dumps
Heap dumps can be very useful when troubleshooting high memory utilization or OutOfMemoryErrors since they show exactly which objects are consuming most of the heap.
The best time to do this would be while you are getting messages from GCInspector.java about the heap being full, like the following:
WARN [ScheduledTasks:1] 2012-10-22 12:14:49,889 GCInspector.java (line 145) Heap is 0.9941227313009479 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
Automatic Heap Dumps
Cassandra is configured to trigger a heap dump at the occurrence of an OutOfMemoryError, but unless you have set your CASSANDRA_HEAPDUMP_DIR in cassandra-env.sh, the file will be written in the Cassandra process's working directory, or may not be written at all due to permission and/or space issues. When DSE is started as a service on packaged installs, the working directory is typically the root directory (/).
Manual Heap Dumps Using JMX
Sometimes jmap may not be able to successfully take a heap dump. It may create an invalid heap dump file, or throw errors like this:
Unable to open socket file: target process not responding or HotSpot VM not loaded The -F option can be used when the target process is not responding
In this case, you can trigger a heap dump by using JMX to invoke the dumpHeap method on the com.sun.management:type=HotSpotDiagnostic MBean.
If you don't already have a favorite JMX client, you can download jmxsh from http://code.google.com/p/jmxsh/. Copy the jar to your node where you want to take the heap dump.
To take the heap dump using jmxsh, enter the following command:
echo 'jmx_invoke -m com.sun.management:type=HotSpotDiagnostic dumpHeap /path/to/heapdump.hprof false' | java -jar jmxsh-R5.jar -h localhost -p 7199
Change /path/to/heapdump.hprof to the location where you want to save the heap dump. The cassandra process must have permission to write to this location, and the file must not already exist. If you have changed your RMI port from the default of 7199, change the port in the command accordingly.
Manual Heap Dumps Using jmap
You can use the following command to manually take a heap dump:
sudo -u <user> jmap -dump:file=heapdump.hprof,format=b <pid>
Replace <pid> with the process ID of Cassandra.
Replace <user> with the process owner. For standard installs this will usually be cassandra.
Using the correct user is very important; otherwise, jmap will either fail or produce an invalid heap dump.
Analyzing Heap Dumps
The recommended tool for analyzing heap dumps is Eclipse Memory Analyzer, available here: http://www.eclipse.org/mat/downloads.php. It is available as a stand-alone application or as a plugin for Eclipse. Analyzing a large heap dump will require a lot of memory, and you will probably also need to increase the Xmx in MemoryAnalyzer.ini. For a large heap dump, the initial analysis phase can take a long time. After it finishes analyzing the heap dump, you can view the dominator tree to see which objects are retaining the most heap.
If you need help understanding the heap dump, feel free to open a ticket and we'll take a look.