Use the attached multidump.sh script to take thread dumps and generate top -H output at regular intervals while your nodes are experiencing high CPU utilization. The multidump.sh script must be run as the same user that owns the DSE java process; otherwise, the jstack command will not work. The usage is:
$ multidump.sh pid interval count
NOTE: I would like to reiterate that you must run this as the same user that owns the dse process. For example, if the process owner is cassandra, you can run the script using the following syntax:
$ sudo -u cassandra multidump.sh interval count
How do you find the DSE process ID (pid)?
If DSE is running as a service, use the pid from /var/run/dse/dse.pid.
If DSE is started from the command line, use 'ps -ef | grep java' to determine the pid.
The recommended parameters to use with multidump.sh are an interval of 5 seconds and a count of 60, which would run for a total of 5 minutes. This will generate two files, jstack.out and top.out, which you should send to DataStax Support.
If you have difficulties using the multidump.sh script where jstack is not installed:
./multidump.sh: line 15: jstack: command not found
you can use the multidump-alt.sh script instead. The usage is the same, but it uses kill -3 to take the thread dumps instead of jstack. This will not kill the process; the JVM interprets the -3 signal as a request to print a thread dump. In order for the thread dump to be logged to a file, you will need to follow these steps:
1. Add these lines to /etc/dse/cassandra/jvm.options file:
-XX:+UnlockDiagnosticVMOptions
-XX:+LogVMOutput
-XX:LogFile=/var/log/cassandra/jvm.log
Note: LogFile can be set to any file, the file does not have to be created in advance.
2. Restart DSE.
3. Run multidump-alt.sh and find the thread dump output in the file you specified in LogFile.
After capturing the thread dump, please send DataStax Support the LogFile & top.out files.