Use the attached multidump.sh script to take thread dumps and generate top -H output at regular intervals while your nodes are experiencing high CPU utilization. The multidump.sh script must be run as the same user that owns the DSE java process; otherwise, the jstack command will not work. The usage is:
$ multidump.sh pid interval count
NOTE: I would like to reiterate that you must run this as the same user that owns the dse process. For example, if the process owner is cassandra, you can run the script using the following syntax:
$ sudo -u cassandra multidump.sh interval count
How do you find the DSE process ID (pid)?
If DSE is running as a service, use the pid from /var/run/dse/dse.pid.
If DSE is started from the command line, use 'ps -ef | grep java' to determine the pid.
The recommended parameters to use with multidump.sh are an interval of 5 seconds and a count of 60, which would run for a total of 5 minutes. This will generate two files, jstack.out and top.out, which you should send to DataStax Support.
If you have difficulties using the multidump.sh script where jstack is not installed:
./multidump.sh: line 15: jstack: command not found
you can use the multidump-alt.sh script instead. The usage is the same, but it uses kill -3 to take the thread dumps instead of jstack. This will not kill the process; the JVM interprets the -3 signal as a request to print a thread dump. In order for the thread dump to be logged to a file, you will need to follow these steps:
1. Add these lines to /etc/dse/cassandra/jvm.options file:
-XX:+UnlockDiagnosticVMOptions
-XX:+LogVMOutput
-XX:LogFile=/var/log/cassandra/jvm.log
Note: LogFile can be set to any file, the file does not have to be created in advance.
2. Restart DSE.
3. Run multidump-alt.sh and find the thread dump output in the file you specified in LogFile.
After capturing the thread dump, please send DataStax Support the LogFile & top.out files.
Comments
9 comments
To save you some time, If you're using a modern version of java you probably won't have the jstack utility (as it appears to be depricated http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstack.html) and will have to use the multidump-alt.sh script.
jstack is only included in the JDK, not the JRE. It is still available as of JDK 7, which is the version that Cassandra currently supports.
I stand corrected, thanks J.B.
Attached a version of the script which merges the original scripts
Usage: ./multidump.sh -i <interval> -c <count> -pid <PID> -pgm [jstack|kill]
Default interval: 5 secs
Default count : 60
Default PID : DSE java PID
Default PGM : jstack
I have changed the code of the script in order to get the information we need
Attached is a newer version of the script. It needs the following in order to run
On /etc/password make sure the cassandra user is set to use /bin/bash rather than /bin/false
We need to use Oracle jstack, it will use the command resulting from this search:
Become root and then cassandra, then execute:
A succesful execution would look like this
We've also run in to several instances of this error:
Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
The following fixed it (along with editing multidump.sh to include):
sudo -u cassandra jstack -J-d64 -l <PID> >> jstack.out
Where does the thread dump output go when using multidump-alt.sh?
The multidump-alt.sh script uses kill -3 to generate thread dumps. This output was expected to be in /var/log/cassandra/output.log but is not seen there. Follow these steps to capture the thread dumps using multidump-alt.sh (kill -3):
1. Add these lines to /etc/dse/cassandra/jvm.options file:
-XX:+UnlockDiagnosticVMOptions
-XX:+LogVMOutput
-XX:LogFile=/var/log/cassandra/jvm.log
Note: LogFile can be set to any file, the file does not have to be created in advance.
2. Restart DSE.
3. Run multidump-alt.sh and find the thread dump output in the file you specified in LogFile.
Some discussion on using forced mode (-f), in cases where the jvm doesn't respond to jstack:
https://stackoverflow.com/questions/26140182/running-jmap-getting-unable-to-open-socket-file
Easy way to get DSE PID so you don't have to sift through ps:
cat /var/run/dse/dse.pid
Please sign in to leave a comment.