Use the attached multidump.sh script to take thread dumps and generate top -H output at regular intervals while your nodes are experiencing high CPU utilization. The multidump.sh script must be run as the same user that owns the DSE java process; otherwise, the jstack command will not work. The usage is:
$ multidump.sh pid interval count
If DSE is running as a service, use the pid from /var/run/dse.pid. If DSE is started from the command line, use 'ps -ef | grep java' to determine the pid. The recommended parameters are an interval of 5 seconds and a count of 60, which would run for a total of 5 minutes. This will generate two files, jstack.out and top.out, which you should send to us.
If you have difficulties using the multidump script, you can use the multidump-alt script instead. The usage is the same, but it uses kill -3 to take the thread dumps instead of jstack. This will not kill the process; the JVM interprets the -3 signal as a request to print a thread dump on stdout. When DSE is run as a service, stdout is redirected to /var/log/cassandra/output.log file, so send us that as well as top.out. If DSE is run as a standalone process, find out where its output is redirected and capture the thread dumps from there.
Comments
9 comments
To save you some time, If you're using a modern version of java you probably won't have the jstack utility (as it appears to be depricated http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstack.html) and will have to use the multidump-alt.sh script.
jstack is only included in the JDK, not the JRE. It is still available as of JDK 7, which is the version that Cassandra currently supports.
I stand corrected, thanks J.B.
Attached a version of the script which merges the original scripts
Usage: ./multidump.sh -i <interval> -c <count> -pid <PID> -pgm [jstack|kill]
Default interval: 5 secs
Default count : 60
Default PID : DSE java PID
Default PGM : jstack
I have changed the code of the script in order to get the information we need
Attached is a newer version of the script. It needs the following in order to run
On /etc/password make sure the cassandra user is set to use /bin/bash rather than /bin/false
We need to use Oracle jstack, it will use the command resulting from this search:
Become root and then cassandra, then execute:
A succesful execution would look like this
We've also run in to several instances of this error:
Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
The following fixed it (along with editing multidump.sh to include):
sudo -u cassandra jstack -J-d64 -l <PID> >> jstack.out
Where does the thread dump output go when using multidump-alt.sh?
The multidump-alt.sh script uses kill -3 to generate thread dumps. This output was expected to be in /var/log/cassandra/output.log but is not seen there. Follow these steps to capture the thread dumps using multidump-alt.sh (kill -3):
1. Add these lines to /etc/dse/cassandra/jvm.options file:
-XX:+UnlockDiagnosticVMOptions
-XX:+LogVMOutput
-XX:LogFile=/var/log/cassandra/jvm.log
Note: LogFile can be set to any file, the file does not have to be created in advance.
2. Restart DSE.
3. Run multidump-alt.sh and find the thread dump output in the file you specified in LogFile.
Some discussion on using forced mode (-f), in cases where the jvm doesn't respond to jstack:
https://stackoverflow.com/questions/26140182/running-jmap-getting-unable-to-open-socket-file
Easy way to get DSE PID so you don't have to sift through ps:
cat /var/run/dse/dse.pid
Please sign in to leave a comment.