Example

ERROR [MemtableFlushWriter:271] 2019-01-19 19:46:43,018 HeapUtils.java:66 - The heap histogram could not be generated due to the following error:
java.io.IOException: Cannot run program "jcmd": error=12, Cannot allocate memory
 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
 at java.lang.Runtime.exec(Runtime.java:620)
 at java.lang.Runtime.exec(Runtime.java:485)
 at org.apache.cassandra.utils.HeapUtils.logHeapHistogram(HeapUtils.java:62)
 at org.apache.cassandra.utils.JVMStabilityInspector$GlobalHandler.handleError(JVMStabilityInspector.java:140)
 at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:102)
 at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:75)
 at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1186)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: error=12, Cannot allocate memory

What does this ERROR message mean?

When an OutOfMemoryError occurs, a heap histogram will be generated. If there is an exception thrown during the generation, this error will be written to the logs with the exception details.

Why does this ERROR occur?

Several actions are performed when generating a heap histogram:

• Logs INFO message Trying to log the heap histogram using jcmd
• The Cassandra/DSE Java process ID is obtained for jcmd
• Find the jcmd path:
• Search JAVA_HOME/bin for jcmd
• If JAVA_HOME is not set, it defaults to simply using jcmd in the hope that it is in the path
• Executes jcmd <PID> GC.class_histogram

If any of these actions throw an exception, this error is written to the log with the exception details.

Note: -Dcassandra.printHeapHistogramOnOutOfMemoryError does not need to be set with DSE as the default is true. However, OSS Cassandra default is false, so this property needs to be explicitly set to true for a heap histogram to be written during an OutOfMemoryError. Ref: CASSANDRA-13006

How do you fix this ERROR?

java.io.IOException: Cannot run program "jcmd": error=12, Cannot allocate memory

This exception indicates there was not enough memory available to run jcmd. Recommend reviewing system memory allocations and either free up memory or add additional memory as needed.

java.io.IOException: Cannot run program "jcmd": error=2, No such file or directory

This exception is due to either jcmd cannot be located or the Cassandra/DSE service account does not have permissions to the jcmd binary. Things to look for are:

• JAVA_HOME is set to the JDK path.
• JAVA_HOME/bin contains jcmd and execute permissions set for the Cassandra/DSE service account.
• If JAVA_HOME is not set, jcmd needs to be in the system path.
• Test executing jcmd using the Cassandra/DSE service account.