Full ERROR Message Example

ERROR [SharedPool-Worker-25] 2016-12-27 10:14:28,100 SEPWorker.java:141 - Failed to execute task, unexpected exception killed worker: {}
 java.lang.IllegalStateException: Shutdown in progress
 at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.8.0_74]
 at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[na:1.8.0_74]
 at org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:764) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:68) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:168) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
at 
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
at 
java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]

What does this ERROR message mean?

The ERROR message means the spark worker was shut down and thus the task cannot be completed.

Why does this ERROR occur?

The ERROR occurs because the JVM for the spark worker was killed. This alert generally follows another alert. For example:

ERROR [SharedPool-Worker-25] 2016-12-27 10:14:26,100 JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting forcefully due to:java.lang.OutOfMemoryError: Java heap space
 at java.util.Arrays.copyOf(Arrays.java:3181) ~[na:1.8.0_74]
at org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:112) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.db.DeletionInfo.copy(DeletionInfo.java:104) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:217) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.db.Memtable.put(Memtable.java:210) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1230) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_74]
 at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]

ERROR [SharedPool-Worker-25] 2016-12-27 10:14:28,100 SEPWorker.java:141 - Failed to execute task, unexpected exception killed worker: {}
 java.lang.IllegalStateException: Shutdown in progress
 at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.8.0_74]
 at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[na:1.8.0_74]
 at org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:764) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:68) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
 at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:168) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
at 
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
at 
java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]

From the above, the JVM ran out of heap space, thus, killing the JVM. When the worker attempts to run the task, the task fails because the worker was already killed.

How do you fix this ERROR?

Given the situation above, you would increase the heap size of the worker JVM, by increasing the spark.executor.memory or the initial_spark_worker_resources in the dse.yaml file or SPARK_WORKER_MEMORY in the spark-env.sh file. However, the solution will be based on what the ERROR preceding the alert is.