FAQ - Warning message "java.net.BindException: Address already in use" when launching Spark shell

Overview

This article explains the warning messages relating to ports already in use when launching a Spark shell.

Background

Each SparkContext has an associated web interface (SparkUI) which displays useful information about Spark jobs and configuration. The SparkUI is accessible by browsing http://<node_ip>:4040.

By default, the SparkUI is bound to port 4040. If there are multiple SparkContexts running on the same node, then the SparkUI will bind to the next available port, e.g. port 4041.

Spark shell

When launching a Spark shell, a SparkContext is automatically created and hence a Spark web UI gets started bound to port 4040.

To illustrate with an example, given a Spark shell which spawns a java process ID 11919:

$ lsof -i -n | grep LISTEN | grep 11919
java    11919 automaton  347u  IPv4  58510      0t0  TCP *:52154 (LISTEN)
java    11919 automaton  378u  IPv4  58356      0t0  TCP 127.0.0.1:38747 (LISTEN)
java    11919 automaton  379u  IPv4  58357      0t0  TCP *:55823 (LISTEN)
java    11919 automaton  380u  IPv4  58878      0t0  TCP *:4040 (LISTEN)
java    11919 automaton  399u  IPv4  59271      0t0  TCP *:43170 (LISTEN)

Launching a second Spark shell, the following warning message is displayed:

WARN  2015-12-17 02:16:45 org.spark-project.jetty.util.component.AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
...
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) ~[na:1.7.0_45]
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[na:1.7.0_45]
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187) ~[spark-core_2.10-1.4.1.3.jar:1.4.1.3]
...
at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:228) [spark-core_2.10-1.4.1.3.jar:1.4.1.3]
at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:238) [spark-core_2.10-1.4.1.3.jar:1.4.1.3]
at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:238) [spark-core_2.10-1.4.1.3.jar:1.4.1.3]
at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1991) [spark-core_2.10-1.4.1.3.jar:1.4.1.3]
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) [scala-library-2.10.5.jar:na]
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1982) [spark-core_2.10-1.4.1.3.jar:1.4.1.3]
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:238) [spark-core_2.10-1.4.1.3.jar:1.4.1.3]
at org.apache.spark.ui.WebUI.bind(WebUI.scala:117) [spark-core_2.10-1.4.1.3.jar:1.4.1.3]
at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:448) [spark-core_2.10-1.4.1.3.jar:1.4.1.3]
at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:448) [spark-core_2.10-1.4.1.3.jar:1.4.1.3]

Further on, it reports:

WARN  2015-12-17 02:16:45 org.apache.spark.util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.

Similar to the first Spark shell, assuming that the second Spark shell spawned a java process ID 12403, the output below shows the process is bound to port 4041:

$ lsof -i -n | grep LISTEN | grep 12403
java    12403 automaton  347u  IPv4  63010      0t0  TCP *:37127 (LISTEN)
java    12403 automaton  378u  IPv4  61103      0t0  TCP 127.0.0.1:33946 (LISTEN)
java    12403 automaton  379u  IPv4  61104      0t0  TCP *:45943 (LISTEN)
java    12403 automaton  380u  IPv4  61106      0t0  TCP *:4041 (LISTEN)
java    12403 automaton  399u  IPv4  64573      0t0  TCP *:58217 (LISTEN)

Conclusion

The warning message is generated by Spark and is important since it alerts the user to the fact that other SparkContexts are in use and that the Spark web UI is running on a different port. This avoids confusion in situations where a user is attempting to monitor jobs in a different SparkContext.

See also

Spark doc - Using the shell, Spark Programming Guide

Spark doc - Web Interfaces, Spark Monitoring and Instrumentation