Summary
A change in the Java driver (on which the Spark Connector depends) can affect how the connection pool into DSE is managed, when using joinWithCassandraTable
and saveToCassandra
methods.
Applies to
DSE 5.0.14
Spark connector 1.6.11
Java driver 3.0.4
Symptoms
The user may see the following error when running applications that previously worked correctly:
ERROR 2018-06-15 07:43:49,617 Logging.scala:95 - org.apache.spark.executor.Executor: Exception in task 1.0 in stage 3.0 (TID 3) java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: myhost1/10.1.2.3:9042(com.datastax.driver.core.exceptions.BusyPoolException: [myhost1/10.1.2.3] Pool is busy (no available connection and the queue has reached its max size 256)), myhost2/10.1.2.4:9042(com.datastax.driver.core.exceptions.BusyPoolException: [myhost2/10.1.2.4] Pool is busy (no available connection and the queue has reached its max size 256)), myhost3/10.1.2.5:9042(com.datastax.driver.core.exceptions.BusyPoolException: [myhost3/10.1.2.5] Pool is busy (no available connection and the queue has reached its max size 256)) )
Cause
There was a change in the Java driver between versions 3.0.3 and 3.0.4 as outlined in JAVA-893
This means that the connection pool becomes fully non-blocking. Therefore instead of the application requesting more connection threads and using the blocking nature to implicitly queue it, the pool will immediately return the BusyPoolException
if there are no more connection threads available. This of course has implications on any dependant code which might use the Java driver connection pool.
For spark users this affected joinWithCassandraTable
and saveToCassandra
see SPARKC-503
Workaround
For spark jobs using the 1.6.11 connector you can download version 1.6.12 from maven here
Then replace the one in your DSE installation. usually under /usr/share/dse/spark/lib
Solution
Upgrade to DSE 5.0.15 (when available) or later
See Also
Java driver upgrade docs: https://docs.datastax.com/en/developer/java-driver/3.0/upgrade_guide/#3-0-4