Summary

A change in the Java driver (on which the Spark Connector depends) can affect how the connection pool into DSE is managed, when using joinWithCassandraTable and saveToCassandra methods.

Applies to

DSE 5.0.14
Spark connector 1.6.11
Java driver 3.0.4

Symptoms

The user may see the following error when running applications that previously worked correctly:

ERROR 2018-06-15 07:43:49,617 Logging.scala:95 - org.apache.spark.executor.Executor: Exception in task 1.0 in stage 3.0 (TID 3) 
java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed 
(tried: 
myhost1/10.1.2.3:9042(com.datastax.driver.core.exceptions.BusyPoolException: [myhost1/10.1.2.3] Pool is busy (no available connection and the queue has reached its max size 256)), 
myhost2/10.1.2.4:9042(com.datastax.driver.core.exceptions.BusyPoolException: [myhost2/10.1.2.4] Pool is busy (no available connection and the queue has reached its max size 256)),
myhost3/10.1.2.5:9042(com.datastax.driver.core.exceptions.BusyPoolException: [myhost3/10.1.2.5] Pool is busy (no available connection and the queue has reached its max size 256))
)

Cause

There was a change in the Java driver between versions 3.0.3 and 3.0.4 as outlined in JAVA-893
This means that the connection pool becomes fully non-blocking. Therefore instead of the application requesting more connection threads and using the blocking nature to implicitly queue it, the pool will immediately return the BusyPoolException if there are no more connection threads available. This of course has implications on any dependant code which might use the Java driver connection pool.

For spark users this affected joinWithCassandraTable and saveToCassandrasee SPARKC-503

Workaround

For spark jobs using the 1.6.11 connector you can download version 1.6.12 from maven here

Then replace the one in your DSE installation. usually under /usr/share/dse/spark/lib

Solution

Upgrade to DSE 5.0.15 (when available) or later

See Also

Java driver upgrade docs: https://docs.datastax.com/en/developer/java-driver/3.0/upgrade_guide/#3-0-4