TransportException on Java driver when running nodetool drain

Summary

While running "nodetool drain" on one node in a DataCenter, a TransportException is raised on the client driver

Applies to

DSE Java driver 1.6.8
DSE 6.0.1
Java 1.8.0_181

Symptoms

While running "nodetool drain" on a node in a Cassandra cluster, the error below is raised in the java driver

com.datastax.driver.core.exceptions.TransportException: [/<IP>:<port>] Connection has been closed  
at com.datastax.driver.core.exceptions.TransportException.copy(TransportException.java:29)  
at com.datastax.driver.core.exceptions.TransportException.copy(TransportException.java:15)  
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:28)  
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:236)  
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:59)  
at com.test.cassandra.CassandraTest.connect(CassandraTest.java:64)  
at com.test.cassandra.CassandraTest.main(CassandraTest.java:113)  
Caused by: com.datastax.driver.core.exceptions.TransportException: [/<IP>:<port>] Connection has been closed  
at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1231)  
at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1216)  
at com.datastax.driver.core.Connection.defunct(Connection.java:453)  
at com.datastax.driver.core.Connection$Dispatcher.exceptionCaught(Connection.java:1149)  
at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:280)  
...

Cause

The query fails from the client due to the shutdown of the node that is initiated by the server during the nodetool drain process.
By default, in a case of ConnectionException, the driver defaults to stating any query that failed due to this error is at risk of non-idempotence and will not retry it.
The TransportException error is an extension of the ConnectionException and falls under that scenario. Retry policy can be considered a risk depending on the type of query executed
See link below for further information:

https://docs.datastax.com/en/developer/java-driver-dse/1.6/manual/idempotence/

https://docs.datastax.com/en/developer/java-driver-dse/1.6/manual/retries/

Workaround

In the scenario where the query is idempotent and that the exception should be avoided, a silent retry can be set by one of two means:
- using the following parameter in the cluster building section of the code: withQueryOptions(new QueryOptions().setDefaultIdempotence(true))
- adding setIdempotence for individual queries

Solution

An enhancement has been logged under reference DB-2367 to provide a notification to the client during a controlled shutdown (ie: stopping a node, drain...) and will be considered in the future for implementation.