Summary
While running "nodetool drain" on one node in a DataCenter, a TransportException is raised on the client driver
Applies to
DSE Java driver 1.6.8
DSE 6.0.1
Java 1.8.0_181
Symptoms
While running "nodetool drain" on a node in a Cassandra cluster, the error below is raised in the java driver
com.datastax.driver.core.exceptions.TransportException: [/<IP>:<port>] Connection has been closed at com.datastax.driver.core.exceptions.TransportException.copy(TransportException.java:29) at com.datastax.driver.core.exceptions.TransportException.copy(TransportException.java:15) at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:28) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:236) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:59) at com.test.cassandra.CassandraTest.connect(CassandraTest.java:64) at com.test.cassandra.CassandraTest.main(CassandraTest.java:113) Caused by: com.datastax.driver.core.exceptions.TransportException: [/<IP>:<port>] Connection has been closed at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1231) at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1216) at com.datastax.driver.core.Connection.defunct(Connection.java:453) at com.datastax.driver.core.Connection$Dispatcher.exceptionCaught(Connection.java:1149) at com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:280) ...
Cause
The query fails from the client due to the shutdown of the node that is initiated by the server during the nodetool drain process.
By default, in a case of ConnectionException, the driver defaults to stating any query that failed due to this error is at risk of non-idempotence and will not retry it.
The TransportException error is an extension of the ConnectionException and falls under that scenario. Retry policy can be considered a risk depending on the type of query executed
See link below for further information:
https://docs.datastax.com/en/developer/java-driver-dse/1.6/manual/idempotence/
https://docs.datastax.com/en/developer/java-driver-dse/1.6/manual/retries/
Workaround
In the scenario where the query is idempotent and that the exception should be avoided, a silent retry can be set by one of two means:
- using the following parameter in the cluster building section of the code: withQueryOptions(new QueryOptions().setDefaultIdempotence(true))
- adding setIdempotence for individual queries
Solution
An enhancement has been logged under reference DB-2367 to provide a notification to the client during a controlled shutdown (ie: stopping a node, drain...) and will be considered in the future for implementation.