Summary
This article describes an issue where Kafka Connector channel tasks fail when a node is bootstrapping or otherwise joining a ring.
Applies to:
- Kafka Connector 1.0
- Kafka Connector 1.1.0
- DataStax Java Driver for DSE 4.0
- DataStax Java Driver for DSE 4.0.1
Note: Driver connections to DSE 6.7 clusters are not impacted.
Symptoms
When a DSE 5.x/6.x node is added to a running ring, the Kafka Connector will throw one of several exceptions:
Caused by: java.lang.IllegalArgumentException: Missing rpc_address or native in system row, can't refresh node at com.datastax.oss.driver.internal.core.metadata.DefaultTopologyMonitor.asNodeInfo(DefaultTopologyMonitor.java:260)
Caused by: java.lang.NullPointerException: address can't be null at java.util.Objects.requireNonNull(Objects.java:228)
Cause
This error is caused by the driver incorrectly reading in a null
value in system.peers
on the control connection host, which is the DSE node providing information to the driver context about the ring topology and state of nodes in the ring. This null
value is expected for a joining node and should be ignored by the driver but is not ignored in the affected driver releases.
Workaround
Administrators can safely delete the bootstrapping node in the system.peers
table of all nodes:
cqlsh> DELETE FROM system.peers WHERE peer = '192.168.0.10';
By deleting the offending row on all nodes, the cluster topology is refreshed for the Kafka Connector, and applications built on the affected driver version. The information about the joining node remains in gossip, and the system.peers
table is updated by all nodes in the ring after the bootstrapping node transitions to UN
state.
Solution
This solution is being tracked on JAVA-2280 and internal defect ID KAF-126.