Kafka Connector fails tasks while node is bootstrapping

Summary

This article describes an issue where Kafka Connector channel tasks fail when a node is bootstrapping or otherwise joining a ring.

Applies to:

• Kafka Connector 1.0
• Kafka Connector 1.1.0
• DataStax Java Driver for DSE 4.0
• DataStax Java Driver for DSE 4.0.1

Note: Driver connections to DSE 6.7 clusters are not impacted.

Symptoms

When a DSE 5.x/6.x node is added to a running ring, the Kafka Connector will throw one of several exceptions:

Caused by: java.lang.IllegalArgumentException: Missing rpc_address or native in system row, can't refresh node
        at com.datastax.oss.driver.internal.core.metadata.DefaultTopologyMonitor.asNodeInfo(DefaultTopologyMonitor.java:260)

Caused by: java.lang.NullPointerException: address can't be null
        at java.util.Objects.requireNonNull(Objects.java:228)

Cause

This error is caused by the driver incorrectly reading in a null value in system.peers on the control connection host, which is the DSE node providing information to the driver context about the ring topology and state of nodes in the ring. This null value is expected for a joining node and should be ignored by the driver but is not ignored in the affected driver releases.

Workaround

Administrators can safely delete the bootstrapping node in the system.peers table of all nodes:

cqlsh> DELETE FROM system.peers WHERE peer = '192.168.0.10';

By deleting the offending row on all nodes, the cluster topology is refreshed for the Kafka Connector, and applications built on the affected driver version. The information about the joining node remains in gossip, and the system.peers table is updated by all nodes in the ring after the bootstrapping node transitions to UN state.

Solution

This solution is being tracked on JAVA-2280 and internal defect ID KAF-126.