Driver connections to an EC2 cluster appears to be delayed by several seconds (or slow).
Driver connections appear to have a 10-second delay and are slower than expected when connecting to clusters with data centres deployed across multiple EC2 regions.
This issue is due to the way the client (e.g. Java driver or Ruby driver) establishes connections to nodes in the cluster.
cluster() initialisation occurs in several phases. The contact points from the
hosts parameter are the nodes that a driver will try to initiate the first connection to the cluster. Once the connection is established, the driver retrieves the addresses of the rest of the nodes in the cluster and it will start establishing connections to those.
The 10-second delay experienced in multi-region EC2 clusters is a result of the driver establishing connections to remote nodes using private IPs which eventually timeout.
For clusters with simple networking configurations, the generic address resolution policy used by the driver works but for EC2 clusters with nodes deployed across multiple regions, the driver needs to be configured to use the
EC2MultiRegion address resolution policy.
Example - Ruby driver
cluster = Cassandra.cluster( hosts: ['22.214.171.124'],
:address_resolution => :ec2_multi_region )
Example - Java driver
Cluster cluster = Cluster.builder() .addContactPoint("126.96.36.199") .withAddressTranslator(new EC2MultiRegionAddressTranslator()) .build();
In the address retrieval phase, the EC2 multi-region resolution policy uses EC2 API to retrieve the addresses of the nodes to find an address that "works", i.e. private IP for nodes in the DC "local" to the client and public IP for nodes in another region.
In addition to the above, the nodes must be configured to broadcast their public IP address as documented in Configuring cassandra.yaml for cross-region communication.
For a more detailed explanation, including a diagram of a sample client lookup, see the Address resolution page of the Ruby driver documentation.
Ruby Driver doc - Address resolution and
Ruby Driver API -
EC2MultiRegion policy class
Java Driver doc - Address resolution and
DataStax doc - Configuring