Summary
Driver connections to an EC2 cluster appears to be delayed by several seconds (or slow).
Symptoms
Driver connections appear to have a 10-second delay and are slower than expected when connecting to clusters with data centres deployed across multiple EC2 regions.
Cause
This issue is due to the way the client (e.g. Java driver or Ruby driver) establishes connections to nodes in the cluster.
The cluster()
initialisation occurs in several phases. The contact points from the hosts
parameter are the nodes that a driver will try to initiate the first connection to the cluster. Once the connection is established, the driver retrieves the addresses of the rest of the nodes in the cluster and it will start establishing connections to those.
The 10-second delay experienced in multi-region EC2 clusters is a result of the driver establishing connections to remote nodes using private IPs which eventually timeout.
Solution
For clusters with simple networking configurations, the generic address resolution policy used by the driver works but for EC2 clusters with nodes deployed across multiple regions, the driver needs to be configured to use the EC2MultiRegion
address resolution policy.
Example - Ruby driver
cluster = Cassandra.cluster(
hosts: ['1.2.3.4'],
:address_resolution => :ec2_multi_region
)
Example - Java driver
Cluster cluster = Cluster.builder() .addContactPoint("1.2.3.4") .withAddressTranslator(new EC2MultiRegionAddressTranslator()) .build();
In the address retrieval phase, the EC2 multi-region resolution policy uses EC2 API to retrieve the addresses of the nodes to find an address that "works", i.e. private IP for nodes in the DC "local" to the client and public IP for nodes in another region.
Broadcast address
In addition to the above, the nodes must be configured to broadcast their public IP address as documented in Configuring cassandra.yaml for cross-region communication.
For a more detailed explanation, including a diagram of a sample client lookup, see the Address resolution page of the Ruby driver documentation.
See also
Ruby Driver doc - Address resolution and EC2MultiRegion
policy
Ruby Driver API - EC2MultiRegion
policy class
Java Driver doc - Address resolution and AddressTranslator
interface
DataStax doc - Configuring EC2MultiRegionSnitch