Driver connection to cluster on EC2 experiencing delays or appears slow

Summary

Driver connections to an EC2 cluster appears to be delayed by several seconds (or slow).

Symptoms

Driver connections appear to have a 10-second delay and are slower than expected when connecting to clusters with data centres deployed across multiple EC2 regions.

Cause

This issue is due to the way the client (e.g. Java driver or Ruby driver) establishes connections to nodes in the cluster.

The cluster() initialisation occurs in several phases. The contact points from the hosts parameter are the nodes that a driver will try to initiate the first connection to the cluster. Once the connection is established, the driver retrieves the addresses of the rest of the nodes in the cluster and it will start establishing connections to those.

The 10-second delay experienced in multi-region EC2 clusters is a result of the driver establishing connections to remote nodes using private IPs which eventually timeout.

Solution

For clusters with simple networking configurations, the generic address resolution policy used by the driver works but for EC2 clusters with nodes deployed across multiple regions, the driver needs to be configured to use the EC2MultiRegion address resolution policy.

Example - Ruby driver

cluster = Cassandra.cluster(
    hosts: ['1.2.3.4'],
    :address_resolution => :ec2_multi_region
)

Example - Java driver

Cluster cluster = Cluster.builder()
    .addContactPoint("1.2.3.4")
    .withAddressTranslator(new EC2MultiRegionAddressTranslator())
    .build();

In the address retrieval phase, the EC2 multi-region resolution policy uses EC2 API to retrieve the addresses of the nodes to find an address that "works", i.e. private IP for nodes in the DC "local" to the client and public IP for nodes in another region.

Broadcast address

In addition to the above, the nodes must be configured to broadcast their public IP address as documented in Configuring cassandra.yaml for cross-region communication.

For a more detailed explanation, including a diagram of a sample client lookup, see the Address resolution page of the Ruby driver documentation.

See also

Ruby Driver doc - Address resolution and EC2MultiRegion policy

Ruby Driver API - EC2MultiRegion policy class

Java Driver doc - Address resolution and AddressTranslatorinterface

DataStax doc - Configuring EC2MultiRegionSnitch