Bootstrap fails with "Unable to gossip with any seeds" yet new node can connect to seed nodes

Summary

When bootstrapping a node, the bootstrap fails even though communications from the new node into the cluster appears to be ok

Symptoms

When bootstrapping the new node the following error is seen in the /var/log/cassandra/system.log

ERROR [main] 2016-03-03 19:19:44,911 CassandraDaemon.java:579 - Exception encountered during startup 
java.lang.RuntimeException: Unable to gossip with any seeds 

Checking the communications from the new node it is observed that the seed nodes can be reached ok (i.e. using ping). Even when checking using nc or telnet the default gossip port of 7000 (or 7001 if using SSL) the new node can reach the existing seed nodes. 

Cause

Gossip communications must be bi-directional. The new node could not be reached from the existing seed nodes or any other nodes for that matter, thus preventing the bootstrap.

Testing

With the DSE process on the new node shutdown a simple test using nc is as follows:

On the new node run nc to listen on port 7000

nc -l 10.1.2.3 7000

On the existing seed node(s) run the following nc command

nc -v -w2 10.1.2.3 7000

A good connection:

Ncat: Connected to 10.1.2.3:7000. 

A refused connection:

Ncat: Connection refused.

Solution

Investigation of the network connectivity between the existing nodes and the new node will need to be carried out. The bootstrap will only work correctly once the communications can be achieved on both directions.

Further reading

The following docs outline the ports used by Apache Cassandra:

https://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureFireWall_r.html