DataStax Help Center

Intermittent gossip issues when using with GossipingPropertyFileSnitch


Nodes are intermittently unable to gossip properly when configured with the GossipingPropertyFileSnitch.


One symptom is nodes randomly going up and down for no apparent reason. 

INFO  [GossipTasks:1] 2016-04-29 02:47:32,559 - InetAddress / is now DOWN
INFO  [GossipTasks:1] 2016-04-29 02:50:47,123 - InetAddress / is now DOWN
INFO  [GossipTasks:1] 2016-04-29 02:54:59,640 - InetAddress / is now DOWN
INFO  [SharedPool-Worker-2] 2016-04-29 03:01:23,828 - InetAddress / is now UP
INFO  [SharedPool-Worker-1] 2016-04-29 03:01:59,432 - InetAddress / is now UP
INFO  [SharedPool-Worker-7] 2016-04-29 03:02:01,839 - InetAddress / is now UP

Similarly, different nodes appear to be down in the nodetool status output depending on where it was ran, for example:

Datacenter: Cassandra
|/ State=Normal/Leaving/Joining/Moving
--  Address   Load       Tokens  Owns    Host ID                               Rack
DN  8.97 GB    256     ?       a50dfef5-229d-4d15-89d9-971bec01094b  rack1
UN  8.9 GB     256     ?       a16b71a2-9b95-4669-a6bd-d7326bd279e2  rack1
DN  9.09 GB    256     ?       ac01b6f9-3cb9-47ff-83c6-0404836386eb  rack1
UN  10.65 GB   256     ?       9c0ef3a2-aad7-4d06-b015-f32ddccac750  rack1


The problem has only been seen in a very small number of clusters and is still under investigation. However, it has been identified that the problem occurs when the exists while nodes are configured with GossipingPropertyFileSnitch:

INFO  [main] 2016-04-29 15:31:26,039 - Loaded for compatibility

It is important to note that the issue is very intermittent and not all vectors which trigger the problem are known yet.


By design, the GossipingPropertyFileSnitch falls back on the PropertyFileSnitch's as a means to allow clusters to be migrated to GossipingPropertyFileSnitch.

If the cluster is already on GossipingPropertyFileSnitch, ensure that has been removed or does not exist even if there are no issues with the nodes to ensure the cluster does not encounter problems in the future.

See also

DataStax doc - Snitch - GossipingPropertyFileSnitch

Cassandra JIRA - CASSANDRA-11508 GPFS property file should more clearly explain the relationship with PFS

Was this article helpful?
5 out of 5 found this helpful
Have more questions? Submit a request


Powered by Zendesk