When using different network interfaces for rpc_address and listen_address in Cassandra, OpsCenter can't connect to the cluster. This problem is evident from releases of OpsCenter 5.2.0 to 5.2.1
The OpsCenter can't connect to the cluster. OpsCenter dashboard shows "Cannot connect to cluster". In the node tab, OpsCenter shows one of the agents is not connected.
In opscenterd.log, you can find:
2015-09-15 18:12:21-0700 [Test_Cluster] INFO: Recognizing new node 127.0.0.2 ('-5380354053572803323')
In this example, there is only one node in the cluster, 127.0.0.2 is the rpc address of the node. If there are multiple nodes in the cluster, you will see OpsCenter recognizes one node with rpc address, but the rest of the nodes with listen address.
Later on, there are messages about connection refused:
2015-09-15 18:12:21-0700 [Test_Cluster] DEBUG: Performing HTTP request (GET): http://127.0.0.2:61621/load-stomp-conf?, body: None 2015-09-15 18:12:21-0700 [Test_Cluster] DEBUG: HTTP request http://127.0.0.2:61621/load-stomp-conf? failed: Connection was refused by other side: 111: Connection refused. 2015-09-15 18:12:21-0700 [Test_Cluster] INFO: Automatic agent setup failed for (127.0.0.2): Connection refused.
listen_address: 127.0.0.3 seed_provider: - seeds: "127.0.0.3" rpc_address: 127.0.0.2
seed_hosts = 127.0.0.2
OpsCenter 5.2.0 switches to native transport protocol and relies on the python driver to gather the cluster information. Driver queries system.peers table to get listen/rpc address which doesn't include data for the node that it is querying.
Before the issue is fixed, users with different network interfaces for rpc and listen address should stay in OpsCenter 5.1.x, which uses thrift protocol.
The fix will be included in OpsCenter 5.2.2.
OPSC-6659: OpsCenter given incorrect broadcast address when connecting to node with different listen/rpc addresses