Full ERROR Message Example

ERROR [pool-10-thread-1] 2020-07-25 11:15:26,946 FailureDetector.java:281 - Unknown endpoint /10.1.5.101

What does this ERROR message mean

It means the endpoint status value of the peer node (10.1.5.101 in the example above) is empty/NULL, when the node you observe the error message, is trying to get the endpoint status as part of the Gossip protocol.

Why does this ERROR occur

This error is triggered when the node is trying to get a peer’s endpoint status through gossip, and if the status is empty or NULL. This usually happens after a node has recently been decommissioned/removed from the cluster. It is expected that nodetool status will not show the peer's IP address as part of the cluster anymore.

How do you fix this ERROR

Step 1: run nodetool status and "odetool gossipinfo to get the cluster status and gossip information
Step 2: If the peer does not appear in any of the above nodetool commands in the cluster, check system.peers to see if they are still in the table:

cqlsh> SELECT * FROM system.peers;

Step 3: If they are showing up in the system.peers, and confirmed they are not supposed to be in the cluster, then can remove them from each node. Please double check the IP address before issuing the delete against table system.peers

cqlsh> delete from system.peers where peer = '<IP address>';

Note: If you see the same error message on Cassandra 4.0 or DSE 6.7.x or later and get stuck, please contact DataStax support for further investigation, because the delete operation can only be performed on Cassandra version 3.x or earlier (DSE 6.0.x or earlier), starting from Cassandra 4.0 (and DSE 6.7.x or later), the deletion is not allowed, below error will be triggered since virtual table does not support "DELETE", more details see Datastax: About virtual keyspaces and tables and CASSANDRA-7622:

cassandra@cqlsh> DELETE from system.peers where peer='<ip address>';
InvalidRequest: Error from server: code=2200 [Invalid query] message="System views don't support DELETE statements"