Excessive DNS lookups from the DSE performance service – DataStax Support

Summary

DSE nodes observed to be creating a large number of DNS PTR (reverse) lookups

Symptoms

A large number of reverse DNS queries were observed to be originating from DSE nodes, the following query was useful for showing this:

$ sudo netstat -antp | awk 'NR==2;/:53/'
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
udp 0 0 10.0.101.2:34002 10.0.102.5:53 ESTABLISHED 21836/java
udp 0 0 10.0.101.2:48475 10.0.102.5:53 ESTABLISHED 21836/java
udp 0 0 10.0.101.2:52361 10.0.102.5:53 ESTABLISHED 21836/java
udp 0 0 10.0.101.2:54805 10.0.102.5:53 ESTABLISHED 21836/java
udp 0 0 10.0.101.2:33304 10.0.102.5:53 ESTABLISHED 21836/java

Cause

When the performance service traces client connections, it unnecessarily makes a call to obtain the client's hostname instead of just its address. The calls with getHostName() was inadvertently causing DSE nodes to make excessive DNS reverse lookups.

This is outlined in the following bug

DSP-7252 - getHostName causes extensive reverse DNS lookup

Workaround

The hosts can be placed in /etc/hosts to avoid the DNS lookups

Solution

Upgrade to the following DSE releases or later, when available:

4.5.10, 4.6.12, 4.7.6, 4.8.3