Summary
DSE nodes observed to be creating a large number of DNS PTR (reverse) lookups
Symptoms
A large number of reverse DNS queries were observed to be originating from DSE nodes, the following query was useful for showing this:
$ sudo netstat -antp | awk 'NR==2;/:53/' Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name udp 0 0 10.0.101.2:34002 10.0.102.5:53 ESTABLISHED 21836/java udp 0 0 10.0.101.2:48475 10.0.102.5:53 ESTABLISHED 21836/java udp 0 0 10.0.101.2:52361 10.0.102.5:53 ESTABLISHED 21836/java udp 0 0 10.0.101.2:54805 10.0.102.5:53 ESTABLISHED 21836/java udp 0 0 10.0.101.2:33304 10.0.102.5:53 ESTABLISHED 21836/java
Cause
When the performance service traces client connections, it unnecessarily makes a call to obtain the client's hostname instead of just its address. The calls with getHostName() was inadvertently causing DSE nodes to make excessive DNS reverse lookups.
This is outlined in the following bug
DSP-7252 - getHostName causes extensive reverse DNS lookup
Workaround
The hosts can be placed in /etc/hosts to avoid the DNS lookups
Solution
Upgrade to the following DSE releases or later, when available:
4.5.10, 4.6.12, 4.7.6, 4.8.3
Comments