Summary
This article discusses an issue where a firewall configured with an idle connection timeout can close connections to local nodes and nodes in other data centers during low traffic intervals.
Applies to
Installations on Linux operating systems for:
- DSE 6.7
- DSE 6.0
- DSE 5.1
- DDAC
Cause
The TCP keep alive may not be appropriately configured for your environment. The default idle connection timeout is usually 60 minutes.
Solution
1. Run this command to get a list of available kernel variables:
sysctl -A | grep net.ipv4
The following variables configure keepalive support in Linux:
- net.ipv4.tcp_keepalive_time
Time of connection inactivity after which the first keep alive request is sent.
- net.ipv4.tcp_keepalive_probes
Number of keep alive requests retransmitted before the connection is considered broken.
- net.ipv4.tcp_keepalive_intvl
Time interval between keep alive probes.
2. Configure these settings as appropriate for your environment. For example, to configure TCP keepalive timeout to 60 seconds with 3 probes, 10 seconds gap between each:
sudo sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10
This command sets keepalive to detect dead TCP connections after 90 seconds (60 + 10 + 10 + 10). Additional traffic is negligible so it is safe to persist these settings.
Tip: Instead of changing the system TCP settings, you can prevent reset connections during streaming by tuning the streaming_keep_alive_period_in_secs setting in cassandra.yaml. Be sure to apply recommended settings for Linux platforms.