If you find your nodes frequently going down, one of the first things to check is whether you have OutOfMemoryErrors in your system.log. The presence of these errors indicate that your heap size is insufficient for your node's data volume and workload. It is important to understand that your heap space requirements are a function of your memtable size (configured in cassandra.yaml), the node's role, your key cache size, and possibly your row cache size. For background information, please refer to Tuning Java Heap Size in the Cassandra documentation.
If you would like tech support to assist you with correctly sizing your heap, please open a ticket and provide them with the following information:
First, attach the output of the following commands:
- nodetool cfstats
- nodetool info
- 'show schema;' command via the cassandra-cli
- free -m
- java -version
Also attach the following files:
And finally answer the following questions:
- What is your average key size for keys that are cached? This information is unfortunately not available via a Cassandra tool, but should be known by your developers.
- If row cache is enabled, what is your average row size? If you are not sure, please send the output of 'nodetool cfhistograms' for each column family on which you have enabled row caching.
Calculating Heap Requirements
If you want to calculate the heap size for yourself, read on for more details.
Starting with Cassandra 1.1, cache management is fairly straightfoward. Memory caches for column families are now managed globally instead of at the individual column family level, simplifying configuration and tuning. Cassandra automatically distributes memory for various column families based on the overall workload and specific column family usage. Two new configuration parameters, key_cache_size_in_mb and row_cache_size_in_mb replace the per column family cache sizing options. Administrators can choose to include or exclude column families from being cached via the caching parameter that is used when creating or modifying column families.
In Cassandra 1.0 and earlier, estimating the size of your key cache and row cache can be tricky because both settings are managed per column family. The formulas for determining the key cache size are documented here. To make these calculations easier, you may use the Heap Space Calculator tool, available on GitHub. When running the Heap Space Calculator, provide the following command line arguments:
- The path a file containing the
- The heapsize (in GB) for this cluster (can be found via
- The average key size (in bytes) (known by the user)
- The average row size (in bytes) (as found via
- Optionally: The memtable size (in MB). Only to be set if changed from the defaults of 1/3 of the heap space. See
If you do not know the average row size, you can use the accompanying avg_row_size_calculator tool, which takes the path to a file containing the
nodetool cfhistograms <keyspace> <cfname> output for all column familes, and calculates the average row size based on this data.
Here is an example run of the heapspace_calculator:
$ heapspace_calculator cfstats_177811528.log 4 46 0
Memtable size: 1.3 GB
Key Cache Used: 3.5 GB
Key Cache Estimate: 19.4 GB
Bloom Filter Size: 135.5 MB
75% of the Heap Size: 2.80 GB
Estimated Java Heap Size (Currently Used): 5.92 GB
Estimated Java Heap Size (Full Capacity): 21.89 GB
The tool parses your cfstats output, sums up the "Key cache size" (currently used) for all column families, and calculates the "Estimated Java Heap Size (Currently Used)" based on the total. It also sums up the "Key cache capacity" entries for each column family and calculates the "Estimated Java Heap Size (Full Capacity)" based on this total.
The "Full Capacity" number tells you how much heap space would be consumed if the key caches grew to the maximum size allowed by the keys_cached setting on each column family. To prevent this, you can reduce the keys_cached setting on your column families. This is usually a better option than your heap size beyond 8GB because (in most cases) the ability of Java 6 to gracefully handle garbage collection above 8GB quickly diminishes.