DataStax Help Center

Configuring Logging in Apache Cassandra

Apache Cassandra's logging configuration is controlled by log4j-server.properties in the /etc/dse/cassandra (deb/rpm installs) or $DSE_HOME/resources/cassandra/conf (tarball installs).

Log Rotation

system.log

For system.log, the maximum log file size and number of backup copies are controlled by the following lines:

log4j.appender.R.maxFileSize=20MB
log4j.appender.R.maxBackupIndex=50

The defaults are to rotate the logs at 20MB and to keep 50 backup copies. Once the log file reaches 20MB, the current log file will be renamed to system.log.1 and a new system.log will be started. Any previous backups are renumbered from system.log.n to system.log.n+1 (which means the higher the number, the older the file is). When the maximum number of backups has been reached, the oldest file is deleted.

If you're using the default settings, you may have up to 50 backup log files, so if an issue occurred but has already been rotated out of the current system.log, check to see if it is captured in an older backup. If you want to keep more history, you can either increase the maxFileSize, maxBackupIndex, or both, but make sure you have enough space to store all the logs that you are keeping.

output.log

The output.log captures the stdout of the cassandra process and therefore its rotation is not controllable via log4j. However, it can be rotated using the standard Linux logrotate facility.  To configure logrotate to work with cassandra, create a file called /etc/logrotate.d/cassandra with the following contents:

/var/log/cassandra/output.log {
  size 10M
  rotate 9
  missingok
  copytruncate
  compress
}

The settings can be changed as desired, but the copytruncate directive is critical because it allows the log to be rotated without any support from Cassandra for closing and reopening the file.  Please refer to the logrotate man page for more information.

Logging Levels

log4j also allows you to control the logging level. The 6 different logging levels, from most to least verbose are:

TRACE,
DEBUG,
INFO,
WARN,
ERROR and
FATAL

The default logging level is determined by the following line:

log4j.rootLogger=INFO,stdout,R

You can also exert more fine-grained control over your logging by specifying a different logging level for specific categories. The categories usually correspond to the package and class name of the code doing the logging.

For example, the following setting will log debug messages from all classes in the org.apache.cassandra.db package:

log4j.logger.org.apache.cassandra.db=DEBUG

Whereas the following setting will cause debug messages specifically from the StorageProxy class in the org.apache.cassandra.service package:

log4j.logger.org.apache.cassandra.service.StorageProxy=DEBUG

To find out what category a particular message in the log belongs to, you need to change the following line:

log4j.appender.R.layout.ConversionPattern=%5p [%t] %d{ISO8601} %F (line %L) %m%n

If you add %c at the beginning of the Conversion pattern as follows, each message logged will be prefixed with the category:

log4j.appender.R.layout.ConversionPattern=%c %5p [%t] %d{ISO8601} %F (line %L) %m%n

After letting Cassandra run for a while, you can use the following command to determine which categories are logging the most messages:

cat system.log.* | egrep 'TRACE|DEBUG|INFO|WARN|ERROR|FATAL' | awk '{ print $1 }' | sort | uniq -c | sort -n

If you determine that a certain class is logging too many messages, you can set a less verbose logging level for that particular class by adding a line to your log4j-server.properties for that particular class, using the following format:

log4j.logger.package.class=WARN

For example a busy Solr node can log quite a lot of INFO messages from the SolrCore, LogUpdateProcessorFactory, and SolrIndexSearcher classes. To suppress these messages, you could add the following lines:

log4j.logger.org.apache.solr.core.SolrCore=WARN
log4j.logger.org.apache.solr.update.processor.LogUpdateProcessorFactory=WARN
log4j.logger.org.apache.solr.search.SolrIndexSearcher=WARN

After completing this exercise you will probably want to remove %c from the ConversionPattern to rever the messages back to the default format.

Was this article helpful?
4 out of 4 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk