How to set global system properties for the Spark Cassandra connector

Overview

This article discusses how to configure global properties in DataStax Enterprise Spark.

Configuration

Global system properties by definition are set for the whole system and shared by all components.

Depending on the version of Spark Cassandra Connector, these properties are not configurable inside a SparkConf object.

In order to override the system defaults, set the property by adding it to <spark_home>/conf/spark-defaults.conf.

For example, the keep_alive_ms property cannot be set in SparkConf in Spark Cassandra Connector v1.1.0 so to change the default period to keep unused connections open, add the following line to spark-defaults.conf:

spark.cassandra.connection.keep_alive_ms     30000

See also

For information on other global system properties, see the Connecting to Cassandra documentation for the Spark Cassandra Connector v1.1.0 on GitHub.