This article provides a solution for a situation where OpsCenter agents are unable to store metrics data on a remote cluster due to incorrect configuration.

Symptoms

When loading or storing OpsCenter data, the agents report that the OpsCenter keyspace defined in <cluster_name>.conf does not exist despite the keyspace already existing in the remote cluster. Below are sample errors reported in a node's agent.log:

 ERROR [clojure-agent-send-off-pool-270] 2016-01-07 18:22:57,457 There was an error when attempting to load stored rollups.
 clojure.lang.ExceptionInfo: throw+: {:type :opsagent.cassandra/keyspaces-does-not-exist, :message "The OpsCenter storage keyspace, \"OpsC_remote_KS\", does not exist yet."} {:object {:type :opsagent.cassandra/keyspaces-does-not-exist, :message "The OpsCenter storage keyspace, \"OpsC_remote_KS\", does not exist yet."}, :environment {conn #<SessionManager com.datastax.driver.core.SessionManager@7e9e946>, ks-to-set "\"OpsC_remote_KS\"", current-ks nil, e #<InvalidQueryException com.datastax.driver.core.
exceptions.InvalidQueryException: Keyspace 'OpsC_remote_KS' does not exist>}}
 at opsagent.cassandra$set_ks.invoke(cassandra.clj:44)
 at opsagent.cassandra$get_storage_conn.invoke(cassandra.clj:50)
 at opsagent.cassandra$scan_pdps.invoke(cassandra.clj:271)
...

ERROR [cassandra-processor-2] 2016-01-07 18:23:01,688 Error when proccessing cassandra callclojure.lang.ExceptionInfo: throw+: {:type :opsagent.cassandra/keyspaces-does-not-exist, :message "The OpsCenter storage keyspace, \"OpsC_remote_KS\", does not exist yet."} {:object {:type :opsagent.cassandra/keyspaces-does-not-exist, :message "The OpsCenter storage keyspace, \"OpsC_remote_KS\", does not exist yet."}, :environment {conn #, ks-to-set "\"OpsC_remote_KS\"", current-ks nil, e #}}

Cause

The errors above can be a little misleading since the OpsCenter keyspace does exist on the remote storage cluster.

It does however provide a clue, i.e. the agent is able to connect to a Cassandra cluster but the cluster that it is connected to does not have the OpsCenter keyspace so the agent is either unable to load metrics or store data.

Solution

In the scenario described above, the problem is due to agent misconfiguration. Follow these steps to identify the offending configuration.

Check 1 - Confirm that the cluster's name is the respective configuration. For example, if configuring OpsCenter for cluster_name: Music then ensure that the OpsCenter configuration file is Music.conf.

Check 2 - Confirm that the correct seeds have been provided in the storage_cassandra directive of the <cluster_name>.conf.

For example, ensure that the seeds nodes are for the remote cluster nodes and not the local cluster. Otherwise, the agents will attempt to connect to local nodes in the cluster.

Check 3 - Confirm that the agents' address.yaml is not configured with the hosts directive.

The hosts directive tells the agents where the OpsCenter data is stored. If this directive constains a list of local nodes, then the agent will try to save the data to the local cluster instead of the remote cluster.

To avoid duplication, delete the hosts directive from all nodes and configure the remote cluster in <cluster_name>.conf as above.

See also

DSE doc - Storing collection data on a separate cluster

DSE doc - DataStax Agent configuration