Summary
This article provides a solution for a situation where OpsCenter agents are unable to store metrics data on a remote cluster due to incorrect configuration.
Symptoms
When loading or storing OpsCenter data, the agents report that the OpsCenter keyspace defined in <cluster_name>.conf
does not exist despite the keyspace already existing in the remote cluster. Below are sample errors reported in a node's agent.log
:
ERROR [clojure-agent-send-off-pool-270] 2016-01-07 18:22:57,457 There was an error when attempting to load stored rollups.
clojure.lang.ExceptionInfo: throw+: {:type :opsagent.cassandra/keyspaces-does-not-exist, :message "The OpsCenter storage keyspace, \"OpsC_remote_KS\", does not exist yet."} {:object {:type :opsagent.cassandra/keyspaces-does-not-exist, :message "The OpsCenter storage keyspace, \"OpsC_remote_KS\", does not exist yet."}, :environment {conn #<SessionManager com.datastax.driver.core.SessionManager@7e9e946>, ks-to-set "\"OpsC_remote_KS\"", current-ks nil, e #<InvalidQueryException com.datastax.driver.core.
exceptions.InvalidQueryException: Keyspace 'OpsC_remote_KS' does not exist>}}
at opsagent.cassandra$set_ks.invoke(cassandra.clj:44)
at opsagent.cassandra$get_storage_conn.invoke(cassandra.clj:50)
at opsagent.cassandra$scan_pdps.invoke(cassandra.clj:271)
...
ERROR [cassandra-processor-2] 2016-01-07 18:23:01,688 Error when proccessing cassandra callclojure.lang.ExceptionInfo: throw+: {:type :opsagent.cassandra/keyspaces-does-not-exist, :message "The OpsCenter storage keyspace, \"OpsC_remote_KS\", does not exist yet."} {:object {:type :opsagent.cassandra/keyspaces-does-not-exist, :message "The OpsCenter storage keyspace, \"OpsC_remote_KS\", does not exist yet."}, :environment {conn #, ks-to-set "\"OpsC_remote_KS\"", current-ks nil, e #}}
Cause
The errors above can be a little misleading since the OpsCenter keyspace does exist on the remote storage cluster.
It does however provide a clue, i.e. the agent is able to connect to a Cassandra cluster but the cluster that it is connected to does not have the OpsCenter keyspace so the agent is either unable to load metrics or store data.
Solution
In the scenario described above, the problem is due to agent misconfiguration. Follow these steps to identify the offending configuration.
Check 1 - Confirm that the cluster's name is the respective configuration. For example, if configuring OpsCenter for cluster_name: Music
then ensure that the OpsCenter configuration file is Music.conf
.
Check 2 - Confirm that the correct seeds have been provided in the storage_cassandra
directive of the <cluster_name>.conf
.
For example, ensure that the seeds nodes are for the remote cluster nodes and not the local cluster. Otherwise, the agents will attempt to connect to local nodes in the cluster.
Check 3 - Confirm that the agents' address.yaml
is not configured with the hosts
directive.
The hosts
directive tells the agents where the OpsCenter data is stored. If this directive constains a list of local nodes, then the agent will try to save the data to the local cluster instead of the remote cluster.
To avoid duplication, delete the hosts
directive from all nodes and configure the remote cluster in <cluster_name>.conf
as above.
See also
DSE doc - Storing collection data on a separate cluster
DSE doc - DataStax Agent configuration