DataStax Help Center

DSE Analytics node will not start with the error "There was a problem with the Cassandra Hive MetaStore: null"

Summary

A DataStax Enterprise node does not start in Analytics mode with a HiveMetaStore error when most of the nodes in the cluster is unavailable or down despite the node being able to start in real-time Cassandra mode.

Symptoms

An attempt to start a DSE node in Analytics mode records the following example entries in the system.log:

WARN [main] 2015-09-30 11:22:10,353 JobTrackerManager.java (line 246) Failed to retrieve jobtracker locations at CL.QUORUM (Cannot achieve consistency level QUORUM)
WARN [main] 2015-09-30 11:22:10,353 JobTrackerManagerPlugin.java (line 86) Failed to retrieve JobTracker location. This may be caused by some nodes being down.

Eventually, DSE fails to start with the following example exception:

ERROR [main] 2015-09-30 12:57:46,707 DseDaemon.java (line 492) Unable to start DSE server.
com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStoreException: There was a problem with the Cassandra Hive MetaStore: null
        at com.datastax.bdp.hadoop.hive.metastore.MetaStorePersister.find(MetaStorePersister.java:150)

 

Cause

When starting a DSE node in Analytics mode, it attempts to access the Hive metastore keyspace. By default the HiveMetaStore keyspace is read at Quorum. If there are not enough replica nodes with this keyspace on to satisfy Quorum then the read request will fail preventing the node from starting up.

This error was identified in the following internal jiras:

DSP-4464 - Can't start analytics node when rest of cluster is stopped (Fixed in DSE 4.6.1)

DSP-4632 - Cannot start node in Analytics mode because hive is blocking it (Fixed in DSE 4.7.0)

Workaround

There are 2 options available to get around the issue.
 
OPTION 1 - Start quorum nodes in the data centre as Cassandra-only mode, then start the remaining nodes in Analytics mode. Then perform a rolling restart of the Cassandra-only nodes in Analytics mode.
 
OPTION 2 - Start quorum nodes simultaneously in Analytics mode so that enough replicas are available to form a quorum. However, this option may not be practical when there is a large number of nodes in the ring.

Solution

Upgrade to DSE4.7.3 or later

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk