A DataStax Enterprise node does not start in Analytics mode with a HiveMetaStore error when most of the nodes in the cluster is unavailable or down despite the node being able to start in real-time Cassandra mode.
An attempt to start a DSE node in Analytics mode records the following example entries in the system.log:
WARN [main] 2015-09-30 11:22:10,353 JobTrackerManager.java (line 246) Failed to retrieve jobtracker locations at CL.QUORUM (Cannot achieve consistency level QUORUM) WARN [main] 2015-09-30 11:22:10,353 JobTrackerManagerPlugin.java (line 86) Failed to retrieve JobTracker location. This may be caused by some nodes being down.
Eventually, DSE fails to start with the following example exception:
ERROR [main] 2015-09-30 12:57:46,707 DseDaemon.java (line 492) Unable to start DSE server. com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStoreException: There was a problem with the Cassandra Hive MetaStore: null at com.datastax.bdp.hadoop.hive.metastore.MetaStorePersister.find(MetaStorePersister.java:150)
When starting a DSE node in Analytics mode, it attempts to access the Hive metastore keyspace. By default the HiveMetaStore keyspace is read at Quorum. If there are not enough replica nodes with this keyspace on to satisfy Quorum then the read request will fail preventing the node from starting up.
This error was identified in the following internal jiras:
DSP-4464 - Can't start analytics node when rest of cluster is stopped (Fixed in DSE 4.6.1)
DSP-4632 - Cannot start node in Analytics mode because hive is blocking it (Fixed in DSE 4.7.0)
Upgrade to DSE4.7.3 or later