Summary
If there is not enough available memory for the chunk cache then this could cause DSE to fail to start
Applies to
- DataStax Enterprise 6.0
- DataStax Enterprise 6.7
- DataStax Enterprise 6.8
Symptoms
Upon starting the node the following error will be seen in the output.log
(package versions) or std out (tarball versions). Also this error will echo to the system.log
and debug.log
Exception (java.lang.UnsupportedOperationException) encountered during startup: Async read is not supported without caching.
java.lang.UnsupportedOperationException: Async read is not supported without caching.
at org.apache.cassandra.io.util.BufferManagingRebufferer.rebuffer(BufferManagingRebufferer.java:141)
at org.apache.cassandra.io.tries.Walker.<init>(Walker.java:66)
at org.apache.cassandra.io.sstable.format.trieindex.PartitionIndex$Reader.<init>(PartitionIndex.java:269)
at org.apache.cassandra.io.sstable.format.trieindex.PartitionIndex.openReader(PartitionIndex.java:221)
at org.apache.cassandra.io.sstable.format.trieindex.TrieIndexSSTableReader.getExactPosition(TrieIndexSSTableReader.java:440)
at org.apache.cassandra.io.sstable.format.AsyncPartitionReader.prepareRow(AsyncPartitionReader.java:310)
at org.apache.cassandra.io.sstable.format.AsyncPartitionReader.access$100(AsyncPartitionReader.java:81)
at org.apache.cassandra.io.sstable.format.AsyncPartitionReader$PartitionReader.performRead(AsyncPartitionReader.java:381)
at org.apache.cassandra.io.sstable.format.AsyncPartitionReader.readWithRetry(AsyncPartitionReader.java:246)
at org.apache.cassandra.io.sstable.format.AsyncPartitionReader.access$1000(AsyncPartitionReader.java:81)
at org.apache.cassandra.io.sstable.format.AsyncPartitionReader$PartitionReader.requestNext(AsyncPartitionReader.java:417)
at org.apache.cassandra.utils.flow.FlowSource.requestFirst(FlowSource.java:41)
at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.requestFirst(FlatMap.java:158)
at org.apache.cassandra.utils.flow.FlatMap.onNext(FlatMap.java:122)
at org.apache.cassandra.utils.flow.Flow$TakeWhile.onNext(Flow.java:423)
at org.apache.cassandra.utils.flow.Flow$IteratorSubscription.requestNext(Flow.java:948)
at org.apache.cassandra.utils.flow.FlowSource.requestFirst(FlowSource.java:41)
at org.apache.cassandra.utils.flow.FlowTransformNext.requestFirst(FlowTransformNext.java:46)
at org.apache.cassandra.utils.flow.FlatMap.requestFirst(FlatMap.java:76)
at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.requestFirst(FlatMap.java:158)
at org.apache.cassandra.utils.flow.FlatMap.onNext(FlatMap.java:122)
at org.apache.cassandra.utils.flow.Flow$IteratorSubscription.requestNext(Flow.java:948)
at org.apache.cassandra.utils.flow.FlatMap.requestNext(FlatMap.java:93)
at org.apache.cassandra.utils.flow.Flow$RequestLoopFlow.requestInLoop(Flow.java:1102)
at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.onComplete(FlatMap.java:218)
at org.apache.cassandra.utils.flow.FlowTransformBase.onComplete(FlowTransformBase.java:44)
at org.apache.cassandra.utils.flow.FlatMap.onComplete(FlatMap.java:138)
at org.apache.cassandra.utils.flow.Flow$IteratorSubscription.requestNext(Flow.java:946)
at org.apache.cassandra.utils.flow.FlatMap.requestNext(FlatMap.java:93)
at org.apache.cassandra.utils.flow.Flow$RequestLoopFlow.requestInLoop(Flow.java:1102)
at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.onComplete(FlatMap.java:218)
at org.apache.cassandra.utils.flow.Flow$5.requestFirst(Flow.java:2017)
at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.requestFirst(FlatMap.java:158)
at org.apache.cassandra.utils.flow.FlatMap.onNext(FlatMap.java:122)
at org.apache.cassandra.utils.flow.Flow$IteratorSubscription.requestNext(Flow.java:948)
at org.apache.cassandra.utils.flow.FlowSource.requestFirst(FlowSource.java:41)
at org.apache.cassandra.utils.flow.FlatMap.requestFirst(FlatMap.java:76)
at org.apache.cassandra.utils.flow.FlowTransformNext.requestFirst(FlowTransformNext.java:46)
at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.requestFirst(FlatMap.java:158)
at org.apache.cassandra.utils.flow.FlatMap.onNext(FlatMap.java:122)
at org.apache.cassandra.utils.flow.Flow$IteratorSubscription.requestNext(Flow.java:948)
at org.apache.cassandra.utils.flow.FlowSource.requestFirst(FlowSource.java:41)
at org.apache.cassandra.utils.flow.FlatMap.requestFirst(FlatMap.java:76)
at org.apache.cassandra.utils.flow.FlatMap.requestFirst(FlatMap.java:76)
at org.apache.cassandra.utils.flow.FlowTransform.requestFirst(FlowTransform.java:44)
at org.apache.cassandra.utils.flow.FlowTransformNext.requestFirst(FlowTransformNext.java:46)
at org.apache.cassandra.utils.flow.Flow$1DoOnClose.requestFirst(Flow.java:569)
at org.apache.cassandra.utils.flow.FlowTransformNext.requestFirst(FlowTransformNext.java:46)
at org.apache.cassandra.utils.flow.FlowTransformNext.requestFirst(FlowTransformNext.java:46)
at org.apache.cassandra.utils.flow.Flow$1DoOnClose.requestFirst(Flow.java:569)
at org.apache.cassandra.db.filter.DataLimits$1Truncate.requestFirst(DataLimits.java:383)
at org.apache.cassandra.utils.flow.Flow$1Using.requestFirst(Flow.java:2225)
at org.apache.cassandra.utils.flow.FlatMap.requestFirst(FlatMap.java:76)
at org.apache.cassandra.utils.flow.FlatMap.requestFirst(FlatMap.java:76)
at org.apache.cassandra.utils.flow.FlowTransform.requestFirst(FlowTransform.java:44)
at org.apache.cassandra.utils.flow.Flow.requestFirst(Flow.java:98)
at org.apache.cassandra.utils.flow.Flow$ReduceSubscriber.start(Flow.java:1175)
at org.apache.cassandra.utils.flow.Flow$1SingleFromFlow.subscribeActual(Flow.java:1554)
at io.reactivex.Single.subscribe(Single.java:3603)
at io.reactivex.internal.operators.single.SingleMap.subscribeActual(SingleMap.java:34)
at io.reactivex.Single.subscribe(Single.java:3603)
at io.reactivex.internal.operators.single.SingleMap.subscribeActual(SingleMap.java:34)
at io.reactivex.Single.subscribe(Single.java:3603)
at io.reactivex.Single.blockingGet(Single.java:2834)
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspaces(SchemaKeyspace.java:1785)
at org.apache.cassandra.schema.SchemaManager.merge(SchemaManager.java:893)
at org.apache.cassandra.schema.SchemaManager.mergeAndAnnounceVersion(SchemaManager.java:877)
at org.apache.cassandra.schema.MigrationManager.lambda$announce$7(MigrationManager.java:345)
at io.reactivex.internal.operators.completable.CompletableFromRunnable.subscribeActual(CompletableFromRunnable.java:35)
at io.reactivex.Completable.subscribe(Completable.java:2302)
at io.reactivex.internal.operators.completable.CompletableDefer.subscribeActual(CompletableDefer.java:43)
at io.reactivex.Completable.subscribe(Completable.java:2302)
at io.reactivex.internal.operators.completable.CompletableResumeNext.subscribeActual(CompletableResumeNext.java:41)
at io.reactivex.Completable.subscribe(Completable.java:2302)
at io.reactivex.internal.operators.completable.CompletableAndThenCompletable.subscribeActual(CompletableAndThenCompletable.java:35)
at io.reactivex.Completable.subscribe(Completable.java:2302)
at io.reactivex.internal.operators.completable.CompletableAndThenCompletable.subscribeActual(CompletableAndThenCompletable.java:35)
at io.reactivex.Completable.subscribe(Completable.java:2302)
at io.reactivex.internal.operators.completable.CompletableAndThenCompletable.subscribeActual(CompletableAndThenCompletable.java:35)
at io.reactivex.Completable.subscribe(Completable.java:2302)
at io.reactivex.Completable.blockingAwait(Completable.java:1219)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1322)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:806)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:730)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:405)
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:539)
at org.apache.cassandra.service.CassandraDaemon.activate0(CassandraDaemon.java:733)
at org.apache.cassandra.service.CassandraDaemon.access$100(CassandraDaemon.java:85)
at org.apache.cassandra.service.CassandraDaemon$3.run(CassandraDaemon.java:694)
Suppressed: org.apache.cassandra.utils.flow.Flow$FlowException: Flow call chain:
IteratorSubscription
flatMap org.apache.cassandra.utils.flow.Concat$$Lambda$1330/626043493@39a415b6(Concat.java:31)
flatMap org.apache.cassandra.db.SinglePartitionReadCommand$$Lambda$713/1352495089@54c4d084(SinglePartitionReadCommand.java:967)
Reduce org.apache.cassandra.db.SinglePartitionReadCommand$$Lambda$714/1699745871@6ffe35a2(SinglePartitionReadCommand.java:968)
Map org.apache.cassandra.db.SinglePartitionReadCommand$$Lambda$715/1743013735@557aaa35(SinglePartitionReadCommand.java:969)
DoOnClose org.apache.cassandra.db.SinglePartitionReadCommand$$Lambda$716/174912123@197385fd(SinglePartitionReadCommand.java:528)
Map org.apache.cassandra.db.ReadCommand$$Lambda$597/1801732455@5603270a(ReadCommand.java:488)
Map org.apache.cassandra.db.ReadCommand$$Lambda$668/1817791313@f19fa58(ReadCommand.java:405)
DoOnClose org.apache.cassandra.db.ReadCommand$$Lambda$669/459043063@644ac759(ReadCommand.java:449)
Truncate
using org.apache.cassandra.db.ReadCommand$$Lambda$575/377038949@334770e4(ReadCommand.java:305)
flatMap org.apache.cassandra.db.rows.FlowablePartitions$$Lambda$585/1422157000@632215da(FlowablePartitions.java:660)
flatMap org.apache.cassandra.cql3.statements.SelectStatement$$Lambda$586/377640279@322e24a4(SelectStatement.java:1315)
Reduce org.apache.cassandra.utils.flow.Flow$$Lambda$583/140398477@814eefa(Flow.java:1707)
ReduceToSingle org.apache.cassandra.utils.flow.Flow$$Lambda$588/1534105471@42e6031c(Flow.java:1566)
at org.apache.cassandra.utils.flow.Flow.wrapException(Flow.java:2639)
at org.apache.cassandra.utils.flow.Flow$ReduceSubscriber.onError(Flow.java:1230)
at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
at org.apache.cassandra.utils.flow.FlatMap.onError(FlatMap.java:133)
at org.apache.cassandra.utils.flow.FlatMap.onError(FlatMap.java:133)
at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
at org.apache.cassandra.utils.flow.FlatMap.onError(FlatMap.java:133)
at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.onError(FlatMap.java:185)
at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.onError(FlatMap.java:185)
at org.apache.cassandra.io.sstable.format.AsyncPartitionReader$PartitionReader.onError(AsyncPartitionReader.java:407)
at org.apache.cassandra.io.sstable.format.AsyncPartitionReader.readWithRetry(AsyncPartitionReader.java:275)
... 80 more
Cause
Although the error looks quite convoluted, what this means is that there is no memory in the chunk cache to read in the SSTable(s). Typically this is when a too high heap setting is (perhaps incorrectly) set for the node and there is not sufficient space left for the cache. In this instance 50GB heap was set on a 64GB node.
Solution
Simply setting a more sensible heap would suffice here. We could also change the chunk cache settings but this isn’t always necessary (Chunk cache settings are discussed in the docs for recommended production settings
Further info
An internal jira DB-4755 was raised to track an improvement to add a more intuitive error message here.