Summary

If there is not enough available memory for the chunk cache then this could cause DSE to fail to start

Applies to

• DataStax Enterprise 6.0
• DataStax Enterprise 6.7
• DataStax Enterprise 6.8

Symptoms

Upon starting the node the following error will be seen in the output.log (package versions) or std out (tarball versions). Also this error will echo to the system.log and debug.log

Exception (java.lang.UnsupportedOperationException) encountered during startup: Async read is not supported without caching.
java.lang.UnsupportedOperationException: Async read is not supported without caching.
        at org.apache.cassandra.io.util.BufferManagingRebufferer.rebuffer(BufferManagingRebufferer.java:141)
        at org.apache.cassandra.io.tries.Walker.<init>(Walker.java:66)
        at org.apache.cassandra.io.sstable.format.trieindex.PartitionIndex$Reader.<init>(PartitionIndex.java:269)
        at org.apache.cassandra.io.sstable.format.trieindex.PartitionIndex.openReader(PartitionIndex.java:221)
        at org.apache.cassandra.io.sstable.format.trieindex.TrieIndexSSTableReader.getExactPosition(TrieIndexSSTableReader.java:440)
        at org.apache.cassandra.io.sstable.format.AsyncPartitionReader.prepareRow(AsyncPartitionReader.java:310)
        at org.apache.cassandra.io.sstable.format.AsyncPartitionReader.access$100(AsyncPartitionReader.java:81)
        at org.apache.cassandra.io.sstable.format.AsyncPartitionReader$PartitionReader.performRead(AsyncPartitionReader.java:381)
        at org.apache.cassandra.io.sstable.format.AsyncPartitionReader.readWithRetry(AsyncPartitionReader.java:246)
        at org.apache.cassandra.io.sstable.format.AsyncPartitionReader.access$1000(AsyncPartitionReader.java:81)
        at org.apache.cassandra.io.sstable.format.AsyncPartitionReader$PartitionReader.requestNext(AsyncPartitionReader.java:417)
        at org.apache.cassandra.utils.flow.FlowSource.requestFirst(FlowSource.java:41)
        at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.requestFirst(FlatMap.java:158)
        at org.apache.cassandra.utils.flow.FlatMap.onNext(FlatMap.java:122)
        at org.apache.cassandra.utils.flow.Flow$TakeWhile.onNext(Flow.java:423)
        at org.apache.cassandra.utils.flow.Flow$IteratorSubscription.requestNext(Flow.java:948)
        at org.apache.cassandra.utils.flow.FlowSource.requestFirst(FlowSource.java:41)
        at org.apache.cassandra.utils.flow.FlowTransformNext.requestFirst(FlowTransformNext.java:46)
        at org.apache.cassandra.utils.flow.FlatMap.requestFirst(FlatMap.java:76)
        at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.requestFirst(FlatMap.java:158)
        at org.apache.cassandra.utils.flow.FlatMap.onNext(FlatMap.java:122)
        at org.apache.cassandra.utils.flow.Flow$IteratorSubscription.requestNext(Flow.java:948)
        at org.apache.cassandra.utils.flow.FlatMap.requestNext(FlatMap.java:93)
        at org.apache.cassandra.utils.flow.Flow$RequestLoopFlow.requestInLoop(Flow.java:1102)
        at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.onComplete(FlatMap.java:218)
        at org.apache.cassandra.utils.flow.FlowTransformBase.onComplete(FlowTransformBase.java:44)
        at org.apache.cassandra.utils.flow.FlatMap.onComplete(FlatMap.java:138)
        at org.apache.cassandra.utils.flow.Flow$IteratorSubscription.requestNext(Flow.java:946)
        at org.apache.cassandra.utils.flow.FlatMap.requestNext(FlatMap.java:93)
        at org.apache.cassandra.utils.flow.Flow$RequestLoopFlow.requestInLoop(Flow.java:1102)
        at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.onComplete(FlatMap.java:218)
        at org.apache.cassandra.utils.flow.Flow$5.requestFirst(Flow.java:2017)
        at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.requestFirst(FlatMap.java:158)
        at org.apache.cassandra.utils.flow.FlatMap.onNext(FlatMap.java:122)
        at org.apache.cassandra.utils.flow.Flow$IteratorSubscription.requestNext(Flow.java:948)
        at org.apache.cassandra.utils.flow.FlowSource.requestFirst(FlowSource.java:41)
        at org.apache.cassandra.utils.flow.FlatMap.requestFirst(FlatMap.java:76)
        at org.apache.cassandra.utils.flow.FlowTransformNext.requestFirst(FlowTransformNext.java:46)
        at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.requestFirst(FlatMap.java:158)
        at org.apache.cassandra.utils.flow.FlatMap.onNext(FlatMap.java:122)
        at org.apache.cassandra.utils.flow.Flow$IteratorSubscription.requestNext(Flow.java:948)
        at org.apache.cassandra.utils.flow.FlowSource.requestFirst(FlowSource.java:41)
        at org.apache.cassandra.utils.flow.FlatMap.requestFirst(FlatMap.java:76)
        at org.apache.cassandra.utils.flow.FlatMap.requestFirst(FlatMap.java:76)
        at org.apache.cassandra.utils.flow.FlowTransform.requestFirst(FlowTransform.java:44)
        at org.apache.cassandra.utils.flow.FlowTransformNext.requestFirst(FlowTransformNext.java:46)
        at org.apache.cassandra.utils.flow.Flow$1DoOnClose.requestFirst(Flow.java:569)
        at org.apache.cassandra.utils.flow.FlowTransformNext.requestFirst(FlowTransformNext.java:46)
        at org.apache.cassandra.utils.flow.FlowTransformNext.requestFirst(FlowTransformNext.java:46)
        at org.apache.cassandra.utils.flow.Flow$1DoOnClose.requestFirst(Flow.java:569)
        at org.apache.cassandra.db.filter.DataLimits$1Truncate.requestFirst(DataLimits.java:383)
        at org.apache.cassandra.utils.flow.Flow$1Using.requestFirst(Flow.java:2225)
        at org.apache.cassandra.utils.flow.FlatMap.requestFirst(FlatMap.java:76)
        at org.apache.cassandra.utils.flow.FlatMap.requestFirst(FlatMap.java:76)
        at org.apache.cassandra.utils.flow.FlowTransform.requestFirst(FlowTransform.java:44)
        at org.apache.cassandra.utils.flow.Flow.requestFirst(Flow.java:98)
        at org.apache.cassandra.utils.flow.Flow$ReduceSubscriber.start(Flow.java:1175)
        at org.apache.cassandra.utils.flow.Flow$1SingleFromFlow.subscribeActual(Flow.java:1554)
        at io.reactivex.Single.subscribe(Single.java:3603)
        at io.reactivex.internal.operators.single.SingleMap.subscribeActual(SingleMap.java:34)
        at io.reactivex.Single.subscribe(Single.java:3603)
        at io.reactivex.internal.operators.single.SingleMap.subscribeActual(SingleMap.java:34)
        at io.reactivex.Single.subscribe(Single.java:3603)
        at io.reactivex.Single.blockingGet(Single.java:2834)
        at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspaces(SchemaKeyspace.java:1785)
        at org.apache.cassandra.schema.SchemaManager.merge(SchemaManager.java:893)
        at org.apache.cassandra.schema.SchemaManager.mergeAndAnnounceVersion(SchemaManager.java:877)
        at org.apache.cassandra.schema.MigrationManager.lambda$announce$7(MigrationManager.java:345)
        at io.reactivex.internal.operators.completable.CompletableFromRunnable.subscribeActual(CompletableFromRunnable.java:35)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableDefer.subscribeActual(CompletableDefer.java:43)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableResumeNext.subscribeActual(CompletableResumeNext.java:41)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableAndThenCompletable.subscribeActual(CompletableAndThenCompletable.java:35)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableAndThenCompletable.subscribeActual(CompletableAndThenCompletable.java:35)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableAndThenCompletable.subscribeActual(CompletableAndThenCompletable.java:35)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.Completable.blockingAwait(Completable.java:1219)
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1322)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:806)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:730)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:405)
        at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:539)
        at org.apache.cassandra.service.CassandraDaemon.activate0(CassandraDaemon.java:733)
        at org.apache.cassandra.service.CassandraDaemon.access$100(CassandraDaemon.java:85)
        at org.apache.cassandra.service.CassandraDaemon$3.run(CassandraDaemon.java:694)
        Suppressed: org.apache.cassandra.utils.flow.Flow$FlowException: Flow call chain:
        IteratorSubscription
        flatMap             org.apache.cassandra.utils.flow.Concat$$Lambda$1330/626043493@39a415b6(Concat.java:31)
        flatMap             org.apache.cassandra.db.SinglePartitionReadCommand$$Lambda$713/1352495089@54c4d084(SinglePartitionReadCommand.java:967)
        Reduce              org.apache.cassandra.db.SinglePartitionReadCommand$$Lambda$714/1699745871@6ffe35a2(SinglePartitionReadCommand.java:968)
        Map                 org.apache.cassandra.db.SinglePartitionReadCommand$$Lambda$715/1743013735@557aaa35(SinglePartitionReadCommand.java:969)
        DoOnClose           org.apache.cassandra.db.SinglePartitionReadCommand$$Lambda$716/174912123@197385fd(SinglePartitionReadCommand.java:528)
        Map                 org.apache.cassandra.db.ReadCommand$$Lambda$597/1801732455@5603270a(ReadCommand.java:488)
        Map                 org.apache.cassandra.db.ReadCommand$$Lambda$668/1817791313@f19fa58(ReadCommand.java:405)
        DoOnClose           org.apache.cassandra.db.ReadCommand$$Lambda$669/459043063@644ac759(ReadCommand.java:449)
        Truncate
        using               org.apache.cassandra.db.ReadCommand$$Lambda$575/377038949@334770e4(ReadCommand.java:305)
        flatMap             org.apache.cassandra.db.rows.FlowablePartitions$$Lambda$585/1422157000@632215da(FlowablePartitions.java:660)
        flatMap             org.apache.cassandra.cql3.statements.SelectStatement$$Lambda$586/377640279@322e24a4(SelectStatement.java:1315)
        Reduce              org.apache.cassandra.utils.flow.Flow$$Lambda$583/140398477@814eefa(Flow.java:1707)
        ReduceToSingle      org.apache.cassandra.utils.flow.Flow$$Lambda$588/1534105471@42e6031c(Flow.java:1566)

                at org.apache.cassandra.utils.flow.Flow.wrapException(Flow.java:2639)
                at org.apache.cassandra.utils.flow.Flow$ReduceSubscriber.onError(Flow.java:1230)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlatMap.onError(FlatMap.java:133)
                at org.apache.cassandra.utils.flow.FlatMap.onError(FlatMap.java:133)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlowTransformBase.onError(FlowTransformBase.java:38)
                at org.apache.cassandra.utils.flow.FlatMap.onError(FlatMap.java:133)
                at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.onError(FlatMap.java:185)
                at org.apache.cassandra.utils.flow.FlatMap$FlatMapChild.onError(FlatMap.java:185)
                at org.apache.cassandra.io.sstable.format.AsyncPartitionReader$PartitionReader.onError(AsyncPartitionReader.java:407)
                at org.apache.cassandra.io.sstable.format.AsyncPartitionReader.readWithRetry(AsyncPartitionReader.java:275)
                ... 80 more

Cause

Although the error looks quite convoluted, what this means is that there is no memory in the chunk cache to read in the SSTable(s). Typically this is when a too high heap setting is (perhaps incorrectly) set for the node and there is not sufficient space left for the cache. In this instance 50GB heap was set on a 64GB node.

Solution

Simply setting a more sensible heap would suffice here. We could also change the chunk cache settings but this isn’t always necessary (Chunk cache settings are discussed in the docs for recommended production settings

Further info

An internal jira DB-4755 was raised to track an improvement to add a more intuitive error message here.