Summary
This article discusses a scenario where updates to a materialized view fails with an assertion error.
Applies to
- DataStax Enterprise 5.0.0 and 5.0.1
Symptoms
In this example, a DSE 5.0.1 node fails to startup as a result of failures to replay the commit logs. Here is a sample entry from the system.log
:
ERROR [main] 2018-06-25 12:34:56,789 CassandraDaemon.java:698 - Exception encountered during startup java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: We shouldn't have got there is the base row had no associated entry at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:50) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:372) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:630) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:519) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:414) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:148) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] ... Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: We shouldn't have got there is the base row had no associated entry ... Caused by: java.lang.AssertionError: We shouldn't have got there is the base row had no associated entry at org.apache.cassandra.db.view.ViewUpdateGenerator.computeLivenessInfoForEntry(ViewUpdateGenerator.java:455) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.db.view.ViewUpdateGenerator.updateEntry(ViewUpdateGenerator.java:273) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.db.view.ViewUpdateGenerator.addBaseTableUpdate(ViewUpdateGenerator.java:127) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.db.view.TableViews.addToViewUpdateGenerators(TableViews.java:403) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.db.view.TableViews.generateViewUpdates(TableViews.java:236) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.db.view.TableViews.pushViewReplicaUpdates(TableViews.java:140) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:485) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.db.Keyspace.applyFromCommitLog(Keyspace.java:394) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:616) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] ...
Cause
The issue occurs when updating the materialized view (MV) when the row in the base table contains a null value, i.e. the clustering column has been deleted (CASSANDRA-12247). Instead of updating the existing row in the materialized view, no update should take place on the MV.
Workaround
In the case where the replay of the commit logs is preventing DSE startup:
Step 1 - Move the contents of the commitlog
directory to another directory.
Step 2 - Start DSE.
Step 3 - Repair the node to synchronise with other replicas.
NOTE - This will only allow DSE to be started on the node. The underlying issue with the MV updates will continue in the background and get repeatedly reported in the logs.
Solution
The fix for CASSANDRA-12247 is included in DSE 5.0.2. Upgrade to the latest version of DSE 5.0 to take advantage of all the available fixes.