Full ERROR Message Example
There are 2 kinds of ERRORs
ERROR1
ERROR [main] 2020-08-26 13:14:24,254 LogTransaction.java:477 - Unexpected disk state: failed to read transaction log [mc_txn_flush_05dn3n40-13cb-11e8-b3dc-effd8e1239f3.log in /var/lib/cassandra/data/data/system/sstable_activity-5a1ff267ac3edsf342dsfd34wm103c65e] Files and contents follow: /var/lib/cassandra/data/data/system/sstable_activity-5a1ff267ac3edsf342dsfd34wm103c65e/mc_txn_flush_05dn3n40-13cb-11e8-b3dc-effd8e1239f3.log ADD:[/var/lib/cassandra/data/data/system/sstable_activity-5a1ff267ac3edsf342dsfd34wm103c65e/mc-7342-big,0,8][3244895344] COMMIT:[,0,0][2613695470] ***This record should have been the last one in all replicas ABORT:[,0,0][737434448]
ERROR2
ERROR [main] 2020-09-02 14:14:24,989 LogTransaction.java (line 492) Unexpected disk state: failed to read transaction log [mc_txn_compaction_69307590-e080-11e9-9e4d-89535f6a3331.log in /var/local/cassandra/data/example/table-3fdd6e00b8addfkdsf4fdsf45652k3a41] Files and contents follow: /var/local/cassandra/data/data/0/example/table-3fdd6e00b8addfkdsf4fdsf45652k3a41/mc_txn_compaction_69307590-e080-11e9-9e4d-89535f6a3331.log ADD:[/var/lib/cassandra/data/example/table-3fdd6e00b8addfkdsf4fdsf45652k3a41/mc-4353-big,0,8][2595327302] REMOVE:[/var/lib/cassandra/data/example/table-3fdd6e00b8addfkdsf4fdsf45652k3a41/mc-5432-big,1569517542000,8][3604952] REMOVE:[/var/lib/cassandra/data/example/table-3fdd6e00b8addfkdsf4fdsf45652k3a41/mc-4355-big,1569517645500,8][1703752985] REMOVE:[/var/lib/cassandra/data/example/table-3fdd6e00b8addfkdsf4fdsf45652k3a41/mc-4358-big,1569517745000,8][717261549] ***Unexpected files detected for sstable [mc-86650-big-]: last update time [17:09:22] should have been [17:09:33] <===================== REMOVE:[/var/lib/cassandra/data/example/table-3fdd6e00b8addfkdsf4fdsf45652k3a41/mc-5500-big,1569517734000,8][1388813582] COMMIT:[,0,0][26132333470]
What does this ERROR message mean?
This error relates to the transaction log used for tracking the compaction status of a sstable. The error above indicates cassandra cannot read the transaction log, there may have some problematic transaction logs.
In addition, transaction logs are read during the start up, to avoid the file system errors, such as disk corruption, a failed read on transaction log will always prevent the node to start.
Why does this error occur?
ERROR1:
This is due to a bug that transaction log can only either end with the keyword COMMIT or ABORT, but not both.
ERROR2:
Some components of the sstable that reports `Unexpected files detected for sstable` error are missing.
How do you fix this ERROR?
ERROR1:
This is fixed in DSE versions 5.0.14+, 5.1.8+
ERROR2:
Run a ls -l under that table data directory, and observe the components of the problem sstable and check if there are any missing files.
Remove the rest of files associated to that sstable, then start node and observe if the error goes away.
If the node still fails to start, please collect the logs and contact DataStax Support to investigate further.
Comments
0 comments
Please sign in to leave a comment.