DataStax Help Center

Node startup after DSE 4.7 upgrade returns "SSTable Data is not mmapped, is compression turned off?"

Summary

After upgrading nodes to DataStax Enterprise 4.7, while loading tables during startup some return a java.lang.IllegalArgumentException.

Symptoms

After upgrading a node(s) from DSE 4.0.x/4.5.x/4.6.x to DSE 4.7, the following errors are reported in the system.log while opening the data files for some tables:

INFO 05:10:46 Opening /var/lib/cassandra/data/appKS/bids/appKS-bids-jb-123 (4567 bytes) 
ERROR 05:10:46 Table appKS.bids uses MemoryOnlyStrategy and should have `compression` set to '' (got LZ4Compressor). mlock will be disabled
ERROR 05:10:46 Table appKS.account_last_indexed_events uses MemoryOnlyStrategy and should have `compression` set to '' (got LZ4Compressor). mlock will be disabled
ERROR 05:10:46 There was a problem mlocking a segment for file SSTableReader(path='/var/lib/cassandra/data/appKS/bids/appKS-bids-jb-123-Data.db')
ERROR 05:10:46 Exception:
java.lang.IllegalArgumentException: SSTable Data is not mmapped, is compression turned off? CompressedPoolingSegmentedFile(path='/var/lib/cassandra/data/appKS/bids/appKS-bids-jb-123-Data.db', length=56789)
at org.apache.cassandra.db.compaction.MemoryOnlyStrategy.getDataFile(MemoryOnlyStrategy.java:251) ~[dse-4.7.0.jar:2.1.5.469]
at org.apache.cassandra.db.compaction.MemoryOnlyStrategy.addSSTable(MemoryOnlyStrategy.java:367) ~[dse-4.7.0.jar:2.1.5.469]
at org.apache.cassandra.db.compaction.WrappingCompactionStrategy.startup(WrappingCompactionStrategy.java:331) [cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.db.compaction.WrappingCompactionStrategy.reloadCompactionStrategy(WrappingCompactionStrategy.java:165) [cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.db.compaction.WrappingCompactionStrategy.<init>(WrappingCompactionStrategy.java:52) [cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:330) [cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:479) [cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:450) [cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:328) [cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:279) [cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:121) [cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:98) [cassandra-all-2.1.5.469.jar:2.1.5.469]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324) [cassandra-all-2.1.5.469.jar:2.1.5.469]
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:329) [dse-4.7.0.jar:4.7.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:536) [cassandra-all-2.1.5.469.jar:2.1.5.469]
at com.datastax.bdp.DseModule.main(DseModule.java:75) [dse-4.7.0.jar:4.7.0]

Cause

The affected tables are in-memory tables. DSE 4.0 introduced the in-memory option for storing and accessing data exclusively from memory for specific use cases where a table is relatively small, has very high traffic and requires extremely low latency.

In DSE 4.7, one of the enhancements is the implementation of the MemoryOnlyStrategy to use mmap() based file access (enhancement ID DSP-4650). But in the current implementation, compressed SSTables are not supported for in-memory tables in DSE 4.7 so compression will need to be disabled.

Disabling compression is a prerequisite to upgrading as documented in Upgrading to DataStax Enterprise 4.7.

Workaround

Follow these steps to recover the affected tables:

Step 1 - For each in-memory table, disable compression as follows:

cqlsh>ALTER TABLE <keyspace>.<table> WITH compression = {'sstable_compression' : ''} ;

Step 2 - For each in-memory table, uncompress the existing data files by forcing a rewrite of the SSTables with upgradesstables:

$ nodetool upgradesstables -a <keyspace> <table>

Solution

Remember to always read the upgrade documentation for instructions specific to your cluster's configuration.

In this case despite having the workaround, it is best to disable compression and upgrade the SSTables before proceeding with the software upgrade.

See also

In-memory feature in DataStax Enterprise 4.7.

Version-specific instructions for Upgrading to DataStax Enterprise 4.7.

Release notes for DataStax Enterprise 4.7.

Was this article helpful?
2 out of 2 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk