DataStax Help Center

Streaming operations throw "java.lang.AssertionError: Memory was freed" error

Summary

An error may sometimes be observed during streaming operations as outlined below.

Symptoms

The following error may be seen in the /var/log/cassandra/system.log:

ERROR [STREAM-OUT-/10.1.2.3] 2016-06-24 13:44:06,041 StreamSession.java:505 - [Stream #35785710-39e2-11e6-b857-ef46ce3e2503] Streaming error occurred 
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:97) ~[cassandra-all-2.1.14.1272.jar:2.1.14.1272]
at org.apache.cassandra.io.util.Memory.getLong(Memory.java:249) ~[cassandra-all-2.1.14.1272.jar:2.1.14.1272]
at org.apache.cassandra.io.compress.CompressionMetadata.getTotalSizeForSections(CompressionMetadata.java:247) ~[cassandra-all-2.1.14.1272.jar:2.1.14.1272]
at org.apache.cassandra.streaming.messages.FileMessageHeader.size(FileMessageHeader.java:112) ~[cassandra-all-2.1.14.1272.jar:2.1.14.1272]
at org.apache.cassandra.streaming.StreamSession.fileSent(StreamSession.java:546) ~[cassandra-all-2.1.14.1272.jar:2.1.14.1272]
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50) ~[cassandra-all-2.1.14.1272.jar:2.1.14.1272]
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41) ~[cassandra-all-2.1.14.1272.jar:2.1.14.1272]
at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45) ~[cassandra-all-2.1.14.1272.jar:2.1.14.1272]
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:358) ~[cassandra-all-2.1.14.1272.jar:2.1.14.1272]
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:338) ~[cassandra-all-2.1.14.1272.jar:2.1.14.1272]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]

Cause

For Cassandra operations such as bootstrap or repair, streaming sessions could be prematurely marked ​as failed ​by the source node if the stream does not complete within the streaming_socket_timeout_in_ms period. Common causes for these are very large sstables and/or slow ​transfers.

​In a streaming session, the source node sends the data files to a receiving node and at the same time starts calculating ​the header size for the metrics. When the streaming timeout is reached, the source node ​marks the session as failed and the reference to the session subsequently gets garbage collected in the JVM since it is already obsolete. As a consequence, the header calculation fails with Memory was freed as reported in CASSANDRA-11345.

Workaround

If the streams are failing as a result of larger sstables or if the network is determined to be slow, a workaround can be to temporarily increase the streaming_socket_timeout_in_ms in the cassandra.yaml to a higher value to allow the transfer to complete.

In some cases with larger files or slower network bandwidth a timeout of 48 hours (172800000 ms) has been known to work.

Solution

At the time of writing this note. This fix is not yet in DSE4.8.x. The following internal jira covers this:

DSP-9660 - AE "Memory was freed" during decommission

For up to date information on fixes in DSE4.8.x see the Cassandra release notes

See also

How to performance tune data streaming activities like repair and bootstrap

FAQ - How to reduce the impact of streaming errors or failures

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk