DataStax Help Center

Errors in Apache Cassandra, DSE, and OpsCenter Due to Insufficent Resource Limit Settings

Insufficient resource limits can result in a number of errors in Apache Cassandra, DSE, and OpsCenter, such as the following:

In Cassandra due to insufficient as (address space) or memlock setting:

ERROR [SSTableBatchOpen:1] 2012-07-25 15:46:02,913 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[SSTableBatchOpen:1,5,main] java.io.IOError: java.io.IOException: Map failed  at 

In Cassandra due to insufficient memlock settings:

WARN [main] 2011-06-15 09:58:56,861 CLibrary.java (line 118) Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.

In Cassandra due to insufficient nofiles setting:

WARN 05:13:43,644 Transport error occurred during acceptance of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files 

In Cassandra due to insufficient nofiles setting:

ERROR [MutationStage:11] 2012-04-30 09:46:08,102 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MutationStage:11,5,main] java.lang.OutOfMemoryError: unable to create new native thread 

In OpsCenter, due to insufficient nofiles:

2012-08-13 11:22:51-0400 [] INFO: Could not accept new connection (EMFILE)

These settings can be permanently set via the /etc/security/limits.conf file. You can view the limits using the 'ulimit -a' command, and they can also be temporarily set using this command.  The following are recommended limits settings for your limits.conf file when using Cassandra and OpsCenter:

* soft nofile 32768
* hard nofile 32768
root soft nofile 32768
root hard nofile 32768
* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited
* soft as unlimited
* hard as unlimited
root soft as unlimited
root hard as unlimited

In addition, you may need to run the following command to enable more mapping (at the command line; do not add to limits.conf):

sysctl -w vm.max_map_count = 1048575

 The following error can indicate needing to increase max_map_count:

Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed. 
 
 
Was this article helpful?
5 out of 5 found this helpful
Have more questions? Submit a request

Comments

  • Avatar
    Ravinder Matte

    Is there any command to find out present cassandra map count? We have vm.max_map_count set to 131072, but still failing with following error message.
    at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:200)
    at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:168)
    at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:345)
    at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:335)
    at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:126)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: java.io.IOException: Map failed
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:907)
    at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192)
    ... 9 more
    Caused by: java.lang.OutOfMemoryError: Map failed
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:904)
    ... 10 more
    ERROR [StreamReceiveTask:17414] 2015-12-02 00:43:50,881 StorageService.java (line 372) Stopping gossiper
    WARN [StreamReceiveTask:17414] 2015-12-02 00:43:50,881 StorageService.java (line 280) Stopping gossip by operator request
    INFO [StreamReceiveTask:17414] 2015-12-02 00:43:50,881 Gossiper.java (line 1330) Announcing shutdown
    ERROR [StreamReceiveTask:17414] 2015-12-02 00:43:52,882 StorageService.java (line 377) Stopping RPC server
    INFO [StreamReceiveTask:17414] 2015-12-02 00:43:52,882 ThriftServer.java (line 141) Stop listening to thrift clients
    ERROR [StreamReceiveTask:17414] 2015-12-02 00:43:52,884 StorageService.java (line 382) Stopping native transport

Powered by Zendesk