Insufficient resource limits can result in a number of errors in Apache Cassandra, DSE, and OpsCenter, such as the following:
In Cassandra due to insufficient as (address space) or memlock setting:
ERROR [SSTableBatchOpen:1] 2012-07-25 15:46:02,913 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[SSTableBatchOpen:1,5,main] java.io.IOError: java.io.IOException: Map failed at
In Cassandra due to insufficient memlock settings:
WARN [main] 2011-06-15 09:58:56,861 CLibrary.java (line 118) Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
In Cassandra due to insufficient nofiles setting:
WARN 05:13:43,644 Transport error occurred during acceptance of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files
In Cassandra due to insufficient nofiles setting:
ERROR [MutationStage:11] 2012-04-30 09:46:08,102 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MutationStage:11,5,main] java.lang.OutOfMemoryError: unable to create new native thread
In OpsCenter, due to insufficient nofiles:
2012-08-13 11:22:51-0400 [] INFO: Could not accept new connection (EMFILE)
These settings can be permanently set via the /etc/security/limits.conf file. You can view the limits using the 'ulimit -a' command, and they can also be temporarily set using this command. The following are recommended limits settings for your limits.conf file when using Cassandra and OpsCenter:
* soft nofile 32768
* hard nofile 32768
root soft nofile 32768
root hard nofile 32768
* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited
* soft as unlimited
* hard as unlimited
root soft as unlimited
root hard as unlimited
In addition, you may need to run the following command to enable more mapping (at the command line; do not add to limits.conf):
sysctl -w vm.max_map_count = 1048575
The following error can indicate needing to increase max_map_count:
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
Comments
1 comment
Is there any command to find out present cassandra map count? We have vm.max_map_count set to 131072, but still failing with following error message.
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:200)
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:168)
at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:345)
at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:335)
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:126)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:907)
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192)
... 9 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:904)
... 10 more
ERROR [StreamReceiveTask:17414] 2015-12-02 00:43:50,881 StorageService.java (line 372) Stopping gossiper
WARN [StreamReceiveTask:17414] 2015-12-02 00:43:50,881 StorageService.java (line 280) Stopping gossip by operator request
INFO [StreamReceiveTask:17414] 2015-12-02 00:43:50,881 Gossiper.java (line 1330) Announcing shutdown
ERROR [StreamReceiveTask:17414] 2015-12-02 00:43:52,882 StorageService.java (line 377) Stopping RPC server
INFO [StreamReceiveTask:17414] 2015-12-02 00:43:52,882 ThriftServer.java (line 141) Stop listening to thrift clients
ERROR [StreamReceiveTask:17414] 2015-12-02 00:43:52,884 StorageService.java (line 382) Stopping native transport
Please sign in to leave a comment.