DataStax Help Center

Running tools such as sstable2json causes gossip and commitlog permission issues


This article discusses an issue where the use of offline tools can cause gossip issues and commit logs to be written with the wrong permissions.


Certain nodes in a cluster have trouble gossiping with other nodes with the following entries reported in the system.log:

WARN [GossipTasks:1] 2016-01-11 16:59:38,294 - Gossip stage has 290 pending tasks; skipping status check (no nodes will be marked down) 
WARN [GossipTasks:1] 2016-01-11 16:59:39,395 - Gossip stage has 292 pending tasks; skipping status check (no nodes will be marked down) 
WARN [GossipTasks:1] 2016-01-11 16:59:40,495 - Gossip stage has 296 pending tasks; skipping status check (no nodes will be marked down)

Further investigation identified gossip failures as a result of permission issues on commit logs, i.e. commit logs could not be flushed to disk preventing updates to system tables:

ERROR [COMMIT-LOG-ALLOCATOR] 2016-01-11 16:20:48,886 - Failed managing commit log segments. Commit disk failure policy is stop; terminating thread /var/lib/cassandra/commitlog/CommitLog-4-1452554366748.log (Permission denied)
        at org.apache.cassandra.db.commitlog.CommitLogSegment.( ~[cassandra-all-]
        at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$ ~[cassandra-all-]
        at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$ ~[cassandra-all-]
        at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow( ~[cassandra-all-]
        at [cassandra-all-]
        at [na:1.8.0_51]
Caused by: /var/lib/cassandra/commitlog/CommitLog-4-1452554366748.log (Permission denied)
        at Method) ~[na:1.8.0_51]
        at ~[na:1.8.0_51]
        at ~[na:1.8.0_51]
        at org.apache.cassandra.db.commitlog.CommitLogSegment.( ~[cassandra-all-]
        ... 5 common frames omitted


There is a known issue with running tools such as sstable2json which can inadvertently generate commit logs with incorrect permissions or owned by the root user as reported in CASSANDRA-8616.

When the tools load the schema, it has the side-effect of updating the schema version on the node which in turn triggers a commit log to be written. If a commit log is generated with root permissions, it blocks flushing since the DSE/Cassandra process does not have permission to access the affected logs.


To bring an affected node back online, follow these steps:

Step 1 - Ensure no DSE/Cassandra process is running, otherwise shut it down.

Step 2 - Change the ownership/permission of all affected logs in the commitlog/ directory. For example:

$ cd /var/lib/data/commitlog
$ sudo chown cassandra:cassandra CommitLog*

Step 3 - Start DSE.

See also

Cassandra JIRA - CASSANDRA-8616 sstable tools may result in commit log segments to be written

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request


Powered by Zendesk