DataStax Help Center

Nodetool clearsnapshot fails on OpsCenter generated snapshots

Background

When a backup is run from OpsCenter it creates a snapshot of the specified keyspaces.  If nodetool clearsnapshot is used to clear snapshots it will fail to delete some snapshot files and reports a failure to clear the snapshots with an error message similar to below.

Requested clearing snapshot for: all keyspaces 

Exception in thread "main" FSWriteError in /raid0/cassandra/data/system/batchlog/snapshots/opscenter_adhoc_2013-09-05-08-25-58-UTC/cf-info.txt 
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:112) 
at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:340) 
at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:336) 
at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:336) 
at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:426) 
at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1633) 
at org.apache.cassandra.db.Table.clearSnapshot(Table.java:243) 
at org.apache.cassandra.service.StorageService.clearSnapshot(StorageService.java:2253) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) 
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) 
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) 
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) 
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) 
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) 
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) 
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1454) 
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:74) 
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1295) 
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1387) 
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:818) 
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303) 
at sun.rmi.transport.Transport$1.run(Transport.java:159) 
at java.security.AccessController.doPrivileged(Native Method) 
at sun.rmi.transport.Transport.serviceCall(Transport.java:155) 
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) 
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) 
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) 
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) 
at java.lang.Thread.run(Thread.java:662) 
Caused by: java.io.IOException: Failed to delete /raid0/cassandra/data/system/batchlog/snapshots/opscenter_adhoc_2013-09-05-08-25-58-UTC/cf-info.txt 

 

Cause

When the snaphshots are created by OpsCenter the files "schema.json and cf-info.txt" are owned by the opscenter-agent user and group.  The Cassandra user won't have write access to these files as they are not owned by Cassandra and Cassandra is not in the opscenter-agent group.

Solution

To get around this issue you would add the 'cassandra' user to the 'opscenter-agent' group.  This will give write access to the files for the 'cassandra' user and allow the snapshot to be cleared.  After adding the 'cassandra' user to the  'opscenter-agent' group you may need to restart DSE so that the new permissions are applied for the DSE process.

 

*Note - This solution assumes that the DSE process is owned by the 'cassandra' user and that the the datastax agent process is run by the 'opscenter-agent' user belonging to the 'opscenter-agent' group.  If you have a alternative setup of users then you would need to apply privileges as required by your custom setup. 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk