In some circumstances it has been observed that datastax-agent is unable to delete the commitlog after the specified retention time.
This error is logged in the agent.log:
ERROR [clus] 2016-07-04 13:03:41,060 Unhandled route Exception: java.io.IOException: Couldn't delete /usr/share/datastax-agent/archive_commitlog.sh
This error shows that the agent tried to delete the commtilog files from "/usr/share/datastax-agent/" directory, which is the datastax-agent installation directory.
Another symptom of the problem is that "/usr/share/datastax-agent/" directory is empty of any regular file.
The cause of this issue is in the missing directive in the <cluster_name>.conf file:
If this directive is empty, the agent will try to delete the commitlog from its own installation directory, deleting all files in "/usr/share/datastax-agent/".
Datastax documentation explicitly says to configure this directive before enabling the commitlog backup with the retention time. However it could happen that a user forgets to do it or other third parties Software Configuration Management systems replace the file with an empty or old one.
Set the directive "backup_staging_dir" into the <cluster_name>.conf file before enabling commitlog backup and removal.
The internal jira OPSC-9393 has been opened in relation to this problem to improve the logging and the error message or warn the user for missing "backup_staging_dir" directive.