Summary
Opscenter tables settings and backup_reports are still repaired using the incremental repair method. This can generate I/O overhead because of the additional anti-compaction phase included necessary for the validation in the incremental repair.
Incremental repair is not the recommended way to repair tables in Cassandra, because subrange repair is more performant and has less impact on the main operations of production clusters.
Scenario
For historical reasons, the settings and backup_reports tables have always been configured for incremental repair. This is hardcoded in the Opscenter python files.
It is not possible to change this settings without recompling the code, however it is possible to disable incremental repair for those tables using the incremental repair properties in cluster_name.conf.
Procedure
In order to disable settings and backup_reports for incremental repair, the user will have to explicitly set the property
[repair_service] incremental_repair_tables=
in either cluster_name.conf file or opscenterd.conf. By leaving the property blank, no tables will be repaired incrementally and all the tables will be repaired using subrange repair.
One important thing is to unset the RepairedAt flag in each sstable of the tables settings and backup_reports by running:
nodetool mark_unrepaired "OpsCenter" backup_reports --force nodetool mark_unrepaired "OpsCenter" settings --force
In this way the tables are marked unrepaired and the subrange repair can start repairing them.
After the changes are effected, proceed to
- restart opscenter
- disable repair service
- enable repair service again.
Additional steps
In order to limit the amount of logging that incremental repair does, even if there are no tables to repair incrementally, users should also include this block in the [repair_service] section of the cluster_name.conf or opscenterd.conf as added in the first part of the Procedure above.
# Stop incremental repairs from spamming the event log # Sleep 1 day after each cycle incremental_sleep = 86400
This article is applicable to all 6.x versions of Opscenter. A jira is tracking the request for changing this behaviour in Opscenter: OPSC-15858 - “change repair service property incremental_repair_tables default to be empty”