Summary
Opscenter tables settings
and backup_reports
are still repaired using the incremental repair method. This can generate I/O overhead because of the additional anti-compaction phase included necessary for the validation in the incremental repair.
Incremental repair is not the recommended way to repair tables in Cassandra, because subrange repair is more performant and has less impact on the main operations of production clusters.
Scenario
For historical reasons, the settings
and backup_reports
tables have always been configured for incremental repair. This is hardcoded in the Opscenter python files.
It is not possible to change this settings without recompling the code, however it is possible to disable incremental repair for those tables using the incremental repair properties in cluster_name.conf
.
Procedure
In order to disable settings
and backup_reports
for incremental repair, the user will have to explicitly set the property
[repair_service] incremental_repair_tables=
in either cluster_name.conf
file or opscenterd.conf
. By leaving the property blank, no tables will be repaired incrementally and all the tables will be repaired using subrange repair.
One important thing is to unset the RepairedAt
flag in each sstable of the tables settings
and backup_reports
by running:
nodetool mark_unrepaired "OpsCenter" backup_reports --force nodetool mark_unrepaired "OpsCenter" settings --force
In this way the tables are marked unrepaired and the subrange repair can start repairing them.
After the changes are effected, proceed to
- restart opscenter
- disable repair service
- enable repair service again.
Additional steps
In order to limit the amount of logging that incremental repair does, even if there are no tables to repair incrementally, users should also include this block in the [repair_service]
section of the cluster_name.conf
or opscenterd.conf
as added in the first part of the Procedure above.
# Stop incremental repairs from spamming the event log # Sleep 1 day after each cycle incremental_sleep = 86400
This article is applicable to all 6.x versions of Opscenter. A jira is tracking the request for changing this behaviour in Opscenter: OPSC-15858 - “change repair service property incremental_repair_tables
default to be empty”