OpsCenter error “Repair Service will not complete within the specified repair window” – DataStax Support

Summary

This technical note address troubleshoting of false-positives of the error found in OpsCenter logs
Based on current repair throughput, it appears that the Repair Service will not complete within the specified repair window

Applies to

DataStax OpsCenter 6.7 (potentially all versions)

Symptoms

Reviewing the opscenterd.log or historical ones (opscenterd.1.log…), the following error is found in the logs on a regular basis:

2020-05-31 03:21:46,983 [cluster] ERROR: Based on current repair throughput, it appears that the Repair Service will not complete within the specified repair window. 
Tasks remaining to repair: 674.63 GB. 
Repair must complete by Tue Jun 09 03:11:23 UTC 2020, which requires throughput of: 910.4 KB/s; however, the actual throughput is: 71.6 KB/s. 
Estimated completion is Tue Sep 22 12:42:13 UTC 2020. 
For more information on tuning the repair service, see http://www.datastax.com/documentation/opscenter/help/repair_services_advanced.html (MainThread)

Cause

During the start of a new cycle of repair, or after a schema change occur, as well as in case of a slow down of repair, this error can appear during a warm up period of 30 minutes or more following any pause/new start of Repair Service and lead to a false-positive

Analysis

The following script allows to

find if the Repair Service finished a cycle and is starting a new one, looking for the following
Rolling subrange repair success, starting a new run
identify if a schema change occured or other operations occured around repair via search of the Repair Service string, which leads to a pause of repair
Parse the error message, to trim and show only relevant information (sed command) as such:
remain: 31.85 GB tocomplete: Wed Jul 01 14:39:34 CEST 2020 throuputreq: 3.51 MB/s throughputactual: 522.9 KB/s estimate: Thu Jul 02 05:49:08 CEST 2020

cd to the opscenter log folder prior to running the command below

for i in $(find . -name 'opscenterd.*log'| sort -V -r); \
 do echo ++++ $i ++++; \
 grep -E 'Based on current repair|Rolling subrange repair success, starting a new run|Repair Service' $i | \
 sed -E 's_(2020.*ERROR):.* repair: ([0-9\.]{1,} [TGKM]B).*complete by (.*2020).*requires throughput of: ([0-9.]{1,} [TGKM]B\/s).*actual throughput is: ([0-9.]{1,} [TGKMb].*B?\/s).*Estimated completion is (.*20[0-9]{2}).*_\1\t remain: \2\t tocomplete: \3\t throuputreq: \4\t throughputactual: \5\t estimate: \6_g'; \
 done | less

Based on this, we can identify the scenarios where a schema change or a new cycle led to a false positive. Note the timestamp at the beginning and ensuing error

Scenarios

1- End of a Repair cycle and start of a new one

Error shows for the first 30mn following a new cycle

2020-06-18 06:44:08,027 [cluster]  INFO: Rolling subrange repair success, starting a new run (MainThread)
2020-06-18 07:44:42,287 [cluster] ERROR    remain: 105.22 GB       tocomplete: Sat Jun 27 06:49:07 UTC 2020        throuputreq: 142.5 KB/s         throughputactual: 9.5 KB/s      estimate: Fri Oct 30 08:01:00 UTC 2020
2020-06-19 08:10:48,771 [cluster] ERROR    remain: 79.99 GB        tocomplete: Sat Jun 27 06:49:07 UTC 2020        throuputreq: 122.2 KB/s         throughputactual: 23.5 KB/s     estimate: Thu Jul 30 14:54:31 UTC 2020
2020-06-20 08:11:05,127 [cluster] ERROR    remain: 43.01 GB        tocomplete: Sat Jun 27 06:49:07 UTC 2020        throuputreq: 75.2 KB/s  throughputactual: 43.6 KB/s     estimate: Thu Jul 02 07:20:40 UTC 2020
2020-06-20 23:59:16,846 [cluster]  INFO: Rolling subrange repair success, starting a new run (MainThread)

Error repeats after every repair cycle (low gc_grace leading to multiple run of repair in a day)

++++ ./opscenterd.log ++++
2020-07-02 10:26:46,480 [cluster]  INFO: Rolling subrange repair success, starting a new run (MainThread)
2020-07-02 10:38:13,926 [cluster] ERROR         remain: 1.24 GB         tocomplete: Thu Jul 02 15:08:46 CEST 2020       throuputreq: 143.5 KB/s         throughputactual: 33.2 KB/s     estimate: Thu Jul 02 23:28:30 CEST 2020
2020-07-02 11:12:35,688 [cluster]  INFO: Rolling subrange repair success, starting a new run (MainThread)
2020-07-02 11:23:55,749 [cluster] ERROR         remain: 1.33 GB         tocomplete: Thu Jul 02 15:54:36 CEST 2020       throuputreq: 153.7 KB/s         throughputactual: 54.3 KB/s     estimate: Thu Jul 02 20:30:40 CEST 2020
2020-07-02 12:20:35,824 [cluster]  INFO: Rolling subrange repair success, starting a new run (MainThread)
2020-07-02 12:23:12,117 [cluster] ERROR         remain: 32.64 GB        tocomplete: Thu Jul 02 17:02:36 CEST 2020       throuputreq: 3.49 MB/s  throughputactual: 1.07 MB/s     estimate: Thu Jul 02 23:05:54 CEST 2020
2020-07-02 13:03:44,105 [cluster]  INFO: Rolling subrange repair success, starting a new run (MainThread)

2- Schema change lead to pause of repair and error follows (full cycle over 7 days):

++++ ./opsclogs/var/log/opscenter/opscenterd.5.log ++++
2020-04-23 18:13:50,176 [cluster]  INFO: Rolling subrange repair success, starting a new run (MainThread)
2020-04-23 18:36:08,101 [cluster] ERROR    remain: 981.52 GB       tocomplete: Sat May 02 18:17:27 UTC 2020        throuputreq: 1.29 MB/s  throughputactual: 359.5 KB/s    estimate: Tue May 26 21:48:40 UTC 2020
2020-04-23 23:43:45,335 [cluster]  INFO: Detected a keyspace changed schema change. The Repair Service will pause for 5 minutes then the Repair Service will activate again. This period of time is configurable with [repair_service].restart_period. (MainThread)
2020-04-23 23:56:01,217 [cluster] ERROR    remain: 1.3 TB  tocomplete: Sat May 02 23:52:12 UTC 2020        throuputreq: 1.76 MB/s  throughputactual: 419.2 KB/s    estimate: Mon Jun 01 13:43:35 UTC 2020
2020-04-24 23:56:06,993 [cluster] ERROR    remain: 559.09 GB       tocomplete: Sat May 02 23:52:12 UTC 2020        throuputreq: 848.4 KB/s         throughputactual: 22.6 KB/s     estimate: Thu Feb 18 19:12:42 UTC 2021
2020-04-26 01:23:57,832 [cluster] ERROR    remain: 202.95 GB       tocomplete: Sat May 02 23:52:12 UTC 2020        throuputreq: 355.1 KB/s         throughputactual: 1.3 KB/s      estimate: Sun Sep 07 16:56:18 UTC 2025
2020-04-29 09:33:23,899 [cluster]  INFO: Rolling subrange repair success, starting a new run (MainThread)

Solution

The scenario above indicates false-positives of the error and can be ignored
In case of doubt or if the OpsCenter Repair Service fail to go through the 100% cycle, tweaking of the Repair Service may be necessary. Please contact Support if need be.