Summary
This article shows how to check the outcome of a scheduled (e.g. cron) nodetool repair from the logs.
Repair session
A repair session, otherwise known as "anti-entropy", is identified by a session UUID in the Cassandra system.log. To determine the start of a repair session, search for a RepairSession.java entry with the phrase new session:, for example:
INFO [AntiEntropySessions:1] 2015-03-01 10:02:53,261 RepairSession.java (line 244) [repair #2e7009b0-c03d-11e4-9012-99a64119c9d8] new session: will sync myCluster/10.1.2.3, /10.1.2.4, /10.1.2.5 on range (0,4611686018427387904] for myKS.myCF
In this example, the session UUID is 2e7009b0-c03d-11e4-9012-99a64119c9d8. To view all entries in the log relating to this session, filter the log for this particular session ID.
Note that depending on how repairs are managed, there could be multiple sessions in the logs.
Completion
To check if the session completed successfully, filter the log with the appropriate session UUID and the last entry will have the phrase session completed successfully.
From the example above, the corresponding completed repair session log entry is as follows:
INFO [AntiEntropySessions:1] 2015-03-01 10:02:56,245 RepairSession.java (line 282) [repair #2e7009b0-c03d-11e4-9012-99a64119c9d8] session completed successfully
Additional information
For more information on investigating an unsuccessful repair session, see Troubleshooting hanging repairs.
Comments
1 comment
Thank you Erick !! We were running repairs sequentially on all our nodes with -pr using ansible (--forks 1). For one of the nodes, we got a "Failed to connect to the host via ssh" in our ansible logs but the process happily went onto to kick off repair on the next node. We were concerned if the repair finished for the node for which we got this ssh error. Grep'ing for session id(s) in the system.log on that node (and also keeping the time stamp in account), we were able to verify that repairs are complete.
Thanks,
Rohit
Please sign in to leave a comment.