Skip to content

rbd: check local image state during promote operation#2684

Merged
mergify[bot] merged 1 commit intoceph:develfrom
Madhu-1:dr-check-image-health
Dec 1, 2021
Merged

rbd: check local image state during promote operation#2684
mergify[bot] merged 1 commit intoceph:develfrom
Madhu-1:dr-check-image-health

Conversation

@Madhu-1
Copy link
Collaborator

@Madhu-1 Madhu-1 commented Nov 30, 2021

rbd mirroring CLI calls are async and it doesn't wait for the operation to be completed. ex:- rbd mirror image enable
it will enable the mirroring on the image but it doesn't ensure that the image is mirroring enabled and healthy
primary. The same goes for the promote volume also. This commits adds a check-in PromoteVolume to make sure the image is in a healthy state i.e up+stopped.

note:-

Signed-off-by: Madhu Rajanna madhupr007@gmail.com

@mergify mergify bot added the component/rbd Issues related to RBD label Nov 30, 2021
@Madhu-1 Madhu-1 force-pushed the dr-check-image-health branch from 073166d to 0ed6411 Compare November 30, 2021 06:40
Copy link
Contributor

@yati1998 yati1998 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Madhu-1 Madhu-1 force-pushed the dr-check-image-health branch from 0ed6411 to 8bd0a7b Compare November 30, 2021 07:23
@Madhu-1
Copy link
Collaborator Author

Madhu-1 commented Nov 30, 2021

below is the output from VR condition when the image is not healthy primary

conditions:
    - lastTransitionTime: "2021-11-30T06:40:24Z"
      message: ""
      observedGeneration: 8
      reason: FailedToPromote
      status: "False"
      type: Completed
    - lastTransitionTime: "2021-11-30T06:40:24Z"
      message: ""
      observedGeneration: 8
      reason: Error
      status: "True"
      type: Degraded
    - lastTransitionTime: "2021-11-30T06:35:58Z"
      message: ""
      observedGeneration: 8
      reason: NotResyncing
      status: "False"
      type: Resyncing
    lastCompletionTime: "2021-11-30T06:35:58Z"
    lastStartTime: "2021-11-26T05:50:08Z"
    message: 'rpc error: code = Internal desc = replicapool/csi-vol-b2f4ee8a-4e7c-11ec-9b91-0242ac110005
      is not healthy. Image state is up=false, state="unknown"'

@Madhu-1 Madhu-1 requested review from a team, ShyamsundarR and yati1998 November 30, 2021 07:25
@Madhu-1 Madhu-1 requested a review from nixpanic November 30, 2021 12:31
@Madhu-1 Madhu-1 force-pushed the dr-check-image-health branch from 8bd0a7b to 293ba6f Compare November 30, 2021 16:10
nixpanic
nixpanic previously approved these changes Nov 30, 2021
@Madhu-1 Madhu-1 requested review from a team and nixpanic December 1, 2021 05:10
@Madhu-1 Madhu-1 force-pushed the dr-check-image-health branch from 293ba6f to 5149f30 Compare December 1, 2021 05:22
@Madhu-1 Madhu-1 requested a review from humblec December 1, 2021 05:22
@mergify mergify bot dismissed nixpanic’s stale review December 1, 2021 05:23

Pull request has been modified.

func checkHealthyPrimary(ctx context.Context, rbdVol *rbdVolume) error {
mirrorStatus, err := rbdVol.getImageMirroringStatus()
if err != nil {
return err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return err -> return fmt.Errorf("failed to get mirror status: %w", err) ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getImageMirroringStatus already returning the same error message. doing this will add a duplicate error message.

@Madhu-1 Madhu-1 force-pushed the dr-check-image-health branch from 5149f30 to 39c70f7 Compare December 1, 2021 05:57
@Madhu-1 Madhu-1 requested a review from yati1998 December 1, 2021 05:57
Copy link
Contributor

@yati1998 yati1998 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Madhu-1
Copy link
Collaborator Author

Madhu-1 commented Dec 1, 2021

/retest all

rbd mirroring CLI calls are async and it doesn't wait
for the operation to be completed. ex:- `rbd mirror image enable`
it will enable the mirroring on the image but it doesn't
ensure that the image is mirroring enabled and healthy
primary. The same goes for the promote volume also.
This commits adds a check-in PromoteVolume to make sure
the image in a healthy state i.e `up+stopped`.

note:- not considering any intermediate states to make
sure the image is completely healthy before responding
success to the RPC call.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
@Madhu-1 Madhu-1 force-pushed the dr-check-image-health branch from 39c70f7 to 9cdd669 Compare December 1, 2021 18:21
@mergify mergify bot merged commit 64ce5e0 into ceph:devel Dec 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/rbd Issues related to RBD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants