Skip to content

Add update-target-stage gate annotation - #1913

Open
ciecierski wants to merge 1 commit into
openstack-k8s-operators:mainfrom
ciecierski:staged-update
Open

Add update-target-stage gate annotation#1913
ciecierski wants to merge 1 commit into
openstack-k8s-operators:mainfrom
ciecierski:staged-update

Conversation

@ciecierski

@ciecierski ciecierski commented May 5, 2026

Copy link
Copy Markdown
Contributor

Introduce a core.openstack.org/update-target-stage annotation on
OpenStackVersion. When set, the update controller completes all
stages up to and including the named stage, marks the next stage as
blocked (FalseCondition/Gated), and pauses reconciliation.
Removing the annotation or advancing it to a later stage resumes the
update.

AI-assisted: Cursor (Claude Sonnet 4.6 by Anthropic)

@ciecierski
ciecierski requested a review from stuggi May 5, 2026 06:48
@openshift-ci
openshift-ci Bot requested review from rabi and rebtoor May 5, 2026 06:48
@openshift-ci

openshift-ci Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ciecierski
Once this PR has been reviewed and has the lgtm label, please assign stuggi for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions

github-actions Bot commented May 5, 2026

Copy link
Copy Markdown

OpenStackControlPlane CRD Size Report

Metric Value
CRD JSON size 337535 bytes (330KB)
Base branch size 337535 bytes
Change +0.00%
Status yellow — growing
Threshold reference
Color Range Meaning
🟢 green < 300KB Comfortable
🟡 yellow 300–400KB Growing
🟠 orange 400–750KB Concerning
🔴 red > 750KB Approaching 1.5MB etcd limit (cut in half to allow space for update)

Comment thread api/core/v1beta1/conditions.go Outdated
Comment thread test/functional/ctlplane/openstackversion_controller_test.go Outdated
Comment thread test/functional/ctlplane/openstackversion_controller_test.go Outdated

@stuggi stuggi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might want to add webhook validation for the annotation values, like validating the annotation value in the ValidateUpdate webhook on OpenStackVersion . If someone sets target-stage=tyop, the webhook rejects the update immediately with a clear error, rather than silently ignoring it and not would run a full update.

The OpenStackVersion already has a webhook. Adding a check like:

  if stage, ok := r.Annotations[MinorUpdateTargetStageAnnotation]; ok {
      validStages := map[string]bool{...}
      if !validStages[stage] {
          return Forbidden("invalid target stage")
      }
  }

Comment thread api/core/v1beta1/conditions.go Outdated
Comment thread api/core/v1beta1/openstackversion_types.go Outdated
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/f690a18c992143429f1e0c967f1684a0

openstack-k8s-operators-content-provider FAILURE in 7m 30s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ openstack-operator-docs-preview SUCCESS in 2m 58s
⚠️ openstack-operator-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

@ciecierski
ciecierski force-pushed the staged-update branch 2 times, most recently from 49cecea to 1ed02a9 Compare May 12, 2026 16:19
@ciecierski ciecierski changed the title Add minor update target-stage gate annotation Add update-target-stage gate annotation May 12, 2026
@ciecierski

Copy link
Copy Markdown
Contributor Author

/test functional

1 similar comment
@ciecierski

Copy link
Copy Markdown
Contributor Author

/test functional

@ciecierski
ciecierski requested review from abays and stuggi May 18, 2026 13:39
@abays

abays commented May 18, 2026

Copy link
Copy Markdown
Contributor

What happens if...

  1. A previous annotation is present for, say, stage D, which the controller reached, paused and set the associated condition to True
  2. But then the user changes the annotation to target, say, stage C?

Stage D's condition would be reset to False, right? Is that okay? Even if it is, do we want to block that via the webhook anyhow?

Comment thread api/core/v1beta1/openstackversion_types.go
@ciecierski

Copy link
Copy Markdown
Contributor Author

What happens if...

  1. A previous annotation is present for, say, stage D, which the controller reached, paused and set the associated condition to True
  2. But then the user changes the annotation to target, say, stage C?

Stage D's condition would be reset to False, right? Is that okay? Even if it is, do we want to block that via the webhook anyhow?

We have two alternatives either let user set annotation for stage C and let user fix it with fixed oc patch. Or as you mentioned set webook to block user from making this kind of harmless mistake(setting annotations with stage C won't rollback updated containers in stage D) .

@ciecierski

Copy link
Copy Markdown
Contributor Author

/retest-required

@ciecierski
ciecierski force-pushed the staged-update branch 3 times, most recently from fae4854 to 6bda39d Compare May 28, 2026 13:44
@ciecierski
ciecierski requested a review from stuggi August 18, 2026 18:29
@ciecierski

Copy link
Copy Markdown
Contributor Author

on comment inline and if I have not missed it, #1913 (comment) is still open?

all done

Introduce a core.openstack.org/update-target-stage annotation on
OpenStackVersion. When set, the update controller completes all
stages up to and including the named stage, marks the next stage as
blocked (FalseCondition/Gated), and pauses reconciliation.

Removing the annotation or advancing it to a later stage resumes the
update.

Includes stage-name constants, the gated-message format string,
controller logic for all seven stages, functional tests for block/resume/
advance scenarios, webhooks and updated operator documentation.

AI-assisted: Cursor (Claude Sonnet 4.6 by Anthropic)
@openshift-ci

openshift-ci Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

@ciecierski: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/openstack-operator-build-deploy-kuttl-4-18 e5ffe26 link true /test openstack-operator-build-deploy-kuttl-4-18

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@ciecierski

ciecierski commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

@dprince CI failure for functional is:
• [FAILED] [42.053 seconds]
DataplaneDeployment Webhook when a running deployment is deleted without the confirm annotation [It] should reject the delete request
/go/src/github.com/openstack-k8s-operators/operator/test/functional/dataplane/openstackdataplanedeployment_webhook_test.go:75

When I run locally the same test:
make ginkgo-run GINKGO_TESTS=./test/functional/dataplane GINKGO_ARGS='--focus="DataplaneDeployment Webhook when a running deployment is deleted without the confirm annotation" it is pass.

Running Suite: DataPlane Controller Suite - /openstack-operator/test/functional/dataplane
======================================================================================================================
Random Seed: 1787079305

Will run 1 of 98 specs
Running in parallel across 3 processes
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•

Ran 1 of 98 Specs in 126.079 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 97 Skipped

coverage: 5.0% of statements
composite coverage: 5.0% of statements

Ginkgo ran 1 suite in 2m12.05405s
Test Suite Passed

@ciecierski

Copy link
Copy Markdown
Contributor Author

/test functional

@ciecierski

Copy link
Copy Markdown
Contributor Author

re kuttl - failure: WARN[2026-07-15T09:42:42Z] Found 1 conditions for ClusterClaim:
*[2026-07-15T08:42:43Z]NoClusters: No clusters in pool are ready to be claimed

@ciecierski

Copy link
Copy Markdown
Contributor Author

/test openstack-operator-build-deploy-kuttl-4-18

@ciecierski

Copy link
Copy Markdown
Contributor Author

/test openstack-operator-build-deploy-kuttl-4-20

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/cc1a2bf4a73e4b029c6ec1f4896a9465

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 15m 33s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 34m 36s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 53m 07s
openstack-operator-tempest-multinode NODE_FAILURE Node(set) request 099-0000173815 failed in 0s
✔️ openstack-operator-docs-preview SUCCESS in 3m 38s
openstack-operator-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000173816 failed in 0s

@ciecierski

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/e9aa756bd40f458aa5c02bb25430e4b9

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 42m 17s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 37m 49s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 45m 00s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 51m 37s
✔️ openstack-operator-docs-preview SUCCESS in 4m 44s
openstack-operator-edpm-baremetal-minor-update FAILURE in 2h 24m 08s

@stuggi stuggi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@ciecierski

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/c35a010f1ff3488d9c1a051b3a668bbf

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 12m 49s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 40m 06s
cifmw-crc-podified-edpm-baremetal NODE_FAILURE Node(set) request 099-0000174813 failed in 0s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 55m 10s
✔️ openstack-operator-docs-preview SUCCESS in 3m 11s
openstack-operator-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000174815 failed in 0s

@ciecierski

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/739ca2d19582469c923d8f4267db530f

openstack-k8s-operators-content-provider NODE_FAILURE Node(set) request 100-0000175383 failed in 0s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ openstack-operator-docs-preview SUCCESS in 3m 07s
⚠️ openstack-operator-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

@stuggi

stuggi commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/980df730b7b94bcaa49e1ce51895ebd1

openstack-k8s-operators-content-provider NODE_FAILURE Node(set) request 100-0000175835 failed in 0s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ openstack-operator-docs-preview SUCCESS in 3m 19s
⚠️ openstack-operator-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

@ciecierski

ciecierski commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

There is problem with upstream Zuul and CI team is investigating root cause at the moment.

@ciecierski

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/12c125abda824ebf9778f672434519d8

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 42m 38s
podified-multinode-edpm-deployment-crc POST_FAILURE in 2h 01m 20s
cifmw-crc-podified-edpm-baremetal FAILURE in 2h 02m 15s
openstack-operator-tempest-multinode RETRY_LIMIT Host unreachable in 1h 45m 56s
✔️ openstack-operator-docs-preview SUCCESS in 3m 15s
openstack-operator-edpm-baremetal-minor-update FAILURE in 2h 02m 44s

@ciecierski

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/8f9fbb408ab04ecd8a6504f641f23d9e

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 40m 12s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 26m 52s
cifmw-crc-podified-edpm-baremetal FAILURE in 35m 52s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 51m 58s
✔️ openstack-operator-docs-preview SUCCESS in 3m 05s
✔️ openstack-operator-edpm-baremetal-minor-update SUCCESS in 2h 20m 23s

@ciecierski

Copy link
Copy Markdown
Contributor Author

recheck

1 similar comment
@ciecierski

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/dace64994dd9492ea83418ce99133fd8

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 00m 07s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 29m 07s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 45m 52s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 53m 49s
✔️ openstack-operator-docs-preview SUCCESS in 3m 19s
openstack-operator-edpm-baremetal-minor-update FAILURE in 2h 20m 05s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants