Skip to content

OTA-1861: cmd/cluster-version-operator/render: Add --cluster-version-manifest-path option#1315

Open
wking wants to merge 1 commit intoopenshift:mainfrom
wking:cluster-version-overrides-on-render
Open

OTA-1861: cmd/cluster-version-operator/render: Add --cluster-version-manifest-path option#1315
wking wants to merge 1 commit intoopenshift:mainfrom
wking:cluster-version-overrides-on-render

Conversation

@wking
Copy link
Member

@wking wking commented Feb 9, 2026

Like a3a6a16 (#1078), but for ClusterVersion spec.overrides. We need this in place to avoid bootstrapping failure when the CVO renders the ClusterImagePolicy despite OPENSHIFT_INSTALL_EXPERIMENTAL_DISABLE_IMAGE_POLICY having been set to trigger the installer to set an override waiving the ClusterImagePolicy:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-serial-1of2/2020129461281755136/artifacts/e2e-aws-ovn-serial/ipi-install-install/artifacts/log-bundle-20260207143404.tar | tar -tvz | grep 'cluster.*image.*polic'
-rw-r--r-- core/core          1678 2026-02-07 06:34 log-bundle-20260207143404/rendered-assets/openshift/cvo-bootstrap/manifests/0000_90_openshift-cluster-image-policy.yaml
-rw-r--r-- core/core          1678 2026-02-07 06:34 log-bundle-20260207143404/rendered-assets/openshift/manifests/0000_90_openshift-cluster-image-policy.yaml

The rendered ClusterImagePolicy is consumed by the bootstrap machine-config operator, and it breaks the ability of unsigned nightly control-plane nodes to launch:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-serial-1of2/2020129461281755136/artifacts/e2e-aws-ovn-serial/ipi-install-install/artifacts/log-bundle-20260207143404.tar | tar -tvz | grep 'control-plane.*journal.log.gz'
-rw-r--r-- core/core         98143 2026-02-07 06:34 log-bundle-20260207143404/control-plane/10.0.110.120/journals/journal.log.gz
-rw-r--r-- core/core        101064 2026-02-07 06:34 log-bundle-20260207143404/control-plane/10.0.5.2/journals/journal.log.gz
-rw-r--r-- core/core         96700 2026-02-07 06:34 log-bundle-20260207143404/control-plane/10.0.71.227/journals/journal.log.gz
$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-serial-1of2/2020129461281755136/artifacts/e2e-aws-ovn-serial/ipi-install-install/artifacts/log-bundle-20260207143404.tar | tar -xOz log-bundle-20260207143404/control-plane/10.0.5.2/journals/journal.log.gz | zgrep 'signature was required' | head -n4
Sat 2026-02-07 13:51:08 UTC ip-10-0-5-2 machine-config-daemon-pull.service[2026]: Error: Source image rejected: A signature was required, but no signature exists
Sat 2026-02-07 13:51:08 UTC ip-10-0-5-2 machine-config-daemon-pull.service[2026]: 2026-02-07 13:51:08.129594235 +0000 UTC m=+0.399222333 image pull-error  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4b3cc836d480ea7156ea797712e9af742fb73e5fb56a9cf9e63aeae11875315 Source image rejected: A signature was required, but no signature exists
Sat 2026-02-07 13:51:09 UTC ip-10-0-5-2 machine-config-daemon-pull.service[2037]: Error: Source image rejected: A signature was required, but no signature exists
Sat 2026-02-07 13:51:09 UTC ip-10-0-5-2 machine-config-daemon-pull.service[2037]: 2026-02-07 13:51:09.533298454 +0000 UTC m=+0.389622944 image pull-error  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4b3cc836d480ea7156ea797712e9af742fb73e5fb56a9cf9e63aeae11875315 Source image rejected: A signature was required, but no signature exists

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Feb 9, 2026
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Feb 9, 2026

@wking: This pull request references OTA-1861 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Like a3a6a16 (#1078), but for ClusterVersion spec.overrides. We need this in place to avoid bootstrapping failure when the CVO renders the ClusterImagePolicy despite OPENSHIFT_INSTALL_EXPERIMENTAL_DISABLE_IMAGE_POLICY having been set to trigger the installer to set an override waiving the ClusterImagePolicy:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-serial-1of2/2020129461281755136/artifacts/e2e-aws-ovn-serial/ipi-install-install/artifacts/log-bundle-20260207143404.tar | tar -tvz | grep 'cluster.*image.*polic'
-rw-r--r-- core/core          1678 2026-02-07 06:34 log-bundle-20260207143404/rendered-assets/openshift/cvo-bootstrap/manifests/0000_90_openshift-cluster-image-policy.yaml
-rw-r--r-- core/core          1678 2026-02-07 06:34 log-bundle-20260207143404/rendered-assets/openshift/manifests/0000_90_openshift-cluster-image-policy.yaml

The rendered ClusterImagePolicy is consumed by the bootstrap machine-config operator, and it breaks the ability of unsigned nightly control-plane nodes to launch:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-serial-1of2/2020129461281755136/artifacts/e2e-aws-ovn-serial/ipi-install-install/artifacts/log-bundle-20260207143404.tar | tar -tvz | grep 'control-plane.*journal.log.gz'
-rw-r--r-- core/core         98143 2026-02-07 06:34 log-bundle-20260207143404/control-plane/10.0.110.120/journals/journal.log.gz
-rw-r--r-- core/core        101064 2026-02-07 06:34 log-bundle-20260207143404/control-plane/10.0.5.2/journals/journal.log.gz
-rw-r--r-- core/core         96700 2026-02-07 06:34 log-bundle-20260207143404/control-plane/10.0.71.227/journals/journal.log.gz
$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-serial-1of2/2020129461281755136/artifacts/e2e-aws-ovn-serial/ipi-install-install/artifacts/log-bundle-20260207143404.tar | tar -xOz log-bundle-20260207143404/control-plane/10.0.5.2/journals/journal.log.gz | zgrep 'signature was required' | head -n4
Sat 2026-02-07 13:51:08 UTC ip-10-0-5-2 machine-config-daemon-pull.service[2026]: Error: Source image rejected: A signature was required, but no signature exists
Sat 2026-02-07 13:51:08 UTC ip-10-0-5-2 machine-config-daemon-pull.service[2026]: 2026-02-07 13:51:08.129594235 +0000 UTC m=+0.399222333 image pull-error  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4b3cc836d480ea7156ea797712e9af742fb73e5fb56a9cf9e63aeae11875315 Source image rejected: A signature was required, but no signature exists
Sat 2026-02-07 13:51:09 UTC ip-10-0-5-2 machine-config-daemon-pull.service[2037]: Error: Source image rejected: A signature was required, but no signature exists
Sat 2026-02-07 13:51:09 UTC ip-10-0-5-2 machine-config-daemon-pull.service[2037]: 2026-02-07 13:51:09.533298454 +0000 UTC m=+0.389622944 image pull-error  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4b3cc836d480ea7156ea797712e9af742fb73e5fb56a9cf9e63aeae11875315 Source image rejected: A signature was required, but no signature exists

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link

coderabbitai bot commented Feb 9, 2026

Walkthrough

Adds a new CLI flag to provide a ClusterVersion manifest path, parses that manifest to extract component overrides, and threads those overrides through the payload rendering pipeline so included manifests can be adjusted accordingly.

Changes

Cohort / File(s) Summary
CLI Flag & invocation
cmd/cluster-version-operator/render.go
Added clusterVersionManifestPath string to renderOpts, registered --cluster-version-manifest-path Cobra flag, and updated the payload.Render call to include the new clusterVersionManifestPath argument (shifting subsequent args).
Rendering pipeline & helpers
pkg/payload/render.go
Updated Render signature to accept clusterVersionManifestPath. Added parseClusterVersionManifest(clusterVersionManifestPath string) ([]configv1.ComponentOverride, error). Extended renderDir signature to accept overrides []configv1.ComponentOverride and passed overrides into manifest inclusion and downstream calls; parsing errors are now propagated.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

No actionable comments were generated in the recent review. 🎉

🧹 Recent nitpick comments
pkg/payload/render.go (1)

193-220: Well-implemented, mirrors existing parseFeatureGateManifest pattern.

The function correctly validates the manifest structure and extracts overrides. Consider adding a log statement (similar to Lines 252-256 in parseFeatureGateManifest) to log when overrides are parsed—this would help with debugging bootstrapping issues like the one motivating this PR.


Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 9, 2026
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@pkg/payload/render.go`:
- Around line 193-220: The function parseClusterVersionManifest has a mismatched
parameter name: the signature declares featureGateManifestPath but the body
references clusterVersionManifestPath; rename the parameter in
parseClusterVersionManifest's signature to clusterVersionManifestPath (or
alternatively update all uses in the body to featureGateManifestPath) so the
identifier matches, ensuring the function compiles and the checks/manifest
loading (manifests :=
manifest.ManifestsFromFiles([]string{clusterVersionManifestPath}) etc.) use the
corrected parameter name.

wking added a commit to wking/openshift-installer that referenced this pull request Feb 9, 2026
…r-version-manifest-path

Like 9bbe172 (data/bootstrap/files/usr/local/bin/bootkube: Pass CVO
render --feature-gate-manifest-path, 2024-08-07, openshift#8813), but for the
ClusterVersion manifest laid down by
pkg/asset/ignition/bootstrap/cvoignore.go.  To populate the new flag
provided by [1].

[1]: openshift/cluster-version-operator#1315
@wking wking force-pushed the cluster-version-overrides-on-render branch from ba1ac8d to a837056 Compare February 9, 2026 18:16
…ath option

Like a3a6a16 (cmd/render: Add --feature-gate-manifest-path option,
2024-08-07, openshift#1078), but for ClusterVersion spec.overrides.  We need
this in place to avoid bootstrapping failure when the CVO renders the
ClusterImagePolicy despite
OPENSHIFT_INSTALL_EXPERIMENTAL_DISABLE_IMAGE_POLICY having been set to
trigger the installer to set an override waiving the
ClusterImagePolicy [1]:

  $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-serial-1of2/2020129461281755136/artifacts/e2e-aws-ovn-serial/ipi-install-install/artifacts/log-bundle-20260207143404.tar | tar -tvz | grep 'cluster.*image.*polic'
  -rw-r--r-- core/core          1678 2026-02-07 06:34 log-bundle-20260207143404/rendered-assets/openshift/cvo-bootstrap/manifests/0000_90_openshift-cluster-image-policy.yaml
  -rw-r--r-- core/core          1678 2026-02-07 06:34 log-bundle-20260207143404/rendered-assets/openshift/manifests/0000_90_openshift-cluster-image-policy.yaml

The rendered ClusterImagePolicy is consumed by the bootstrap
machine-config operator, and it breaks the ability of unsigned nightly
control-plane nodes to launch [1]:

  $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-serial-1of2/2020129461281755136/artifacts/e2e-aws-ovn-serial/ipi-install-install/artifacts/log-bundle-20260207143404.tar | tar -tvz | grep 'control-plane.*journal.log.gz'
  -rw-r--r-- core/core         98143 2026-02-07 06:34 log-bundle-20260207143404/control-plane/10.0.110.120/journals/journal.log.gz
  -rw-r--r-- core/core        101064 2026-02-07 06:34 log-bundle-20260207143404/control-plane/10.0.5.2/journals/journal.log.gz
  -rw-r--r-- core/core         96700 2026-02-07 06:34 log-bundle-20260207143404/control-plane/10.0.71.227/journals/journal.log.gz
  $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-serial-1of2/2020129461281755136/artifacts/e2e-aws-ovn-serial/ipi-install-install/artifacts/log-bundle-20260207143404.tar | tar -xOz log-bundle-20260207143404/control-plane/10.0.5.2/journals/journal.log.gz | zgrep 'signature was required' | head -n4
  Sat 2026-02-07 13:51:08 UTC ip-10-0-5-2 machine-config-daemon-pull.service[2026]: Error: Source image rejected: A signature was required, but no signature exists
  Sat 2026-02-07 13:51:08 UTC ip-10-0-5-2 machine-config-daemon-pull.service[2026]: 2026-02-07 13:51:08.129594235 +0000 UTC m=+0.399222333 image pull-error  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4b3cc836d480ea7156ea797712e9af742fb73e5fb56a9cf9e63aeae11875315 Source image rejected: A signature was required, but no signature exists
  Sat 2026-02-07 13:51:09 UTC ip-10-0-5-2 machine-config-daemon-pull.service[2037]: Error: Source image rejected: A signature was required, but no signature exists
  Sat 2026-02-07 13:51:09 UTC ip-10-0-5-2 machine-config-daemon-pull.service[2037]: 2026-02-07 13:51:09.533298454 +0000 UTC m=+0.389622944 image pull-error  quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4b3cc836d480ea7156ea797712e9af742fb73e5fb56a9cf9e63aeae11875315 Source image rejected: A signature was required, but no signature exists

[1]: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.22-e2e-aws-ovn-serial-1of2/2020129461281755136
@QiWang19
Copy link
Member

/verified by cluster-bot

/lgtm

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Feb 12, 2026
@openshift-ci-robot
Copy link
Contributor

@QiWang19: This PR has been marked as verified by cluster-bot.

Details

In response to this:

/verified by cluster-bot

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 12, 2026
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 12, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: QiWang19, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 13, 2026

@wking: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-techpreview a837056 link unknown /test e2e-aws-ovn-techpreview
ci/prow/e2e-agnostic-ovn a837056 link unknown /test e2e-agnostic-ovn

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@wking
Copy link
Member Author

wking commented Feb 13, 2026

@wking
Copy link
Member Author

wking commented Feb 13, 2026

UserDefinedPrimaryNetworks and HAProxy router failures are unrelated to this pull:

/override ci/prow/ee2e-aws-ovn-techpreview

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 13, 2026

@wking: /override requires failed status contexts, check run or a prowjob name to operate on.
The following unknown contexts/checkruns were given:

  • ci/prow/ee2e-aws-ovn-techpreview

Only the following failed contexts/checkruns were expected:

  • CodeRabbit
  • ci/prow/e2e-agnostic-operator
  • ci/prow/e2e-agnostic-ovn
  • ci/prow/e2e-agnostic-ovn-techpreview-serial
  • ci/prow/e2e-agnostic-ovn-upgrade-into-change
  • ci/prow/e2e-agnostic-ovn-upgrade-out-of-change
  • ci/prow/e2e-aws-ovn-techpreview
  • ci/prow/e2e-hypershift
  • ci/prow/e2e-hypershift-conformance
  • ci/prow/gofmt
  • ci/prow/images
  • ci/prow/lint
  • ci/prow/okd-scos-images
  • ci/prow/unit
  • ci/prow/verify-deps
  • ci/prow/verify-update
  • pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-operator
  • pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn
  • pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-techpreview-serial
  • pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-upgrade-into-change
  • pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn-upgrade-out-of-change
  • pull-ci-openshift-cluster-version-operator-main-e2e-aws-ovn-techpreview
  • pull-ci-openshift-cluster-version-operator-main-e2e-hypershift
  • pull-ci-openshift-cluster-version-operator-main-e2e-hypershift-conformance
  • pull-ci-openshift-cluster-version-operator-main-gofmt
  • pull-ci-openshift-cluster-version-operator-main-images
  • pull-ci-openshift-cluster-version-operator-main-lint
  • pull-ci-openshift-cluster-version-operator-main-okd-scos-images
  • pull-ci-openshift-cluster-version-operator-main-unit
  • pull-ci-openshift-cluster-version-operator-main-verify-deps
  • pull-ci-openshift-cluster-version-operator-main-verify-update
  • tide

If you are trying to override a checkrun that has a space in it, you must put a double quote on the context.

Details

In response to this:

UserDefinedPrimaryNetworks and HAProxy router failures are unrelated to this pull:

/override ci/prow/ee2e-aws-ovn-techpreview

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants