ci: publish signed cluster OCI artifact - #2163
Conversation
konflate — summaryNote ✅ No rendered changes. konflate · rendered |
|
Warning Review limit reached
Next review available in: 28 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe PR adds a GitHub Actions workflow for Kubernetes cluster artifact releases. The workflow publishes an OCI artifact to GHCR, signs and verifies its digest with Cosign, and promotes the verified digest to ChangesCluster artifact release
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant FluxCLI
participant GHCR
participant Cosign
GitHubActions->>GitHubActions: Validate main ref and prepare tools
GitHubActions->>GHCR: Authenticate
GitHubActions->>FluxCLI: Push Kubernetes OCI artifact
FluxCLI->>GHCR: Store artifact
GHCR-->>GitHubActions: Return digest
GitHubActions->>Cosign: Sign and verify digest with OIDC identity
Cosign->>GHCR: Store and validate signature
GitHubActions->>FluxCLI: Apply latest tag to verified digest
FluxCLI->>GHCR: Promote digest to latest
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
The artifact layout is right — copying into Two things about the rollout gate itself: Expect the first run to fail at The gate does not currently cover the Nothing under |
|
Addressed the review feedback:
Rebased and force-updated all downstream stack branches so their PR diffs remain 6 / 60 / 86 files. |
|
The artifact layout is right — copying into Add to the rollout gate. A GHCR package created by
If neither happens, PR 2 lands an Nits, none blocking:
|
|
Addressed the second review round:
I also confirmed the live cluster runs All downstream stack branches were rebased and force-updated. |
|
Nothing under I did verify the artifact layout against what steps 2–4 will need, since getting it wrong here would be invisible until the source swap:
Three things flagged inline; the cosign signature-format one is the one I would resolve before merging step 2, since the rollout gate as written cannot detect it. One more, non-blocking: |
|
Addressed this review round:
The downstream branches were rebased and force-updated again. I also linked #2163, #2164, #2165, and #2161 using GitHub’s stacked pull request feature as stack #2166. |
| cosign sign --yes --new-bundle-format=true "${OCI_REPOSITORY}@${DIGEST}" | ||
| cosign verify \ | ||
| --new-bundle-format=true \ | ||
| --certificate-identity "${GITHUB_SERVER_URL}/${GITHUB_WORKFLOW_REF}" \ | ||
| --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \ | ||
| "${OCI_REPOSITORY}@${DIGEST}" |
There was a problem hiding this comment.
--new-bundle-format=true will very likely produce a signature Flux cannot verify, which breaks step 2 of the stack.
With the new bundle format, cosign stores the signature as an OCI 1.1 referrer (artifactType: application/vnd.dev.sigstore.bundle.v0.3+json) attached to the subject digest. Flux's OCIRepository cosign verifier goes through cosign.VerifyImageSignatures with the legacy discovery path — it resolves the sha256-<digest>.sig tag and does not set NewBundleFormat in CheckOpts. No .sig tag is written when the new format is used, so verify.provider: cosign + matchOIDCIdentity (the pattern every existing OCIRepository in this repo uses, e.g. kubernetes/apps/kube-system/reloader/app/ocirepository.yaml) would fail with "no matching signatures".
The cosign verify on line 83 will not catch this — cosign auto-detects the format it just wrote. So the rollout gate in the PR description ("confirm the Cluster Release workflow succeeds") gives false confidence about whether Flux can consume the artifact.
Note that on cosign v3.x the new bundle format is already the default, so this has to be turned off explicitly:
| cosign sign --yes --new-bundle-format=true "${OCI_REPOSITORY}@${DIGEST}" | |
| cosign verify \ | |
| --new-bundle-format=true \ | |
| --certificate-identity "${GITHUB_SERVER_URL}/${GITHUB_WORKFLOW_REF}" \ | |
| --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \ | |
| "${OCI_REPOSITORY}@${DIGEST}" | |
| cosign sign --yes --new-bundle-format=false "${OCI_REPOSITORY}@${DIGEST}" | |
| cosign verify \ | |
| --certificate-identity "${GITHUB_SERVER_URL}/${GITHUB_WORKFLOW_REF}" \ | |
| --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \ | |
| "${OCI_REPOSITORY}@${DIGEST}" |
Alternatively run cosign sign twice (once with each format) if you want the modern bundle available too. Either way, worth confirming against a real OCIRepository with verify.provider: cosign before merging step 2, rather than relying on the in-workflow verify.
(The --certificate-identity construction itself is right: $GITHUB_SERVER_URL/$GITHUB_WORKFLOW_REF → https://github.com/jfroy/flatops/.github/workflows/cluster-release.yaml@refs/heads/main, which is what the step-2 matchOIDCIdentity.subject regex will need to match.)
| - name: Setup Flux CLI | ||
| uses: fluxcd/flux2/action@889be9d6cc8afa8ed639e1e1ba4ab678e3b38d8c # v2.9.4 | ||
| with: | ||
| token: ${{ github.token }} | ||
| version: 2.9.4 | ||
|
|
||
| - name: Setup Cosign | ||
| uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2 | ||
| with: | ||
| cosign-release: v3.1.3 |
There was a problem hiding this comment.
These two pinned tool versions are outside Renovate's reach and will silently drift.
.renovate/customManagers.json5 scopes both custom managers to managerFilePatterns: ["/^(kubernetes|talos)/.+\\.yaml$/"], and Renovate's built-in github-actions manager only reads uses: refs — it does not look inside with: inputs. So helpers:pinGitHubActionDigestsToSemver will keep bumping the action digests while version: 2.9.4 and cosign-release: v3.1.3 stay frozen. After the next fluxcd/flux2/action bump you'd be running the new action to install an old flux CLI.
Two ways out:
- Drop
version:/cosign-release:and let each action install its default (latest) release — simplest, and matches the "no unnecessary knobs" style of the other two workflows. - Keep the pins and add
# renovate:annotations plus a.github/workflows/entry in the custom manager'smanagerFilePatterns, e.g.:
with:
token: ${{ github.token }}
# renovate: datasource=github-releases depName=fluxcd/flux2
version: 2.9.4
|
Two inline comments, one of which I think blocks the next PR in the stack rather than this one. Signature format vs. Flux — Renovate coverage — The rest checks out:
Two smaller notes:
One thing I could not verify without network access in this environment: that |
|
Follow-up to the latest review:
I did not add a second render/build gate here; Konflate remains responsible for rendered PR validation, and the earlier redundant artifact-build step stays removed. The force-pushed branches remain linked in GitHub stack #2166, in positions #2163 → #2164 → #2165 → #2161. |
| # renovate: datasource=github-releases depName=fluxcd/flux2 | ||
| version: 2.9.4 |
There was a problem hiding this comment.
The version: value here is unprefixed, but the custom manager you just pointed at .github/workflows/** will very likely rewrite it with a v prefix and break this step.
github-releases yields v-prefixed versions for fluxcd/flux2, and the custom manager's versioningTemplate falls back to semver-coerced, whose getNewValue returns the new version verbatim. Both existing github-releases annotations in the repo bear this out — planeVersion: v1.4.1 (kubernetes/apps/plane/plane/app/helmrelease.yaml:20) and the v0.14.0 release URL in kubernetes/apps/cnpg-system/barman-cloud/app/kustomization.yaml:7. The capture group v?\d[-a-zA-Z0-9._+]* happily matches 2.9.4 on read and won't stop v2.9.5 from being written back.
Since :automergeMinor is on, that bump auto-merges and the next Cluster Release run fails at Setup Flux CLI (the action interpolates version into the v${version} release path). The cosign-release: v3.1.3 annotation below is unaffected because cosign-installer wants the prefix.
Cleanest fix is a packageRule so the datasource hands back an unprefixed version:
{
matchDatasources: ["github-releases"],
matchPackageNames: ["fluxcd/flux2"],
extractVersion: "^v(?<version>.+)$",
}Worth checking before merge whether fluxcd/flux2/action tolerates a leading v — if it does, this is cosmetic; if it doesn't, the workflow silently stops publishing after the first Renovate bump, which is the failure mode the rollout gate won't catch.
| - name: Require main ref | ||
| if: github.ref != 'refs/heads/main' | ||
| run: | | ||
| echo "Cluster releases must run from refs/heads/main" >&2 | ||
| exit 1 |
There was a problem hiding this comment.
This guards the ref but not the commit, so it doesn't stop latest from moving backwards.
Re-running an earlier run ("Re-run all jobs" / "Re-run failed jobs") keeps that run's original GITHUB_SHA while github.ref is still refs/heads/main, so this check passes. The checkout is the old commit, the artifact is built from the old kubernetes/ tree, and Promote artifact re-points latest at it. That's harmless today, but once step 3/4 of the stack lands it is a silent cluster rollback — and it's a realistic path, since re-running a run that failed at Sign and verify artifact after main has advanced is exactly what someone would do.
Consider asserting that the checked-out commit is still the tip of main before promoting:
- name: Require main tip
env:
GH_TOKEN: ${{ github.token }}
run: |
tip="$(gh api "repos/${GITHUB_REPOSITORY}/commits/main" --jq .sha)"
if [[ "${tip}" != "${GITHUB_SHA}" ]]; then
echo "${GITHUB_SHA} is not the tip of main (${tip}); refusing to promote" >&2
exit 1
fiPlacing it just before Promote artifact still lets the immutable SHA tag and its signature be (re)published, and only blocks the latest move.
|
Two inline comments on the workflow. Beyond those, some notes on the rollout gate itself, since a few things can only fail on the first real run: Add package visibility to the gate. Two CLI surfaces worth eyeballing in the first run's logs:
The The Ordering (publish → sign → verify → promote) is right — |
|
Addressed this round:
I also checked the common alternatives for stale deployment prevention:
The API form from the review is the simplest fit here because |
| - name: Require main tip | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| tip="$(gh api "repos/${GITHUB_REPOSITORY}/commits/main" --jq .sha)" | ||
| if [[ "${tip}" != "${GITHUB_SHA}" ]]; then | ||
| echo "${GITHUB_SHA} is not the tip of main (${tip}); refusing to promote" >&2 | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
The strict tip equality check interacts badly with the paths: filter above and will block legitimate promotions.
paths restricts the trigger to kubernetes/** and this workflow file, so not every commit on main starts a release run — but every commit on main moves the tip. Sequence:
- Commit A touches
kubernetes/apps/default/foo/→ release run starts, publishes and signs:A. - Commit B lands while A is still running and touches only
talos/talconfig.yaml,docs/,.renovate/, orREADME.md→ no release run (filtered out). - A's tip check sees
tip == B != A→exit 1.
latest never gets A's content even though A is the current state of kubernetes/, and it stays stale until the next kubernetes/** push (or a manual workflow_dispatch). Given how often Renovate lands talos/ and .renovate/ commits here, this will fire regularly — and once step 2 of the stack points Flux at latest, a stale latest means the cluster silently stops picking up manifest changes with nothing to alert on.
Secondary issue: even in the genuine race (two kubernetes/** pushes back to back), exit 1 leaves a red run on main for a commit that was correctly superseded. That's a skip, not a failure.
Scoping the comparison to release-relevant paths and downgrading supersession to a skip handles both:
- name: Resolve promotion
id: promote
env:
GH_TOKEN: ${{ github.token }}
run: |
cmp="$(gh api "repos/${GITHUB_REPOSITORY}/compare/${GITHUB_SHA}...main")"
case "$(jq -er .status <<< "${cmp}")" in
identical)
echo "promote=true" >> "${GITHUB_OUTPUT}"
;;
ahead)
# Only a newer commit that changes the artifact should block promotion;
# docs/talos/renovate commits move the tip without triggering a release.
if jq -e '[(.files // [])[].filename]
| any(startswith("kubernetes/")
or . == ".github/workflows/cluster-release.yaml")' \
<<< "${cmp}" >/dev/null; then
echo "superseded by a newer release commit; skipping promotion"
echo "promote=false" >> "${GITHUB_OUTPUT}"
else
echo "promote=true" >> "${GITHUB_OUTPUT}"
fi
;;
*)
echo "${GITHUB_SHA} is not an ancestor of main; refusing to promote" >&2
exit 1
;;
esacwith if: steps.promote.outputs.promote == 'true' on Promote artifact. This also keeps the behind/diverged cases as hard failures, which is the property the current check was presumably after.
(Note the compare API caps files at 300 — for a range that large you'd want the != 0 file-count fallback, though that's unlikely between two consecutive main commits.)
|
One blocking-ish issue in the promotion gate (inline on the Require main tip step) — the A few rollout notes for the gate in the PR description: GHCR package visibility. The Renovate changes don't take effect in this PR. Minor: the flux version is now two independent deps — the action digest ( Also: nothing here validates the manifests, despite what the CodeRabbit summary claims. Not a regression — Flux tracks main directly today and konflate gates PRs — but a green Cluster Release shouldn't be read as a manifest gate once The rest checks out, and a couple of the non-obvious choices are right:
|
1 participant
Summary
kubernetes/tree as an OCI artifact on pushes tomainlatestworkflow_dispatchruns from non-main refsRollout gate
This is step 1 of the OCI migration. After merge, confirm the Cluster Release workflow succeeds and
ghcr.io/jfroy/flatops/cluster:latestexists before merging the next PR. This seeds the signed artifact without changing Flux.Stack
Summary by CodeRabbit
latestreference.