feat(flux): bootstrap verified OCI source - #2164
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
konflate — summaryNote +2 added · 0 changed · −3 removed — 5 resources · 1 app Caution 1 render failure
konflate · rendered |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| resources: | ||
| - apiVersion: source.toolkit.fluxcd.io/v1 | ||
| kind: GitRepository | ||
| name: flux-system | ||
| - apiVersion: kustomize.toolkit.fluxcd.io/v1 | ||
| kind: Kustomization | ||
| kind: OCIRepository | ||
| name: flux-system |
There was a problem hiding this comment.
Dropping GitRepository/flux-system (and Kustomization/flux-system) from resources leaves the cluster with no push-triggered reconciliation at all for the duration of the stack.
After this PR:
Receiver/github-webhookis gone, so the webhook path registered in GitHub repo settings (sha256(github-webhook + flux-system + token)) 404s, and the new Receiver can't accept it anyway — it'sgeneric-oidc, notgithub/HMAC.- The only resource this Receiver pokes is
OCIRepository/flux-system, which nothing consumes yet:kubernetes/cluster/ks.yaml(cluster-apps,cluster-vap) and the FluxInstancesync(kind: GitRepository,interval: 1h) all still source from Git.
So from merge until stack step 4 lands, every change to main waits on the 1h sync interval plus each child Kustomization's own 1h interval instead of reconciling in seconds.
Cheap fix — keep the Git source in the list until the source migration completes, then remove it:
| resources: | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: GitRepository | |
| name: flux-system | |
| - apiVersion: kustomize.toolkit.fluxcd.io/v1 | |
| kind: Kustomization | |
| kind: OCIRepository | |
| name: flux-system | |
| resources: | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: OCIRepository | |
| name: flux-system | |
| # TODO: drop once the sync source migrates to OCIRepository (stack step 4). | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: GitRepository | |
| name: flux-system |
| # Keep the source in place when FluxInstance takes over its management. | ||
| kustomize.toolkit.fluxcd.io/prune: Disabled | ||
| spec: | ||
| interval: 5m |
There was a problem hiding this comment.
interval: 5m deviates from every other OCIRepository in this repo (app-template, flux-instance, flux-operator all use 1h) and from the FluxInstance sync interval. Because ref.tag: latest is mutable, this re-resolves the tag against GHCR 12×/hour, and each resolution runs keyless cosign verification against Fulcio/Rekor. The Receiver added in this PR is the push path, so the poll interval is only a fallback — 1h matches convention and keeps sigstore traffic down.
| interval: 5m | |
| interval: 1h |
| for attempt in $(seq 1 30); do | ||
| if curl --fail-with-body --silent --show-error \ | ||
| --request POST \ | ||
| --header "Authorization: Bearer ${token}" \ | ||
| "${FLUX_RECEIVER_URL}"; then | ||
| exit 0 | ||
| fi | ||
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | ||
| sleep 10 |
There was a problem hiding this comment.
The retry loop treats every non-2xx the same, so a permanent rejection burns the full 5 minutes and then fails with no useful signal. That is the likely failure mode here: if any of the three CEL validations doesn't match, notification-controller returns 403, and if the hardcoded path drifts from status.webhookPath it returns 404 forever. Both are indistinguishable from "Receiver not rolled out yet".
Consider capturing the status code and only retrying on connection errors / 5xx (and 404 for the first few attempts), failing fast and loudly on 403:
code="$(curl --silent --output /dev/stderr --write-out '%{http_code}' \
--request POST --header "Authorization: Bearer ${token}" "${FLUX_RECEIVER_URL}")"
case "${code}" in
2*) exit 0 ;;
403) echo "Receiver rejected the OIDC token (CEL validation failed)"; exit 1 ;;
esac
ReviewVerified independently:
Three comments inline. The one that matters is the Receiver's Two things I could not verify from this environment (no cluster or network access), and which your rollout gate should cover explicitly:
Neither is blocking on its own — nothing depends on |
| resources: | ||
| - apiVersion: source.toolkit.fluxcd.io/v1 | ||
| kind: GitRepository | ||
| name: flux-system | ||
| - apiVersion: kustomize.toolkit.fluxcd.io/v1 | ||
| kind: Kustomization | ||
| kind: OCIRepository | ||
| name: flux-system |
There was a problem hiding this comment.
This drops push-triggered reconciliation of the Git source, and nothing consumes OCIRepository/flux-system yet.
Every Kustomization in the repo still has sourceRef: {kind: GitRepository, name: flux-system} — cluster-apps and cluster-vap (kubernetes/cluster/ks.yaml), flux-instance, and the FluxInstance sync itself (sync.kind: GitRepository, interval: 1h in instance/ks/helm-values.yaml). The old github-webhook Receiver poked GitRepository/flux-system and Kustomization/flux-system; the new one pokes only the OCI source. So from the moment this merges until PRs 3 and 4 land, every merge to main waits on the 1h GitRepository interval.
That also creates a bootstrap gap for this PR itself: Receiver/cluster-release only exists after flux-instance reconciles the renamed manifest, which is now up to an hour after merge — well past the 5-minute retry window in the workflow. The Notify Flux step on this PR's own merge commit will almost certainly exhaust its 30 attempts unless you flux reconcile source git flux-system && flux reconcile ks flux-instance manually.
Since the release workflow runs on push to main, keeping the Git resources in the trigger list restores the old behaviour for free and can be removed in PR 4:
| resources: | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: GitRepository | |
| name: flux-system | |
| - apiVersion: kustomize.toolkit.fluxcd.io/v1 | |
| kind: Kustomization | |
| kind: OCIRepository | |
| name: flux-system | |
| resources: | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: OCIRepository | |
| name: flux-system | |
| # Transitional: most Kustomizations still source from Git until the | |
| # source migration completes. Remove once the migration lands. | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: GitRepository | |
| name: flux-system | |
| - apiVersion: kustomize.toolkit.fluxcd.io/v1 | |
| kind: Kustomization | |
| name: flux-system |
| # Must match status.webhookPath for Receiver cluster-release/flux-system. | ||
| FLUX_RECEIVER_URL: https://flux-webhook.kantai.xyz/hook/ef1ac25eeff8057bf82070418a3eabb755c4821b29ff32377c75863bc1692ab5 |
There was a problem hiding this comment.
This digest can't be validated from Git, and this PR changes both inputs that feed it.
status.webhookPath is computed by notification-controller from the Receiver object itself, and this PR renames it (github-webhook → cluster-release) and removes secretRef — so the path will not be whatever the currently-deployed github-webhook Receiver reports, and it isn't the sha256 of the old token either. git log -S shows this hash is new in this PR, so it wasn't read off a live object.
Please confirm it against the live object after the Receiver reconciles, before relying on it:
kubectl -n flux-system get receiver cluster-release -o jsonpath='{.status.webhookPath}'Two follow-ons worth noting since the value is a literal:
- notification-controller derives the path partly from Receiver-instance identity, so any prune-and-recreate of
Receiver/cluster-release(namespace rebuild, moving the manifest between Kustomizations,specimmutable-field churn) rotates the path and silently breaks releases with no signal in Git. APrometheusRuleon the workflow failing, or akubectllookup at the start of theNotify Fluxstep, would catch it. - there is no
Receiver/github-webhook→cluster-releasemigration for the GitHub-side webhook. The repo/org webhook still points at the old/hook/<old-digest>path and will start returning 404 on every push once the old Receiver is pruned. It should be deleted in the GitHub UI as part of this rollout.
| for attempt in $(seq 1 30); do | ||
| if curl --fail-with-body --silent --show-error \ | ||
| --request POST \ | ||
| --header "Authorization: Bearer ${token}" \ | ||
| "${FLUX_RECEIVER_URL}"; then | ||
| exit 0 | ||
| fi | ||
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | ||
| sleep 10 | ||
| done | ||
| exit 1 |
There was a problem hiding this comment.
The retry loop can't distinguish "Receiver not rolled out yet" from "permanently misconfigured", so both cost 5 minutes and produce the same misleading log line.
--fail-with-body makes curl fail on any 4xx/5xx. A wrong FLUX_RECEIVER_URL digest → 404, a rejected OIDC token (bad audience, or a validations expression that doesn't match) → 401/403. None of those become truthy on retry, but each burns 30 attempts and prints Flux Receiver unavailable, which points the reader at the cluster rather than at the token or the URL.
Capturing the status code lets you retry only what's actually transient and surface the CEL rejection message from the Receiver on the terminal cases:
| for attempt in $(seq 1 30); do | |
| if curl --fail-with-body --silent --show-error \ | |
| --request POST \ | |
| --header "Authorization: Bearer ${token}" \ | |
| "${FLUX_RECEIVER_URL}"; then | |
| exit 0 | |
| fi | |
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | |
| sleep 10 | |
| done | |
| exit 1 | |
| for attempt in $(seq 1 30); do | |
| body="$(mktemp)" | |
| code="$(curl --silent --show-error \ | |
| --output "${body}" --write-out '%{http_code}' \ | |
| --request POST \ | |
| --header "Authorization: Bearer ${token}" \ | |
| "${FLUX_RECEIVER_URL}")" | |
| case "${code}" in | |
| 2*) | |
| exit 0 | |
| ;; | |
| 404|502|503|504) | |
| echo "Flux Receiver unavailable (HTTP ${code}, attempt ${attempt}/30); retrying in 10 seconds" | |
| ;; | |
| *) | |
| echo "Flux Receiver rejected the notification (HTTP ${code}):" | |
| cat "${body}" | |
| exit 1 | |
| ;; | |
| esac | |
| sleep 10 | |
| done | |
| exit 1 |
Note that 404 stays in the retry set on purpose — notification-controller returns 404 both for an unknown path and for a Receiver it hasn't indexed yet, so it's ambiguous. That makes the status.webhookPath check in my other comment the thing that actually distinguishes them.
| annotations: | ||
| # Keep the source in place when FluxInstance takes over its management. | ||
| kustomize.toolkit.fluxcd.io/prune: Disabled | ||
| spec: | ||
| interval: 5m | ||
| ref: | ||
| tag: latest | ||
| url: oci://ghcr.io/jfroy/flatops/cluster |
There was a problem hiding this comment.
Two things to confirm on this source, both of which the rollout gate would catch but are cheaper to check up front:
No pull credentials. There's no secretRef / serviceAccountName / provider, and there is no registry secret anywhere under kubernetes/apps/flux-system/ — every other OCIRepository in the repo targets a public upstream (bjw-s-labs, controlplaneio-fluxcd). This only works if the jfroy/flatops/cluster GHCR package is public. Packages published with GITHUB_TOKEN inherit repo visibility, so it probably is, but worth verifying on the package settings page since a private package fails both the pull and the cosign signature lookup with a 401 that reads like a verification failure.
prune: Disabled needs the file removed in PR 4, not just left in place. The annotation stops kustomize-controller from deleting the object, but it does not stop it from applying it. Once FluxInstance switches to sync.kind: OCIRepository it will manage OCIRepository/flux-system too, and if this file is still in kubernetes/cluster/ both controllers will apply competing specs on every reconcile. Deleting the file in the final PR is what makes the annotation do its job — the object survives unowned and flux-operator adopts it.
|
The signing/verification chain lines up: Four things flagged inline. The two that gate the rollout:
The other two are smaller: the retry loop can't tell a 404/403 misconfiguration from a slow rollout, and the new One addition to the rollout gate in the description: the GitHub-side repo webhook pointing at the old |
| # Must match status.webhookPath for Receiver cluster-release/flux-system. | ||
| FLUX_RECEIVER_URL: https://flux-webhook.kantai.xyz/hook/ef1ac25eeff8057bf82070418a3eabb755c4821b29ff32377c75863bc1692ab5 |
There was a problem hiding this comment.
This literal path can't be right yet, and it's the one thing here that can't be validated before merge.
status.webhookPath is generated by notification-controller as a SHA-256 digest that includes the Receiver's UID (plus the token, empty here). This PR renames the Receiver — github-webhook → cluster-release — so kustomize-controller will create a brand-new object with a brand-new UID, and its path will not be whatever was read out of the cluster to produce this value. The only way this digest is already correct is if Receiver/cluster-release was applied to the cluster out-of-band and Flux then adopts that same object (same UID); if it ever gets pruned and recreated, the path silently changes again.
Failure mode is quiet: the merge run publishes and promotes the artifact fine, then Notify Flux POSTs to a path notification-controller doesn't serve, gets 404 for 30 attempts, and fails the job after 5 minutes — while the cluster keeps running old state until the 1h Git interval fires.
Please confirm after this reconciles:
kubectl -n flux-system get receiver cluster-release -o jsonpath='{.status.webhookPath}'And consider moving the URL out of the tree into a repository variable (vars.FLUX_RECEIVER_URL), so a Receiver recreation is a settings change rather than a commit that re-triggers the very workflow it's fixing.
| token="$(curl --fail-with-body --silent --show-error \ | ||
| --header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ | ||
| "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \ | ||
| | jq -er '.value')" | ||
| for attempt in $(seq 1 30); do | ||
| if curl --fail-with-body --silent --show-error \ | ||
| --request POST \ | ||
| --header "Authorization: Bearer ${token}" \ | ||
| "${FLUX_RECEIVER_URL}"; then | ||
| exit 0 | ||
| fi | ||
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | ||
| sleep 10 | ||
| done |
There was a problem hiding this comment.
The ID token is minted once, outside a loop that can run for 5 minutes. GitHub Actions OIDC tokens are short-lived (exp - iat is on the order of minutes), so in exactly the scenario this retry exists for — Receiver not yet reconciled — the later attempts can start failing 401 on an expired token rather than 404 on a missing path, and the log line will still say "Flux Receiver unavailable". Minting inside the loop is cheap and removes the ambiguity:
| token="$(curl --fail-with-body --silent --show-error \ | |
| --header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ | |
| "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \ | |
| | jq -er '.value')" | |
| for attempt in $(seq 1 30); do | |
| if curl --fail-with-body --silent --show-error \ | |
| --request POST \ | |
| --header "Authorization: Bearer ${token}" \ | |
| "${FLUX_RECEIVER_URL}"; then | |
| exit 0 | |
| fi | |
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | |
| sleep 10 | |
| done | |
| run: | | |
| get_token() { | |
| curl --fail-with-body --silent --show-error \ | |
| --header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ | |
| "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \ | |
| | jq -er '.value' | |
| } | |
| for attempt in $(seq 1 30); do | |
| token="$(get_token)" | |
| if curl --fail-with-body --silent --show-error \ | |
| --request POST \ | |
| --header "Authorization: Bearer ${token}" \ | |
| "${FLUX_RECEIVER_URL}"; then | |
| exit 0 | |
| fi | |
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | |
| sleep 10 | |
| done |
Separately: a 403 from a failed CEL validation is permanent, and retrying it 30 times buys nothing but 5 minutes of confusing logs. Worth breaking out of the loop on 4xx-other-than-404 if you want the misconfiguration to surface fast.
| resources: | ||
| - apiVersion: source.toolkit.fluxcd.io/v1 | ||
| kind: GitRepository | ||
| name: flux-system | ||
| - apiVersion: kustomize.toolkit.fluxcd.io/v1 | ||
| kind: Kustomization | ||
| kind: OCIRepository | ||
| name: flux-system |
There was a problem hiding this comment.
Dropping GitRepository/flux-system (and Kustomization/flux-system) from resources leaves the Git source with no notification path for the duration of the migration.
Everything still consuming Git today — the FluxInstance sync GitRepository (interval: 1h), cluster-apps, cluster-vap, flux-instance — only picks up a merge on its 1h poll once the old github Receiver is pruned. That's the window in which stack PRs 3 and 4 land, so those two merges go from near-instant to up-to-an-hour reconcile unless you flux reconcile kustomization cluster-apps --with-source by hand each time.
Cheap fix while the stack is in flight — the release workflow already fires on every kubernetes/** push to main, so one notification can poke both sources:
| resources: | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: GitRepository | |
| name: flux-system | |
| - apiVersion: kustomize.toolkit.fluxcd.io/v1 | |
| kind: Kustomization | |
| kind: OCIRepository | |
| name: flux-system | |
| resources: | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: OCIRepository | |
| name: flux-system | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: GitRepository | |
| name: flux-system |
Then drop the GitRepository entry in PR 4 when the Git source goes away.
| spec: | ||
| interval: 5m | ||
| ref: | ||
| tag: latest | ||
| url: oci://ghcr.io/jfroy/flatops/cluster |
There was a problem hiding this comment.
No secretRef, so source-controller pulls anonymously. GHCR packages created by a GITHUB_TOKEN push default to private — unlike the other ghcr.io/jfroy/* images in this repo, which are all pulled without pull secrets because they've been flipped to public. If ghcr.io/jfroy/flatops/cluster was auto-created by the artifact-publisher run and nobody changed its visibility, this will sit in 401 Unauthorized rather than Ready.
Your rollout gate ("OCIRepository/flux-system is Ready and SourceVerified") catches it, but worth confirming the package visibility before merging rather than after.
|
Reviewed the OCI bootstrap. The pieces that matter check out: Four things flagged inline, one of which I would resolve before merging:
Two bits of housekeeping the diff implies but does not cover: the GitHub repository webhook pointing at the old |
| namespace: flux-system | ||
| annotations: | ||
| # Keep the source in place when FluxInstance takes over its management. | ||
| kustomize.toolkit.fluxcd.io/prune: Disabled |
There was a problem hiding this comment.
Disabled is capitalized here. Flux's documented value for this annotation is disabled, and kustomize-controller passes kustomizev1.DisabledValue ("disabled") as an exact-match exclusion when computing the prune set — every other occurrence in this repo uses lowercase (kubernetes/components/common/kustomization.yaml:15, the eight objectbucketclaim.yaml files).
If the match is case-sensitive, this annotation is a no-op, which defeats its stated purpose: when PR 4 deletes this file, cluster-apps/root flux-system will garbage-collect OCIRepository/flux-system — the source the root Kustomization is by then reading from — instead of leaving it for FluxInstance to adopt.
| kustomize.toolkit.fluxcd.io/prune: Disabled | |
| kustomize.toolkit.fluxcd.io/prune: disabled |
| resources: | ||
| - apiVersion: source.toolkit.fluxcd.io/v1 | ||
| kind: GitRepository | ||
| name: flux-system | ||
| - apiVersion: kustomize.toolkit.fluxcd.io/v1 | ||
| kind: Kustomization | ||
| kind: OCIRepository | ||
| name: flux-system |
There was a problem hiding this comment.
This drops push-triggered reconciliation for the Git source, which nothing consumes the OCI source yet to replace.
After this merges, OCIRepository/flux-system is the only notified resource, but FluxInstance.spec.sync is still kind: GitRepository (instance/ks/helm-values.yaml) and every Kustomization — cluster-vap, cluster-apps, and all app ks.yaml — still has sourceRef.kind: GitRepository. The old github-webhook Receiver and its path are gone, so for the duration of PRs 3–4 a push to main only reconciles at interval: 1h instead of within seconds.
Since the Cluster Release workflow already fires on the same push: main + kubernetes/** trigger, adding the Git source to this Receiver restores that for free until the migration completes:
| resources: | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: GitRepository | |
| name: flux-system | |
| - apiVersion: kustomize.toolkit.fluxcd.io/v1 | |
| kind: Kustomization | |
| kind: OCIRepository | |
| name: flux-system | |
| resources: | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: OCIRepository | |
| name: flux-system | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: GitRepository | |
| name: flux-system |
| token="$(curl --fail-with-body --silent --show-error \ | ||
| --header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ | ||
| "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \ | ||
| | jq -er '.value')" | ||
| for attempt in $(seq 1 30); do | ||
| if curl --fail-with-body --silent --show-error \ | ||
| --request POST \ | ||
| --header "Authorization: Bearer ${token}" \ | ||
| "${FLUX_RECEIVER_URL}"; then | ||
| exit 0 | ||
| fi | ||
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | ||
| sleep 10 | ||
| done | ||
| exit 1 |
There was a problem hiding this comment.
The OIDC token is minted once, outside a loop that can run for 5 minutes (30 × 10s). GitHub Actions ID tokens are short-lived, so the later attempts can start failing on an expired exp rather than on the Receiver being unavailable — and since every non-2xx is treated identically, the step would burn the full window and then fail for a reason the log message misattributes.
Minting inside the loop keeps the retry meaningful:
| token="$(curl --fail-with-body --silent --show-error \ | |
| --header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ | |
| "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \ | |
| | jq -er '.value')" | |
| for attempt in $(seq 1 30); do | |
| if curl --fail-with-body --silent --show-error \ | |
| --request POST \ | |
| --header "Authorization: Bearer ${token}" \ | |
| "${FLUX_RECEIVER_URL}"; then | |
| exit 0 | |
| fi | |
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | |
| sleep 10 | |
| done | |
| exit 1 | |
| - name: Notify Flux | |
| run: | | |
| for attempt in $(seq 1 30); do | |
| token="$(curl --fail-with-body --silent --show-error \ | |
| --header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ | |
| "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \ | |
| | jq -er '.value')" | |
| if curl --fail-with-body --silent --show-error \ | |
| --request POST \ | |
| --header "Authorization: Bearer ${token}" \ | |
| "${FLUX_RECEIVER_URL}"; then | |
| exit 0 | |
| fi | |
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | |
| sleep 10 | |
| done | |
| exit 1 |
| url: oci://ghcr.io/jfroy/flatops/cluster | ||
| verify: | ||
| provider: cosign | ||
| matchOIDCIdentity: | ||
| - issuer: ^https://token[.]actions[.]githubusercontent[.]com$ | ||
| subject: ^https://github[.]com/jfroy/flatops/[.]github/workflows/cluster-release[.]yaml@refs/heads/main$ |
There was a problem hiding this comment.
Two things to confirm before merging, both of which fail as "not Ready" on the root source rather than anywhere diagnosable:
-
No
secretRef. There is no image pull secret on anyOCIRepositoryin this repo, so this only works ifghcr.io/jfroy/flatops/clusteris public. GHCR packages published byGITHUB_TOKENare created private by default and do not inherit repository visibility — the package's visibility has to be flipped manually once. Worth adding to the rollout gate. -
--new-bundle-format=truein the sign step (line 84). That stores the signature as a sigstore bundle rather than the classicsha256-<digest>.sigtag. Please confirm the deployed source-controller's cosign verifier resolves it;cosign verifyin the same job passing does not prove Flux can, since it uses a pinned cosign library version. IfSourceVerifieddoesn't go true, dropping--new-bundle-formatfrom bothcosign signandcosign verifyis the fallback.
The identity regexes themselves match what the job asserts via --certificate-identity "${GITHUB_SERVER_URL}/${GITHUB_WORKFLOW_REF}".
|
Reviewed against AGENTS.md. Four inline comments; the annotation casing one is the only thing I'd call a bug. Verified as correct:
Housekeeping once the old Receiver is gone: the GitHub repository webhook pointing at the old |
| resources: | ||
| - apiVersion: source.toolkit.fluxcd.io/v1 | ||
| kind: GitRepository | ||
| name: flux-system | ||
| - apiVersion: kustomize.toolkit.fluxcd.io/v1 | ||
| kind: Kustomization | ||
| kind: OCIRepository | ||
| name: flux-system |
There was a problem hiding this comment.
This replaces the push trigger before its replacement is consumed by anything. After this PR, the only Receiver in the repo notifies OCIRepository/flux-system — but nothing sources from it yet: cluster-apps and cluster-vap (kubernetes/cluster/ks.yaml), flux-instance (kubernetes/apps/flux-system/instance/ks.yaml), and every other ks.yaml still use sourceRef.kind: GitRepository.
Net effect for the window between this PR and stack step 4:
GitRepository/flux-systemandKustomization/flux-systemlose their webhook, so a push tomainreconciles only on the FluxInstance syncinterval: 1h(ks/helm-values.yaml).- The GitHub repo webhook still points at the old
/hook/<sha256(github-webhook+flux-system+token)>path, which no longer exists → 404s on every delivery.
Since Receivers are additive, consider keeping the github-webhook Receiver (and its externalsecret.yaml) in place alongside this one and deleting both in step 4, once the Kustomizations actually consume the OCI source.
| token="$(curl --fail-with-body --silent --show-error \ | ||
| --header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ | ||
| "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \ | ||
| | jq -er '.value')" | ||
| for attempt in $(seq 1 30); do | ||
| if curl --fail-with-body --silent --show-error \ | ||
| --request POST \ | ||
| --header "Authorization: Bearer ${token}" \ | ||
| "${FLUX_RECEIVER_URL}"; then | ||
| exit 0 | ||
| fi | ||
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | ||
| sleep 10 | ||
| done | ||
| exit 1 |
There was a problem hiding this comment.
Two issues with the retry loop:
-
The ID token is minted once, outside the loop, but the loop spans ~5 minutes (30 × 10s). GitHub Actions OIDC tokens are short-lived, and notification-controller validates
exp, so the later attempts can fail on token expiry rather than on Receiver availability — i.e. the retry never actually helps in the tail of the window. Move the token request inside the loop. -
It retries indiscriminately. A permanent rejection — a CEL validation failure (403) from a claim mismatch, or a 404 from a webhook-path drift — burns the full 5 minutes before failing, which reads as "Receiver unavailable" rather than "misconfigured". Capturing the HTTP status (
--write-out '%{http_code}'/-o /dev/null) and bailing out on 4xx other than 404-during-rollout would make the failure mode legible.
| url: oci://ghcr.io/jfroy/flatops/cluster | ||
| verify: | ||
| provider: cosign | ||
| matchOIDCIdentity: | ||
| - issuer: ^https://token[.]actions[.]githubusercontent[.]com$ | ||
| subject: ^https://github[.]com/jfroy/flatops/[.]github/workflows/cluster-release[.]yaml@refs/heads/main$ |
There was a problem hiding this comment.
No secretRef/serviceAccountName, so this relies on ghcr.io/jfroy/flatops/cluster being an anonymously pullable package — same as ghcr.io/jfroy/charts/zfs-static-csi. GHCR packages published via GITHUB_TOKEN are private on first publish regardless of repository visibility; visibility has to be flipped manually once. Worth adding to the rollout gate explicitly, since the symptom is OCIRepository stuck on a 401 with a UNAUTHORIZED message rather than an obvious auth-config error.
|
Wiring and Flux plumbing check out on the mechanical points:
Three comments inline. The ordering one on I could not independently verify the |
| resources: | ||
| - apiVersion: source.toolkit.fluxcd.io/v1 | ||
| kind: GitRepository | ||
| name: flux-system | ||
| - apiVersion: kustomize.toolkit.fluxcd.io/v1 | ||
| kind: Kustomization | ||
| kind: OCIRepository | ||
| name: flux-system |
There was a problem hiding this comment.
Dropping GitRepository/flux-system here leaves the cluster with no push-triggered git reconciliation for the whole duration of the migration.
Every ks.yaml in the repo — plus the FluxInstance-generated flux-system Kustomization itself (helm-values.yaml: sync.kind: GitRepository, interval: 1h) — still resolves sourceRef to GitRepository/flux-system. Steps 3 and 4 of the stack are what move those over. Until then, the only thing that poked the git source was the old github-webhook Receiver, and this PR deletes it, so the GitHub push webhook configured in repo settings starts hitting an unknown /hook/<old-hash> path (404). Net effect after merge: kubernetes/** changes land on the cluster up to an hour late, and the new Receiver only refreshes an OCIRepository nothing consumes yet.
Since the release workflow runs on every push to main that touches kubernetes/**, listing both sources keeps the fast path alive through the migration window:
| resources: | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: GitRepository | |
| name: flux-system | |
| - apiVersion: kustomize.toolkit.fluxcd.io/v1 | |
| kind: Kustomization | |
| kind: OCIRepository | |
| name: flux-system | |
| resources: | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: OCIRepository | |
| name: flux-system | |
| - apiVersion: source.toolkit.fluxcd.io/v1 | |
| kind: GitRepository | |
| name: flux-system |
The GitRepository entry can then be removed in step 4 along with the git source itself.
| token="$(curl --fail-with-body --silent --show-error \ | ||
| --header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ | ||
| "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \ | ||
| | jq -er '.value')" | ||
| for attempt in $(seq 1 30); do | ||
| if curl --fail-with-body --silent --show-error \ | ||
| --request POST \ | ||
| --header "Authorization: Bearer ${token}" \ | ||
| "${FLUX_RECEIVER_URL}"; then | ||
| exit 0 | ||
| fi | ||
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | ||
| sleep 10 | ||
| done |
There was a problem hiding this comment.
The OIDC token is minted once, outside a loop that can run for ~5 minutes (30 × 10s). GitHub Actions ID tokens are short-lived, so in exactly the scenario this retry exists for — a Receiver that stays unavailable through the initial rollout — the later attempts can start failing on an expired exp rather than on the Receiver, turning a recoverable wait into a permanent failure with a misleading Flux Receiver unavailable message. Minting per attempt is cheap and removes the coupling between the retry budget and the token lifetime.
| token="$(curl --fail-with-body --silent --show-error \ | |
| --header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ | |
| "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \ | |
| | jq -er '.value')" | |
| for attempt in $(seq 1 30); do | |
| if curl --fail-with-body --silent --show-error \ | |
| --request POST \ | |
| --header "Authorization: Bearer ${token}" \ | |
| "${FLUX_RECEIVER_URL}"; then | |
| exit 0 | |
| fi | |
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | |
| sleep 10 | |
| done | |
| for attempt in $(seq 1 30); do | |
| token="$(curl --fail-with-body --silent --show-error \ | |
| --header "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ | |
| "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=notification-controller" \ | |
| | jq -er '.value')" | |
| if curl --fail-with-body --silent --show-error \ | |
| --request POST \ | |
| --header "Authorization: Bearer ${token}" \ | |
| "${FLUX_RECEIVER_URL}"; then | |
| exit 0 | |
| fi | |
| if [[ "${attempt}" -lt 30 ]]; then | |
| echo "Flux Receiver unavailable (attempt ${attempt}/30); retrying in 10 seconds" | |
| sleep 10 | |
| fi | |
| done | |
| exit 1 |
(The -lt 30 guard also drops the pointless 10s sleep before the final exit 1.)
Separately, worth considering: the loop retries indiscriminately, so a genuine misconfiguration — a CEL validation that never matches, or a stale FLUX_RECEIVER_URL — burns the full five minutes as 401/404 before failing. A fast-fail on those statuses would make a broken Receiver config much more obvious than a wall of "unavailable" lines.
|
Reviewed the OCI bootstrap wiring. The pieces that are easy to get wrong here check out:
Two inline comments, one of which I would treat as a merge blocker for the migration window: the Receiver no longer pokes One forward-looking note on Also flagging that the rollout gate is doing real work here: nothing consumes |
1 participant
Summary
OCIRepository/flux-systemalongside the existing Git sourcecluster-release.yamlonmaingeneric-oidcReceiver restricted to the exact repository, ref, and workflowThe bootstrap source carries
kustomize.toolkit.fluxcd.io/prune: Disabledso it remains available when FluxInstance takes over ownership in the final migration PR.Rollout gate
Merge only after the artifact-publisher PR has completed successfully. Before advancing the stack, confirm:
OCIRepository/flux-systemis Ready and SourceVerifiedReceiver/cluster-releaseis ReadyThe deployed Receiver CRD was checked and supports
generic-oidcwithoutsecretRef.Stack