feat(flux): complete cluster OCI migration - #2161
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
konflate — summaryNote +541 added · 126 changed · −1 removed — 668 resources · 101 apps · 168 not shown Blast radius
Warning ⚠ Cautions
Caution 51 render failures
Image changes
konflate · rendered |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| sourceRef: | ||
| kind: GitRepository | ||
| kind: OCIRepository | ||
| name: flux-system | ||
| namespace: flux-system |
There was a problem hiding this comment.
Flipping cluster-apps/cluster-vap to OCIRepository in the same commit that flips the FluxInstance sync creates a bootstrap deadlock on merge.
At merge time the live Flux is still syncing from GitRepository/flux-system. The root sync Kustomization applies this file, so cluster-apps and cluster-vap are updated to sourceRef.kind: OCIRepository — but OCIRepository/flux-system does not exist yet, because the only thing that creates it is the updated flux-instance HelmRelease values, which are delivered through cluster-apps. Both root Kustomizations stall on "OCIRepository/flux-system not found" and the FluxInstance change is never applied. The 148 child Kustomizations keep running off their currently-applied (Git-sourced) specs, so the cluster silently freezes at the pre-merge revision until someone intervenes by hand.
There's also a race even if the source did exist: cluster-release.yaml only runs on push to main, so the artifact is being built at the same moment Flux is reconciling the switch.
Suggested sequencing:
- Land
cluster-release.yamlalone, run it (orworkflow_dispatchonmain), and confirmoci://ghcr.io/jfroy/flatops/cluster:latestexists, is signed, and is pullable by source-controller. - Land the
FluxInstancesync switch soOCIRepository/flux-systemis created and Ready. - Only then land the 148
sourceRefflips.
If the plan is instead to do step 2 by hand before merging, please note that in the PR description — as written this PR is not self-applying.
| sync: | ||
| kind: GitRepository | ||
| url: https://github.com/jfroy/flatops | ||
| ref: refs/heads/main | ||
| kind: OCIRepository | ||
| url: oci://ghcr.io/jfroy/flatops/cluster | ||
| ref: latest | ||
| path: kubernetes/cluster | ||
| interval: 1h | ||
| interval: 5m |
There was a problem hiding this comment.
No pull credentials are configured for the new source. spec.sync has no pullSecret, and the only patch on OCIRepository/flux-system adds spec.verify — nothing adds secretRef/serviceAccountName.
GHCR packages are private on first push, and ghcr.io/jfroy/flatops/cluster is new (the existing ghcr.io/jfroy/flatops/* packages don't cover it). If the package inherits private visibility, source-controller gets 401 Unauthorized on every poll and the entire cluster loses its source of truth — same failure mode as the ordering issue in kubernetes/cluster/ks.yaml, but permanent.
Either set the package to public (the artifact only contains manifests and ExternalSecret references, no plaintext secrets, so that's viable) or add a pullSecret with a GHCR read token. Worth stating which one was chosen, since it's cluster-fatal and not visible from the repo.
Also note the signature verification is doing double duty here: with a public package, spec.verify is what keeps an unrelated pusher from being trusted, so it's load-bearing rather than defence-in-depth. Please confirm kustomize.patches actually reach the generated sync source in your flux-operator version (i.e. the live OCIRepository shows spec.verify after rollout) — kustomize silently no-ops a patches entry whose target doesn't match, so a wrong target here fails open rather than loudly.
| on: | ||
| workflow_dispatch: |
There was a problem hiding this comment.
workflow_dispatch is unguarded, so this can be dispatched from any branch. The run will flux push artifact a :<sha> tag and then cosign sign it with that branch's OIDC identity before cosign verify (pinned to @refs/heads/main) fails the job. Promotion to latest is correctly not reached, so the cluster is safe, but you end up with signed non-main artifacts and Rekor entries plus a red run.
Cheap fix — guard the job:
jobs:
release:
if: github.ref == 'refs/heads/main'
| # The generic-oidc Receiver path is SHA-256(receiver name + namespace). | ||
| FLUX_RECEIVER_URL: https://flux-webhook.kantai.xyz/hook/ef1ac25eeff8057bf82070418a3eabb755c4821b29ff32377c75863bc1692ab5 |
There was a problem hiding this comment.
This hash is a magic constant duplicated from live cluster state, and the Receiver was just renamed (github-webhook → cluster-release), so it had to be recomputed. Please confirm it matches status.webhookPath on the reconciled Receiver — a stale value makes every release fail at the Notify Flux step (curl --fail-with-body on a 404) even though the artifact published fine, and the failure is indistinguishable from a real problem.
Longer term the coupling is unfortunate: renaming the Receiver silently breaks releases. Not blocking, but the comment should probably also say "must match status.webhookPath of Receiver cluster-release in flux-system" rather than restating the formula, since the formula is an implementation detail of notification-controller.
|
Two things I think need resolving before this merges, both left as inline comments:
Smaller items:
What I did verify:
|
jfroy
changed the title
|
Split the migration into a staged stack to remove the bootstrap dependency and keep every review below 100 files:
This PR now targets |
| # Only reconcile cluster state from artifacts signed by this workflow on main. | ||
| - patch: |- | ||
| - op: add | ||
| path: /spec/verify | ||
| value: | ||
| provider: cosign | ||
| matchOIDCIdentity: | ||
| - issuer: ^https://token[.]actions[.]githubusercontent[.]com$ | ||
| subject: ^https://github[.]com/jfroy/flatops/[.]github/workflows/cluster-release[.]yaml@refs/heads/main$ | ||
| target: | ||
| group: source.toolkit.fluxcd.io | ||
| kind: OCIRepository | ||
| name: flux-system |
There was a problem hiding this comment.
Moving spec.verify from the deleted kubernetes/cluster/ocirepository.yaml into a JSON6902 patch on the flux-operator-generated sync source makes signature enforcement fail open: kustomize silently no-ops a patch whose target matches nothing, so if flux-operator ever changes the generated resource's name/group (or this stanza gets a typo), the cluster keeps reconciling unsigned artifacts with no error anywhere.
Worth confirming after rollout that the field actually landed:
kubectl -n flux-system get ocirepository flux-system -o jsonpath='{.spec.verify}'
(and ideally asserting it in the release workflow or a VAP, since this is the only thing standing between a GHCR push and cluster-wide apply).
|
Source flip looks complete and consistent — no
Nothing reconciles until the first successful run of this workflow.
The artifact-content guard only checks one path. |
| kind: OCIRepository | ||
| url: oci://ghcr.io/jfroy/flatops/cluster | ||
| ref: latest | ||
| path: kubernetes/cluster | ||
| interval: 1h | ||
| interval: 5m |
There was a problem hiding this comment.
spec.sync has no pullSecret, so ghcr.io/jfroy/flatops/cluster must be anonymously pullable. Konflate's check on this PR says it isn't (yet):
OCIRepository flux-system/flux-system resolve latest:
HEAD "https://ghcr.io/v2/jfroy/flatops/cluster/manifests/latest": ...
403: denied: requested access to the resource is denied
That's the same anonymous path source-controller will take. GHCR packages are private by default and cluster-release.yaml never sets visibility or links the package to the repo, so if this 403 is package visibility rather than "artifact not published from main yet", the root source goes NotReady the moment FluxInstance takes over and there is no in-band way to fix it (see the deadlock note on the deleted kubernetes/cluster/ocirepository.yaml).
Please confirm ghcr.io/jfroy/flatops/cluster is public — and that a Konflate re-run is green — before merging, or add pullSecret here.
| namespace: flux-system | ||
| annotations: | ||
| # Keep the source in place when FluxInstance takes over its management. | ||
| kustomize.toolkit.fluxcd.io/prune: Disabled |
There was a problem hiding this comment.
Removing this file is only safe if the live object's prune-disable annotation is actually honored, and the value here is Disabled with a capital D. Every other use in this repo is lowercase:
kubernetes/components/common/kustomization.yaml:15: kustomize.toolkit.fluxcd.io/prune: disabled
kubernetes/apps/database/cnpg/pg18vc/objectbucketclaim.yaml:8: kustomize.toolkit.fluxcd.io/prune: disabled
... (7 more)
kustomize-controller passes kustomize.toolkit.fluxcd.io/prune: disabled as a delete exclusion and compares the annotation value against that exact string, so Disabled likely does not match.
If it doesn't match, this is not a benign leftover — it's a deadlock:
flux-system(still Git-sourced at this point) applieskubernetes/cluster, and prunesOCIRepository/flux-systembecause the file is gone from the artifact.cluster-vap/cluster-apps— repointed toOCIRepository/flux-systemin this same commit — now have no source.- The FluxInstance values change that would recreate that OCIRepository is delivered by
flux-system/flux-instance, which sits undercluster-apps. It never reconciles, so the source is never recreated.
Recovery would require a manual kubectl apply, which AGENTS.md forbids without explicit authorization.
Suggested sequencing: land a one-line fix to lowercase disabled (and confirm kubectl get ocirepository flux-system -n flux-system -o jsonpath='{.metadata.annotations}' reflects it) before dropping the file, or keep the file for one more release and delete it only after spec.verify and FluxInstance ownership are confirmed on the live object.
| # Only reconcile cluster state from artifacts signed by this workflow on main. | ||
| - patch: |- | ||
| - op: add | ||
| path: /spec/verify | ||
| value: | ||
| provider: cosign | ||
| matchOIDCIdentity: | ||
| - issuer: ^https://token[.]actions[.]githubusercontent[.]com$ | ||
| subject: ^https://github[.]com/jfroy/flatops/[.]github/workflows/cluster-release[.]yaml@refs/heads/main$ | ||
| target: | ||
| group: source.toolkit.fluxcd.io | ||
| kind: OCIRepository | ||
| name: flux-system |
There was a problem hiding this comment.
This moves the cluster's only signature gate from a first-class manifest into a Kustomize patch, which fails open: Kustomize silently no-ops a patch whose target selects nothing. If flux-operator ever renames or re-kinds the generated sync source, spec.verify just disappears and the FluxInstance still reports Ready — nothing surfaces the loss.
Two things worth adding:
- Confirm after rollout that the generated object really carries it:
kubectl get ocirepository flux-system -n flux-system -o jsonpath='{.spec.verify}'(and thatSourceVerifiedis on the status). - Consider a
kubernetes/vap/ValidatingAdmissionPolicy requiringspec.verify.provider == "cosign"onOCIRepository/flux-systeminflux-system, so the gate can't be dropped silently.cluster-vapalready runs beforecluster-apps, so it's the right place for this invariant.
|
Mechanical part checks out: all 174 Three things I would want resolved before this lands, all of them about the ownership handoff rather than the mechanical swap. Details inline:
One more question, not blocking: the description says Flate/Konflate aliases bootstrap |
| # Only reconcile cluster state from artifacts signed by this workflow on main. | ||
| - patch: |- | ||
| - op: add | ||
| path: /spec/verify | ||
| value: | ||
| provider: cosign | ||
| matchOIDCIdentity: | ||
| - issuer: ^https://token[.]actions[.]githubusercontent[.]com$ | ||
| subject: ^https://github[.]com/jfroy/flatops/[.]github/workflows/cluster-release[.]yaml@refs/heads/main$ | ||
| target: | ||
| group: source.toolkit.fluxcd.io | ||
| kind: OCIRepository | ||
| name: flux-system |
There was a problem hiding this comment.
Worth confirming this patch actually reaches the sync-generated OCIRepository before kubernetes/cluster/ocirepository.yaml goes away, because a patches entry whose target selects nothing is a silent no-op in kustomize — no error, no warning.
The failure mode is unusually easy to miss here. The live OCIRepository/flux-system already carries spec.verify from the previous kustomize-controller apply, and SSA field ownership for it stays with kustomize-controller once this repo stops managing the object — flux-operator won't strip a field it doesn't own. So a no-op patch looks perfectly healthy on this cluster and only surfaces on a rebuild or if the object is ever recreated, at which point the cluster root reconciles latest with no signature verification at all.
Cheap check after the FluxInstance takes over: kubectl get ocirepository flux-system -n flux-system -o yaml and confirm spec.verify is claimed by a flux-operator entry in metadata.managedFields, not just inherited from kustomize-controller.
| sync: | ||
| kind: GitRepository | ||
| url: https://github.com/jfroy/flatops | ||
| ref: refs/heads/main | ||
| kind: OCIRepository | ||
| url: oci://ghcr.io/jfroy/flatops/cluster | ||
| ref: latest | ||
| path: kubernetes/cluster | ||
| interval: 1h | ||
| interval: 5m |
There was a problem hiding this comment.
After this lands there is no path into the cluster that doesn't go through ghcr.io/jfroy/flatops/cluster:latest with signature verification enforced. Anything that makes the artifact unpullable or unverifiable — Sigstore trust-root or bundle-format change, GHCR outage, an accidental flux tag artifact --tag latest onto a bad digest — freezes all 176 Kustomizations, and the fix cannot be delivered through Git, because the source definition now exists only in these values, which are themselves delivered by that source.
Recovery becomes a manual kubectl patch fluxinstance or a hand-applied replacement source, i.e. exactly the live mutation AGENTS.md tells agents not to perform without authorization. Since this PR already updates AGENTS.md for the new sync model, it'd be worth documenting the break-glass procedure there too — or keeping a suspended GitRepository + Kustomization in kubernetes/cluster/ as a dormant fallback.
| @@ -11,8 +11,6 @@ spec: | |||
| eventSources: | |||
| - kind: FluxInstance | |||
There was a problem hiding this comment.
Dropping GitRepository here is correct, but the sibling Alert in this same component — ../github-status/alert.yaml, wired in via alerts/kustomization.yaml — still watches Kustomization * at eventSeverity: info and feeds the github Provider pointed at https://github.com/jfroy/flatops.
GitHub commit statuses need a git SHA from the event's revision metadata. Once every Kustomization is OCI-sourced, that metadata is the OCIRepository artifact revision, latest@sha256:<oci-digest>, not main@sha1:<commit>. The --revision "${GITHUB_REF_NAME}@sha1:${GITHUB_SHA}" in cluster-release.yaml is artifact annotation metadata and does not become status.artifact.revision, so notification-controller has no commit to attach to and will fail on each Kustomization event — in every namespace this component is applied to, since the alert is namespace-scoped.
Worth resolving in the change that removes the last GitRepository: either drop ./github-status from kubernetes/components/common/alerts/kustomization.yaml, or leave a note that it's expected-dead so the error volume isn't mistaken for a regression later.
|
Kustomize wiring and Flux plumbing check out. Verified:
Three things flagged inline, none of them a broken manifest:
|
| # Only reconcile cluster state from artifacts signed by this workflow on main. | ||
| - patch: |- | ||
| - op: add | ||
| path: /spec/verify | ||
| value: | ||
| provider: cosign | ||
| matchOIDCIdentity: | ||
| - issuer: ^https://token[.]actions[.]githubusercontent[.]com$ | ||
| subject: ^https://github[.]com/jfroy/flatops/[.]github/workflows/cluster-release[.]yaml@refs/heads/main$ | ||
| target: | ||
| group: source.toolkit.fluxcd.io | ||
| kind: OCIRepository | ||
| name: flux-system |
There was a problem hiding this comment.
With kubernetes/cluster/ocirepository.yaml deleted, this patch becomes the only declaration of Cosign verification for the cluster root source — and a Kustomize patches entry whose target matches nothing is a silent no-op, not an error.
So if the target ever stops matching (flux-operator changes the generated source's name, or someone later sets spec.sync.name), flux-operator re-applies OCIRepository/flux-system with force ownership and strips spec.verify from the live object. The cluster then reconciles whatever latest points to, with no signature check and no warning — a fail-open on the one source that controls everything.
Two ways to make it fail closed, either is cheap:
- Add a VAP under
kubernetes/vap/requiringspec.verify.provider == "cosign"onOCIRepositorynamedflux-systeminflux-system. That fits the existingcluster-vap→cluster-appsordering and would reject the unverified apply outright. - At minimum, after merge assert the operator-generated source actually carries the block and reports
SourceVerified, not justReady—Readyalone is satisfied by an unverified pull.
|
The mechanical part of the migration is complete and consistent — all 176 Three things worth acting on: 1. Verification is now fail-open — see the inline comment on 2.
Committed Git state is no longer the precondition — the Cluster Release workflow has to publish and promote 3. Failed release + advancing Nothing here blocks the merge order described in the PR body. |
| - patch: |- | ||
| - op: add | ||
| path: /spec/verify | ||
| value: | ||
| provider: cosign | ||
| matchOIDCIdentity: | ||
| - issuer: ^https://token[.]actions[.]githubusercontent[.]com$ | ||
| subject: ^https://github[.]com/jfroy/flatops/[.]github/workflows/cluster-release[.]yaml@refs/heads/main$ | ||
| target: | ||
| group: source.toolkit.fluxcd.io | ||
| kind: OCIRepository | ||
| name: flux-system |
There was a problem hiding this comment.
With kubernetes/cluster/ocirepository.yaml deleted, this patch is the only thing that puts spec.verify on the source that every one of the 176 sourceRef entries now depends on — and a Kustomize patches entry whose target selects nothing is a silent no-op, not an error. Two coupled assumptions are load-bearing and neither is asserted anywhere:
spec.synchas noname, so the generated source is named by flux-operator's default (flux-system) — which thistarget.namehardcodes, as do all theks.yamlsourceRefs.- flux-operator has to run
spec.kustomize.patchesover the sync-generated objects, not just the controller Deployments.
If either changes, the OCIRepository is still created and still Ready — just unverified, pulling whatever latest points at. Worse, the live object already carries spec.verify from the bootstrap manifest under kustomize-controller's SSA field ownership, so a non-matching patch would look fine indefinitely on the existing cluster and only surface on a rebuild.
Worth pinning down before merge:
# after the FluxInstance flips, confirm the patch actually landed and is owned by flux-operator
kubectl get ocirepository flux-system -n flux-system \
-o jsonpath='{.spec.verify}{"\n"}{range .metadata.managedFields[*]}{.manager}{" "}{end}{"\n"}'A VAP under kubernetes/vap/ requiring spec.verify.provider == "cosign" on OCIRepository/flux-system would make it fail closed permanently.
| namespace: flux-system | ||
| annotations: | ||
| # Keep the source in place when FluxInstance takes over its management. | ||
| kustomize.toolkit.fluxcd.io/prune: Disabled |
There was a problem hiding this comment.
This annotation is what keeps the live source alive when this file leaves the tree, and it is the odd one out in this repo: the other nine uses (components/common/kustomization.yaml, all the objectbucketclaim.yaml files) all spell it lowercase disabled.
The ordering makes the blast radius worse than a normal orphaned object. On merge, the still-Git-backed flux-system Kustomization reconciles this commit and does two things in one pass: drops this object from its inventory, and updates the flux-instance HelmRelease values. GC runs immediately; flux-operator only recreates the source after the HelmRelease upgrade lands and the FluxInstance reconciles. If Disabled isn't honored, cluster-apps, cluster-vap and every child Kustomization lose their source in that window. Non-destructive (a missing source stalls reconciliation, it doesn't trigger pruning) and it should self-heal, but it's an avoidable flap on the one change that has no Git fallback left.
Cheap to de-risk — either normalize the casing in PR 2 before this lands, or confirm on the live object first:
kubectl get ocirepository flux-system -n flux-system \
-o jsonpath='{.metadata.annotations}{"\n"}'
| ## Cluster | ||
|
|
||
| This is **kantai**, a Kubernetes cluster running Talos Linux, with a mix of bare-metal and virtual nodes, managed entirely through GitOps via FluxCD and Flux Operator. All services live under `kantai.xyz`. Current repo configuration targets Talos `v1.13.2` and Kubernetes `v1.36.1`. Flux syncs `refs/heads/main` from `https://github.com/jfroy/flatops` at `kubernetes/cluster`. | ||
| This is **kantai**, a Kubernetes cluster running Talos Linux, with a mix of bare-metal and virtual nodes, managed entirely through GitOps via FluxCD and Flux Operator. All services live under `kantai.xyz`. Current repo configuration targets Talos `v1.13.2` and Kubernetes `v1.36.1`. Flux syncs `kubernetes/cluster` from the `latest` artifact at `oci://ghcr.io/jfroy/flatops/cluster`. GitHub Actions publishes and keylessly signs the artifact from `refs/heads/main`; Flux verifies its Cosign identity before reconciliation. |
There was a problem hiding this comment.
The Cluster section is updated, but two things this PR changes are left undocumented, and both are exactly the kind of invariant an agent can't recover from the tree:
The flux-system OCIRepository is no longer a checked-in manifest. This PR deletes kubernetes/cluster/ocirepository.yaml, so grep -rn 'kind: OCIRepository' kubernetes/cluster/ now comes back empty while 176 sourceRef entries point at it. It is generated by FluxInstance.spec.sync and its Cosign verify block exists only as a Kustomize patch in kubernetes/apps/flux-system/instance/ks/helm-values.yaml. Worth a sentence saying so, since the natural place to look for it no longer has it.
"Maintenance Commands" (line 33) is now wrong. It still says:
Flux reconciliation, when appropriate and after Git state is ready:
Committed Git state is no longer the precondition — Cluster Release has to publish and promote latest first. Until it does, flux reconcile kustomization cluster-apps --with-source re-pulls the previous artifact and reads as a successful no-op, which is a genuinely misleading failure mode for anyone debugging "my commit didn't apply". Suggest rewording to "after the Cluster Release workflow has promoted the artifact" and adding flux reconcile source oci flux-system, since that is the source under cluster-apps now.
|
The mechanical swap holds up:
Konflate is red on this PR — 403 while resolving |
1 participant
Summary
OCIRepository/flux-systemoci://ghcr.io/jfroy/flatops/cluster:latestPrerequisites
Merge only after the first three PRs have landed and the bootstrap OCI source is Ready and SourceVerified. At that point all default applications already consume OCI, while the temporary source and existing Git root keep this final ownership transfer recoverable.
Flate/Konflate explicitly aliases bootstrap
OCIRepositorysources to the PR working tree, so rendered PR diffs continue to use each PR head rather than the mutablelatestartifact.Stack