Skip to content

refactor(flux): migrate default apps to OCI source - #2165

Open
jfroy wants to merge 1 commit into
oci-bootstrapfrom
oci-refs-1
Open

refactor(flux): migrate default apps to OCI source#2165
jfroy wants to merge 1 commit into
oci-bootstrapfrom
oci-refs-1

Conversation

@jfroy

@jfroy jfroy commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Summary

  • migrate all default-namespace Flux Kustomizations from GitRepository/flux-system to the already-bootstrapped OCIRepository/flux-system
  • keep the change mechanical and below the automated review file limit

This is safe to merge only after the bootstrap OCI source is Ready and verified. FluxInstance and the root Kustomizations remain Git-backed at this stage, providing a rollback path while application sources move to OCI.

Stack

  1. Publish and seed the signed artifact
  2. Bootstrap the verified OCI source and Receiver
  3. This PR: migrate default-namespace Kustomizations
  4. Complete the source migration and transfer source ownership to FluxInstance

@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 29f11ae0-8d9b-493a-991f-770021dd643f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@daddy-ro

daddy-ro Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

konflate — summary

Note

+0 added · 50 changed · −540 removed — 590 resources · 99 apps · 90 not shown

Blast radius

  • Kustomization default/immich — 1 dependent (Kustomization default/immich-pet-tagger)
  • Kustomization default/netbox-valkey — 1 dependent (Kustomization default/netbox)
  • Kustomization flux-system/immich — 1 dependent (Kustomization flux-system/immich-pet-tagger)
  • Kustomization flux-system/netbox-valkey — 1 dependent (Kustomization flux-system/netbox)

Warning

⚠ Cautions

  • PersistentVolumeClaim default/buildkit-root-amd64 — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/buildkit-root-arm64 — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/crd-schema-publisher — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/gluetun-update — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/immich — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/immich-pet-tagger — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/jellyfin-cache — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/karakeep-cache — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/karakeep-meili — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/photon — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/readur — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • StatefulSet default/seerr — removed StatefulSet — its PersistentVolumeClaims and data may be deleted
  • PersistentVolumeClaim default/stash-data — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/stash-plugins — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/stash-sais-data — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/stash-scrapers — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • StatefulSet default/zot — removed StatefulSet — its PersistentVolumeClaims and data may be deleted
  • PersistentVolumeClaim default/bookorbit — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/browserr — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/changedetection — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/dawarich — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/docker-registry-ui — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/homebox — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/houndarr — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/photos — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/immichframe — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/jellyfin — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/karakeep — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/komga — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/labby — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/livesync — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/mealie — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/media1 — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/media2 — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/memos — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/netbox — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/paperless — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/pgadmin — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/plex — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/plex-local — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/pocket-id — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/qbittorrent — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/qui — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/radarr — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/recyclarr — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/sabnzbd — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/seerr — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/shelfmark — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/sonarr — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/spoolman — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/stash — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • PersistentVolumeClaim default/ytptube — removed PersistentVolumeClaim — the bound volume's data may be reclaimed
  • 590 resources · 99 apps — large change set — more ground to cover than a typical PR; review with extra care
  • Kustomization default/immich — removed, but still declared in spec.dependsOn by Kustomization default/immich-pet-tagger — those will wedge on the missing dependency
  • Kustomization default/netbox-valkey — removed, but still declared in spec.dependsOn by Kustomization default/netbox — those will wedge on the missing dependency
  • Kustomization flux-system/immich — removed, but still declared in spec.dependsOn by Kustomization flux-system/immich-pet-tagger — those will wedge on the missing dependency
  • Kustomization flux-system/netbox-valkey — removed, but still declared in spec.dependsOn by Kustomization flux-system/netbox — those will wedge on the missing dependency

Caution

51 render failures

Image changes

image from to
docker.io/busybox 1.38.0
docker.io/curlimages/curl sha256:7c12af72ceb3…
docker.io/dpage/pgadmin4 sha256:2f4ce946ddf8…
docker.io/electh/nextflux sha256:310f230fc975…
docker.io/getmeili/meilisearch sha256:d36e713e8f89…
docker.io/library/couchdb sha256:b80216f643e9…
docker.io/rancher/kubectl v1.36.2
ghcr.io/arabcoders/ytptube sha256:95e0be28fefb…
ghcr.io/autobrr/qui sha256:3285c52f0258…
ghcr.io/av1155/houndarr sha256:475cf515388a…
ghcr.io/bookorbit/bookorbit sha256:e131834be597…
ghcr.io/browserless/chromium sha256:8f6a3937c574…
ghcr.io/calibrain/shelfmark sha256:bd314405a9ca…
ghcr.io/connorgallopo/tracearr sha256:3d57d9b032b4…
ghcr.io/dgtlmoon/changedetection.io sha256:5438423d5e90…
ghcr.io/diced/zipline sha256:bfd5b0f7b5b8…
ghcr.io/donkie/spoolman sha256:f17489666719…
ghcr.io/eznix86/docker-registry-ui sha256:f30a167cd060…
ghcr.io/flaresolverr/flaresolverr sha256:139dfee1c6f8…
ghcr.io/gethomepage/homepage sha256:a0b71c8e7572…
ghcr.io/gotson/komga sha256:c4f9885fc077…
ghcr.io/home-operations/esphome sha256:b23f64c1a975…
ghcr.io/home-operations/plex sha256:235402dacb4c…
ghcr.io/home-operations/postgres-init sha256:ebd9d30add17…
ghcr.io/home-operations/prowlarr sha256:ce5d6bdd5be6…
ghcr.io/home-operations/qbittorrent sha256:4fcf15b7f265…
ghcr.io/home-operations/radarr sha256:260469d70761…
ghcr.io/home-operations/sabnzbd sha256:457be5fad7b8…
ghcr.io/home-operations/sonarr sha256:01db3e6a923f…
ghcr.io/immich-app/immich-machine-learning v3.1.0-cuda
ghcr.io/immich-app/immich-server v3.1.0
ghcr.io/immichframe/immichframe sha256:edae2c0b9ab6…
ghcr.io/instrumentisto/rsync-ssh alpine3.22
ghcr.io/janpuc/browserr sha256:3ba025efc6dd…
ghcr.io/jellyfin/jellyfin sha256:45f648c382a0…
ghcr.io/jellyfin/jellyfin-vue sha256:49d8694bb84e…
ghcr.io/jfroy/etampe sha256:cc5ac861aa0e…
ghcr.io/jfroy/gluetun sha256:970fdef9eedb…
ghcr.io/jfroy/stash v0.31.1-cudajellyfin
ghcr.io/jfroy/vuetorrent sha256:7fe6d12a1d0b…
ghcr.io/karakeep-app/karakeep sha256:5467873df817…
ghcr.io/mealie-recipes/mealie sha256:36c28f0642fb…
ghcr.io/miniflux/miniflux 2.3.3
ghcr.io/netbox-community/netbox v4.6.7
ghcr.io/paperless-ngx/paperless-ngx sha256:65a4cabf0169…
ghcr.io/project-zot/zot v2.1.18
ghcr.io/readur/readur sha256:bb6356cf930f…
ghcr.io/recyclarr/recyclarr sha256:73303ba5ee64…
ghcr.io/samuelloranger/labby sha256:424a06937118…
ghcr.io/seerr-team/seerr v3.4.0
ghcr.io/sholdee/crd-schema-publisher v2026.721.161952
ghcr.io/skier233/stash-ai-server 0.9.3
ghcr.io/sysadminsmedia/homebox sha256:b1ad7e3c63f7…
ghcr.io/tedornitier/immich-pet-tagger sha256:bae28707d3dd…
ghcr.io/usememos/memos sha256:71a5b4738d1b…
ghcr.io/valkey-io/valkey sha256:3acc0687f2a2…
mirror.gcr.io/freikin/dawarich 1.11.0
mirror.gcr.io/freikin/dawarich sha256:4e3c55c4cd57…
moby/buildkit sha256:28a898719c18…
registry.kantai.xyz/jfroy/photon-docker sha256:8a516fc9ecdb…
registry.kantai.xyz/nams sha256:2215b72a676b…
registry.kantai.xyz/pocket-id/pocket-id sha256:72808e68b4c7…
registry.kantai.xyz/sais-frontend sha256:eac1c7c1aee0…
registry.kantai.xyz/stash/bulk-transcode sha256:ef916bda7158…

View the full rendered diff →

konflate · rendered e37a5b1 · advisory, not a gate

@claude

claude Bot commented Aug 8, 2026

Copy link
Copy Markdown

Checked the mechanical migration and it looks correct — no blocking issues.

What I verified:

  • Coverage is complete. All 68 sourceRef blocks across the 60 kubernetes/apps/default/*/ks.yaml files now say OCIRepository; grep -rn GitRepository kubernetes/apps/default/ returns nothing. The multi-doc files match their diff counts (immich 4, paperless 3 including the two commented-out paperless-ai/paperless-gpt blocks, netbox 2), so no stale GitRepository is left behind in a commented block that would come back wrong when uncommented.
  • The target source exists and is in scope. OCIRepository/flux-system in flux-system is defined in kubernetes/cluster/ocirepository.yaml, which the FluxInstance root Kustomization (path: kubernetes/cluster) applies, and it carries kustomize.toolkit.fluxcd.io/prune: Disabled.
  • Paths still resolve against the artifact root. cluster-release.yaml builds the artifact from a dir containing only kubernetes/, so path: ./kubernetes/apps/default/<app>/app resolves, and the ../../../../components/{kopiur/backup,envoy-gateway-oidc} component refs land on kubernetes/components/... — inside the artifact, not above it. The workflow guard grep -q 'kubernetes/cluster/ks.yaml$' keeps that layout honest.
  • Anonymous pull is exercised by the workflow, so the app Kustomizations need no secretRef, and the cosign verify identity matches the publishing workflow.

Two non-blocking operational notes for the stack, both consequences of this step rather than defects in it:

  1. These 60 Kustomizations keep prune: true but now read from the mutable latest tag, while cluster-apps still reads Git. If a cluster-release run for a commit fails — or gets dropped by cancel-in-progress: true and the follow-up run also fails — latest silently pins to an older commit. Default-namespace apps then reconcile stale content (and can prune resources a newer commit added) with no signal other than the artifact revision, while every other namespace moves forward from Git. Worth an alert on cluster-release failure, or on OCIRepository revision age, before PR 4 removes the Git fallback.
  2. Landing this on main should be paired with a successful cluster-release for the same commit. Content under default/*/app is untouched here so the window is harmless for this PR, but if PRs 3 and 4 land together with real app changes, the flip and the republish need to be ordered.

@claude

claude Bot commented Aug 8, 2026

Copy link
Copy Markdown

Blocking: oci://ghcr.io/jfroy/flatops/cluster is not anonymously pullable, and the source has no credentials

konflate on this PR reports 51 render failures, all with the same root cause:

OCIRepository flux-system/flux-system resolve latest:
HEAD "https://ghcr.io/v2/jfroy/flatops/cluster/manifests/latest":
GET "https://ghcr.io/token?scope=repository:jfroy/flatops/cluster:pull&service=ghcr.io":
response status code 403: denied: requested access to the resource is denied

kubernetes/cluster/ocirepository.yaml has no secretRef and no serviceAccountName, and kubernetes/apps/flux-system/instance/ks/helm-values.yaml adds no image-pull credentials to source-controller — so source-controller pulls that artifact anonymously, exactly as konflate did. ghcr.io/jfroy/flatops/cluster is a newly created GHCR package and defaults to private; cluster-release.yaml authenticates with docker/login-action before flux push, so the publish path never exercises an anonymous pull. Other oci://ghcr.io/jfroy/... sources in this repo (e.g. kubernetes/apps/kube-system/zfs-static-csi/app/ocirepository.yaml) work without a secretRef because those packages are public.

Consequence of merging as-is: all 68 sourceRef blocks across the 60 kubernetes/apps/default/*/ks.yaml files go Ready=False with 403 denied. Running workloads keep running, but the entire default namespace stops reconciling — no drift correction, and no later PR touching a default app takes effect — until the source can be pulled. prune: true is not a hazard here (Flux does not prune on a source/build failure), and the konflate "removed PersistentVolumeClaim" and "removed, but still declared in spec.dependsOn" cautions are artifacts of the failed render, not real removals.

Two ways to fix, either is fine:

  • Flip the flatops/cluster package to public in GHCR package settings (consistent with jfroy/charts/*), or
  • Add a secretRef to kubernetes/cluster/ocirepository.yaml pointing at a dockerconfigjson Secret in flux-system sourced from 1Password via external-secrets. The verify: cosign block also needs registry read access, so it has to use the same credential.

Worth confirming before merge rather than after, since the failure mode is namespace-wide:

crane manifest ghcr.io/jfroy/flatops/cluster:latest   # must succeed with no auth

If the package was made public after konflate last ran, re-run it so the status check is green — it is currently the only pre-merge signal for this class of failure, and it is red.


Correcting my earlier comment on this PR: the claim that "anonymous pull is exercised by the workflow" was wrong — the workflow logs in to GHCR first, so it proves nothing about unauthenticated pulls. There is also no grep -q 'kubernetes/cluster/ks.yaml$' guard in cluster-release.yaml; I misattributed that. The rest of that comment still holds: coverage is complete, path: resolves correctly against the artifact root, the ../../../../components/... refs stay inside the artifact, and the latest-tag staleness and merge-ordering notes stand.

One addition to the staleness note, now that this source is the only input for these apps: once this lands, a PR that adds a new default app both registers the child Kustomization (via cluster-apps, from Git, applied at merge) and adds its app/ directory (visible only after cluster-release publishes). The new Kustomization will error on a missing path for the duration of the artifact build and then self-heal. Transient, but expect it.

prune: true
sourceRef:
kind: GitRepository
kind: OCIRepository

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anchoring the blocking issue here since it applies identically to all 68 sourceRef blocks in this PR: OCIRepository/flux-system resolves to oci://ghcr.io/jfroy/flatops/cluster:latest, which currently returns 403 denied for an unauthenticated pull, and neither kubernetes/cluster/ocirepository.yaml nor source-controller carries a credential. konflate failed to render 51 of these Kustomizations for exactly that reason. Make the GHCR package public or add a secretRef before merging — details in the top-level comment.

@claude

claude Bot commented Aug 8, 2026

Copy link
Copy Markdown

Mechanical correctness checks out. Verified:

  • All 64 active sourceRef blocks under kubernetes/apps/default/*/ks.yaml are now kind: OCIRepository / name: flux-system / namespace: flux-system; zero GitRepository refs remain in that tree. The 2 commented-out blocks (paperless-ai, paperless-gpt) were swapped too, so they stay consistent when re-enabled.
  • Diff is exactly 66 one-line changes — no accidental edits to path, prune, components, dependsOn, or postBuild.
  • Artifact layout supports the unchanged paths: cluster-release.yaml does cp -a kubernetes/. "${ARTIFACT_DIR}/kubernetes/", so the tarball root contains kubernetes/ and path: ./kubernetes/apps/default/<app>/app resolves. The ../../../../components/kopiur/backup component refs land exactly on kubernetes/components/ — nothing escapes the artifact root (checked: no ../ chain anywhere under kubernetes/ goes five levels up).
  • No files under kubernetes/ match flux push artifact's default exclude set (no .png/.jpg/.zip/.sops.yaml, no .sourceignore), so the artifact is content-complete for these 64 Kustomizations.
  • Changing sourceRef.kind in place does not recreate the Kustomization objects, so inventories are preserved and no app resources get pruned. No PVC/volsync target renames here.

Three operational notes, none of them a defect in the diff:

1. latest must exist before this lands, or all 64 default apps stall at once. kubernetes/cluster/ocirepository.yaml pins ref.tag: latest with cosign matchOIDCIdentity on cluster-release.yaml@refs/heads/main. If cluster-release has never completed a flux tag artifact --tag latest on main, every default-namespace Kustomization goes ArtifactFailed the moment this merges. The PR body already gates on "bootstrap OCI source is Ready" — worth literally confirming flux get source oci flux-system -n flux-system is Ready and that a default app has reconciled against it before merging. Failure mode is stall, not prune, so nothing is destroyed either way.

2. The swap itself can take up to an hour to take effect. These child Kustomization objects are still applied by cluster-apps, which reads GitRepository/flux-system. The FluxInstance sync sets interval: 1h (instance/ks/helm-values.yaml), and instance/ks/receiver.yaml notifies only OCIRepository/flux-system — there is no webhook on the Git source. So after merge you will likely want flux reconcile kustomization cluster-apps --with-source rather than waiting.

3. Spec-vs-content source skew is now real for future commits. Child Kustomization specs come from Git while their rendered content comes from the OCI artifact, which only appears after the cluster-release workflow finishes (~1-3 min). For this PR that is harmless — only ks.yaml changed, and that file lives outside every spec.path. But any later commit that changes both a ks.yaml and the app/ content it points at will briefly apply the new spec against the previous artifact. Mostly benign, except where a new postBuild.substitute variable is introduced alongside the manifest that consumes it: new spec + old content can produce a var-not-found failure, and old spec + new content can leave ${VAR} unsubstituted. Sequencing the substitution variable one commit ahead of its consumer avoids that until step 4 moves cluster-apps to OCI as well.

Unrelated to this PR, but noticed while checking coverage: autobrr/ks.yaml is migrated here but - ./autobrr/ks.yaml is absent from kubernetes/apps/default/kustomization.yaml, so that Kustomization is not reconciled at all. Pre-existing — flagging only in case it is unintentional rather than a deliberate disable like the commented-out entries.

@claude

claude Bot commented Aug 8, 2026

Copy link
Copy Markdown

The sweep itself checks out: all 66 sourceRef blocks under kubernetes/apps/default/*/ks.yaml are now OCIRepository/flux-system/flux-system, zero GitRepository references remain in that tree, and the multi-Kustomization files are fully covered (immich ×4, netbox ×2, paperless ×3 including the two commented-out blocks). OCIRepository/flux-system exists in kubernetes/cluster/ocirepository.yaml, and cluster-release.yaml stages the artifact under a kubernetes/ prefix, so every path: ./kubernetes/apps/default/... and every spec.components: ../../../../components/kopiur/... (immichframe, netbox) resolves inside the artifact root. Both dependsOn edges in this namespace (immich-pet-taggerimmich, netboxnetbox-valkey) stay same-source, so kustomize-controller's "dependency is not updated yet" revision check is preserved rather than silently downgraded to a bare Ready check.

Three things worth acting on before this goes further.

1. "Ready" is not the merge precondition you want — "current" is.

The PR body says this is safe to merge once the OCI source is Ready and verified. But cluster-apps still reads Git while these 60 Kustomizations now read oci://ghcr.io/jfroy/flatops/cluster:latest, and cluster-release only publishes after a merge commit lands on main. Every one of these Kustomizations has prune: true (except media-pv). If latest lags main by more than the content-identical commit immediately preceding the merge, the switchover applies the stale artifact and prunes whatever is in the cluster but not in it — HelmReleases, HTTPRoutes, ExternalSecrets deleted, then re-created minutes later when the artifact catches up. Before the stack merges, confirm flux get source oci flux-system -n flux-system reports main@sha1:<current main HEAD> and that the latest Cluster Release run on main succeeded — not just that the source is Ready.

2. Nothing alerts on a Ready-but-stale artifact.

components/common is applied to flux-system and its Alertmanager Alert already lists kind: OCIRepository, so hard failures (cosign verify, GHCR pull) will page. The gap this PR opens is the quiet case: if cluster-release fails or never fires, the OCIRepository stays happily Ready on an old digest and all 60 default apps freeze on stale manifests while every other namespace keeps tracking Git in real time — mixed-revision cluster state with no signal. There is no gotk_reconcile_* PrometheusRule covering this; instance/ks/prometheusrule.yaml only has FluxInstanceAbsent/FluxInstanceNotReady. Worth adding a staleness rule (OCIRepository status.artifact.revision vs. the GitRepository revision) or making Cluster Release a required check, ideally as part of step 4.

3. Confirm konflate still renders these before step 4.

konflate is the only pre-merge validation this repo has (no test suite, per AGENTS.md), and its config knows only repo: github://jfroy/flatops. If it maps Kustomizations onto the PR checkout by matching sourceRef.kind: GitRepository, the whole default namespace silently drops out of PR render/diff coverage — and that loss is invisible, since a PR with no rendered diffs looks identical to a PR with no manifest changes. Worth verifying on the next PR that touches a kubernetes/apps/default/*/app file, before migrating the remaining namespaces.

Minor: autobrr/ks.yaml and openspoolman/ks.yaml are not referenced from kubernetes/apps/default/kustomization.yaml at all, not even as commented entries, so those two edits are inert. Pre-existing rather than introduced here, but a mechanical sweep is a good moment to notice it.

Also expected-to-work, but worth an eyeball post-merge: the github-status Alert from components/common watches kind: Kustomization, name: "*" and posts GitHub commit statuses keyed off the source revision. For an OCI source that only resolves via the artifact's origin revision, which cluster-release.yaml does set correctly (--revision "${GITHUB_REF_NAME}@sha1:${GITHUB_SHA}"). If per-app commit statuses stop showing up for default apps after this lands, that annotation plumbing is where to look, not the Alert.

@claude

claude Bot commented Aug 8, 2026

Copy link
Copy Markdown

Checked this as a mechanical migration and it holds up. Notes from verifying it rather than from reading it:

Migration is complete and self-consistent

  • grep -rn "kind: GitRepository" kubernetes/apps/default/ returns nothing, and every changed line is a sourceRef.kind (64 live + 2 inside the commented-out paperless-ai/paperless-gpt blocks — good that those were kept in sync, so re-enabling them later doesn't silently reintroduce a Git ref). No name:/namespace:/path: line moved.
  • OCIRepository/flux-system exists on the base branch (kubernetes/cluster/ocirepository.yaml), and OCIRepository is a valid sourceRef.kind for kustomize.toolkit.fluxcd.io/v1.
  • Paths still resolve: cluster-release.yaml stages kubernetes/. under ${ARTIFACT_DIR}/kubernetes/, so the artifact root has the same layout as the repo root. Both path: ./kubernetes/apps/default/<app>/app and spec.components: ../../../../components/kopiur/backup land where they did under Git. There are no symlinks, dotfiles, or flux push-excluded extensions (.png/.jpg/.zip/.tar.gz) anywhere under kubernetes/, so nothing gets dropped from the artifact.
  • The cross-namespace source ref is unchanged in character — the NamespaceTransformer puts these Kustomizations in default, and they already referenced flux-system cross-namespace before.
  • dependsOn in immich (immich) and netbox (netbox-valkey) are same-namespace and source-agnostic; nothing depends on the source kind.
  • Commit statuses survive: because the workflow pushes with --revision "${GITHUB_REF_NAME}@sha1:${GITHUB_SHA}", kustomize-controller can propagate originRevision from org.opencontainers.image.revision, which is what the github-status Provider needs. Had that flag been omitted, the provider would have received an OCI digest instead and every default-app status check would have failed.
  • Alerting is covered — components/common includes an eventSources entry for OCIRepository "*", and kubernetes/apps/flux-system/kustomization.yaml applies that component, so failures on OCIRepository/flux-system do surface.

One consequence to expect, not a defect

cluster-apps still renders ./kubernetes/apps from Git, so child Kustomization specs land the moment a commit hits main, while their content now comes from latest. That's the correct ordering for this step (no chicken-and-egg — the artifact doesn't need to contain the OCI refs yet), but it opens a skew window: a future PR that adds a new app directory and registers it in kustomization.yaml in the same merge will have its child Kustomization created from Git before cluster-release has published an artifact containing that directory. Expect a transient kustomize build failed: path not found for the duration of build + cosign sign + cosign verify + flux tag, which will fire the alertmanager Alert (eventSeverity: error) before self-healing. If that noise is unwelcome, adding a path not found entry to the Alert exclusionList in components/common/alerts/alertmanager/alert.yaml is the cheap fix; otherwise it's just worth knowing before the next new-app PR. It resolves itself in step 4 when cluster-apps moves to OCI too.

Nit, pre-existing

autobrr/ks.yaml and openspoolman/ks.yaml were migrated but neither is listed in kubernetes/apps/default/kustomization.yaml — not even commented out, unlike docling/glance/ollama/etc. They're dead files, so the edits are harmless but also inert. Worth deleting the two directories in a follow-up rather than carrying them through the rest of the stack.

@claude

claude Bot commented Aug 8, 2026

Copy link
Copy Markdown

The mechanical part checks out: all 66 sourceRef blocks under kubernetes/apps/default/ now point at OCIRepository/flux-system (grep finds zero remaining kind: GitRepository in that tree), indentation and surrounding fields are untouched, and nothing else in the repo keys off the source kind for these apps. Path resolution against the new artifact is fine too — cluster-release.yaml stages kubernetes/ at the artifact root, so every spec.path: ./kubernetes/apps/default/... and every components: ../../../../components/... (resolving to kubernetes/components/...) stays inside the artifact root, same as with the Git source.

One operational risk worth a decision before this lands, because it is created by this PR and only removed by step 4:

The Git and OCI sources refresh on very different schedules, so ks.yaml definitions and the app manifests they parameterize are no longer applied atomically.

Receiver/cluster-release lists only OCIRepository/flux-system in spec.resources, so a merge to main pushes the new artifact into the cluster within seconds. The root GitRepository/flux-system (FluxInstance sync.interval: 1h, plus reconcileArtifactEvery: 1h) has no Receiver entry, and cluster-apps is itself interval: 1h. After this PR the ordering inverts: app manifests land immediately, while the child Kustomizations that configure them lag by up to an hour.

Concretely, for any merge touching both an app's manifests and its ks.yaml:

  • A new postBuild.substitute var introduced alongside the manifest that consumes it — the manifest applies right away with the variable still undefined. Flux's envsubst resolves undefined vars to the empty string rather than failing, so this is silent: size: ${VOLSYNC_CAPACITY} renders as an invalid PVC, ${APP_SUBDOMAIN:-${APP}}.kantai.xyz can render as a bare domain, etc.
  • A changed spec.path, or a renamed/deleted app directory — the stale child Kustomization points at a path that no longer exists in the fresh artifact and goes red for up to an hour. Not destructive (kustomize-controller will not prune when it cannot build the source), but it pages alertmanager and reports failing commit statuses through the github-status Alert.

Cheapest fix while the stack is in flight is to add the Git source to the same Receiver so both refresh off the one webhook:

  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1
      kind: OCIRepository
      name: flux-system
    - apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
      name: flux-system

Otherwise this is acceptable as long as step 4 follows quickly and interim merges avoid combining manifest changes with spec.path / postBuild.substitute changes.

Two smaller notes:

  • Base branch is oci-bootstrap, so this must not be retargeted at main before that PR merges — the OCIRepository/flux-system these 66 refs depend on is defined there (kubernetes/cluster/ocirepository.yaml), and if it were absent all 63 default-namespace Kustomizations would go NotReady at once. The failure mode is at least safe: no source → no build → no prune.
  • Worth confirming after the first merge that the github-status provider still resolves commit statuses from an OCI-sourced Kustomization. It should — the workflow passes --revision "${GITHUB_REF_NAME}@sha1:${GITHUB_SHA}", the format notification-controller parses — but it is a behavior change across all 63 apps and is cheap to eyeball once.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant