Repository navigation
Add reference-aware cleanup for runnable-image staging trees - #129
Merged
Bruno Borges (brunoborges) merged 3 commits intoSep 29, 2026
Merged
Bruno Borges (brunoborges) merged 3 commits into
Bruno Borges (brunoborges) merged 3 commits into
Conversation
Co-authored-by: brunoborges <129743+brunoborges@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Fix unreferenced runnable-image staging trees accumulation
Add reference-aware cleanup for runnable-image staging trees
Sep 28, 2026
Run `brewlet stage-gc` periodically from the operator-managed provisioner DaemonSet, configured through new Helm stageGC values (enabled, 5m interval, 24h minimum age). Existing nodes stay blocked until stageGC.upgradeAcknowledged is set, and each compatible node persists a compatibility record. - Ship the brewlet CLI in the provisioner image as brewlet-stage-gc - Supervise each sweep in its own process group; cancel on TERM/INT - Skip sweeps on inconclusive API reads instead of crashing the worker - Use a fixed /tmp/brewlet-runnable default stage root on Linux - Add tier 17 e2e coverage and update docs and specification Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bruno Borges (brunoborges)
marked this pull request as ready for review
September 29, 2026 12:13
Bruno Borges (brunoborges)
deleted the
copilot/fix-unreferenced-stage-accumulation
branch
September 29, 2026 20:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Published runnable-image stages accumulate indefinitely, retaining extracted payloads and packed layers after workloads and image metadata disappear. Image churn can exhaust node storage.
brewlet stage-gcwith dry-run support and a configurable age floor, defaulting to 24 hours. Preserve stages referenced by containerd image/content metadata across all namespaces or by live per-stage mounts; fail closed on incomplete reference checks.metrics.enabled. New Helm values:stage GC blockeduntil older unguarded shims and stage-dependent bundles are retired andstageGC.upgradeAcknowledged=trueis rolled out. Each compatible node persists/opt/brewlet/.stage-gc-compatible, tied to both shim copies and the stage root, so the acknowledgment can be reset afterward./usr/local/bin/brewlet-stage-gc./tmp/brewlet-runnableregardless ofTMPDIR, matching the reaper even when containerd setsTMPDIR.brewlet_runnable_stage_bytes, backed by a read-only host-stage mount in the operator's metrics exporter. Sweep counts, reclaimed bytes, and last success are logged by the provisioner.# Manual use outside the provisioner sudo brewlet stage-gc --dry-run sudo brewlet stage-gc --min-age 24hTests
nsenterflags.integration-tests/e2e/tier17-stage-gc.sh): installs the chart on kind to verify defaults, the blocked upgrade gate, acknowledgment and record persistence, stage protection while referenced or mounted, and reclamation after image removal.