Skip to content

Add reference-aware cleanup for runnable-image staging trees - #129

Merged
Bruno Borges (brunoborges) merged 3 commits into
mainfrom
copilot/fix-unreferenced-stage-accumulation
Sep 29, 2026
Merged

Bruno Borges (brunoborges) merged 3 commits into
mainfrom
copilot/fix-unreferenced-stage-accumulation

Conversation

Copilot AI commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Published runnable-image stages accumulate indefinitely, retaining extracted payloads and packed layers after workloads and image metadata disappear. Image churn can exhaust node storage.

  • Conservative eviction: Add brewlet stage-gc with dry-run support and a configurable age floor, defaulting to 24 hours. Preserve stages referenced by containerd image/content metadata across all namespaces or by live per-stage mounts; fail closed on incomplete reference checks.
  • Lifecycle safety: Coordinate cleanup with resolution and mount creation through shared/exclusive locking. Give exported runnable bundles independent payload copies so later eviction cannot invalidate them.
  • Automatic cleanup, enabled by default: The operator-managed provisioner DaemonSet now runs the reaper periodically after provisioning, independently of metrics.enabled. New Helm values:
    stageGC:
      enabled: true
      interval: 5m
      minAge: 24h
      upgradeAcknowledged: false
    Brewlet follows kubelet/containerd image retention rather than duplicating it: kubelet removes unused images, containerd collects their content, and Brewlet then reclaims the orphaned stages. It does not read or change kubelet GC settings or apply its own disk thresholds.
  • Upgrade gate: Fresh nodes (no shim, no record, empty stage root) activate automatically. Existing nodes log stage GC blocked until older unguarded shims and stage-dependent bundles are retired and stageGC.upgradeAcknowledged=true is rolled out. Each compatible node persists /opt/brewlet/.stage-gc-compatible, tied to both shim copies and the stage root, so the acknowledgment can be reset afterward.
  • Supervision: Sweeps run in the host PID/mount namespaces, in their own process group, never overlap, and use the configured interval plus up to 10% jitter. Node ownership is rechecked before every sweep. API read failures skip that sweep instead of crashing the worker, and TERM/INT cancels the active sweep. The provisioner image now ships the CLI, installed on the host as /usr/local/bin/brewlet-stage-gc.
  • Consistent stage root: On Linux, the shim and CLI default to /tmp/brewlet-runnable regardless of TMPDIR, matching the reaper even when containerd sets TMPDIR.
  • Visibility: Expose brewlet_runnable_stage_bytes, backed by a read-only host-stage mount in the operator's metrics exporter. Sweep counts, reclaimed bytes, and last success are logged by the provisioner.
  • Docs and spec: Updated the specification (§5.2.1 cleanup lifecycle, §6.3, error codes, paths), configuration, installation (activation procedure), CLI reference, runtime metrics, chart/provisioner/exporter READMEs, and third-party notices.
# Manual use outside the provisioner
sudo brewlet stage-gc --dry-run
sudo brewlet stage-gc --min-age 24h

Tests

  • Go: reaper, CLI SIGTERM cancellation, operator config validation and DaemonSet environment, chart rendering.
  • Provisioner shell tests: activation/compatibility, invalidation, retry, ownership rechecks, inconclusive API reads, jitter, process-group shutdown, and a real PID-namespace check of the nsenter flags.
  • New E2E tier 17 (integration-tests/e2e/tier17-stage-gc.sh): installs the chart on kind to verify defaults, the blocked upgrade gate, acknowledgment and record persistence, stage protection while referenced or mounted, and reclamation after image removal.

Co-authored-by: brunoborges <129743+brunoborges@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix unreferenced runnable-image staging trees accumulation Add reference-aware cleanup for runnable-image staging trees Sep 28, 2026
Run `brewlet stage-gc` periodically from the operator-managed provisioner
DaemonSet, configured through new Helm stageGC values (enabled, 5m interval,
24h minimum age). Existing nodes stay blocked until stageGC.upgradeAcknowledged
is set, and each compatible node persists a compatibility record.

- Ship the brewlet CLI in the provisioner image as brewlet-stage-gc
- Supervise each sweep in its own process group; cancel on TERM/INT
- Skip sweeps on inconclusive API reads instead of crashing the worker
- Use a fixed /tmp/brewlet-runnable default stage root on Linux
- Add tier 17 e2e coverage and update docs and specification

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@brunoborges
Bruno Borges (brunoborges) marked this pull request as ready for review September 29, 2026 12:13
@brunoborges
Bruno Borges (brunoborges) merged commit bc9ce4f into main Sep 29, 2026
26 of 29 checks passed
@brunoborges
Bruno Borges (brunoborges) deleted the copilot/fix-unreferenced-stage-accumulation branch September 29, 2026 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runnable-image staging trees under BREWLET_RUNNABLE_STAGE accumulate without eviction for unreferenced stages

2 participants