Runnable-image staging trees under BREWLET_RUNNABLE_STAGE accumulate without eviction for unreferenced stages #115

Summary

runnableStageDir (core/internal/artifact/blobs.go:372-387) publishes every runnable image into an immutable per-image staging tree under BREWLET_RUNNABLE_STAGE (default os.TempDir()/brewlet-runnable), keyed immutable-v2/<manifest-hex>. Each tree retains the gunzipped app tree, the uncompressed layer tars, and the retained-content copy that lets later replicas start after containerd garbage-collects its packed layers — on the order of 2x the image's layer bytes per distinct image, per node.

There is no eviction or garbage-collection path for the published trees. The only os.RemoveAll in the staging pipeline (blobs.go:220) cleans the transient pending directory during atomic publish — nothing ever removes a published immutable-v2/<hex> tree. docs/runnable-image.md states the consequence directly:

legacy stages and staging directories left by abruptly terminated processes are not automatically garbage-collected ... Allow extra disk capacity for the retained compressed bytes per image.

So on a long-lived shared node, every distinct runnable image ever started accumulates in /tmp indefinitely, and the documented operational answer is "allow extra disk". The retention contract has one hard rule — "Do not remove staging trees while workloads still reference them" — which implies unreferenced trees are safe to remove, but nothing ever removes them.

Repro

Start two different runnable images on one node, stop the workloads (and let containerd drop the images), then inspect the staging root:

$ ls /tmp/brewlet-runnable/immutable-v2/
<manifest-a-hex>/   <manifest-b-hex>/
$ du -sh /tmp/brewlet-runnable/immutable-v2/*

Both trees remain — structure and lifecycle verified by reading runnableStageDir, stageLayerTars, and the retained-verified-blob logic at HEAD a367aac; the doc quotes are from docs/runnable-image.md (rollout notes). Repeating with more distinct images grows the root monotonically; /tmp is on the node root filesystem by default, so the growth eventually surfaces as ENOSPC for new stage operations (and everything else sharing that filesystem).

Suggested fix

A reference-aware reaper for unreferenced stage trees:

• candidate: any immutable-v2/<hex> whose manifest digest is no longer present in containerd's image/content metadata AND holds no live mount (a tree referenced by a running workload carries an active read-only bind mount — per-tree mount check via /proc/mounts, or an open-handle check on the retained blob set);
• never touch trees with live references (the documented rule);
• an age floor (only reap trees older than N hours) to tolerate start storms and upgrade windows where legacy stages must survive;
• expose remaining-bytes as a brewlet-metrics-exporter gauge and ship the reaper as a node-level command so operators can also run it manually against BREWLET_RUNNABLE_STAGE.

Impact framing

Availability/resource hardening on the operator's own node, not a tenant-boundary issue. Disk exhaustion from accumulated staging degrades every workload that later needs to stage an image on that node; plain fleet churn (daily image turnover) reaches it before any hostile input does.