fix: stop the four-hourly prune from deleting live containers

[arunim2405]

Summary

prune() runs two sweeps. The first is age-guarded; the second has no filter at all:

docker system prune -f --filter "until=4h"   # age-guarded, safe
docker system prune --volumes -f             # ← removes ANY non-running container

An unfiltered docker system prune removes every non-running container regardless of age — including one in created state that was made milliseconds ago. setup_cron() installs this to run every 4 hours at minute 0, so when the sweep overlaps the ECS agent creating a task container, the container is deleted between docker create and docker start.

How it got here

The guard was never meant to be dropped. 4e59eb0 (2024-01-21, "fix volumes") split a single filtered call into two and did not carry the filter onto the volume half:

-  reclaimed=$(... system prune --volumes -f \
-    --filter "until=4h" \
+  reclaimed_containers_images=$(... system prune -f \
+    --filter $prune_filter \
 ...
+  reclaimed_volumes=$(... system prune --volumes -f \    ← no filter

Full timeline of the cron and the prune command:

Date	Commit	Change
2023-07-31	bc52bde	setup_cron() introduced — doctor only, no prune
2023-07-31	8411701	prune() born + cron 0 0 * * * (daily), filter until=<10 days ago>
2023-08-22	2836ef5	clean_cron() added; schedule still daily
2023-11-02	12fe6a8	filter → until=24h; --volumes added to the single filtered call
2023-11-15	bd887e3	--volumes removed
2024-01-14	dc530fa	--volumes re-added — one call, still filtered, safe
2024-01-14	49819b9	schedule 0 0 * * * → 0 */4 * * *; filter → until=4h
2024-01-21	4e59eb0	split into two calls; volume half left unfiltered — this bug
2025-03-14	919a298	typo fix only
2026-06-18	f54a4c1	reformat + update_diagnostic; commented out setup_cron/clean_cron; bug untouched

Live since 2024-01-21, ~31 months. 49819b9 a week earlier had already moved the schedule from daily to four-hourly, so the two compounded: 6x the prune runs, one of them now unguarded.

The fix

- docker system prune --volumes -f
+ docker volume prune -f

docker volume prune reclaims the same anonymous volumes and cannot delete a container at all, so this removes the capability rather than re-guarding it. Restoring --filter until=4h would also work, but leaves the same footgun for the next edit — which is precisely how 4e59eb0 happened.

Evidence — this fired in prod

Workflow run /wfruns/fw325defo4rs (/orgs/rheinenergie), 2026-08-25 14:00 UTC, on private runner private_ansible_iop:

level=info  msg="Created docker container for task"  dockerId="76e662ad…"  elapsed=40.7ms
level=info  msg="Starting container"                 runtimeID="76e662ad…"
level=error msg="Error transitioning container"
            error="Error response from daemon: No such container: 76e662ad…"

The runner host is UTC+2, so the 0 */4 cron fired at 16:00 local = 14:00 UTC. The container was created 17 seconds later. Corroborating:

• Host completely idle — load 0.02/CPU, 33% memory, /var at 24.3%, no kernel flags.
• Docker daemon healthy — created and started another container successfully 5 seconds later.
• No StopTask in CloudTrail — the removal was local, not API-driven.
• The tagged image stayed cached (61 ms pull 15 min later), exactly right for a default system prune, which evicts only dangling images.

Test plan

Adds test/unit/prune.bats (5 tests) and a volume case to the docker mock.

• prune age-guards the container and image sweep
• prune never issues a system prune without an age filter — the regression guard
• prune reclaims volumes without a container-deleting sweep
• prune records the filter it actually applied to the guarded sweep
• prune records reclaimed space for both sweeps

Both new assertions confirmed RED before the fix, GREEN after. Local make test-unit failure count unchanged at the pre-existing baseline of 31 (api_call, patch_json, ecs_exec_deps — also failing on main, macOS-tooling related); all green in CI.

Follow-ups — not in this PR

1. This does not remediate already-registered runners. setup_cron was commented out in f54a4c1 (2026-06-18), stopping new registrations from getting the cron — but clean_cron is commented out too, and main.sh has no self-update. Every runner registered between 2024-01-21 and 2026-06-18 keeps running 0 */4 * * * main.sh prune against its on-disk copy, indefinitely. The affected runner registered 2025-11-04. Those hosts need the crontab entry actively removed.
2. README.md:161 says prune cleans "everything older than 10 days" — true of 8411701 in 2023, wrong since 12fe6a8. Pre-existing, left alone here.
3. system.docker.prune_filter reported until=4h while the destructive sweep had none — actively misleading during this investigation. Accurate now as a side effect.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[arunim2405]

Correcting the attribution in the original description: the unfiltered sweep was introduced by 4e59eb0 (2024-01-21, "fix volumes"), which split one filtered call into two and did not carry the filter onto the volume half. dc530fa was safe — it added --volumes to the still-filtered single call. Body updated with the full timeline; the fix itself is unchanged.