Skip to content

fix: clean up Docker verifier containers after runs - #28

Open
DeSitterUniverse wants to merge 1 commit into
stevibe:mainfrom
DeSitterUniverse:fix/stop-verifier-containers-after-run
Open

fix: clean up Docker verifier containers after runs#28
DeSitterUniverse wants to merge 1 commit into
stevibe:mainfrom
DeSitterUniverse:fix/stop-verifier-containers-after-run

Conversation

@DeSitterUniverse

@DeSitterUniverse DeSitterUniverse commented Aug 20, 2026

Copy link
Copy Markdown

Summary

Clean up BenchLocal-owned Docker verifier containers when the final run lease is released. This is done to reduce RAM and resource usage from containers that are no longer needed when benchmarking.

Covered paths:

  • full Bench Pack runs;
  • scenario retries;
  • run resumes;
  • verifier/host setup failures;
  • aborts and other run errors.

Implementation

  • Centralize prepared-resource, host-resource, and verifier cleanup in one finalizer.
  • Track active leases per Bench Pack so shared verifier containers stay alive while another run/retry/resume is using them.
  • Make lease release idempotent.
  • Treat cleanup as best-effort: a failed cleanup is logged and cannot replace the original run/setup error.
  • Remove containers only. Images, model files, run artifacts, and logs are preserved.
  • Existing explicit verifier start/stop/status APIs are unchanged.

Why

A verifier container is a short-lived benchmark dependency, but it previously remained running after a run completed. This retains Docker/Node process memory until manually stopped.

Measured local impact:

Workload Verifier memory while active Potential reclaim after cleanup
StructOutput-15, full 15-scenario model run 12.13 MiB peak ~10–12 MiB
CLI-40 live integration sample 18.59 MiB ~18.59 MiB
Three verifier containers held live 46.96 MiB aggregate ~46.96 MiB

The StructOutput-15 measurement used 1,014 Docker memory samples during a 33m44s run. The container peaked at 12.13 MiB, then no container remained after finalization. Potential memory savings are higher than tested.

Validation

  • npm.cmd run build:compile — passed.
  • npm.cmd test — passed after the documented compile step.
  • git diff --check — passed.
  • Normal run — verifier removed.
  • Abort after container startup — expected abort returned; verifier removed.
  • Concurrent same-pack runs — verifier stayed available until the final lease released, then was removed.
  • Setup-failure cleanup — covered by the same finalization path.

@DeSitterUniverse DeSitterUniverse changed the title Stop verifier containers after Bench Pack runs fix: clean up Docker verifier containers after runs Aug 20, 2026
@DeSitterUniverse
DeSitterUniverse marked this pull request as ready for review August 20, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant