Skip to content

test(integration-tests): reap test containers with Ryuk when runs are killed - #519

Merged
dmealing merged 5 commits into
mainfrom
fm/mo-test-container-cleanup
Oct 31, 2026
Merged

dmealing merged 5 commits into
mainfrom
fm/mo-test-container-cleanup

Conversation

@dmealing

@dmealing dmealing commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Why

Test containers outlived the runs that started them. Every helper relied on in-process teardown (afterAll, close(), fixture finalizers, the ci-local EXIT trap), and none used --rm or a reaper. SIGKILL and an out-of-memory killer skip all of that, and so does plain SIGTERM in most runners, so each killed run left its containers running.

Three more causes sat alongside it:

  • The ci-local Postgres sidecar was deliberately kept per checkout. Each gate run uses a new worktree, so nothing ever reused it.
  • C# only looked protected. Testcontainers for .NET 3.10 starts Ryuk 0.6.0, which speaks Docker API 1.43. Docker 29 refuses that, so Ryuk accepted labels, removed nothing and exited.
  • Teardown left volumes behind. docker rm -f without -v keeps the anonymous volumes the Postgres and MySQL images declare.

What stops new leftovers

Ryuk, the Testcontainers reaper, runs in its own container and holds a TCP connection from each client. Ten seconds after that connection closes, for any reason, it removes everything carrying that session's label. The operating system closes a dead process's socket, so no code in the killed run has to execute.

Where How it is covered
Python, Java, Kotlin The docker-CLI helpers stay, and each container carries the Ryuk session label from the Testcontainers library (reaper.py, ReaperLabels)
C# Testcontainers 4.16, which ships Ryuk 0.14.0
TypeScript A private Ryuk per process, started in test-reaper.ts
scripts/ci-local.sh The sidecar is per run, always labelled, and a small background shell holds the Ryuk connection

TypeScript does not use testcontainers-node's getReaper. It adopts any Ryuk already running on the host, a sibling project's included, so a killed run's containers would wait for every other client of that Ryuk to end, with no upper limit. TESTCONTAINERS_RYUK_TEST_LABEL hides only Ryuks carrying the same label, so it does not help. That left testcontainers and @testcontainers/postgresql unused, and they are removed.

Bash cannot mark a descriptor close-on-exec, so ci-local holds the socket in a background shell that polls the script's pid. Otherwise every test JVM and build server would inherit the socket and keep the session open after the run.

The sidecar still serves every lane within a run, which is where its saving is. Only reuse between runs is gone: a warm start saved about 3 seconds on a run of about 26 minutes.

Teardown now uses rm -f -v everywhere.

The regression guard is one test per port that starts a real Postgres helper in a child process, sends SIGKILL, and asserts through Docker that the container is removed within 45 seconds: reaper-kill.test.ts, test_reaper_kill.py, ReaperKillTest.java, ReaperKillTest.kt, ReaperKillTests.cs. A helper that loses its label, or a Ryuk that removes nothing, fails on that deadline. The TypeScript file also kills one process while a second stays alive, and kills a real ci-local run to check its sidecar.

Start paths that remain unprotected

  • TESTCONTAINERS_RYUK_DISABLED=true is still honoured by the Python, Java, Kotlin and C# libraries. A run with it set is unprotected by choice. ci-local and the TypeScript helper no longer read it.
  • A killed Ryuk or a restarted Docker daemon. Ryuk is the only watcher, so nothing removes that session's containers if it dies first.
  • A docker run still in flight when its parent dies. The CLI child can outlive the parent. If it creates the container after Ryuk has pruned, for example after a slow image pull, that container is missed. The window is narrow.
  • Containers started by hand with docker run, by a person or an agent. No code here touches them.
  • contract-oasdiff.test.ts runs a foreground docker run --rm. A killed run leaves it until its short command exits, and --rm then removes it.
  • MO_PG_SIDECAR_NAME pointing at an already-running container. It is used and left alone on purpose, because that container is not the run's to remove.
  • services: containers in local-ci.yml on self-hosted runners. The runner owns them.
  • tools/prerelease/docker-compose.yml is a long-lived registry, not a test container.

If Ryuk cannot start, TypeScript and Python fail loudly rather than run unreaped, and ci-local falls back to per-scenario containers.

Validation

  • The five new SIGKILL tests pass against real Docker, including the TypeScript case where a killed process's container goes while a second process's keeps running.
  • Real test runs were also killed by hand with SIGKILL in every port, and with the cube lane's Docker network: containers were gone in 11 to 13 seconds.
  • A killed run left a concurrent run in a second checkout untouched in ci-local, Java and C#.
  • scripts/ci-local.sh --strict-toolchains, every lane: passed, with no test containers left behind.
  • No container that existed before this work was stopped or removed by it.

This took one no-mistakes run with one fix round, which covered the three review findings below. --skip ci was passed as triage directed, so hosted checks were followed on the PR directly.

  • TypeScript shared a host-wide Ryuk session: replaced with a private Ryuk per process.
  • A git grep gate stood in for proof: deleted, and replaced with the per-port kill tests.
  • ci-local carried switches nothing needed: TESTCONTAINERS_RYUK_DISABLED, TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE and CI_REAPER_FAILED are removed.

The pipeline record listed the gate script's finding as still open because it re-reported that finding after the fix. The script no longer exists.

Pipeline

Updates from git push no-mistakes

A test run's containers were removed only by its own teardown, which a killed
process never reaches. Every killed run left its containers running.

Put every docker-CLI test container under Ryuk, the Testcontainers reaper, and
give ci-local.sh a sidecar per run instead of one kept per checkout.
Testcontainers for .NET 3.10 starts Ryuk 0.6.0. It speaks Docker API 1.43, which
Docker 29 refuses, so it accepted every label, removed nothing and exited: a killed
C# run kept its containers while looking protected. 4.16 ships Ryuk 0.14.0, the one
ci-local uses. The static gate now refuses a version below 4, because no test can
tell the difference.

The Python helpers' docstrings still said they avoid the testcontainers package.
A failed start waited up to 15s again in every lane and left the named Ryuk running
until its own timeout. The Java Postgres failed-start path was also the one teardown
still leaving its anonymous volume.
@dmealing
dmealing merged commit 64e49f8 into main Oct 31, 2026
9 checks passed
@dmealing
dmealing deleted the fm/mo-test-container-cleanup branch October 31, 2026 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant