Repository navigation
test(integration-tests): reap test containers with Ryuk when runs are killed - #519
Merged
Merged
Conversation
A test run's containers were removed only by its own teardown, which a killed process never reaches. Every killed run left its containers running. Put every docker-CLI test container under Ryuk, the Testcontainers reaper, and give ci-local.sh a sidecar per run instead of one kept per checkout.
Testcontainers for .NET 3.10 starts Ryuk 0.6.0. It speaks Docker API 1.43, which Docker 29 refuses, so it accepted every label, removed nothing and exited: a killed C# run kept its containers while looking protected. 4.16 ships Ryuk 0.14.0, the one ci-local uses. The static gate now refuses a version below 4, because no test can tell the difference. The Python helpers' docstrings still said they avoid the testcontainers package.
A failed start waited up to 15s again in every lane and left the named Ryuk running until its own timeout. The Java Postgres failed-start path was also the one teardown still leaving its anonymous volume.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Test containers outlived the runs that started them. Every helper relied on in-process teardown (
afterAll,close(), fixture finalizers, theci-localEXIT trap), and none used--rmor a reaper. SIGKILL and an out-of-memory killer skip all of that, and so does plain SIGTERM in most runners, so each killed run left its containers running.Three more causes sat alongside it:
ci-localPostgres sidecar was deliberately kept per checkout. Each gate run uses a new worktree, so nothing ever reused it.docker rm -fwithout-vkeeps the anonymous volumes the Postgres and MySQL images declare.What stops new leftovers
Ryuk, the Testcontainers reaper, runs in its own container and holds a TCP connection from each client. Ten seconds after that connection closes, for any reason, it removes everything carrying that session's label. The operating system closes a dead process's socket, so no code in the killed run has to execute.
reaper.py,ReaperLabels)test-reaper.tsscripts/ci-local.shTypeScript does not use testcontainers-node's
getReaper. It adopts any Ryuk already running on the host, a sibling project's included, so a killed run's containers would wait for every other client of that Ryuk to end, with no upper limit.TESTCONTAINERS_RYUK_TEST_LABELhides only Ryuks carrying the same label, so it does not help. That lefttestcontainersand@testcontainers/postgresqlunused, and they are removed.Bash cannot mark a descriptor close-on-exec, so
ci-localholds the socket in a background shell that polls the script's pid. Otherwise every test JVM and build server would inherit the socket and keep the session open after the run.The sidecar still serves every lane within a run, which is where its saving is. Only reuse between runs is gone: a warm start saved about 3 seconds on a run of about 26 minutes.
Teardown now uses
rm -f -veverywhere.The regression guard is one test per port that starts a real Postgres helper in a child process, sends SIGKILL, and asserts through Docker that the container is removed within 45 seconds:
reaper-kill.test.ts,test_reaper_kill.py,ReaperKillTest.java,ReaperKillTest.kt,ReaperKillTests.cs. A helper that loses its label, or a Ryuk that removes nothing, fails on that deadline. The TypeScript file also kills one process while a second stays alive, and kills a realci-localrun to check its sidecar.Start paths that remain unprotected
TESTCONTAINERS_RYUK_DISABLED=trueis still honoured by the Python, Java, Kotlin and C# libraries. A run with it set is unprotected by choice.ci-localand the TypeScript helper no longer read it.docker runstill in flight when its parent dies. The CLI child can outlive the parent. If it creates the container after Ryuk has pruned, for example after a slow image pull, that container is missed. The window is narrow.docker run, by a person or an agent. No code here touches them.contract-oasdiff.test.tsruns a foregrounddocker run --rm. A killed run leaves it until its short command exits, and--rmthen removes it.MO_PG_SIDECAR_NAMEpointing at an already-running container. It is used and left alone on purpose, because that container is not the run's to remove.services:containers inlocal-ci.ymlon self-hosted runners. The runner owns them.tools/prerelease/docker-compose.ymlis a long-lived registry, not a test container.If Ryuk cannot start, TypeScript and Python fail loudly rather than run unreaped, and
ci-localfalls back to per-scenario containers.Validation
ci-local, Java and C#.scripts/ci-local.sh --strict-toolchains, every lane: passed, with no test containers left behind.This took one no-mistakes run with one fix round, which covered the three review findings below.
--skip ciwas passed as triage directed, so hosted checks were followed on the PR directly.git grepgate stood in for proof: deleted, and replaced with the per-port kill tests.ci-localcarried switches nothing needed:TESTCONTAINERS_RYUK_DISABLED,TESTCONTAINERS_DOCKER_SOCKET_OVERRIDEandCI_REAPER_FAILEDare removed.The pipeline record listed the gate script's finding as still open because it re-reported that finding after the fix. The script no longer exists.
Pipeline
Updates from git push no-mistakes