[dmealing]

What Changed

• All five ports (Python, Java, Kotlin, C#, TypeScript) raise the SIGKILL reaper test's removal bound from 45s to 120s, and each failure message now includes Docker's status for the container. The comments record the measured removal times. TypeScript's outer test timeouts grow to fit the new bound.
• Python's reaper (server/python/tests/integration/reaper.py) now starts its own Ryuk 0.14.0 and waits for Ryuk's ACK, as the TypeScript helper does. The testcontainers dependency leaves the integration extra (pyproject.toml, uv.lock).
• Python's test_reaper_kill.py adds a refused-removal test, in which a Docker API proxy fails the first container removal, and two socket tests for the ACK. Its failure message also includes Ryuk's log.

Cause

Verdict: a test problem in all five ports, plus a separate reaper gap in Python only.

• Trigger: dockerd cannot report a container's exit until its overlay unmount has synced the whole underlying filesystem. Sampled kernel stacks showed dockerd in ovl_sync_fs -> sync_filesystem -> folio_wait_writeback in 24 of 30 samples between the kill and the container's exit.
• Amplifier: a dirty-page backlog on a slow disk. Without one, the same path takes under a second.
• Symptom: the container stays listed past 45s with status Up while removal is still in progress. Removal then succeeds.

The hosted cause is inferred, not reproduced on hosted hardware. The hosted failure recorded no Docker status and no Ryuk log. The real suite's dirty backlog peaked at 153 MiB in the lab, below the 256 MiB the lab used to pass 45s, and hosted disk speed and dirty settings are unknown. If it recurs, the new messages will show it: a status of Exited or Dead, or a Ryuk log with no removal attempt, would point away from slow exit.

Evidence

Time from SIGKILL to removal on a capped Docker daemon with a 256 MiB dirty cap and a continuous writer:

Disk limit	Ryuk 0.8.1	Ryuk 0.14.0
12 MB/s	33.7-37.5s	33.9-36.7s
6 MB/s	74.3-76.0s	66.8-73.7s
6 MB/s, no writer	10.4-10.8s	not run

• Every removal succeeded and no container was left behind.
• On Ryuk 0.14.0 the real test fails at 45s with the hosted assertion and status Up, and passes at 120s in 90s.
• Without a forced backlog the old reaper removed 71 of 71 containers in 10.3-14.1s, with one client per Ryuk. A missing registration, a second client and a shared Ryuk did not appear in any of those runs.

The Python gap is separate. The testcontainers package runs Ryuk 0.8.1, which tries each removal once, discards the error and never has its ACK read. A proxy that refused one removal left the container running for good while Ryuk logged it as removed; Ryuk 0.14.0 retried and removed it. This is proven by fault injection only and never appeared under real load, so it is not offered as the explanation of the hosted failure. The other ports already ran Ryuk 0.14.0 with an ACK.

Slow removal has no regression test, because reproducing it needs a throttled daemon and a sustained backlog that CI cannot host. The bound's comments carry the measurements instead.

Gate

This took one no-mistakes run. Review found one issue, now fixed: the held-open test asserted only that a connection was returned. The Test step passed with an approved exception, because 2 of 4 live scenarios were driven against Docker. The two ACK tests are deliberately Docker-free unit tests, and real Ryuk's ACK is exercised by the live SIGKILL and retry tests, because the reaper waits for it before any container starts. A mutant check confirmed the held-open test can fail.

Validation

Risk Assessment

✅ Low: Test-only and test-harness changes; the reaper replacement, ACK wait, widened bounds, status-bearing messages and the fixed held-open test all behave correctly on trace, with the inferred-cause caveat already recorded.

Testing

Drove two reaper scenarios live against the local Docker daemon: a SIGKILLed child's container is removed by Ryuk, and a removal that fails once is retried until removed. Both pass. The two Ryuk registration scenarios ran against an in-process fake socket server, not Docker. They pass, and a mutant that closes the connection before returning is caught by the peer-side check, but they are not live product results. The full Python suite passes with xdist (4456 passed, 11 skipped), excluding reaper_kill, which ran separately. The flake root cause from the intent (why the container outlived its process on hosted CI) was not reproduced or confirmed here.

• Live validation: ⚠️ inconclusive - 2 of 4 scenarios driven live against the product

Scenario	Result	Live	Evidence
Killed child process's container is removed by Ryuk within bound	✅ pass	live	pytest tests/integration/test_reaper_kill.py::test_killed_process_removes_its_container
Container removal that fails once is retried until removed	✅ pass	live	pytest tests/integration/test_reaper_kill.py::test_a_removal_that_fails_once_is_retried
Unacknowledged Ryuk registration is not treated as registered	⏸️ untested	no	The prior run used an in-process fake socket server, not Docker, so it does not establish a live result. Live check needs the Ryuk reaper driven against the Docker daemon with an unacknowledged regist…
Acknowledged Ryuk registration stays open from peer side (adversarial: connection closed before return)	⏸️ untested	no	The prior run used a fake socket server and a mutant check, not Docker, so it does not establish a live result. Live check needs a real Ryuk connection held open against the Docker daemon and observed…

• Outcome: ⚠️ 1 warning across 1 run (16m7s)

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info

• ⚠️ server/python/tests/integration/test_reaper_kill.py:215 - test_a_connection_ryuk_acknowledged_is_held_open claims the connection is held open but asserts only conn is not None. The session lifetime depends on that connection staying open. An implementation that closed the socket before returning it (for example with socket.create_connection(...) as conn: in _register) still passes, although Ryuk would then prune the session 10s later. Public behaviour: _register returns a live connection after ACK. Oracle: Ryuk's protocol, where closing the last connection starts the prune. Fix: assert the socket is still open, such as conn.fileno() != -1, or have _one_shot_server confirm that it sees no EOF.
• ℹ️ server/python/tests/integration/test_reaper_kill.py:44 - The hosted failure's cause stays inferred, and the commit message says so: not reproduced on hosted hardware, and the lab's real-suite backlog peak of 153 MiB was below what reached 45s. I traced the candidate paths. Slow removal is covered by the 120s bound in all five ports. A single failed removal is covered by Python's move to Ryuk 0.14.0 with the fault-injection test. An unregistered session is covered by the ACK wait. Java (Testcontainers 2.0.5) and C# (4.16.0) already run Ryuk 0.14.0, so no sibling port keeps the one-try gap. A removal slower than 120s can still fail, and the new messages then show Docker's status. The PR summary should carry the 'inferred' caveat.

🔧 Fix applied.
1 info still open:

• ℹ️ server/python/tests/integration/test_reaper_kill.py:44 - The hosted failure's cause stays inferred, and the commit message says so: not reproduced on hosted hardware, and the lab's real-suite backlog peak of 153 MiB was below what reached 45s. I traced the candidate paths. Slow removal is covered by the 120s bound in all five ports. A single failed removal is covered by Python's move to Ryuk 0.14.0 with the fault-injection test. An unregistered session is covered by the ACK wait. Java (Testcontainers 2.0.5) and C# (4.16.0) already run Ryuk 0.14.0, so no sibling port keeps the one-try gap. A removal slower than 120s can still fail, and the new messages then show Docker's status. The PR summary should carry the 'inferred' caveat.

⚠️ **Test** - 1 warning

• ⚠️ live validation verdict: inconclusive (2 of 4 scenarios were driven live against the product); untested: Unacknowledged Ryuk registration is not treated as registered, Acknowledged Ryuk registration stays open from peer side (adversarial: connection closed before return)
• Live validation: ⚠️ inconclusive - 2 of 4 scenarios driven live against the product

Scenario	Result	Live	Evidence
Killed child process's container is removed by Ryuk within bound	✅ pass	live	pytest tests/integration/test_reaper_kill.py::test_killed_process_removes_its_container
Container removal that fails once is retried until removed	✅ pass	live	pytest tests/integration/test_reaper_kill.py::test_a_removal_that_fails_once_is_retried
Unacknowledged Ryuk registration is not treated as registered	⏸️ untested	no	The prior run used an in-process fake socket server, not Docker, so it does not establish a live result. Live check needs the Ryuk reaper driven against the Docker daemon with an unacknowledged regist…
Acknowledged Ryuk registration stays open from peer side (adversarial: connection closed before return)	⏸️ untested	no	The prior run used a fake socket server and a mutant check, not Docker, so it does not establish a live result. Live check needs a real Ryuk connection held open against the Docker daemon and observed…

• scripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchains
• cd server/python && .venv/bin/python -m pytest tests/integration/test_reaper_kill.py -k "ack or never" -q (2 passed, no Docker)
• Mutant check: _register replaced by a version that closes the socket before returning; test_a_connection_ryuk_acknowledged_is_held_open fails with AssertionError, and the peer-side state is "eof"
• uv sync --all-extras --all-groups in server/python (repo-local .venv, to install fastapi and other extras that were missing)
• .venv/bin/python -m pytest tests -q -n 4 -k 'not reaper_kill' (4456 passed, 11 skipped)
• .venv/bin/python -m pytest tests/integration/test_reaper_kill.py (4 passed, live Docker, Ryuk reaper)

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.