Repository navigation
tests: the handoff driver cannot orphan its decoder on SIGKILL (PR_SET_PDEATHSIG, negative control included, refs #698) - #1524
Conversation
…_PDEATHSIG) tests/kvshare_fd_handoff.py runs an HRX llama-server in its own session and reaps it in a `finally`, which covers a clean exit, Ctrl-C and SIGTERM -- but not SIGKILL (an OOM kill, a CI timeout, `kill -9`). On this box an orphaned llama-server held 51 GiB of GTT for 14 h and knocked out another service, so a test that can leak a decoder holding an HRX context is exactly the wrong kind of test to have. The server is now spawned with PR_SET_PDEATHSIG (SIGTERM), so the kernel takes it down when this process dies whatever the reason. Linux only, which is where the HRX route runs; elsewhere the flag is simply not set. tests/kvshare_orphan_test.sh pins it without a GPU or a model: it spawns a sleep through the driver's own `die_with_parent`, SIGKILLs the parent and asserts the child is gone -- and, as the negative control, that a child spawned without the flag survives the same kill, which is the leak being prevented. Registered as `kvshare_orphan`. Refs #698.
|
#1524 ready for review at |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at d76d6b9110f6d4b804da2fbf936e25f042368c1b
I reviewed d76d6b9 against #698 and the orphaned-server incident on 11-08.
What I ran at this head:
bash tests/kvshare_orphan_test.shpassed four times in a row, with all three rows ok each time, including the negative control. The control is what makes the test meaningful: if the flag were dropped, row 2 would fail while row 3 still passed.- Afterwards, no
time.sleep(120)child is left on the box. py_compilepasses andtools/copyright.py --checkexits 0.shellcheck's one warning (SC2034,goneunused) is a false positive, becausegoneis read inside theevalincheck.
Checked for correctness:
preexec_fnis only safe when no other thread is running at fork time. The driver's only thread (do_share,:198-199) starts after thePopen(:149), so the fork happens single-threaded.PR_SET_PDEATHSIGfires on the death of the parent thread that forked the child. Here that's the main thread, so it lives exactly as long as the driver.- The signal survives
execvebecausellama-serverisn't setuid.
Three small notes, not blocking:
- A fork →
prctlrace. If the driver dies afterforkbut beforeprctlruns, the child is already reparented and no signal ever comes. The standard close is to saveos.getpid()beforePopen, then indie_with_parent, after theprctl, exit withos._exit(1)ifos.getppid()no longer matches. - SIGTERM vs SIGKILL. SIGTERM starts
llama-server's graceful shutdown. If that teardown ever stalls (HRX has had teardown and import stalls before), the decoder stays alive and keeps its GTT, which is the case this PR is closing.SIGKILLis the safer death signal for a test driver. prctl's return value isn't checked. Through ctypes, a failure returns -1 rather than raising, so theexceptnever sees it. Fine on Linux, but the comment "elsewhere the flag is simply not set" is accurate only by accident.
I didn't re-run the device row. The test this PR changes is driven without a GPU, and the author's 25 s gpu-hold run is reported above.
…s thread-safe (refs #1624, #698) (#1637) * tests: the handoff driver's provider dies with it too, not just the server #1524 gave the shared-KV driver's *server* PR_SET_PDEATHSIG, so a SIGKILL of the driver cannot leave an HRX decoder holding a context. Its **provider** was still spawned with `subprocess.run(...)` and no such hook -- and that is the one that runs for up to 1800 s on a 16k prompt, so a driver killed inside that window left an orphan with the region mapped. Same class as engine#1624 (kill_tree recomputing descendants after the root died), in this lane's own file. The provider now spawns as `Popen` with `start_new_session=True` and `preexec_fn=die_with_parent`, then `wait(timeout=1800)`; on the deadline it reaps **the group**, which `run()` could not do (it blocks and leaves the child in the driver's process group, which the teardown does not signal). argv, env, `pass_fds` and the redirected log are unchanged, so the provider sees exactly the invocation it saw before. One thing the multi-threaded shape forced: the driver runs a socket-listener thread, so the fork that spawns a child happens in a threaded process, where anything the child does before `exec` that allocates or imports can deadlock on a lock another thread held. `die_with_parent` therefore resolves libc and the signal number once at **import** time and leaves the child a single C call that allocates nothing (it used to `import ctypes` inside the forked child). tests/kvshare_orphan_test.sh gains the source assertion for both spawns -- the functional path needs an HRX server and a model, so a grep is the only device-free way to see that a later edit did not drop one of the two hooks -- and stays at 2.5 s, 4/4 checks. Gate: `kvshare_orphan` 4/4 and `ctest` green (142/142) on this tree at low load; the route-test reds reappear at load 62 and pass in isolation, which is the #1485 contention class and not this diff. The **device** e2e was not re-run: its two inputs (the HRX llama-server and the reference provider) went with the build trees in the 11-08 disk cleanup, and rebuilding them is ~40 minutes of CPU; the invocation is byte-identical apart from the process attributes, so what could differ there is the lifecycle this test already pins. Refs #1624 (the same orphan class), #698 (the gate), #1524 (the server-side half). * tests: the handoff driver stops picking a port and reads the one the server bound (engine#1485) The driver picked its port with the bind(0)-and-release one-liner, so between closing that socket and llama-server binding it any other test on a busy box could take it -- engine#1485's race, which every other serve test has already migrated away from (#1525, #1545). Its own census lists this file as covered by #1637, which until now was true only of the orphan fix and not of the port. It is now the other shape: `--port 0` and read the bound port back. llama-server handles 0 by binding any port (`tools/server/server-http.cpp`) and prints `listening on http://host:port`, so there is no window to lose. `wait_bound_port()` parses that line and the caller gets None -- with the log tail -- if the server dies or never prints it, rather than guessing. `free_port()` is deleted: nothing else in the file used it. Two details the parser had to get right, both measured: the capture is greedy up to the **last** colon and anchored at end-of-line, because the non-greedy form matched the `1` of `http://[::1]:44444`; and a `unix://` ready line matches nothing, which is the correct answer for a socket curl cannot reach. Checked against four synthetic logs (IPv4, IPv6, unix, no line) plus the orphan test. --------- Co-authored-by: t <t@e.com>
tests/kvshare_fd_handoff.py— the driver behindtests/hybrid_kvshare_e2e.sh(#1408) — runs an HRX llama-server in its own session and reaps it in afinally. That covers a clean exit, Ctrl-C and SIGTERM, but not SIGKILL: an OOM kill, a CI timeout, or an operator'skill -9leaves the decoder running with its HRX context mapped. On 2026-11-08 an orphaned llama-server held 51 GiB of GTT for 14 h and knocked out another service; a test that can leak a decoder is exactly the wrong kind of test to have, so the child is now spawned withPR_SET_PDEATHSIG(SIGTERM) and the kernel takes it down whenever the parent dies, for any reason.Linux only, which is where the HRX route runs; elsewhere the flag is not set and the
finallyremains the reaper.Pinned without a GPU or a model
tests/kvshare_orphan_test.sh(ctestkvshare_orphan, ~0.9 s) drives the mechanism directly:PR_SET_PDEATHSIG)The third row is the one that matters: without it the test would pass just as happily if
die_with_parent()were a no-op.Device re-run of the test this changes (no regression)
gpu-hold -l zerocopy -t 15, ticket held 25 s, released clean, no server left behind:Refs #698.
tools/copyright.py --checkclean (notice on the new test),bash -nandpy_compileclean, and the change is 19 lines in the driver plus the test.What this does not fix: a run that is SIGKILLed now leaves nothing, but a run whose provider is killed mid-write still leaves a region the decoder will adopt from — that is the "provider claims more than it wrote" case, which the route already refuses per request (engine#1203) and is not this PR's subject.