Skip to content

tests: the handoff driver cannot orphan its decoder on SIGKILL (PR_SET_PDEATHSIG, negative control included, refs #698) - #1524

Merged
bong-water-water-bong merged 1 commit into
mainfrom
1bit/kvshare-orphan
Nov 8, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
1bit/kvshare-orphan

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

tests/kvshare_fd_handoff.py — the driver behind tests/hybrid_kvshare_e2e.sh (#1408) — runs an HRX llama-server in its own session and reaps it in a finally. That covers a clean exit, Ctrl-C and SIGTERM, but not SIGKILL: an OOM kill, a CI timeout, or an operator's kill -9 leaves the decoder running with its HRX context mapped. On 2026-11-08 an orphaned llama-server held 51 GiB of GTT for 14 h and knocked out another service; a test that can leak a decoder is exactly the wrong kind of test to have, so the child is now spawned with PR_SET_PDEATHSIG (SIGTERM) and the kernel takes it down whenever the parent dies, for any reason.

Linux only, which is where the HRX route runs; elsewhere the flag is not set and the finally remains the reaper.

Pinned without a GPU or a model

tests/kvshare_orphan_test.sh (ctest kvshare_orphan, ~0.9 s) drives the mechanism directly:

check result
the driver spawns a child in its own session ok (pid printed, alive)
SIGKILL the driver → the child goes with it ok (PR_SET_PDEATHSIG)
negative control: a child spawned without the flag survives the same kill ok — this is the leak being prevented, so the test would fail if the flag were dropped

The third row is the one that matters: without it the test would pass just as happily if die_with_parent() were a no-op.

Device re-run of the test this changes (no regression)

gpu-hold -l zerocopy -t 15, ticket held 25 s, released clean, no server left behind:

share ok: share_id=1 regions=1
provider ASan reports=0   provider result: {"n_tokens":271,"import_ms":30.240,...}
adopt ok: {"n_adopted":271,"n_read":0,"timings":{"adopt_ms":0.103}}
decode after adopt: prompt_n=1  text=' The tide comes in twice a day,'
ok x5  →  PASS

Refs #698. tools/copyright.py --check clean (notice on the new test), bash -n and py_compile clean, and the change is 19 lines in the driver plus the test.

What this does not fix: a run that is SIGKILLed now leaves nothing, but a run whose provider is killed mid-write still leaves a region the decoder will adopt from — that is the "provider claims more than it wrote" case, which the route already refuses per request (engine#1203) and is not this PR's subject.

…_PDEATHSIG)

tests/kvshare_fd_handoff.py runs an HRX llama-server in its own session and reaps it in a `finally`, which covers a
clean exit, Ctrl-C and SIGTERM -- but not SIGKILL (an OOM kill, a CI timeout, `kill -9`). On this box an orphaned
llama-server held 51 GiB of GTT for 14 h and knocked out another service, so a test that can leak a decoder holding
an HRX context is exactly the wrong kind of test to have.

The server is now spawned with PR_SET_PDEATHSIG (SIGTERM), so the kernel takes it down when this process dies whatever
the reason. Linux only, which is where the HRX route runs; elsewhere the flag is simply not set.

tests/kvshare_orphan_test.sh pins it without a GPU or a model: it spawns a sleep through the driver's own
`die_with_parent`, SIGKILLs the parent and asserts the child is gone -- and, as the negative control, that a child
spawned without the flag survives the same kill, which is the leak being prevented. Registered as `kvshare_orphan`.

Refs #698.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

#1524 ready for review at d76d6b9110f6d4b804da2fbf936e25f042368c1b — MERGEABLE, 3 files, +86. The handoff driver (#1408) reaped its HRX server in a finally, which misses SIGKILL — the exact shape that left a 51 GiB-GTT llama-server orphaned for 14 h on 11-08. Now PR_SET_PDEATHSIG, and tests/kvshare_orphan_test.sh (ctest kvshare_orphan, 0.9 s, no GPU/model) pins it with a negative control: a child spawned without the flag must survive the same kill, so a no-op die_with_parent() cannot pass. Device re-run of the changed test: PASS (n_adopted 271, n_read 0, prompt_n 1, ASan 0, adopt 0.103 ms), gpu-hold ticket released clean with no server left behind.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at d76d6b9110f6d4b804da2fbf936e25f042368c1b

I reviewed d76d6b9 against #698 and the orphaned-server incident on 11-08.

What I ran at this head:

  • bash tests/kvshare_orphan_test.sh passed four times in a row, with all three rows ok each time, including the negative control. The control is what makes the test meaningful: if the flag were dropped, row 2 would fail while row 3 still passed.
  • Afterwards, no time.sleep(120) child is left on the box.
  • py_compile passes and tools/copyright.py --check exits 0.
  • shellcheck's one warning (SC2034, gone unused) is a false positive, because gone is read inside the eval in check.

Checked for correctness:

  • preexec_fn is only safe when no other thread is running at fork time. The driver's only thread (do_share, :198-199) starts after the Popen (:149), so the fork happens single-threaded.
  • PR_SET_PDEATHSIG fires on the death of the parent thread that forked the child. Here that's the main thread, so it lives exactly as long as the driver.
  • The signal survives execve because llama-server isn't setuid.

Three small notes, not blocking:

  1. A fork → prctl race. If the driver dies after fork but before prctl runs, the child is already reparented and no signal ever comes. The standard close is to save os.getpid() before Popen, then in die_with_parent, after the prctl, exit with os._exit(1) if os.getppid() no longer matches.
  2. SIGTERM vs SIGKILL. SIGTERM starts llama-server's graceful shutdown. If that teardown ever stalls (HRX has had teardown and import stalls before), the decoder stays alive and keeps its GTT, which is the case this PR is closing. SIGKILL is the safer death signal for a test driver.
  3. prctl's return value isn't checked. Through ctypes, a failure returns -1 rather than raising, so the except never sees it. Fine on Linux, but the comment "elsewhere the flag is simply not set" is accurate only by accident.

I didn't re-run the device row. The test this PR changes is driven without a GPU, and the author's 25 s gpu-hold run is reported above.

@bong-water-water-bong
bong-water-water-bong merged commit 8e5155b into main Nov 8, 2026
9 of 10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/kvshare-orphan branch November 8, 2026 19:07
bong-water-water-bong added a commit that referenced this pull request Nov 9, 2026
…s thread-safe (refs #1624, #698) (#1637)

* tests: the handoff driver's provider dies with it too, not just the server

#1524 gave the shared-KV driver's *server* PR_SET_PDEATHSIG, so a SIGKILL of the driver cannot leave an HRX decoder
holding a context. Its **provider** was still spawned with `subprocess.run(...)` and no such hook -- and that is the
one that runs for up to 1800 s on a 16k prompt, so a driver killed inside that window left an orphan with the
region mapped. Same class as engine#1624 (kill_tree recomputing descendants after the root died), in this lane's own
file.

The provider now spawns as `Popen` with `start_new_session=True` and `preexec_fn=die_with_parent`, then
`wait(timeout=1800)`; on the deadline it reaps **the group**, which `run()` could not do (it blocks and leaves the
child in the driver's process group, which the teardown does not signal). argv, env, `pass_fds` and the redirected
log are unchanged, so the provider sees exactly the invocation it saw before.

One thing the multi-threaded shape forced: the driver runs a socket-listener thread, so the fork that spawns a child
happens in a threaded process, where anything the child does before `exec` that allocates or imports can deadlock on
a lock another thread held. `die_with_parent` therefore resolves libc and the signal number once at **import** time
and leaves the child a single C call that allocates nothing (it used to `import ctypes` inside the forked child).

tests/kvshare_orphan_test.sh gains the source assertion for both spawns -- the functional path needs an HRX server
and a model, so a grep is the only device-free way to see that a later edit did not drop one of the two hooks -- and
stays at 2.5 s, 4/4 checks.

Gate: `kvshare_orphan` 4/4 and `ctest` green (142/142) on this tree at low load; the route-test reds reappear at load
62 and pass in isolation, which is the #1485 contention class and not this diff. The **device** e2e was not re-run:
its two inputs (the HRX llama-server and the reference provider) went with the build trees in the 11-08 disk cleanup,
and rebuilding them is ~40 minutes of CPU; the invocation is byte-identical apart from the process attributes, so
what could differ there is the lifecycle this test already pins.

Refs #1624 (the same orphan class), #698 (the gate), #1524 (the server-side half).

* tests: the handoff driver stops picking a port and reads the one the server bound (engine#1485)

The driver picked its port with the bind(0)-and-release one-liner, so between closing that socket and llama-server
binding it any other test on a busy box could take it -- engine#1485's race, which every other serve test has already
migrated away from (#1525, #1545). Its own census lists this file as covered by #1637, which until now was true only of
the orphan fix and not of the port.

It is now the other shape: `--port 0` and read the bound port back. llama-server handles 0 by binding any port
(`tools/server/server-http.cpp`) and prints `listening on http://host:port`, so there is no window to lose.
`wait_bound_port()` parses that line and the caller gets None -- with the log tail -- if the server dies or never
prints it, rather than guessing. `free_port()` is deleted: nothing else in the file used it.

Two details the parser had to get right, both measured: the capture is greedy up to the **last** colon and anchored at
end-of-line, because the non-greedy form matched the `1` of `http://[::1]:44444`; and a `unix://` ready line matches
nothing, which is the correct answer for a socket curl cannot reach. Checked against four synthetic logs (IPv4, IPv6,
unix, no line) plus the orphan test.

---------

Co-authored-by: t <t@e.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant