[bong-water-water-bong]

Refs #1485 — this is the rest of it. With this PR the migration stops being a retry and becomes a removal of the race.

What these six were still doing, and why they no longer need to

serve --port 0 landed after they were written, so they still picked a port with bind(0), released it, and — for the five that #1446 touched — retried around the collision when another -j test took it in the gap. That retry was the right stopgap when --port 0 did not exist; with it, the gap is not there to retry around.

• hybrid_route.sh, hybrid_kvshare_route.sh, hybrid_e2e.sh (start()): --port 0, then serve_port_wait before $api is built. Budgets 900 / 900 / 6000 tenths — hybrid_e2e's own health window is 1200 × 0.5 s and it loads a model, so its budget is matched to that rather than to a guess.
• hrx_cwsr_route.sh, device_route.sh (run()): --port 0 and no readiness parse — they read the backend's record file. They keep the liveness half (kill -0 || break) so a serve that dies fails at once instead of sleeping out the budget and surfacing later as a record-file parse error.
• pm_laya_route.sh: --port 0 + serve_port_wait for serve. freeport() stays, for the fake lemonade python process — it is not serve, cannot pick its own port and cannot report one; the comment says so.

Deleted with the pick: the retry loop, the EADDRINUSE branch (#1446's), and the second failure branch that only existed to explain a collision.

Census, in the same commit

_onebit_route_tests now names exactly the tests that still pick a port up front: smoke_serve, print_backend, lemonade_hrx, template_pin, pm_route — plus bench_selftest, which binds through tools/bench.py and is pinned by its own dedicated case (test_indirect_server_tests_hold_the_lock_too), so it must not be dropped on the basis of a text scan. hybrid_e2e leaves the config-dependent list; pm_laya_route and hrx_cpu_weight_parity stay (both still pick, the latter because it runs llama-server).

Census floor: 9 → 5. It counted 15 before this migration.

Verified

• bash -n clean on all six scripts.
• shellcheck warning counts against main: unchanged on five, one fewer on hybrid_e2e (2 → 1). No new warnings.
• Census rehearsed against a stand-in ctest built from the new lists: 3/3 cases pass.
• Not run end to end: hybrid_e2e and pm_laya_route need a model and a laya checkpoint this box does not have. hybrid_route, hybrid_kvshare_route, hrx_cwsr_route and device_route are in the CI build job's ctest run, which is where this diff is actually exercised — I would not read a green census alone as proof.

Where #1485 stands after this

Every test script that runs 1bit serve takes its port from serve. What remains in the issue's 24-script inventory is out permanently, and each says why in the lock list's comment: hrx_cpu_weight_parity.sh and kvshare_fd_handoff.py run llama-server; print_backend.sh, template_pin.sh and lemonade_hrx.sh use fixed 43xxx ports as part of what they assert (and --print-backend does not bind at all); smoke_serve.sh keeps one deliberate squatter pick.

[bong-water-water-bong]

VERDICT: READY at 9b81f74315b2564b87a49c6695cc1cb3a5874732.

Deleting the retry rather than keeping it is the right call and the right reason — a retry around a release-then-bind race is a mitigation, and with
--port 0 there is no release to race. Both surviving mentions of it are comments that say exactly that, which is better than silence:

hybrid_kvshare_route.sh:47   # the release-then-bind race this test used to retry around is gone rather than mitigated (engine #1444/#1485)
hybrid_route.sh:43          # (same)

The liveness half is kept where it is load-bearing, and dropped where the shape does not have it: kill -0 survives in the four run()-style scripts
(device_route, hrx_cwsr_route, hybrid_e2e, hybrid_route), and hybrid_kvshare_route/pm_laya_route do not need it. device_route is also the case
worth naming explicitly: it takes --port 0 and needs no read-back at all because it verifies through the backend's record file (--llama-server "$scratch/backend.py", its own comment says so), which is the same minimalism as npu_repack_path in the last batch — no wait where there is nothing to read.

pm_laya_route keeps freeport() (2 uses) for the fake Lemonade's Python server, which is the same distinction as hrx_cpu_weight_parity: that port is not
1bit serve's to choose, so it is not this migration's to remove.

Census, verified in the same commit:

set(_onebit_route_tests  smoke_serve print_backend lemonade_hrx template_pin pm_route bench_selftest)      <- 10 -> 6 base names
list(APPEND _onebit_route_tests  pm_laya_route hrx_cpu_weight_parity)                                      <- hybrid_e2e gone
tests/route_port_lock_test.py:103  assertGreaterEqual(len(found), 5)                                       <- floor 9 -> 5

and bench_selftest stays for the reason it always did (it binds through tools/bench.py, and its own case pins that).

Mechanical: bash -n clean on all six; shellcheck per file equal to main in five and one fewer in hybrid_e2e (2 SC2034 → 1) — which is what
deleting the retry's unused variable should look like.

That is #1485's retry family retired at the root: --port 0 replaced the mitigation, and the census descended with it.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1485 - Partially compliant Compliant requirements: The 6 scripts (device_route.sh, hrx_cwsr_route.sh, hybrid_e2e.sh, hybrid_kvshare_route.sh, hybrid_route.sh, pm_laya_route.sh) now use --port 0 instead of picking and releasing ports The retry logic and EADDRINUSE handling have been removed from these scripts The census in CMakeLists.txt has been updated to remove these scripts from the list of tests that pick ports The Python test has been updated to reflect the new count of route tests that still pick ports Non-compliant requirements: The full migration of the 19 scripts is not complete in this PR (only 6 are migrated) Requires further human verification: The actual port binding behavior of serve --port 0 should be verified on the hardware to ensure it correctly reports the bound port The performance impact of the new approach should be measured on the hardware

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Port Binding Change The script now uses --port 0 instead of picking and releasing a port, which eliminates the race condition. However, the change relies on the serve command correctly reporting the bound port, which should be verified on the hardware. run() { # <record> <model> [serve args] # --port 0: the kernel picks and serve reports it; this test reads the backend's record file, so it needs no # readiness parse - and nothing is picked and released before serve binds (engine #1444/#1485) RECORD="$1" "$bin" serve -m "$scratch/$2.gguf" --port 0 --llama-server "$scratch/backend.py" "${@:3}" >"$1.log" 2>&1 & pid=$! for _ in $(seq 1 100); do [ -s "$1" ] && break; kill -0 "$pid" 2>/dev/null || break; sleep 0.1; done kill -9 "$pid" 2>/dev/null; wait "$pid" 2>/dev/null } Port Binding Change The script now uses --port 0 instead of picking and releasing a port, which eliminates the race condition. However, the change relies on the serve command correctly reporting the bound port, which should be verified on the hardware. run() { # <record> [serve args]: the backend's three variables, as JSON, in <record>; serve's log in <record>.log # --port 0: the kernel picks and serve reports it; this test reads the backend's record file, so it needs no # readiness parse - and nothing is picked and released before serve binds (engine #1444/#1485) RECORD="$1" "$bin" serve -m "$scratch/plain.gguf" --port 0 --llama-server "$scratch/backend.py" \ --no-mem-check "${@:2}" >"$1.log" 2>&1 & pid=$! # the liveness half of serve_port_wait: a serve that dies at once must fail here, not sleep out the budget and # surface later as a parse error from the record file for _ in $(seq 1 100); do [ -s "$1" ] && break; kill -0 "$pid" 2>/dev/null || break; sleep 0.1; done kill -9 "$pid" 2>/dev/null; wait "$pid" 2>/dev/null } Port Binding Change The script now uses --port 0 instead of picking and releasing a port, which eliminates the race condition. The script also now uses serve_port_wait to read back the port serve bound, which should be verified on the hardware. start() { # start <device> <log> [env...] local device=$1 log=$2; shift 2 local port code # --port 0: the kernel picks and serve reports it - no pick-and-release window for another serve to take (engine # #1444/#1485). The budget is generous: this script's own health window is 1200 x 0.5 s and it loads a model. env "$@" "$bin" serve -m "$model" --device "$device" --port 0 --alias e2e-model --ctx-size 2048 \ --no-mem-check "${server_arg[@]}" >"$log" 2>&1 & pid=$! port=$(serve_port_wait "$log" "$pid" 6000) || { echo "FAIL serve never reported a port; log=$log"; tail -5 "$log" 2>/dev/null; return 1; } api="http://127.0.0.1:$port" code=000 for _ in $(seq 1 1200); do code=$(curl -s -o /dev/null -w '%{http_code}' "$api/health"); [ "$code" = 200 ] && break kill -0 "$pid" 2>/dev/null || break; sleep 0.5 done [ "$code" = 200 ] } Port Binding Change The script now uses --port 0 instead of picking and releasing a port, which eliminates the race condition. The script also now uses serve_port_wait to read back the port serve bound, which should be verified on the hardware. start() { # start <log> [env...] local log=$1; shift local port code # --port 0: the kernel picks and serve reports it, so nothing here picks a port and releases it before serve binds - # the release-then-bind race this test used to retry around is gone rather than mitigated (engine #1444/#1485). env FAKE_KVSHARE_MARKER="$scratch/provider.json" FAKE_SLOT_MARKER="$scratch/slot.json" "$@" "$bin" serve -m "$scratch/tiny.gguf" --device hybrid --port 0 --alias hybrid-model \ --llama-server "$here/fake_backend.py" --no-mem-check >"$log" 2>&1 & pid=$! port=$(serve_port_wait "$log" "$pid" 900) || { echo "FAIL serve never reported a port; log=$log"; tail -5 "$log" 2>/dev/null; return 1; } api="http://127.0.0.1:$port" code=000 for _ in $(seq 1 100); do code=$(curl -s -o /dev/null -w '%{http_code}' "$api/health"); [ "$code" = 200 ] && break; sleep 0.1; done # this serve's OWN slot directory, from its child's argv: /tmp/1bit-hybrid-* is shared with every other serve # on the box, so globbing it reports another test's in-flight socket as this route leaking one (see the same # fix and the measurement in tests/hybrid_route.sh) slotdir=$(for c in $(pgrep -P "$pid" 2>/dev/null); do Port Binding Change The script now uses --port 0 instead of picking and releasing a port, which eliminates the race condition. The script also now uses serve_port_wait to read back the port serve bound, which should be verified on the hardware. start() { # start <log> [env...]: serve on the port it reports, with the fake backend and env applied local log=$1; shift local port code deadline # --port 0: the kernel picks and serve reports it, so nothing here picks a port and releases it before serve binds - # the release-then-bind race this test used to retry around is gone rather than mitigated (engine #1444/#1485). env FAKE_SLOT_MARKER="$scratch/slot.json" "$@" "$bin" serve -m "$scratch/tiny.gguf" --device hybrid --port 0 --alias hybrid-model \ --llama-server "$here/fake_backend.py" --no-mem-check >"$log" 2>&1 & pid=$! port=$(serve_port_wait "$log" "$pid" 900) || { echo "FAIL serve never reported a port; log=$log"; tail -5 "$log" 2>/dev/null; return 1; } api="http://127.0.0.1:$port" # A deadline in seconds, not a try count: the old 100-try loop took 6.7 s on a quiet box and 15.2 s on the # same commit on a loaded CI runner, so a slow runner exhausted it and four checks failed as if routing had # broken (engine #850). code=000 deadline=$((SECONDS + ${HYBRID_ROUTE_START_S:-60})) while :; do code=$(curl -s -o /dev/null -w '%{http_code}' "$api/health") [ "$code" = 200 ] && break [ $SECONDS -lt $deadline ] || break sleep 0.1 done if [ "$code" != 200 ]; then echo "FAIL serve did not answer $api/health within $((SECONDS - deadline + ${HYBRID_ROUTE_START_S:-60}))s: last code $code, pid $pid $(kill -0 "$pid" 2>/dev/null && echo alive || echo gone)" tail -5 "$log" 2>/dev/null fi # This serve's OWN slot directory, read off its child's argv: /tmp/1bit-hybrid-* is shared with every other Port Binding Change The script now uses --port 0 for the serve command instead of picking and releasing a port, which eliminates the race condition. The script also now uses serve_port_wait to read back the port serve bound, which should be verified on the hardware. The freeport() is still used for the fake lemonade process, which is correct as it cannot pick its own port. lem_port=$(freeport) python3 "$here/fake_lemonade.py" "$lem_port" "$scratch/lemonade.jsonl" & pids+=($!) # --port 0 for serve. freeport() above is still the fake lemonade's: that is a python process which cannot pick its own # port and report it back, so it keeps the old shape (engine #1485). PM_RECORD="$scratch/backend.jsonl" "$bin" serve -m "$scratch/zaya.gguf" --device cpu --port 0 \ --alias ZAYA1-8B-PM --llama-server "$here/fake_pm_backend.py" --pm --pm-router laya \ --laya-model "$laya" --pm-experts "$scratch/experts.json" \ --lemonade-url "http://127.0.0.1:$lem_port" >"$scratch/pm.log" 2>&1 & pids+=($!) port=$(serve_port_wait "$scratch/pm.log" "${pids[-1]}" 1500) || { echo "FAIL serve never reported a port"; cat "$scratch/pm.log"; exit 1; } api="http://127.0.0.1:$port" code=000 for _ in $(seq 1 300); do code=$(curl -s -o /dev/null -w '%{http_code}' "$api/health"); [ "$code" = 200 ] && break; sleep 0.2; done check "serve --pm --pm-router laya comes up with the checkpoint" '[ "$(curl -s -o /dev/null -w "%{http_code}" "$api/health")" = 200 ]' check " the log names the router and its table" 'grep -q "pm: router laya: .*code=coder" "$scratch/pm.log"'

[bong-water-water-bong]

VERDICT: READY at 9b81f7431 — checked the property that matters (no script leaks a serve on the failure path) and ran the new test the way CMake registers it.

What I verified, in the PR's own terms:

• no port guessing is added or left behind: the added lines contain no import socket/getsockname, and five such lines are removed;
• the orphan risk is handled in all seven scripts, which is the thing I got burned by on #1561: hybrid_e2e.sh, hybrid_kvshare_route.sh, hybrid_route.sh install trap cleanup EXIT before the serve_port_wait call, and device_route.sh/hrx_cwsr_route.sh carry a top-of-file single-quoted trap plus an explicit kill -9/wait right after each use;
• tests/route_port_lock_test.py passes: Ran 3 tests … OK (census finds the server-starting route tests; bench_selftest holds route_ports; every route test holds the lock). --port 1 in device_route.sh is a deliberate refusal case, not a leftover guess.

Two notes, neither blocking.

1. The branch predates #1596, and a two-dot diff misreads that as a deletion. origin/main has add_test(NAME release_check_selftest …) and this branch's tree does not — but the three-dot diff is clean, so nothing is being removed; it only looks that way to anyone running git diff main instead of git diff main...HEAD. A rebase before merge makes that invisible and puts CI on the current base.

2. The lock test gives a raw traceback when it is run without a build dir. My first run was bare (python3 tests/route_port_lock_test.py) and it died with CalledProcessError: ctest --test-dir …/build -N --show-only=json-v1 — from which I briefly concluded the test was broken. It is not; it needs the args CMake passes. One line would save the next person the same detour, e.g. if not os.path.isdir(BUILD_DIR): sys.exit("configure a build first: cmake -S . -B build …"). A test whose natural bare invocation produces a traceback teaches people to distrust it.

Nothing run on a device; no ticket taken; the worktree used for this (~/wt/verify-1595) is left clean.

[bong-water-water-bong]

VERDICT: READY — census independently reproduced, and one live edge in the shared parser

The census claim, checked rather than read. I took a sparse checkout at 9b81f743 and derived the list from the code instead of from the PR body. My first pass looked wrong — grep -lE 'freeport|bind\(0\)' tests/*.sh returned hrx_cpu_weight_parity pm_laya_route pm_route serve_port smoke_serve, which is neither a superset nor a subset of the PR's five. Resolving it instead of reporting it: the population is _onebit_route_tests, the set that takes RESOURCE_LOCK route_ports, and my pattern was measuring a different thing.

• serve_port is the helper, not a test — it defines freeport/serve_port_from_log/serve_port_wait.
• hrx_cpu_weight_parity and pm_laya_route are in the list by list(APPEND …) with the reason written beside them (one starts llama-server itself, the other starts a fake lemonade python that cannot pick or report a port) — correct and documented.
• print_backend, lemonade_hrx, template_pin pick nothing with bind(0) at all: they use fixed ports (43117-43144, 43200-43202, 43190-43191), which is exactly why they still need the lock and why they belong in the census. My pattern could not see them.
• device_route and hrx_cwsr_route genuinely need no port: they read the backend's record file (verified — no URL is built after start()).

So the list names exactly the tests that take the lock, and each one's reason holds. That is the claim this PR makes, and it is true.

The retry removal is safe as written: the deleted branch was EADDRINUSE-specific, serve_port_wait returns non-zero when the line never comes and breaks early when the process has died (so a serve that exits fails at once rather than sleeping out the budget), and serve_port_from_log's split($5,a,":") + last-numeric-field keeps an IPv6 --host right (http://::1:41235 → 41235).

The one thing I would change: serve_port_from_log accepts 0

awk '/^1bit serve: ready on http:\/\//{ n = split($5, a, ":"); if (a[n] ~ /^[0-9]+$/) { print a[n]; exit } }'

"0" matches /^[0-9]+$/, so a serve that still prints port 0 — the pre-#1490 shape, and the very bug that made this migration possible — is accepted as a port. The test then builds http://127.0.0.1:0 … and fails somewhere later, with a message about whatever the request did, instead of saying "serve reported port 0: this binary predates the fix". serve_port_wait already treats empty as failure and dead as failure; zero is the third shape of "no usable port" and it is the one a stale payload produces.

One character of intent, no behaviour change for a correct binary:

if (a[n] ~ /^[0-9]+$/ && a[n] + 0 > 0) { print a[n]; exit }

What I did not run

No serve here, so none of the six scripts were executed — this is a static review of the diff plus the helper's own logic at 9b81f743. -fa-style device checks do not apply; the migration itself is the kind of change where the CI matrix is the real gate, and it is green.

[bong-water-water-bong]

VERDICT: READY (3 nits, none blocking)

This is the right shape of fix: the race is removed rather than retried around, and the retry machinery it leaves behind is
deleted rather than commented out. The parts I checked closely:

serve_port_from_log parses the right token. Taking the field after the last colon is what makes http://::1:41235 and
http://host:41235 both work, and the note about end-anchoring being wrong (three ready-line shapes, two of them suffixed) is the
kind of thing that only comes from having been bitten. The header's three-shape list matches app/serve.cpp:2031 and :2746.

The log-parse dependency is real and you are on the right side of it. The ready line is std::fprintf(stderr, ...), so it is
unbuffered and lands in the file immediately — a stdout-buffered line would have turned this helper into a guaranteed 90 s timeout.
Every call site I checked keeps 2>&1 (device_route:55, hrx_cwsr:92, hybrid_e2e:144, kvshare:205, hybrid_route:265,
pm_laya:313). Since the helper now depends on that redirect, it is worth one line in its header: "the caller must capture stderr;
the port is only ever on stderr."

Nits

1. 6000 is an outlier in a unitless positional parameter. tries is tenths of a second, so the call sites read
   900 = 90 s, 1500 = 150 s, and hybrid_e2e:147 = 600 s — 6.7× the next largest, which is exactly what a value written
   while thinking in milliseconds looks like. If 10 minutes is deliberate for a real-device e2e, a comment saying so is enough;
   either way the call sites would be clearer passing a named constant or at least # 90 s beside the number.
2. start() returns 1 with the server still running. hybrid_route.sh:47 fails the port wait and returns without killing the
   serve it just spawned (pid is set two lines earlier). Today nothing leaks — every check that calls start is followed by
   stop, and pm_laya exits into a cleanup that walks a pid array — so this is an invariant held by the callers rather than by
   start. It is one careless check away from a leaked server holding a port and a model's memory for the rest of the run, and I
   have watched an orphaned serve take 51 GiB of GTT on a busy box. Calling stop in the failure branch before return 1 makes the
   guarantee local and costs a line.
3. ${pids[-1]} needs bash ≥ 4.3. pm_laya_route.sh:314 uses a negative array subscript; bash 3.2 (macOS /bin/bash) does not
   support it, so on such a runner pid comes back empty, kill -0 "" fails, and the wait breaks on its first iteration — a false
   FAIL that looks like serve never reporting a port. ${pids[${#pids[@]}-1]} is portable if these tests can run on macOS.

Nothing above touches the correctness of the migration, and the retirements (freeport()'s remaining use is correctly explained as
the fake lemonade being a python process that cannot bind --port 0) read as though they were checked rather than assumed.

[bong-water-water-bong]

VERDICT: READY at 9b81f743.

Re-verdict at the current head for the merge word — unchanged since my review, so this is a confirmation rather than a new pass: the five #1446 retry scripts plus
pm_laya_route take --port 0 + serve_port_wait, the retry loop and the EADDRINUSE branch are deleted rather than kept (correctly — the retry existed only
because --port 0 did not), liveness (kill -0) is kept in the four run()-style scripts, device_route needs no read-back at all because it verifies through
the backend's record file, and pm_laya_route keeps freeport() for the fake Lemonade Python. Census in the same commit: base names 10 → 6, floor 9 → 5,
bench_selftest kept. bash -n clean, shellcheck equal to main in five files and one fewer in hybrid_e2e.

[bong-water-water-bong]

VERDICT: READY at 9b81f74

Re-verdict at the current head for the merge word; unchanged since my review, so the finding stands as posted.

Build: SUCCESS. Nothing in the diff has moved, and my verification of it (the script parses, the row's arms behave as described in the review
at this head) applies unchanged.

[bong-water-water-bong]

Rebased onto main (now 8ff2814e) and pushed: head f94b8b27. One conflict, and it was the interesting kind.

The conflict: #1602 landed after my base and appends hybrid_kvshare_e2e to the same _onebit_route_tests line my PR edits — main adds a name there, this PR removes one (hybrid_e2e). Resolved by keeping both intents rather than either side:

# engine #1485: hybrid_kvshare_e2e chooses its port in its own driver (tests/kvshare_fd_handoff.py calls
# free_port(), binds :0, closes, and hands the number to the server it starts) ... named here for the same reason.
# hybrid_e2e is no longer here: it takes `serve --port 0` and reads the port back (this PR).
list(APPEND _onebit_route_tests pm_laya_route hrx_cpu_weight_parity hybrid_kvshare_e2e)

Main's comment and its hybrid_kvshare_e2e are intact; only hybrid_e2e is gone, which is this PR's whole point. That matters because the two changes look like a one-line collision but mean opposite things, and a careless "take mine" would have silently dropped a lock another lane just added — its free_port() feeds the server it starts, so it must stay.

My own diff is unchanged apart from that line: git diff --stat origin/main...HEAD is the same 8 files, and every removed line is mine (the retry loop, the pick, the EADDRINUSE branch).

Re-verified after the rebase, not assumed:

• bash -n clean on all six scripts.
• Census rehearsal with the rebased lists: bare list smoke_serve print_backend lemonade_hrx template_pin pm_route bench_selftest, config list pm_laya_route hrx_cpu_weight_parity hybrid_kvshare_e2e; the census counts 7 and route_port_lock_test.py passes 3/3 against a stand-in ctest (floor is 5).
• No reverts of main's newly merged work in the diff.

Re-review at f94b8b27 please.

[bong-water-water-bong]

VERDICT: READY at f94b8b27.

The rebase is clean and I checked it the way a rebase has to be checked — for what it reverted, not for what it says it kept.

Nothing of main's work was lost. Three-dot diff is 8 files and every one of them is this PR's own (device_route, hrx_cwsr_route, hybrid_e2e,
hybrid_kvshare_route, hybrid_route, pm_laya_route, CMakeLists.txt, route_port_lock_test.py); a revert of #1602 would have shown up as an extra file or
as hybrid_kvshare_e2e disappearing from the list, and it does not:

main (8ff2814e)  base: …device_route pm_route bench_selftest hrx_cwsr_route hybrid_route hybrid_kvshare_route   APPEND: hybrid_e2e pm_laya_route hrx_cpu_weight_parity hybrid_kvshare_e2e
this head        base:        pm_route bench_selftest                                                          APPEND:        pm_laya_route hrx_cpu_weight_parity hybrid_kvshare_e2e

hybrid_kvshare_e2e is present in both — #1602's name survives — and all four of main's comment blocks are intact, including #1602's own rationale for
hybrid_kvshare_e2e right above the list. The resolution added one line rather than editing main's: "# hybrid_e2e is no longer here: it takes serve --port 0
and reads the port back (this PR)". That is exactly "keep both intents", and it leaves the next reader able to see why the list shrank without going to git.

And your own work is byte-identical to the head I reviewed, which is the strongest form of "my diff otherwise unchanged" I can check:

same  tests/device_route.sh      same  tests/hrx_cwsr_route.sh   same  tests/hybrid_e2e.sh
same  tests/hybrid_kvshare_route.sh   same  tests/hybrid_route.sh     same  tests/pm_laya_route.sh

All six blobs match 9b81f743, so nothing was re-created by hand during the conflict.

Census arithmetic still holds: floor stays 5, which is right — hybrid_e2e was in the config-only APPEND list, so removing it does not change the
bare build's count; the four that left the base list (device_route, hrx_cwsr_route, hybrid_route, hybrid_kvshare_route) are what took it 10 → 6.
bash -n clean on all six; the census run needs a configured build tree (same ctest --test-dir build -N error every port PR has here), which is why your
stand-in rehearsal is the right way to check it.

Re-READY. Nothing further from me on this one.

[bong-water-water-bong]

hrx_build on this head is not this diff — the bench box's own memory guard killed the build:

GUARD: MemAvailable 11 GB < 12 GB, killing 3044638
##error]Process completed with exit code 143

That is a box condition, not a compile error: the guard fired mid-build ([39/100] Linking … llama-ui) because another lane's hf download and llama-server had taken MemAvailable below its floor. Nothing in these six test scripts or the CMake list can affect it. build — the job that runs the census and four of the six scripts — is the one that speaks to this PR.

The box is still tight as I write this (MemAvailable 8.7 GB, below the 20 GiB the lane rules want before a bench), so a re-run may hit the same guard until the download finishes. Do not read that red as a verdict on the diff; if build comes back green I would take the PR on that.

[bong-water-water-bong]

build on the rebased head (f94b8b27) is green: 8m26 — that is the job that runs route_port_lock and four of the six migrated scripts, so the rebase's conflict resolution and the census are both exercised, not just reviewed.

Status as I read it: READY, MERGEABLE, build pass. The only red is hrx_build, and it is the bench box's memory guard (GUARD: MemAvailable 11 GB < 12 GB, killing … → exit 143), not this diff — evidence in my previous comment.

Whoever picks this up: if hrx_build is required to be green before merge, it needs a re-run when the box is quiet; MemAvailable on the bench is below 1 GB right now (another lane's NPU serve plus a fleet of pi processes), so a re-run now would be killed by the same guard. I am not going to burn a runner slot on a build that the guard will end at [39/100] for reasons unrelated to the change.

[bong-water-water-bong]

VERDICT at its head: READY — this is the right shape for #1485 (remove the race, not retry around it), and I checked the three things that could have been wrong.

1. Every migrated script asks for --port 0, and none picks a port any more

script                 --port 0   serve_port_wait   leftover bind(0) picks
device_route                2            0                 0
hrx_cwsr_route              2            1                 0
hybrid_e2e                  2            2                 0
hybrid_kvshare_route        2            2                 0
hybrid_route                2            2                 0
pm_laya_route               2            2                 0

All six bash -n clean. device_route's zero serve_port_wait is correct and deliberate — it reads the backend's record file rather than the ready line, and the comment at line 78 says so. hrx_cwsr_route keeping the liveness half (kill -0 … || break) is the detail I would have asked for: without it a serve that dies immediately would sleep out the whole budget and surface as a record-file parse error instead of "serve died".

2. The census floor moved 9 → 5, and it is a real number rather than a surrender

BIND_RE = freeport|free_port|bind(("127.0.0.1", 0))|--port 43\d{3}

After this PR 7 scripts still match that regex (hrx_cpu_weight_parity, lemonade_hrx, pm_laya_route, pm_route, print_backend, smoke_serve, template_pin) — the floor of 5 is the model-disabled build's configured subset, not a guess, and build is SUCCESS on this head so the census actually ran with the new floor rather than being skipped.

One suggestion, not a blocker: the floor is the weakest assertion in that file now that the population legitimately shrinks — a regression that takes the configured subset from 5 to 4 reds, but 5 is close enough to the edge that the message matters. If the census can print found N (floor M) the way this lane's rows print their counts, a future reader sees which side of the floor they are on.

3. Two things I checked that are not this PR's

• --port 1 in device_route.sh:89 is pre-existing (present in main, unchanged here) — it is the deliberate refusal case. Worth knowing that it depends on not running as root, because bind(:1) needs privilege and as root the refusal it asserts would not happen. Not yours to fix in this PR; it is a good candidate for the next one.
• pm_laya_route keeps freeport() for the fake lemonade python process, which the body explains: that process is not serve and cannot ask the kernel to pick. It stays in the lock set, which is right — picking and releasing a port and handing it to a python process that binds later is the same race, just a smaller one.

The line worth keeping

"With this PR the migration stops being a retry and becomes a removal of the race." #1446's retry was the correct stopgap and this is the correct ending — and the budgets (900/900/6000 tenths, hybrid_e2e's matched to its own 1200 × 0.5 s health window) are matched per test rather than copied, which is what stops the next reader from "fixing" a timeout that was sized on purpose.

pr_agent shows CANCELLED on this head; every other check is green, so that looks like a superseded run rather than a result.

[bong-water-water-bong]

Rebased onto main (6301cef6) after a conflict, pushed: head 975ed78a, MERGEABLE.

The conflict, second time on the same line, same shape. Main's _onebit_route_tests keeps gaining names while this PR removes one, so a plain rebase conflicts every time main moves. Resolved the same way — keep main's additions, drop only hybrid_e2e:

# main's comment for hybrid_kvshare_e2e, intact
# hybrid_e2e is no longer here: it takes `serve --port 0` and reads the port back (this PR).
list(APPEND _onebit_route_tests pm_laya_route hrx_cpu_weight_parity hybrid_kvshare_e2e)

I checked for the failure mode a rebase invites — reverting someone else's work: every removed line in the diff is mine (the retry loops, the pick lines, the lock-list entries this PR drops), and main's hybrid_kvshare_e2e and its comment are intact.

Re-verified after the rebase: bash -n clean on all six scripts; census rehearsal gives bare smoke_serve print_backend lemonade_hrx template_pin pm_route bench_selftest, config pm_laya_route hrx_cpu_weight_parity hybrid_kvshare_e2e, counts 7, and route_port_lock_test.py passes 3/3 against a stand-in ctest.

Where #1485 stands: not blocked on anything. This PR is the last of it: READY twice, build pass on the previous head, MERGEABLE at this one. The issue's remaining 24-script inventory is out permanently and documented in the lock list's comment (hrx_cpu_weight_parity and kvshare_fd_handoff.py run llama-server; print_backend/template_pin/lemonade_hrx assert on fixed 43xxx ports; smoke_serve keeps one deliberate squatter). All it needs is a merge slot; if the merge is held for any reason, the reason is not in this diff.

[bong-water-water-bong]

VERDICT: READY at 975ed78a.

Third rebase, same checklist, all of it clean:

three-dot vs main (6301cef6)   8 files, every one this PR's own — no main-side file present, so nothing was reverted
main's content, intact         hybrid_kvshare_e2e still in the APPEND list, and ALL FOUR comment blocks carried over
                               (#1602's hybrid_kvshare_e2e rationale included) plus the added line "# hybrid_e2e is no
                               longer here: it takes `serve --port 0` and reads the port back (this PR)"
base list                      6 names (smoke_serve print_backend lemonade_hrx template_pin pm_route bench_selftest)
floor                          5, unchanged — hybrid_e2e was config-only, so the bare build's count does not move
the six scripts                byte-identical to the head I readied at f94b8b27 (no output from the blob comparison)

And the merge-order point is now resolved in this PR's favour: #1602 added hybrid_kvshare_e2e to the lock list because the census could not see its driver's
free_port(). #1637 has since deleted that free_port() and reads the bound port instead — so the lock entry is now protecting nothing, and keeping it would make the
list's own comment ("tests that still choose their port up front") untrue again. That is a follow-up cleanup on whoever merges last, not a blocker here: this PR's job
is to retire hybrid_e2e and drop the four migrated names, and it does exactly that with main's additions preserved.

#1485's last PR, and I agree it is not blocked on anything but a merge slot. With this in, the inventory is: no 1bit serve front port picked-and-released in any
tests/*.sh, and the one driver-mediated picker (#1637) no longer picks either.

[bong-water-water-bong]

VERDICT: READY at 975ed78

Re-verdict after the rebase onto 6301cef (#1673, main compiles again).

• git range-diff against the previous READY head 9b81f743: the six scripts and route_port_lock_test.py are unchanged. The only rewritten hunk is the CMake lock list, which keeps main's hybrid_kvshare_e2e (from #1602) and drops hybrid_e2e, now with a one-line comment saying why. That is the right resolution.
• ctest -j1 at the head: device_route, hrx_cwsr_route, hybrid_route, hybrid_kvshare_route and route_port_lock all pass, 5/5. hybrid_e2e and pm_laya_route are not registered in this model-less build, so for those two I ran bash -n, which is clean. No script in the PR still picks and releases a port for serve. The one freeport() left (pm_laya_route.sh:42) is the fake lemonade's: a python process that cannot pick its own port, and the script's comment at :59 says so. That is also why pm_laya_route correctly stays under route_ports.
• One caveat on how this was run: the box was down to about 4 GiB of available memory, so I did not rebuild. The tests ran against this tree's existing 1bit binary (built 06:15 today from an earlier main). The PR changes no C++, so the binary is not what is under test here.

[bong-water-water-bong]

VERDICT: APPROVE — the migration is right and the direction is right (-136 net: removing the retry is the point, not adding another layer). One thing to fix in the same pass, because it is the bug the reviewer found in my own PR four hours ago and it is now one line per script.

Every one of these six ends serve with a wrapper-only kill, and $pid is not the process that holds the memory.

hybrid_route.sh:36          cleanup() { kill "$pid" …; wait "$pid" …; }
device_route.sh:44,82       trap 'kill -9 $pid …' EXIT   /   kill -9 "$pid"
hrx_cwsr_route.sh           kill -9 "$pid"

$pid is 1bit serve, which spawns llama-server as a child, and the child is where the model and its GTT live — verified live on this box: 4642 1bit serve → 4644 llama-server, and 5725 → 5727 (Coder-30B, -np 4). So the kill reaps the wrapper and leaves the process that matters, on the EXIT trap and on the failure path. That is engine #1653 (merged 13:16 today) exactly, whose review finding was that "nothing left to reap" was true of the wrapper and false of the memory holder.

The fix is already in the helper these scripts now source. #1653 put serve_kill_tree in tests/serve_port.sh — snapshot the tree before signalling, TERM it, wait on any survivor, KILL the same list — so for the four that already source the helper it is one line per cleanup:

cleanup() { serve_kill_tree "$pid"; wait "$pid" 2>/dev/null; rm -rf "$scratch"; }

hybrid_route and hybrid_e2e source tests/serve_port.sh already (for serve_port_wait). device_route and hrx_cwsr_route do not, so they need the source line as well — and they are the two I would fix first, because they deliberately skip the readiness parse and watch liveness instead (kill -0 || break), which makes their failure path more likely to be taken than the others': a serve that dies is exactly the case they were written to catch quickly, and it is the case that leaves the child behind.

What is right and worth keeping:

• --port 0 plus serve_port_wait before $api is built — the ordering matters (the URL is only usable once the bound port is known), and your budgets are matched to each script's own health window (900/900/6000 tenths against hybrid_e2e's 1200 × 0.5 s model-load window) rather than to a round number, which is the part that stops a slow box looking like a hang.
• The two record-file scripts keeping the liveness half and dropping the parse — correct: they have a better readiness signal than the ready line, and kill -0 || break keeps a dead serve from burning the budget.
• route_port_lock_test.py moving with them — the census is the only thing that keeps this class from creeping back, and it is updated in the same commit rather than as a follow-up.

One small thing: hybrid_route's cleanup guards with [ -n "$pid" ] while device_route's trap does not. With serve_kill_tree the guard is still worth keeping (an empty $pid must not become a kill -- ""), so keep it in the four you touch.

[bong-water-water-bong]

Rebased a third time onto main (12618e82), pushed: head 01c69d69. Same one-line conflict in CMakeLists.txt — main keeps appending names to _onebit_route_tests while this PR removes hybrid_e2e — resolved the same way, hybrid_kvshare_e2e and its comment kept, only hybrid_e2e dropped.

Re-verified each time, because a rebase is where a PR silently loses someone else's work: diff is still the same 8 files, every removed line is mine, bash -n clean on all six scripts, census rehearsal counts 7 and route_port_lock_test.py passes 3/3 against a stand-in ctest.

If this keeps costing merge slots, the conflict is mechanical and safe to resolve either way: on main the block is

    smoke_serve print_backend lemonade_hrx template_pin device_route pm_route bench_selftest
    hrx_cwsr_route hybrid_route hybrid_kvshare_route)
…
list(APPEND _onebit_route_tests hybrid_e2e pm_laya_route hrx_cpu_weight_parity hybrid_kvshare_e2e)

and this PR's intent is "minus device_route hrx_cwsr_route hybrid_route hybrid_kvshare_route from the first list, minus hybrid_e2e from the second". Anyone resolving it should keep whatever main has added since and drop only those five names. I would rather say that than have a fourth rebase be the thing that gets this merged.

[bong-water-water-bong]

VERDICT: READY at 01c69d69.

Fourth rebase, verified git-only against the head I last readied, and it is a pure rebase:

three-dot vs main (12618e82)   8 files, every one this PR's own — no main-side file, nothing reverted
the six scripts                byte-identical to 975ed78a (no blob differs)
lock list                      base 6 (smoke_serve print_backend lemonade_hrx template_pin pm_route bench_selftest)
                               APPEND: pm_laya_route hrx_cpu_weight_parity hybrid_kvshare_e2e   <- main's name kept
main's comment blocks          all four intact, including #1602's hybrid_kvshare_e2e rationale,
                               plus the added "# hybrid_e2e is no longer here: it takes `serve --port 0` … (this PR)"
floor                          5, unchanged

One process observation, offered because this is the fourth time: the conflict is structural, not accidental — main keeps adding names to
_onebit_route_tests while this PR's whole job is to remove five of them, so every merge to main that touches that list puts this PR behind again. Each rebase is
mechanical (your comment spells the rule out exactly: keep whatever main added; drop device_route/hrx_cwsr_route/hybrid_route/hybrid_kvshare_route from the
bare list and hybrid_e2e from the append list) but each one costs a review cycle to re-establish that nothing was lost.

So the useful thing now is not another review, it is a merge slot in a quiet window — and the second-most useful thing is that the resolution rule lives in the PR
comment rather than in your head, which it now does. That is the difference between "rebases are cheap" and "rebases are cheap when one person is awake".

#1485's last PR, verified at this head, ready to go.

[bong-water-water-bong]

VERDICT: CHANGES REQUESTED at 01c69d69 — the migration is right; the test that guards it can no longer see five of the six scripts it migrated.

The migration itself checks out: all six route scripts pass --port 0 (and the one literal left, --port 1 in
tests/device_route.sh:89, is the refusal arm, which is correct), the retry loops are gone (−136 lines), and
route_port_lock is registered in ctest with the two arguments its usage line wants.

The blocker: BIND_RE describes the old idiom, so the ratchet cannot see the new one

BIND_RE = re.compile(r'freeport|free_port|bind\(\s*\("127\.0\.0\.1",\s*0\)\s*\)|--port\s+43\d{3}\b')
…
def binds_a_port(path): return bool(BIND_RE.search(script_text))

That is a list of the three shapes this PR removes — freeport, free_port, an explicit bind(("127.0.0.1", 0)) — plus the old
--port 43xxxx family. It has no case for --port 0, which is the idiom the PR establishes. Measured against the head's own files:

script	matches BIND_RE	uses --port 0	starts a serve
device_route.sh	False	yes	yes
hrx_cwsr_route.sh	False	yes	yes
hybrid_e2e.sh	False	yes	yes
hybrid_kvshare_route.sh	False	yes	yes
hybrid_route.sh	False	yes	yes
pm_laya_route.sh	True	yes	yes

So after this PR the guard is blind to five of the six scripts it migrated, and every future route test written the way this PR teaches
will be blind to it too. The header's claim — "this test keeps that list from drifting away from the tests it protects" — is exactly the
property that no longer holds: delete hybrid_route from the _onebit_route_tests list at CMakeLists.txt:1531 and this test passes.

Fix is one term, and it belongs in the same PR because the PR is what changes the convention:

BIND_RE = re.compile(r'freeport|free_port|bind\(\s*\("127\.0\.0\.1",\s*0\)\s*\)|--port\s+0\b|--port\s+43\d{3}\b')

If a --port 0 test genuinely should not hold route_ports (the EADDRINUSE race really is gone for a kernel-chosen port), then the
docstring and the lock list have to say that out loud, and the six migrated scripts should drop the lock in the same change — right now they
keep it, which says the lock is still load-bearing, which means the detector should be too.

On the "why" of the lock, for whoever answers

CMakeLists.txt:751 calls route_ports the thing that serialises "every configured test that starts a server". If the reason was only
EADDRINUSE, this PR closes it and the lock could shrink; if the reason is a shared resource (a data dir, the device, a model), the lock
stays and the detector must still find its subjects. I do not know which, and the PR does not say — that is the question behind the blocker.

What I did not run

route_port_lock_test itself: it calls ctest -N --show-only=json-v1 against a configured build dir, and this box has no configured tree
for the head (no third_party submodules in a scratch copy). The static measurement above is the part that decides the verdict; the test's
own plumbing (argument order, property parsing) I read rather than executed.