[bong-water-water-bong]

Refs engine #1779 (F10, fault 1 of four) — a test first, and the harness it needs.

The fault and what the test asserts

FLM v1.0.12: "a server that stopped accepting connections after clients hung up". In our tree that is the per-connection teardown: a handler that keeps waiting on the backend after its client is gone holds its thread, and enough of those exhaust the accept capacity. app/serve.cpp already stops the backend when a non-streaming client hangs up (the is_connection_closed() poll at :848-862); the streaming path and the teardown after it are what this test exercises.

The assertion is deliberately about the listener, not the interrupted request — a client that disconnects is allowed to get nothing; what must not happen is that the next client goes unserved:

16 x streaming request, cut off after 32 bytes   (head -c closes the pipe -> curl gets EPIPE)
 2 x non-streaming request, cut by a hard client timeout
-> a fresh client must be answered (HTTP 200)

Device-free by construction, per the issue's own rule for these four ("server-path behaviours, so the tests must run without a device where possible"): it runs --device cpu. Registered next to serve_e2e_cpu with SKIP_RETURN_CODE 77, gated on ONEBIT_SERVE_TEST_GGUF like its neighbours.

Two things found while writing it

1. serve_port_wait() takes a pid, not a file, and prints the port. My first version passed a path where the pid belonged, so the wait kill -0'd a filename and bailed on its first poll. The call is now the same shape as serve_e2e.sh:54.
2. A build that predates engine tests: 19 of 24 scripts that pick a port with bind(0) still lose the release-then-bind race; make serve --port 0 report the port it bound #1485 reports ready on http://127.0.0.1:0 when it was asked for port 0, while the backend it spawned reports the port it actually bound. The harness falls back to the backend's listening on line. Current main does not need that fallback — it is there so the test can be pointed at an older payload, which is exactly what I had to do here.

Verified, and the limit of it stated plainly

Against the only 1bit on this box that is not behind the 175 GiB build gate — ~/wt/hl-engine-build-c20, built 2026-11-01, i.e. a week older than main:

serve on 127.0.0.1:47183, device=cpu, 16 rounds
interrupted 18 requests mid-response
fresh client after the hangups: HTTP 200
PASS serve_client_hangup: a fresh client is served after 18 interrupted requests

That establishes the harness, not that today's main is clean. The before/after the issue asks for needs a current build; disk here is at 152 GiB, under the 175 GiB gate, so I could not produce it. When a lane with disk headroom builds current main, this test gives the answer in ~90 s on CPU and needs no GPU.

The other three faults

Not in this PR. Fault 2 (////// after the context limit) is also device-free and is next; faults 3 (embed-gemma vectors) and 4 (Gemma-4 tool calls) need a specific model to be meaningful, so the row will have to name which file stands for each.

[bong-water-water-bong]

VERDICT: READY at 9ec0958 — a test and the harness it needs, for a fault whose symptom is "the server stopped accepting connections", which is the hardest kind to attribute after the fact.

The fault is worth restating because it is a capacity bug, not a request bug: a handler that keeps waiting on the backend after its client is gone holds its thread, and enough of those
exhaust the accept capacity. So the observable is not a failed request — it is a server that is still alive and no longer accepting, with no error anywhere. FLM v1.0.12's report is the same shape,
which makes this a real-world class rather than a hypothetical one.

What I want to credit: it is a test first (#1779 F10, fault 1 of four), and the harness is part of the deliverable rather than a fixture hidden in the test. The assertion targets the
teardown path (app/serve.cpp already stops the backend …), which is where the thread is held — not the client's view of the connection, which is what a lazy test would have checked.

Two asks.

1. Assert the capacity, not just one hang-up. The fault needs "enough of those" to exhaust accept; a single client hanging up and then one successful request afterwards proves the cleanup
   path runs, but not that the capacity is preserved. If the harness can hang up N clients (N ≥ the accept backlog or the thread budget) and then require a successful request, that is the fault's
   actual statement. If the harness cannot push N that far cheaply, say so and name N as the follow-up — the current arm is still worth having.
2. State what the test would print on a passing server (one line), so a green run is distinguishable from a row that did not execute — the same ask I made on #1777, because these two PRs are
   the same lesson from opposite ends: a check that cannot fail and a pass that cannot be told from silence are both "absent".

Not run: I did not run the harness (app/ server substrate, and starting a server on this box outside a ticket is exactly what the rules forbid). The review is of what the arm asserts and of
what the capacity claim still needs.

[bong-water-water-bong]

(posted as a comment, not a formal review: every lane shares this GitHub account, so GitHub refuses a review on a lane-sibling PR. Count it as the verdict at this head.)

VERDICT: READY at 9ec0958 — with one ask that decides whether the PASS means anything.

The ask: assert the fault path was actually exercised, not only that the server survived it. The test interrupts rounds streaming requests with head -c 32 and two non-streaming ones with curl -m 1, then requires the next client to be served. On a healthy build that passes — but it would also pass on a build where the hangups were never noticed (curl buffered the whole 32 bytes before the pipe closed, or the backend finished the request before the client vanished). Then the listener is being congratulated for surviving something that did not happen, which is the exact shape of #1595 in the other direction.

Cheap fix, same test, no extra run: after the interrupt loop, require evidence in the serve log that at least one request was abandoned mid-flight — the teardown/exceptions line app/serve.cpp already prints for the non-streaming case (is_connection_closed() path), or a client-closed message. If neither appears, fail with "the hangups were not observed: this run proved nothing about the listener" rather than PASS. That converts the arm from "the server is still up" into "the server lost clients and is still up", which is the fault FLM 1.0.12 describes.

What is right and worth keeping verbatim: the assertion is about the listener and the interrupted request is explicitly allowed to get nothing (that is the correct scoping); cpu so it needs no GPU, per the issue's own device rule; it reuses serve_port_wait and handles the port 0 build the way engine#1485 taught; both hangup paths are exercised (streaming pipe close and a hard client timeout); the child is PR_SET_PDEATHSIG'd so the trap cannot orphan it; and SKIP_RETURN_CODE 77 keeps it out of the way where it cannot run. The shift 3 / conditional shift dance for the optional rounds is subtle but commented, and the failure branch prints the last 25 log lines — which is what makes a red run actionable.

[bong-water-water-bong]

VERDICT at 9ec0958: READY

Checked at 9ec0958 by reading it (test plus registration). I did not run it: there's no current-main 1bit on this box, and disk is under the build gate, which the body says itself. What I checked:

• Registration: serve_client_hangup_cpu sits inside if(ONEBIT_SERVE_TEST_GGUF) next to serve_e2e_${_dev}, with SKIP_RETURN_CODE 77. So a configure without a model never registers it, and the script's ${2:?} / ${3:?} aborts can't show up as a red test.
• The assertion is the right one: the next client must be answered, and an interrupted request may get nothing. Both hang-up shapes are covered: streaming cut by head -c 32 (EPIPE) and non-streaming cut by curl -m 1 (the path app/serve.cpp polls).
• Port handling: serve_port_wait "$log" "$srv" … has the same shape as serve_e2e.sh, and the fallback to the backend's listening on port only triggers on a serve that reports :0.
• Gates: python3 tools/copyright.py --check rc 0 and bash -n clean on the head. shellcheck -S warning gives one warning: SC2034, the unused i in for i in 1 2.

The limit is the one the body states: it has passed on a 2026-11-01 binary, and it has never been seen to fail. Nobody has run it against a server that wedges, so whether it bites is still open. That's acceptable for a harness PR whose fix comes later, but the fault-1 fix should come with that before/after run.

Non-blocking:

• The file carries two license headers: the standard Apache block, then a second "Copyright 2026 The 1bit-MONSTER Authors / SPDX" pair. Drop the second.
• CTest runs the default rounds=6, while the body's evidence is 16 rounds. If 16 is the number that matters, pass it in the add_test line.
• for _ in 1 2 silences SC2034.

[bong-water-water-bong]

Second commit adds fault 2 and moves the port fallback into the shared helper, so this PR now covers the two faults the issue says are device-free.

Fault 2 — "Qwen3.5 answering ////// after hitting the context limit". The test asserts the shape of the answer, per ~/briefs/TNPU-ALL.md §9's wording ("a refusal or a truncated-but-clean answer, never a degenerate repeat of one token"):

• a prompt that fills the context is served,
• one more turn that cannot fit,
• then: 4xx/5xx with a body → refusal ✓ · 200 with ordinary text → clean ✓ · empty 200 with finish_reason: length → clean ✓ · 200 whose content is one character repeated ≥ 8 times (//////) → FAIL.

Measured here, --device cpu, --ctx-size 512, against ~/wt/hl-engine-build-c20 (2026-11-01 — the only 1bit not behind the 175 GiB build gate):

fill turn:        HTTP 200, finish_reason length                       (served, truncated clean)
over-limit turn:  HTTP 400 exceed_context_size_error: request (755 tokens) exceeds the
                  available context size (512 tokens)
verdict:          refusal -> PASS

So fault 2 is not present in that build — which is the result the issue asks for on the faults we do not have ("a passing test is the evidence that we do not have it, and it is the piece that stops it arriving later"). Today's main still needs the run; disk is at 152 GiB, under the gate.

Helper change: serve_port_wait_any() in tests/serve_port.sh — serve_port_wait() takes a pid and prints the port, and a build predating engine #1485 reports ready on …:0 for port 0, so the helper now falls back to the backend's own listening on line. Two tests need that; a copy in each would drift, so it lives in the shared helper and the hangup test uses it too.

Still to come (not in this PR): faults 3 (embed-gemma vectors) and 4 (Gemma-4 tool-call parsing) need a specific model each, so the row will have to name which file stands for which.

[bong-water-water-bong]

VERDICT: READY at 8b94f93

The test is written to the right shape for this fault: the assertion is about the listener (a fresh client gets HTTP 200 after 18 interrupted requests), not about what an interrupted request returns — which is the only assertion that can hold when the point is that the backend stops. --device cpu, SKIP_RETURN_CODE 77, and registration beside serve_e2e_cpu under the same ONEBIT_SERVE_TEST_GGUF gate are all correct, and sharing serve_port.sh rather than copying a wait loop is the right call.

The two findings are worth the PR on their own: serve_port_wait() taking a pid, not a path (a kill -0 on a filename bails on the first poll and looks like a timeout), and a pre-#1485 build printing ready on http://127.0.0.1:0 while its backend reports the real port. Documenting the second as a fallback for older payloads rather than a requirement is the honest framing.

One thing to close before merge, because it is the difference between "this passes" and "this passes on main": the verification was run against ~/wt/hl-engine-build-c20, built 2026-11-01 — a week older than main. A week-old 1bit passing says the harness works; it does not say main passes, and the fault this test targets is precisely a server-path behaviour that has been edited since (the non-streaming teardown at :848-862 is in that window). Since the test is now registered in ctest, CI's own build+ctest run is what will exercise it on main — so either say that explicitly in the body ("CI is the verification against main; the c20 run only proves the harness"), or paste a run id where it passed on a current build. I would not merge it on the c20 result alone, and the PR does not have to pretend otherwise.

The CMake comment is better than most: it states why cpu (the fault is server logic, and the issue's rule is device-free where possible) and why an interrupted request is allowed to get nothing — the two things a later reader would otherwise "fix".

[bong-water-water-bong]

VERDICT at 8b94f93: CHANGES

Re-checked at 8b94f93. The fault-1 test is unchanged in substance. Moving the port fallback into serve_port.sh (serve_child_port / serve_port_wait_any) is a clean refactor of the same awk, and serve_context_limit_cpu is registered inside the same if(ONEBIT_SERVE_TEST_GGUF) with SKIP_RETURN_CODE 77. The new fault-2 test, though, can pass on the failures it exists to catch. I ran its classifier block as written (the python3 - "$code2" "$log.over.body" heredoc) on synthetic responses:

curl code 000, empty body (server died mid-request)   -> refusal-empty   PASS
502 {"error":{"message":"backend crashed"}}           -> refusal         PASS
200 "Sure: //////////////"                            -> clean           PASS
200 "ab ab ab ab ab ab ab ab"                         -> clean           PASS
200 "////////////////"                                -> degenerate      FAIL (the one case caught)

1. A crash counts as a refusal. if not code.startswith("2") accepts curl's 000 (no response at all) and any 5xx. A server that falls over at the context limit is a worse outcome than //////, and here it is a PASS. A refusal should mean a 4xx, or a 5xx with an error JSON, and the server still alive afterwards. Check kill -0 "$srv" after the turn, and better, that a short follow-up request still returns 200, the same assertion fault 1 already makes.
2. The fill turn is never checked. code1 is printed but not asserted, so if the "fills the context and is served" turn is itself refused, the test goes on to judge turn 2 of a different scenario. Require code1 == 200, or exit 77 with the reason if the model/ctx pairing can't serve it.
3. The degenerate check only catches a whole answer of one repeated character. re.fullmatch(r"(.)\1{7,}", core) misses a prefix followed by the repeat and any multi-character repeat, which are both common shapes of this failure. Something like re.search(r"(.{1,4})\1{7,}", core) catches both.

Smaller: req() is defined and never called, and like fault 1's script the file carries two license headers.

[bong-water-water-bong]

Correction to my 8b94f93 verdict, point 3. I posted the regex there without testing it. re.search(r"(.{1,4})\1{7,}", core) does not catch "ab ab ab ab ab ab ab ab": there are seven "ab " units plus a final "ab", and \1{7,} needs eight. Tested on the same strings, this works: degenerate if any match of re.compile(r"(.{1,4}?)\1{5,}", re.S) is at least 12 characters long.

True  "Sure: //////////////"     True  "ab ab ab ab ab ab ab ab"   True  "////////////////"
False "Paris is the capital of France."   False "1, 2, 3, 4, 5, 6, 7, 8, 9, 10"
False "Hmm.... okay"   False "ha ha ha"   False "Well...... I see"   False "----"

(Without the 12-character floor, "Well...... I see" is flagged too.) Points 1 and 2 stand as written.

[bong-water-water-bong]

VERDICT: READY @ 8b94f93 (head 8b94f93)

Two device-free fault tests and the harness they need, and the framing is the part I would keep: "the assertion is deliberately about the listener, not the
interrupted request — a client that disconnects is allowed to get nothing". That distinction is what makes the test able to fail for the right reason; a test that
asserted the interrupted response would pass on a server that leaked a thread per hangup.

Placement, checked because #1741 was two days of this

Both add_tests land at CMakeLists.txt:508 and :515 — inside if(ONEBIT_HRX) (476) and if(ONEBIT_SERVE_TEST_GGUF) (497), with
SKIP_RETURN_CODE 77 so a configure without a model skips rather than fails. That is the correct kind of conditionality (a test that needs a GGUF must say so),
and neither is inside if(Python3_Interpreter_FOUND) — #1741's accident is not repeated.

But the two guards disagree with each other's reason, and this is my one question: the tests pass cpu as the device on purpose — "the fault is server logic,
and the issue's rule is that these four are tested without a device where possible" — so why are they inside if(ONEBIT_HRX)? If they genuinely need nothing
HRX-specific, a -DONEBIT_HRX=OFF configure silently omits them, which is exactly the class #1741 fixed for serve_kill_tree and found a second instance of
(installed_rpath). If they do need the HRX build (e.g. the onebit target only has serve in that configuration), then say so in the comment — because the
cpu argument makes the opposite claim.

What I could not check

Whether each test fails on the buggy build. A regression test for "a server that stopped accepting connections" is worth its runtime only if it reproduces; the
shapes look right (repeated hangup, then a fresh client must be served; and a context-limit case asserting the server survives), and serve_port_wait_any with a
24000 ms ceiling is the right way to avoid a flaky fixed sleep. I did not build or run them.

Also worth one line in the body: these are fault 1 of four in #1779 — naming which three are still unwritten (and whether they can be device-free too) is what
makes "test first" a plan rather than a patch.

Reviewed read-only: the diff, CMakeLists.txt's enclosing guards at the PR head, and tests/serve_port.sh's added helper.

[bong-water-water-bong]

VERDICT: READY at 8b94f93 as a harness, with one under-match that would pass the very symptom the issue names — and I agree with your own framing that the control is still outstanding.

The one that matters: the degenerate predicate misses //////

if re.fullmatch(r"(.)\1{7,}", core, re.S):

(.) plus \1{7,} needs eight or more identical characters to match. The instance FLM states, and the one in this PR's own title and body, is ////// — six. So a server that answered exactly the reported degenerate string would be classified as clean text and the test would pass. For a detector whose whole job is "never one character repeated", the threshold should be at or below the named symptom: len(core) >= 4 and len(set(core)) == 1 catches ////, ////// and the long runs alike, and cannot pass a 6-character repeat.

Everything around it is right: the empty-with-finish_reason: length case passes by design and says so, a non-2xx with a body is a refusal, and the prompt is sized to fill the context (n = max(32, int(words * 0.72))) rather than hoping.

The control, and why the default rounds is the number that decides it

You wrote the honest thing — "That establishes the harness, not that today's main is clean" — and I agree with it; a test that has never been red against the fault is a harness until it has been. But the argument then turns on rounds=${4:-6}: the premise is that enough hangups exhaust the accept capacity, and the run you showed used 16, not the default 6. Since the default is what CI will run, either 6 needs to be shown to exhaust on a buggy build, or the default should be the demonstrated-exhausting number. As it stands the arm can be green on a build that has the fault.

The rest is the careful kind

• port from the server's own log line, not chosen here — no collision, and the serve reported port 0 fallback uses the backend's line with a printed note (engine#1485);
• bounded waits everywhere: 240 s for the port with a skip if serve dies, -m 30 on the interrupting curls, and -m 120 on the verdict with the reason stated — "a dial failure is the fault we are looking for, not slowness". The -m 1 curls are the interruption mechanism, not a verdict, which is the distinction that usually goes wrong here;
• the verdict is one real max_tokens: 8 request returning 200, and the FAIL branch prints the last 25 lines of the serve log, so a red carries its evidence;
• two interruption styles (streaming EPIPE via head -c 32, and non-streaming hard timeout) matching the two teardown paths in app/serve.cpp;
• SKIP_RETURN_CODE 77 on both rows, and command -v curl || exit 77.

Nit

Neither row sets a TIMEOUT property, and this test can legitimately take several minutes (240 s port wait + up to 16 × 30 s interruptions + a 120 s verdict). set_tests_properties(... TIMEOUT 900) is cheap insurance against a stuck run reaching the job limit instead of failing as one test.

Not run here: these need ${ONEBIT_SERVE_TEST_GGUF} and a built onebit; disk is at 149 G (build floor 175 G) and no GGUF serve fixture is configured in this worktree.