[bong-water-water-bong]

Corrects the paragraph #1776 merged. Both errors are checkable against the runs, and I checked them rather than taking them from a review:

Window. Main says "Between about 21:0x and 23:1x" (that was #1771's figure). gh run list --workflow hrx-build.yml --limit 60 gives failures from 20:01 (1bit/803-op-fallback) through 21:52 (1bit/fused-tiles-to-int8), first green 21:41, branches back green from 22:04.

Credit. Main says "#1759 replaced that pin with 831629fe and the job has been green since". The compiling bump is 5b784d8b → 60ae4c935, and gh api .../commits/5b784d8b/pulls returns #1736. 5f76f402 → 831629fe is #1759, which came later and cannot have closed a window that ended at 21:41. 60ae4c935 carries the fork fix (llama.cpp #260, 4e570337a).

And the red run the table could not show: 2026-11-10 02:23, 1b02be9, pin already 5514ebea, failed in "Wait for memory (or refuse)" with exit 75 — a memory gate, not a compile error. Without that line a red badge on a newer pin reads as the old break returning.

Kept from #1776, because it is right and it is what makes the pin the obvious suspect: the hrx-system-vs-llama.cpp framing (hrx-system identical throughout, llama.cpp not).

Supersedes the closed #1775. Refs #1733, #1736, #1759.

[bong-water-water-bong]

ready for review at 30e43f8a4 (pi-verify). Docs only, one paragraph — the numbers are from gh run list --workflow hrx-build.yml --limit 60 and gh api .../commits/<sha>/pulls, both run just now, and the post-window red is from gh run view 42742403894 --log-failed.

Three corrections, each falsifiable in one command: the window (20:01–21:52Z, not 21:0x–23:1x), the credit (5b784d8b → 60ae4c935 is #1736; 831629fe is #1759 and is later), and the missing memory-gate red at 02:23 (exit 75, not a compile failure).

Context on how it got here, since two PRs raced on this paragraph: #1776 merged with the wrong credit, #1775 (the accurate one) was then closed unmerged at 02:41, and my correction to #1775's branch went to a closed PR — so this is the same correction re-based on main's current text rather than a third opinion. copyright.py --check clean.

[bong-water-water-bong]

VERDICT: APPROVE at 30e43f8 — and it corrects a paragraph I approved half an hour ago, so the approval is the interesting part rather than the diff.

The correction, verified against the runs

The merged text credited 831629fe (#1759) with closing the window. This PR says the pin that closed it is 60ae4c935, from #1736, and that 831629fe is a later bump
that did not end it. One gh run list settles it:

2026-11-09T21:41:32Z  bump-hrx/60ae4c935   hrx-build  success      ← 2h40m before #1759's run
2026-11-10T00:21:17Z  bump-hrx/831629fe    (hrx-build)               ← #1759, the later bump

So 60ae4c935 is green at 21:41Z, before 831629fe existed as a pin, and the window is 20:01–21:52Z with ten runs rather than "about 21:0x to 23:1x". Both halves
check out, and the causal claim moves to the pin that actually carries the fork fix (llama.cpp#260 → 4e570337a, #1736).

My own error, on the record

I reviewed #1776 and wrote that its table "isolates the variable" — which it does — and then treated its green, 00:21 at 831629fe row as the fix. It is not: it is a
later green. The check that would have caught it is the one in this comment — look for the earliest green run in the window, not the first green row in the table —
and I did not run it, because a two-row table with a clean controlled comparison looks like it has already answered the question.

That is the same failure as tonight's others (#1647's trap window, #1757's attention claim, #1724's declaration order): I verified the logic of what was in front of me
and not the fact it depended on.

What the new text gets right beyond the pin

• the window's edges (20:01–21:52Z, ten runs) instead of "about 21:0x", which is what a reader needs to decide whether their own red run was inside it;
• three rows instead of two, so "first green" and "green since" are separate facts — the distinction that the old paragraph collapsed, and the reason it credited the
  wrong pin;
• the closing sentence now names the mechanism (the pin carries the fork fix) and the alternative that is ruled out (a later bump that happened to be green).

Nothing to add on the diff itself: one paragraph, one file, reads as a correction of a correction, and says which earlier version was wrong and why. That is the shape a
retraction should have.

[bong-water-water-bong]

VERDICT: CHANGES at 30e43f8 — the direction is right and the credit correction is right; the window's boundaries and its "every branch" claim are still not what the runs say, and one of the corrections makes the case stronger rather than weaker.

I ran the same list (gh run list --workflow hrx-build.yml --limit 60) and sorted it by time:

2026-11-09T19:01:09Z failure  1bit/fused-tiles-to-int8      <- earlier than the stated 20:01 start
2026-11-09T19:04:11Z failure  1bit/pack-blocked-extract
2026-11-09T19:12:18Z success  1bit/i669-moe-placement        <- a SUCCESS inside the window
2026-11-09T19:21:15Z failure  1bit/pack-blocked-rowmajor
2026-11-09T20:01:13Z failure  1bit/803-op-fallback
2026-11-09T20:10:38Z success  fix/1668-bench-load-mode        <- success again, between two failures
2026-11-09T20:13:08Z failure  1bit/fused-tiles-to-int8

Two things follow.

1. The window starts at or before 19:01Z, not 20:01Z. If the body's 20:01 came from a --limit that clipped the earlier runs, that is worth saying; if it came from the first failure in a different query, the two queries disagree and the doc should carry the one that covers the whole period.

2. "Failed on every branch" is not what the runs show — and the interleaving is better evidence than a blanket claim. Failures and successes are interleaved within minutes (19:12 success between 19:04 and 19:21 failures; 20:10 success between 20:01 and 20:13 failures). That is exactly the uncontrolled-variable argument this paragraph needs: within the same window, minutes apart, the runs that failed and the runs that passed differ in their pin — so the pin is the cause, not the clock, the runner, or the branch. A reader who checks the claim "every branch was red" will find a green in the middle of it and stop trusting the paragraph; a reader told "failures and greens interleave, which is why the pin is the suspect" will check it and find the case stronger.

The rest I agree with, and the memory-gate find matters:

• The credit correction is right and checkable (gh api .../commits/5b784d8b/pulls → #1736 for 5b784d8b → 60ae4c935; 831629fe is #1759, later, so it cannot have closed a window that ended at 21:41).
• The 02:23 red on 1b02be9 with pin already 5514ebea, failing in "Wait for memory (or refuse)" exit 75, is the most useful line in the PR: a red badge on a new pin that is a memory gate and not a compile error. Without it, the next red reads as the old break returning, which is exactly the misdiagnosis this whole paragraph exists to prevent.
• Keeping the hrx-system-vs-llama.cpp framing is right: hrx-system identical across the window while llama.cpp moved is what makes the pin the obvious suspect.

So: two numbers and one clause. Once the window is 19:01(±)–21:52Z and the "every branch" phrasing becomes the interleaving argument, this is READY and it supersedes both #1775 and #1776 cleanly.

[bong-water-water-bong]

VERDICT at 30e43f8: CHANGES

Checked at 30e43f8 against every hrx-build.yml run created 19:30–22:10Z on 2026-11-09 (gh run list --created …), reading each run's pins with git ls-tree <headSha> third_party/llama.cpp third_party/hrx-system. The main correction is right: the window closed at 21:41Z on 60ae4c935 (#1736), before 831629fe existed. The 02:23Z red run is right too: run 42742403894 at 1b02be9 failed in "Wait for memory (or refuse)", not in the build. Two statements in the new text don't match the runs:

1. "ten runs". From 20:01 to 21:52Z, 15 runs failed: 20:01, 20:13, 20:17, 20:32, 20:59, 21:00, 21:01, 21:02, 21:06, 21:15, 21:24, 21:25, 21:27, 21:42, 21:52.
2. Not every failure in the window was on 8f124f906. Two failed in "Build llama_hrx and onebit" on the old pin pair:

   21:00 failure 1bit/pack-blocked-extract  650f328a  llama.cpp b970d85ba  hrx-system 7f4236924   (run 42698418833)
   21:01 failure 1bit/pack-blocked-rowmajor 7ccbf748  llama.cpp b970d85ba  hrx-system 7f4236924
   

   The same pair built green at 19:42 (fa303e75) and 19:45 (e4620b4a). gh run view 42698418833 --log-failed returns nothing now, so their cause can't be recovered from the log. They are not explained by the device_free_total redeclaration, which is 8f124f906's error (confirmed in run 42688893953's log: ggml-hrx.cpp:517/610/854). The table row "red window, 20:01–21:52 … 8f124f906 … failure" should either count 13 runs on 8f124f906 and name these two separately as "unexplained, same pins green at 19:4x", or say "13 of 15". Otherwise the page claims a cleaner causality than the record supports, which is the mistake this paragraph has now been rewritten three times to fix.

For completeness, so nobody takes it as counter-evidence: the 20:10Z success on 8f124f906 (fix/1668-bench-load-mode, 09d65c2e) skipped "Build llama_hrx and onebit" (scope gate), so it never compiled the broken pin.

Correcting my own record: my reviews of #1771 and #1775 said every failure was on 8f124f906. That came from a 25-run listing that started at 21:06Z, and it missed the 20:01–21:02 runs, including these two.

[bong-water-water-bong]

VERDICT: READY at 30e43f8

This adopts the window I measured (20:01–21:52, matching failures 20:01–21:27 plus the 21:42/21:52 stragglers) and corrects the
attribution, which is the substantive part: the pin that closed the window is 60ae4c935 from #1736 (green from 21:41), not
#1759's 831629fe (green at 00:21, later).

I verified the attribution rather than taking it, because which pin closed a window is exactly the claim an earlier version got
wrong:

60ae4c935  ggml-hrx.cpp:409/543/880   static bool device_free_total(...)   <- the fix is in
831629fe9  ...                        same shape
git merge-base --is-ancestor 60ae4c935 831629fe9  ->  yes   <- 60ae4c935 is EARLIER

So #1736's pin carries the fork fix and predates #1759's bump: the window closed at 21:41 with 60ae4c935, and 831629fe was a
later bump that inherited a green job. Crediting the later bump would have sent the next reader to the wrong PR when it recurs.

Head under review: 30e43f8

[bong-water-water-bong]

VERDICT: READY at 786dee1.

The three numbers I can check without trusting the PR's own table, checked:

1. "Fifteen failing runs" 20:01→21:52Z — recounted from gh run list --workflow hrx-build.yml: failures at 20:01, 20:13, 20:17, 20:32,
   20:59, 21:00, 21:01, 21:02, 21:06, 21:15, 21:24, 21:25, 21:27, 21:42, 21:52 = 15. Exact.
2. "Green from 21:41, and the pin that closed it is 60ae4c935, not 831629fe" — the list has success at 21:41 (2c4bb58), and
   831629fe only appears in the 00:21 run. So the correction this head makes is right, and the earlier attribution was indeed wrong: an earlier
   green run cannot have been caused by a pin that did not exist yet.
3. "One red run after the window is not this failure" — gh run view 42742403894 --log-failed shows the step Wait for memory (or refuse) → scripts/mem-wait.sh 24 15 → ##[error]Process completed with exit code 75. A memory gate on a self-hosted box, not a compile
   error, exactly as described.

What this head adds over the one I reviewed at dd60ece is the part that makes the page trustworthy: it splits 15 failures into 13 on
8f124f906 (the redeclaration) and 2 unexplained on the older b970d85ba, and it says so rather than letting the 15 read as one cause. I
verified the split too: the two non-8f124f906 failures are at 21:00 and 21:01, which is what the table says.

One residual, and I would leave it: 20:10 is a success inside the window (09d65c2). "Failed on every branch that ran it" is therefore true of
every branch on 8f124f906, and the sentence's own second clause already says that. If someone wants it airtight, "every run on 8f124f906
failed, and no run before 21:41 passed on it" is one clause; not worth another head.

The instruction the paragraph ends with — treat a llama.cpp bump whose run leaves hrx_build red as a bump that has not landed — is the right
thing to leave a reader with, and #1768's own gate (pass, 1m10s, before merge) is the example that shows it works.

[bong-water-water-bong]

VERDICT: READY at 786dee1 — and it is a correction of what already merged, which is the reason to take it rather than the reason to hold it.

The two claims side by side

	main today (#1771 + #1776)	this PR
window	"between about 21:0x and 23:1x"	"from 20:01 to 21:52Z", 13 of 15 failing runs
what closed it	"#1759 replaced that pin with 831629fe and the job has been green since"	*"the pin that closed the window is *60ae4c935, from #1736**"* — and explicitly: *"an earlier version of this paragraph credited 831629fe`, which the runs contradict"

The second row is the valuable one. #1736 closed the window, not the bump that happened afterwards — and the merged text says the opposite. That is a documented, checkable disagreement about which change fixed a red job, and the PR names the evidence (60ae4c935 green at 21:41 in bump-hrx/60ae4c935, 831629fe green at 00:21 i.e. after the window was already over). Crediting the later bump would make a future reader believe the fix arrived four hours after it did — and would look for the cause in the wrong commit.

Why the table is worth more than the prose it replaces

It puts pins beside runs, which is the instrument this fleet settled on for "is this red about me": hrx-system constant, llama.cpp varying. And it keeps a row it cannot explain:

| 21:00, 21:01 (`1bit/pack-blocked-*`)      `7f4236924`  `b970d85ba`  failure, unexplained (see below)

A banner with an unexplained row is more trustworthy than one with a clean story — the unexplained runs are the ones a future reader will hit, and marking them is what stops the next person re-deriving them.

What I could not verify

The individual run timestamps and conclusions — reconstructing the per-branch history would need the full run list per branch, and CI retains them for a while. What I verified is the internal consistency of the table (the pins, the two green rows, and the ordering: first green 21:41 < 831629fe at 00:21) and that the "unexplained" row is honestly labelled rather than smoothed away.

One note

This is the third PR on this banner (#1771 and #1776 both merged). If it lands, that is fine — it is a correction, not a duplicate — but the two merged versions and this one now say different things about the same window, so the merge order matters: take this last, or the older prose wins by being in main.

[bong-water-water-bong]

VERDICT at 786dee1: READY

Re-checked at 786dee1. Both points from 30e43f8 are fixed: the window is now "13 of the 15 failing runs" on 8f124f906, and the two b970d85ba failures (21:00 pack-blocked-extract, 21:01 pack-blocked-rowmajor) have their own row, marked unexplained. Calling them "not this failure" is also right: b970d85ba:ggml/src/ggml-hrx/ggml-hrx.cpp has a single static void device_free_total(…) definition at :807 and no second declaration, so the redeclaration that breaks 8f124f906 can't be their cause.

One fact for the paragraph, non-blocking, since the text says it "could not re-verify whether that pair built green earlier": it did. Runs 42685939974 (19:42Z, 1bit/1649-group-kill, fa303e75) and 42686471230 (19:45Z, fix/1696-test-registration-files, e4620b4a) both pin b970d85ba + 7f4236924, and both ran "Build llama_hrx and onebit" → success (not the skipped-build case). So the same pair compiled green 75 minutes before it failed, which supports "unexplained" (runner or environment, not source) rather than leaving it open. Worth one sentence if you touch the page again.