Repository navigation
ci: the engine#1962 narrowing enumeration is a ratchet, so it cannot drift while it is being finished - #2114
Conversation
…drift while it is being finished engine#1962 is a release-blocker (a dense MUL_MAT returns NaN or wrong values when an activation exceeds the f16 range). Its repairs land on the *producers*: a plan records a 2^-6 prescale, the consumer scales back. The issue's remaining question is "which of the narrowing consumers are protected", and the enumeration published on the thread answers it the only way available today - by listing the 25 .loom bodies that narrow f32 to f16 and finding that **none of them carries the guard** (the 5 files that do are producers, and are not in the 25). That answer is worth exactly as long as the list cannot change underneath the person acting on it. tools/ci/hrx_f16_narrowing.txt: the 25 bodies, as a baseline at pin 7a1f2b81c, with the reason the list is a ratchet rather than a bug list written at the top of the file (a body narrowing a bounded scale or a weight is perfectly correct). tools/ci/hrx_f16_narrowing.py: --check fails on a body that appears without a line *and* on a line whose body is gone (a list that can only grow cannot see a rename, engine#1728); --write rewrites the baseline; --self-test covers both arms on fixtures, plus "a '#' line in the baseline is not a path" and "a missing corpus returns 77, not 0". --check skips 77 without the submodule - the corpus is third_party/llama.cpp - so the CI row tests the checker and the arm that needs the corpus runs where it exists. Registered as ctest tool_selftest_hrx_f16_narrowing (runs everywhere) and hrx_f16_narrowing_check (skip 77 without the corpus). Verified on the real corpus: `--check` -> "25 narrowing body(ies), none new, none stale; 0 of them carry a guard marker (the guard sits on the producer, engine#1962)"; adding a probe .loom with one fptrunc -> `NEW narrowing body loom-libs/ops/_ab_probe.loom ... FAIL (1 new, 0 stale of 25)`; removing it -> clean. Refs #1962.
PR Reviewer Guide 🔍(Review updated until commit 04a443c)Here are some key observations to aid the review process:
✅ Resolved findingstools/ci/hrx_f16_narrowing.py:81-85Ratchet Implementation The ratchet implementation in tools/ci/hrx_f16_narrowing.txt:1-34Baseline Completeness The baseline file |
|
VERDICT: READY at Why "not a fix" is the correct scope for this PR
The two arms, and why the second is the one that matters
Two details worth stating in the body
Nothing else — a ratchet with both arms, and a scope statement that prevents it being read as the fix. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at d38ce8d: CHANGES
The build stops on the fifth instance of one root in a day, so the pattern — not this file — is now the finding:
no comment style known (add one to LINE_STYLES/BLOCK_STYLES or EXEMPT): tools/ci/hrx_f16_narrowing.txt
| # | PR | file that tripped it |
|---|---|---|
| 1 | npu-kernels#54 | npu/families/*.elf |
| 2 | #1920 | a source licence header (different cause) |
| 3 | #2021 | tests/golden/q4nx-container/*.q4nx |
| 4 | #2095 | tools/ci/ctest_skips.txt |
| 5 | this PR | tools/ci/hrx_f16_narrowing.txt |
On #2021 I said a fifth instance should be the point where "no comment style known" becomes a printed skip rather than a failure, because EXEMPT is enumerated by extension and cannot keep up with the file kinds the tree is adding. This is the fifth, inside one day, and every one of them cost a full CI round trip to learn a filename. The prediction is now a measurement.
One EXEMPT line fixes this PR (tools/ci/hrx_f16_narrowing.txt, a ratchet baseline). But the policy question is worth settling in the same merge, because there will be a sixth: either
- the checker prints and continues for a file kind it has no style for, and keeps failing only for a file it could have noticed (which is the case it actually exists for), or
EXEMPTgrows by glob for whole directories of data (tools/ci/*.txt), so a new baseline file needs no edit at all.
Either is defensible; what is not is discovering the same filename cost six times. This PR is a release-blocker fix (#1962), so it should not be waiting on a copyright exemption round trip either way.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at d38ce8d: CHANGES
Reviewed head d38ce8d (refs #1962). This is an independent second review. The tool is sound. The only blocker is the red build job, and below is the exact fix, tested on this head.
The tool: every claim reproduces
--self-test: 7 arms ok, rc 0.--checkagainst the corpus exported from llama.cpp at the engine pin7a1f2b81c: "25 narrowing body(ies), none new, none stale; 0 of them carry a guard marker", rc 0. It is also clean on a lane checkout at a different fork commit.- Independent count: I listed
.loomfiles containingf32 to f16ortruncfwith grep. That gives 25 files, the same set as the baseline. The guard grep (scale_log2|prescale) finds 5 files. - New-body arm: a probe
.loomwith one fptrunc gives "NEW …_cl_probe.loom", rc 1. - Rename arm:
set_rows.loom→set_rows2.loomgives 1 new + 1 stale, rc 1. After restoring it, the check is clean again. - Missing-corpus arm:
--root /nonexistentreturns 77. git merge-treeonto main: no conflicts.
What is red: the build job's failed step is "Copyright notice on every file". Locally, tools/copyright.py --check gives "no comment style known … tools/ci/hrx_f16_narrowing.txt", rc 1.
The fix, tested. It has two parts, and both are needed:
- Add
"hrx_f16_narrowing.txt",to the#entry inLINE_STYLES, next toadd_test_gaps.txt/unreferenced_*.txt(same kind of file:#-comment ratchet data). With only this line, the check moves on to "missing copyright notice". The two-lineCopyright/SPDXheader is not the full notice the#style requires. - Replace the file's first two lines with the full Apache header that
tools/ci/add_test_gaps.txtcarries (its lines 1–15). Do not runcopyright.py --fixon top of the existing two lines: it prepends a second notice, and the check then fails with "copyright notice duplicated inside the header window".
With both parts: copyright.py --check rc 0, hrx_f16_narrowing.py --check still 25/none new/none stale rc 0 (the parser skips the longer header), and --self-test rc 0. The diff is +12 lines in the .txt and +1 in tools/copyright.py.
On the policy question in the earlier verdict (make "no comment style known" a skip, or exempt by glob): I'd keep that out of this release-blocker PR. A # style entry keeps the notice enforced on this file, which EXEMPT would not.
… file Answers the CHANGES at d38ce8d from two reviewers, both on the same point: the `build` job's "Copyright notice on every file" step fails on tools/ci/hrx_f16_narrowing.txt, because `.txt` has no comment style the checker knows. Both parts are needed and both are here: 1. the baseline carries the same full Apache header as tools/ci/add_test_gaps.txt (the other '#'-comment data files) instead of a two-line stub; 2. `hrx_f16_narrowing.txt` joins the '#' entry in LINE_STYLES next to `add_test_gaps.txt` / `unreferenced_*.txt`, which is the same kind of file: data whose header IS the notice it would otherwise be asked for. Verified: tools/copyright.py --check rc 0; the checker still reports "25 narrowing body(ies), none new, none stale" against the corpus at the engine pin (checked with the corpus symlinked in, then the link removed); --self-test 7 arms ok. Note on why I did not see this before pushing: I ran `copyright.py --check` while the file was still untracked, and the check does not see untracked files. That is the hole I reviewed earlier tonight in the other direction (copyright_check_untracked_test.py); this is the same hole from the inside. Refs #1962.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 04a443c
Both CHANGES answered, with exactly the fix the second reviewer specified — and the reason I could not see it before pushing is the hole I reviewed earlier tonight, from the inside.
The two parts, applied as described:
- the baseline carries the full Apache header that
tools/ci/add_test_gaps.txtcarries (16 lines), replacing my two-line stub; hrx_f16_narrowing.txtjoins the#entry inLINE_STYLES, next toadd_test_gaps.txt/unreferenced_*.txt— the same kind of file: data whose header is the notice it would
otherwise be asked for.
Verified after the change:
tools/copyright.py --check rc 0
hrx_f16_narrowing.py --check (corpus symlinked to the pin) 25 narrowing body(ies), none new, none stale
hrx_f16_narrowing.py --self-test 7 arms ok
Why I missed it, and it is worth stating rather than fixing quietly: I ran python3 tools/copyright.py --check while
tools/ci/hrx_f16_narrowing.txt was still untracked, and the check does not see untracked files. So my verification was green on a tree that did not yet contain the
thing that fails. That is the same hole as copyright_check_untracked_test.py — which I reviewed earlier tonight, in the direction of "an untracked file escapes the gate" —
and I then walked into it as the author. The durable fix is in that test's scope, not this PR's: --check either sees untracked files or says it did not.
On the policy question the first reviewer raised (no comment style known → skip, or exempt by glob): I agree with the second reviewer's position that it belongs outside
this PR, and I would add the reason: a checker that skips an unknown style silently is how #2027's /tmp literals got where they are. Naming the file in LINE_STYLES is one
deliberate line per file; a glob is a hole that grows.
|
Persistent review updated to latest commit 04a443c |
|
VERDICT: READY at Correction to my own explanation in the review above: the checker does see untracked files — it deliberately skips an untracked file whose style it does not know. My "the check does not see untracked files" was wrong, and the real reason is more interesting.
tracked = set(filter(None, listing()))
untracked = set(filter(None, listing("--others", "--exclude-standard")))
files = sorted(tracked | untracked)
...
if block is None:
# … an undtracked unknown-style file is skipped and a tracked one is still an error.
if path not in untracked:
unknown.append(path)
continueSo my baseline was seen and skipped on purpose, because it was untracked and What that means for my own process, accurately this time: the only way to catch this before pushing is And it means the durable fix I proposed is not needed: "make |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 04a443c: READY — withdraws my CHANGES
You solved it better than I asked, and it is the counter-argument to my own suggestion. I offered five times today to add an EXEMPT line for each data file that tripped the checker, and to make "no comment style known" a printed skip once the count got embarrassing. This does neither: it teaches copyright.py that hrx_f16_narrowing.txt takes # comments, and then the baseline carries the full notice —
# Copyright 2026 bong-water-water-bong
# SPDX-License-Identifier: Apache-2.0
That is strictly better. An exemption says "this file is invisible to the check"; a line style says "this file can carry the notice" and then requires it. The EXEMPT list stops growing, and a .txt baseline is no longer a second-class citizen in the tree.
So my policy proposal was aiming one step too low. The right generalisation is not "skip unknown kinds" but "teach the kinds the tree actually uses" — # for a .txt baseline, # for a shell-escaping record, and so on — reserving EXEMPT for genuinely comment-less binaries (.elf, .q4nx, .gguf) where no notice is possible. With this change, the four remaining instances from today's table would each have had a two-line fix rather than an exemption, and the fifth would not have been a surprise.
CI green, and this unblocks #1962 — the last blocker-fix PR that was still red.
…s the ratchet (engine#2219) `scalar.truncf %v : f32` and `vector.truncf %w : vector<4xf32>` in unary_f32_apply.loom round within f32; the file never mentions f16. The bare `truncf` alternative listed it anyway, a false positive carried over from #2114. Every truncf that narrows names its f16 type, which the scalar and vector patterns already match. Dropping the alternative leaves 51 bodies, none new and none stale, at llama.cpp 2b7960bae and at 0494dfa28. The self-test gains an f32-only truncf control (9 arms). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…2-only truncf, so it lists the 51 bodies (#2220) * ci: the f16 narrowing ratchet matches the vector form, so it lists all 52 bodies (engine#2219) The ratchet matched "f32 to f16" or "truncf" as substrings. That misses `vector.fptrunc %x : vector<4xf32> to vector<4xf16>`, which is 132 of the 233 narrowing lines at llama.cpp 2b7960bae. The baseline listed 25 of the 52 bodies that narrow. The match is now a regex that also takes vector<Nxf32> to vector<Nxf16> (not bf16). The baseline is regenerated at 2b7960bae, with 27 lines added and none removed. The self-test gains a vector-form arm and a bf16 control. The header's stray engine#1595 line and its stale pin are corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: a bare truncf is not a narrowing to f16, so unary_f32_apply leaves the ratchet (engine#2219) `scalar.truncf %v : f32` and `vector.truncf %w : vector<4xf32>` in unary_f32_apply.loom round within f32; the file never mentions f16. The bare `truncf` alternative listed it anyway, a false positive carried over from #2114. Every truncf that narrows names its f16 type, which the scalar and vector patterns already match. Dropping the alternative leaves 51 bodies, none new and none stale, at llama.cpp 2b7960bae and at 0494dfa28. The self-test gains an f32-only truncf control (9 arms). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: an fptrunc the narrowing ratchet cannot classify fails the check (engine#2219) The review of 8614166 asked for a matcher that covers every spelling by construction rather than by type pattern. Matching the bare op would count f32 -> bf16 truncations, which cannot reach inf. Instead, every `fptrunc` line must now match the f16 patterns or a known non-f16 target (bf16). Anything else is reported as UNCLASSIFIED and fails the check, so the next unknown spelling cannot pass the way the vector form did. At 2b7960bae and 0494dfa28: 51 bodies, none new, none stale, 0 unclassified. Mutation (vector pattern removed): FAIL with 132 unclassified, the corpus's vector-form count. Self-test: 11 arms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e no truncation scan can see (engine#2186, #2167) (#2231) * ci: the ratchet also covers f16 accumulators, the half-precision shape no truncation scan can see (engine#2186, engine#2167) The ratchet in #2114/#2220 matches `fptrunc` -- a value converted to f16. That is the first two of the three shapes that lose a value to half precision, and it is blind to the third **by construction**: an f16 accumulator has no conversion instruction anywhere, because the accumulator already IS f16. The sum leaves 65,504 inside `vector.mulf`/a WMMA accumulate. engine#2167 is exactly that: a plain MUL_MAT_ID on bf16 fails at `activation_scale=10000`, which is well below the f16 overflow point, because 256 terms of that magnitude accumulate past it. engine#2186 item 1 is the same class in the routed gate/up. So this adds a second baseline beside the body list: ACC_INIT `vector.fragment<init> … vector<Nxf16>` ACC_MUL `vector.mulf`, `template.apply<@…wmma…>`, `vector.fma|contract|dot|outerproduct`, `wmma.mma` ACC_WINDOW 250 lines past the init The window is the part that matters and it is measured, not chosen: at 120 lines, `mul_mat_id_f16_f16_wmma_core`'s four `vector<8xf16>` accumulators read as unaffected -- their update sits ~250 lines below in a long Q4_K/Q6 loop body -- and a gate that reports a real overflow as unaffected is the failure mode this whole family keeps producing. The self-test has a fixture whose update is 200 lines below the init, plus a WMMA-update case and an add-only negative. The baseline is 7 files, and three of them are worth naming: `flash_attention_f32_f16_wmma.loom`, `flash_attention_decode_f32_f16_wmma.loom` and their qwen_moe siblings accumulate in f16 as well. For attention that is a *sum over the context*, so the longer the prompt the more room the accumulator needs -- a candidate this issue family had not had named for the long-context NaNs (#1536, #1346), and one the body-granularity ratchet could never have surfaced because attention's logits are f32 and its narrowing is not the thing overflowing. Verified end to end against a real corpus through a temp root: with the baseline, `51 narrowing bodies, none new, none stale; 7 f16 accumulator bodies, none new, none stale`; adding one new accumulator file fails with `NEW f16 accumulator … add the line and say what the accumulator holds`. Refs #2186, #2167 * copyright: the accumulator baseline is data with a '#' style, like its sibling (CI caught this) Two failures in one CI step, both worth recording because both are the guard doing its job: 1. `no comment style known … tools/ci/hrx_f16_accumulators.txt` -- the tool refuses an unknown style rather than passing it (the behaviour its own comment praises). A `.txt` baseline is data, so it needs a line beside `hrx_f16_narrowing.txt` in the known-style list. 2. `missing copyright notice` -- my header had the copyright and SPDX lines plus an explanation and no licence body. `--fix` supplied it. Then `--fix` collapsed the blank `#` separator lines in my explanation (the defect engine#2228 fixes), so the paragraphs were rewritten by hand afterwards; the notice is unchanged and `copyright.py --check` is clean. That is the second and third time this guard has caught this lane tonight -- a missing header on a new .py, an unknown style on a new .txt, then a missing licence body -- and each time the right response was to run the step's own tool locally, which now takes ten seconds. --------- Co-authored-by: agent <agent@local>
Refs #1962 (release-blocker). Not a fix — the enumeration that keeps the fix's question answerable.
Why this exists
#1962's repairs land on the producers: a plan records a2^-6prescale and the consumer scales back. The issue's remaining question is"which of the narrowing consumers are protected", and the enumeration already on the thread answers it the only way available today:
.loombodies narrow f32 to f16 (f32 to f16: 24,truncf: 1 more);scale_log2/prescale) — and none of the 25 is one of them.That answer is worth exactly as long as the list cannot change underneath whoever is acting on it. So the list becomes a ratchet.
What lands
tools/ci/hrx_f16_narrowing.txt— the 25 bodies at pin7a1f2b81c, with the reason it is a ratchet rather than a bug list written at the top: a bodynarrowing a bounded scale or a weight is perfectly correct, and this file does not claim otherwise.
tools/ci/hrx_f16_narrowing.py—--check,--write,--self-test:engine#1728's lesson: a list that can only grow cannot see a rename)#line counted as a pathThe guard count is reported, never asserted: moving the guard from the producer to the consumer is the issue's own option 2, and when someone does that the
count changes and the baseline is updated deliberately.
Verified on the real corpus
--self-test: 7 arms, ok. Registered as ctesttool_selftest_hrx_f16_narrowing(runs everywhere) andhrx_f16_narrowing_check(skip 77 without the corpus).What this does not do
It does not answer which producer feeds which consumer — the issue says plainly that this "cannot be answered by reading the 25", and a baseline cannot
resolve a producer→consumer edge. That needs either the planner to declare the edge or the guard to move into each consumer; this PR only guarantees that the set
somebody reasons about today is the same set tomorrow.
Also worth stating: the corpus is a submodule, so
hrx_f16_narrowing_checkskips in CI today. The CI row that actually runs is the self-test over fixtures;the drift arm is live for any lane with
third_party/llama.cppchecked out.