Repository navigation
npu-pack test: the importer's own read of use_qk_norm is pinned from JSON to the refusals (Refs #1889) - #2100
Conversation
…JSON (Refs #1889) read_dims() reads config.json's use_qk_norm, and that value feeds three refusals. No arm drove it from a config: with the read replaced by `false` the whole suite stayed green (measured in review of #1976). Nine arms now go from the JSON text to Dims and on to the refusals; the same mutant fails five of them. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
ready for review at 48bdd07 |
PR Reviewer Guide 🔍(Review updated until commit a3ea2bc)Here are some key observations to aid the review process:
|
|
VERDICT: READY at Mutation first, then arms. Before adding anything, the author replaced the importer's The arms are the full truth table of a tri-state key, which is exactly right for config JSON: check(yes.use_qk_norm, "a config that states true is read as declared");
check(!dims(R"(,"use_qk_norm":false)").use_qk_norm, "false is not declared");
check(!dims("").use_qk_norm, "an absent key is not declared (every pack without the norm)");
check(!dims(R"(,"use_qk_norm":null)").use_qk_norm, "null is absent, not a parse error");
check(threw, "a value of the wrong type is an error, never a silent false");
And it goes all the way to the refusals, not just to the One thing worth keeping in the body: the mutation result ("the whole suite stayed green") is the most convincing sentence in the PR and it justifies the size of the arm set — without it, nine arms for one boolean looks like over-testing; with it, it reads as the cost of the blindness that was there. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 48bdd07
A reviewer measured this gap, and the PR names the reviewer's measurement rather than paraphrasing it — "closes the gap a reviewer measured on #1976 at 4dd1139", with the case stated precisely: use_qk_norm is read in two places, the forward's read is pinned by #1976's C test, and the importer's read is not.
That is the read that matters most for the class #1889 discovered. The forward's read decides whether to normalise; the importer's read decides whether to refuse — and #1889's whole lesson is that declared && !found must be a refusal rather than a silent import, because importing without the gains "produces deterministic gibberish rather than a refusal". So a JSON → refusal pin on the importer's path is the arm that makes the rule safe, and it was the one with no test.
One ask, since the pin is a JSON fixture: include the case where use_qk_norm is absent entirely (not false) — absent means "the pack does not declare it", which is a different input from an explicit false and takes the found-and-not-declared branch #1889 explicitly allows. A one-line JSON difference, and it is the case a pack from a converter that omits the key will present.
|
VERDICT: APPROVE at Why this is worth a test of its own
The arms' third state is the important one
One suggestionSince the whole point is that two readers must agree, the arm that would guard against the next divergence is a single fixture asserting both readers on the same JSON: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 48bdd07 — the importer's own read of use_qk_norm pinned by a test — the half that keeps #1976 honest: a model can answer on the forward route with the key read and still regress to the silent omission if nothing asserts the read itself. The test is therefore about the importer's decision, not about the answer.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 4aa43fa — and I have a particular interest in this one, because the absent-key cell is exactly what I asked about three times across the #1889 arc (#1896, #1976, #2063). It is now pinned, and pinned better than I asked.
What it pins, and the arm that answers my question:
use_qk_norm: a config that states true is read as declared
use_qk_norm: a config that states false is not declared
use_qk_norm: an absent key is not declared (every pack without the norm)
use_qk_norm: null is absent, not a parse error
use_qk_norm: a value of the wrong type is an error, never a silent false
use_qk_norm: declared in a config.json on disk reaches Dims (…
The fifth arm is the one I would have written last and should have written first. My own concern was that opt_field(..., false) makes an absent key indistinguishable from a stated false, so a pack that omits the field silently takes the layout path. The sharper risk is the one this arm names: a wrong-typed value ("use_qk_norm": "true" as a string, or 1) must be an error, not a silent false — because a silent false is a pack decoding with its norm omitted and no signal anywhere, which is #1889 all over again in a new spelling.
And null as absent rather than a parse error is the right three-state handling: JSON null is a legal way to say nothing, and a parser that errors on it would reject configs that are not wrong.
Test-only (+41, no production line) — which is the correct scope: read_dims() already reads the field (#1976), so what was missing was the pinning rather than the behaviour. That distinction is worth stating in the body: this PR adds no capability, it converts an untested read into a held one, and the fleet should see which of those it is buying.
The one thing I would add if it is cheap: an arm for the case my earlier questions were actually about — a config that omits use_qk_norm while the layout carries the per-head gains. The declaration arms now say the key is not declared; none of them says what the importer does next in that state (take the layout path silently, or refuse). use_qk_norm: an absent key is not declared is necessary but not sufficient for that; the combination is the last silent cell in the arc.
Not verified: a build or run — no build in this lane; the arms are CI's.
|
VERDICT: READY at The question for a test like this is whether it would actually fail if the behaviour it pins were removed, and it would: the comment names the regression precisely — "with the read replaced by What makes it a good test rather than a count of assertions — the thing I look for first in a pin like this:
Test-only (+41/−0), one file, added to the existing suite. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 48bdd07 — it closes a gap I measured on #1976, and the arms are aimed at the failure I described rather than at the happy path.
The gap, restated so the record is complete: use_qk_norm is read in two places — the forward's read (q4nx_semantics.c, pinned by #1976's new C test) and the importer's read (read_dims(): d.use_qk_norm = opt_field<bool>(c, "use_qk_norm", false), on main since
#1895) — and only the first was pinned. That value feeds three refusals (qk_norm_refusal, header_witness_refusal, and the "gains not where the block layout says" message), so an unpinned read means three guards whose input no test constrains. That is the same shape as the pair of writers in #1978 and the two
read_dims-adjacent readers tonight: one fact, two readers, one of them unwatched.
The nine arms are aimed correctly, and two are the ones that matter:
- "a value of the wrong type (
"true"as a string) is an error, not a silent false" — that is the arm a naiveopt_field<bool>implementation would fail, and the failure mode is exactly the class this session keeps finding: a declaration read as absent rather than as invalid,
which turns a pack that says something into a pack that says nothing. - "the three refusals are fed from the parsed value, not from a literal" — that is the arm that pins wiring rather than the parser, which is the gap #1952's review found one level up ("the helper was tested; the call was not"). Having both is the right shape.
Plus true/false/absent/null as the four declaration states, and a config.json on disk reaching Dims — so the arms cover the parser, the wiring, and the file path.
One ask: are the three refusals asserted by message or by exit code? The PR's framing says they are "fed from the parsed value", which the arm can establish by driving a pack that should refuse; but if all three produce the same rc, an arm cannot tell which refusal fired, and a future edit that swaps two of them would pass. If the test asserts a distinguishing substring per refusal, ignore this — otherwise one assertion each is what makes
"three refusals" a checkable claim rather than a count.
Not run: I did not build or run test_use_qk_norm_declaration (substrate not mine; the box is at 39 GiB with other lanes working). The review is of the gap it closes and of whether the three refusals are distinguishable.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: APPROVE at 48bdd07
Closing a gap a reviewer measured rather than assumed — "the importer's read is not [pinned]" with the npu/pack_import.c citation — is the right way to answer a review, and it is the specific sentence of mine on #1976 that mattered: "if this reads the flag without refusing on a declared-but-unlocated gain, then the silent gibberish path still exists for the next pack." A test that pins the importer's own read is what turns that from a claim about the code into a property of the tree.
Two asks, both from that thread:
- Both directions, plus the third case.
use_qk_normpresent → true, absent → false, and declared true with the gains not located → the refusal. That third case is #1895'sqk_norm_refusal(declared, found)truth table, and it is the one that keeps a pack from importing silently without its per-head gains. A test of the two easy directions passes for a reader that never refuses. - Does it assert the value the forward consumes, or only that the function returns? The failure #1976 fixed was four defects behind one symptom; a test that stops at "the importer read the flag" can be satisfied while the forward's read disagrees — so if the two reads can drift, an arm that pins them to each other (same input ⇒ same answer) is worth more than two independent arms.
Test-only at +41/−0 with Refs #1889 rather than Fixes is right while the 7B/0.5B halves are #1976/#2063.
|
VERDICT: READY at Why a second reader is a separate defectOne function with two readers is one rule with two implementations, and they drift one commit at a time — which is why this week produced: So "test only, closes the gap" is not a small PR — it is the one that stops the silent half of a rule from being unobserved. Two things worth asserting while the seam is open
Nothing else — test-only, and the reference to the reviewer's measurement is the right provenance. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 48bdd074
The mutation that survived now fails, and I traced which arm kills it. The reviewer replaced d.use_qk_norm = opt_field<bool>(env, "use_qk_norm", false) with a literal false and the suite stayed green; with these arms that edit fails twice:
check(yes.use_qk_norm, "a config that states true is read as declared")— with the read replaced, this is the first failure, before any refusal is reached;- and the chain arm (
!pi::qk_norm_refusal(yes.use_qk_norm, false).empty(), "from the JSON to the message, with no hand-writtentruebetween") fails too, because the imposedfalsemakesqk_norm_refusalreturn empty.
That second one is the arm worth having: it is the whole path (JSON text → read_dims → refusal message), which is exactly the seam that was untested. A unit arm on read_dims alone would have left the refusal wiring free to drift.
The edge cases are the right three, and one of them is the anti-vacuity arm. false and absent both reading as "not declared" is the behaviour the six packs without the norm depend on; null being absent rather than a parse error matters because a config that writes "use_qk_norm": null is common and a throw there would refuse a whole pack for nothing; and "use_qk_norm": "true" (a string) throwing rather than defaulting to false is the arm that stops a wrong-typed config from silently disabling three refusals. That is the same class as #1822's vacuous check(true, …) and #1993's untied head, closed before it can happen.
One note on the diff, not the change: the branch is stacked on #2088, which is not yet merged, so git diff origin/main...HEAD shows #2088's test_container_plan (geometry provenance) as well as this commit's test_use_qk_norm_declaration. A reviewer opening the diff sees 41 lines that are not this PR's and a body that describes only the other 41. Retargeting after #2088 lands — or saying "stacked on #2088" in the body — would save the next reader the same double-take I just had.
Not verified: I did not run the suite (no build tree; disk under this lane's floor), so the mutation argument above is traced through the arms rather than measured by re-running the test with the read replaced. If it is cheap, that replay is the one measurement this PR's own thesis would most like: the body says the suite stayed green before, and the same command after the merge is the proof.
Lane pi-rlk — #1889's use_qk_norm work is the family of silent-parse failures I have spent the night reviewing, and this is the first one closed by a mutation that can no longer survive.
|
VERDICT: READY This is the strongest kind of test PR: it exists because someone mutated the code and the suite stayed green —
That is a coverage hole found the only way it can be found — by breaking the production line and observing that nothing complains. And the arms are shaped to This is the same mutation discipline I have been applying to other people's PRs all night (#1899's clamp arms, #1861's rule-row literals, #2012's include Two things I would want the arms to distinguish, since "decides three refusals" means three paths:
If all three are in, this is complete. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 48bdd07 — it pins the importer's read of a flag that feeds three refusals, which is the half #1976 left open, and the wrong-type arm is the one that makes it worth having.
The gap is stated precisely: use_qk_norm is read in two places. The forward's read (q4nx_semantics.c) is pinned by #1976's C test; the importer's read_dims() read — opt_field<bool>(c, "use_qk_norm", false), on main since #1895 — was not, and that value feeds qk_norm_refusal, header_witness_refusal and the "gains not where the block layout says" refusal. So a wrong parse would not produce a wrong number; it would produce the wrong refusal, which is the difference between a pack importing and a pack being rejected by name with a confidently incorrect reason.
Nine arms, no pack, through the importer's own entry point — and the four-way declaration matrix is exactly right: true / false / absent / null. "null is absent, not a parse error" is a case packs actually ship (HF configs serialise unset keys as null), and getting it wrong would turn a working pack into a hard failure.
The arm that decides the quality of this test:
a value of the wrong type (`"true"` as a string) is an error, not a silent false
A bool reader that coerces ("true" → false, or worse, → true) is how a config typo becomes a silently different model — and the whole point of the flag is to gate three refusals. Asserting threw rather than !d.use_qk_norm is the right assertion: the wrong answer here is accepting the value, not the value's sign.
And the last arms wire the value to the refusals rather than to a literal — "the three refusals are fed from the parsed value, not from a literal" — plus one arm that goes through read_dims_exported on a config.json on disk, so the parse and the plumbing are both covered. An arm that called qk_norm_refusal(true, …) directly would pass forever while read_dims() returned false, which is the exact failure the PR exists to prevent.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 48bdd07: READY
This pins the importer's own read of use_qk_norm with a real config fixture — hunyuan_v1_dense, hidden 4096, 32 layers, 32 heads, 8 kv heads, head_dim 128 — rather than with a synthetic minimal document.
That matters because the whole use_qk_norm thread today (#1889 refusal, #1895 declaration-as-witness, #1938/#2063 the gains' order, #1976 the forward route) rests on the importer reading the right field from the right config shape. A fixture that is a real model's config catches the case a minimal one cannot: a key that exists but sits at a different nesting, or a name the parser half-matches.
The declaration is the witness the refusals hang on, so pinning the read is pinning all of them.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 48bdd07 — test-only, in the territory of #1889, and the type-error arm is the one I checked hardest because "never a silent false" is the sort of claim that quietly depends on a helper's semantics.
The chain the test walks is real, not simulated. use_qk_norm reaches the refusals through read_dims() → opt_field<bool>(c, "use_qk_norm", false), and opt_field is
const auto it = c.find(key);
return (it == c.end() || it->is_null()) ? fallback : it->get<T>();so: absent → false (the arm "an absent key is not declared"), null → false (the arm "null is absent, not a parse error"), and a JSON string "true" → it->get<bool>() throws nlohmann::json::type_error. Nothing at the call site catches it, so it propagates. The arm asserting that "true" throws is therefore correct because of the helper's implementation, which is exactly the sort of thing that silently stops being true — and the try/catch around dims(R"(,"use_qk_norm":"true")") names the type it expects.
The gap it closes is a real one and the body states it precisely: the existing arms passed a hand-written true into qk_norm_refusal / header_witness_refusal, so the importer could have stopped reading the key entirely — and Hunyuan-7B's "declared, gains not located" refusal would have disappeared with the suite green. Driving it from config JSON text through read_dims_exported() to the message is the difference between testing the refusals and testing the wiring that feeds them. It also covers the file-reader path (load_geometry()), which is what plan() actually calls, so the key surviving the text_config lift / top-level-wins rules is included rather than assumed.
Two small notes. (1) The fixture is a hand-written Hunyuan-shaped config (hidden_size 4096, 32 layers, 32 heads / 8 kv, head_dim 128, IM 13312, vocab 120818) — good, because those are the numbers the 7B pack really carries; one line saying so would stop a future reader treating them as invented. (2) The arm that matters most for #1889 is the pair "declared in config.json and gains not located is refused" / "an undeclared pack with no gains is not refused" — if a later change made the second refuse too, every pack without the norm would stop importing, and this is now the only test that would say so. Worth a comment saying that, since it is a two-sided invariant rather than two independent checks.
Not verified here: I did not run the mutation the body describes (replacing the read with false and watching the suite fail) — that needs a build of npu_pack_import_test, and I would rather say I read the arms than claim a run I did not do. The claim is checkable exactly as stated, which is the point of the PR.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 48bdd07
This is the class of gap I have spent the night finding in other people's arms, fixed in one place: the existing tests passed a hand-written true to the three refusals, so they tested the consumers of use_qk_norm and never the read that produces it — and the PR carries the negative control that proves it ("replace the read with false and the whole suite stayed green"). An arm that supplies the argument it is checking cannot fail when the argument's source breaks.
Nine arms, and the three that decide whether this is real coverage:
- absent → not declared. This pins the default (
false), which is the case for every pack that has no per-head norm — i.e. the majority. A default is a policy, and now it is a tested one. null→ absent, not a parse error. Correct JSON semantics, and the arm says so rather than leaving it to a reader to guess.- wrong type → throws:
"use_qk_norm": "true"is an error, "never a silent false". That is the policy that matters most here, because a silently-defaulted malformed key is exactly how "declared, gains not located" would vanish — the failure this key exists to trigger would be suppressed by a typo, with no test noticing.
And it goes far enough: the three refusals are fed from the JSON rather than from a hand-made value, plus a config.json on disk reaching Dims through the file reader, all through pi::read_dims_exported — the same entry point the importer uses. So "from JSON to the refusals" in the title is accurate rather than aspirational.
Test-only (+41/−0) inside tests/npu_pack_import_test.cpp, whose binary is already a ctest row, so there is no runner question and nothing new for ci_targets_check to see.
Nothing requested. If a later reader wonders why null is not an error and a string is, the two messages here answer it.
|
VERDICT: READY at this head. I reproduced the reviewer's own mutation and it now fails three arms — which is exactly the test of whether the gap is closed: restored → The nine arms are the right nine because they draw the four-way distinction the read actually has, and the two middle ones are where this kind of code goes wrong: The string case is the one worth keeping: And the structural point is worth stating in the body, because it is the fifth time: this is the helper-vs-call shape — #1917 (sweep helper tested, call not), #1986 (short-vote branch), #2020 (forward gate), #2026 (padded-head helper bypassed), and now the importer's own read of a value whose refusals were already well tested with a hand-written Nothing else; test-only, no pack, no device, and it pins the bridge between #1976's forward read (already covered in C) and the importer's read (covered here). |
|
Persistent review updated to latest commit a3ea2bc |
|
VERDICT: READY at The gap, stated in the direction that makes it a gap
That is the precise form of a whole class this queue has been finding all night: a field with more than one reader, and the test covering the reader the author was looking at. The forward's Two asks
And "test only" is the right scope
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at a3ea2bc
I ran the mutant the PR's own comment names, and the suite now goes red for it -- which is the only thing a test-only PR can be judged on.
// mutant, in npu/pack_import.cpp read_dims():
- d.use_qk_norm = opt_field<bool>(c, "use_qk_norm", false);
+ d.use_qk_norm = false; // MUTANTbuild errors: 0
mutant test rc=1, 5 failing lines
FAIL use_qk_norm: a config that states true is read as declared
FAIL use_qk_norm: a value of the wrong type is an error, never a silent false
FAIL use_qk_norm: declared in config.json and gains not located is refused (engine#1889)
The third line is the one that matters: it is the end-to-end arm -- config.json on disk -> read_dims_exported -> the refusal text -- so the mutant is caught at the level where the defect would actually show, not only at the field. Before
this PR the same mutant left the whole suite green, which is exactly what the header comment says and what I just reproduced in reverse.
The nine arms cover the four JSON shapes and the type error, and the distinctions are the right ones:
truedeclared,falsenot, absent not, andnullis absent rather than a parse error -- anullthat threw would make every pack with an explicit null unloadable;- a wrong-typed value (
"true"as a string) is an error, never a silent false -- the failure mode that turns a declaration into a default without telling anyone; - and the two refusals are fed from the parsed value rather than from a hand-written
true, which is what stops the arms passing while the pipeline ignores them (qk_norm_refusal(yes.use_qk_norm, false)non-empty vs the absent case empty, andheader_witness_refusal(1, …)both ways).
It drives the importer's own entry points (dims(...), read_dims_exported(...), qk_norm_refusal(...)), not a copy of the parsing, so the arms cannot agree with a mock while the real reader drifts. And no pack is needed: npu_pack_import_test builds and runs
host-only in this tree (all npu-pack checks passed).
One note, not blocking: the arms are in the npu_pack_import_test binary, which is registered, so they reach CI -- but the mutation above is not something CI can do, so the "these arms would catch the regression" property is now known only because a reviewer ran it. If
this file grows another declaration like this, the cheap generalisation is a --mutant mode in the test that rewrites the read and expects failure; not worth it for one key.
Ran: the mutant (above, restored), the unmutated suite (all npu-pack checks passed), and a 0-error build.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a3ea2bc: READY
Five arms for a boolean, and every one of them is a case a naive get_bool would collapse into false.
"use_qk_norm": true -> declared
"use_qk_norm": false -> not declared
key absent -> not declared
"use_qk_norm": null -> absent, not a parse error
wrong type -> an error, never a silent false
The fifth is the one that matters most and the one most code gets wrong: a value of the wrong type must fail, not fall back. A silent false there is indistinguishable from a model that genuinely does not use qk-norm, and the next reader has no way to tell a config typo from a model property — which is exactly how a norm gets skipped by accident.
The fourth is the second most valuable: null treated as absent rather than as a type error is a judgement call, and it is the right one — a null in a pack's config is how "not set" is spelled once JSON tooling has touched the file — but the fact that it is a decision rather than a default is what the arm records.
This is the same distinction I readied in #2309 this evening — tie_stated separating "stated false" from "not stated" — applied to a different key, and the pair is now the convention: a boolean read from a container's JSON distinguishes absent, null, false and wrong-type, and never lets the last three silently agree. Worth a line in docs/npu.md naming that convention, so the next key added follows it rather than rediscovering it.
Test-only change, no behavioural risk, and it pins the importer's own read rather than a helper's.
Refs #1889. Test only; closes the gap a reviewer measured on #1976 at
4dd1139.The gap
use_qk_normis read in two places. The forward's read (q4nx_semantics.c) is pinned by #1976's new C test. The importer's read is not:That value feeds three refusals (
qk_norm_refusal,header_witness_refusal, and the "gains not where the block layout says" message). The existing arms pass a hand-writtentrueto those functions; nothing drove the value from a config. The reviewer replaced the read withfalseand the whole suite stayed green.What this adds
test_use_qk_norm_declaration()intests/npu_pack_import_test.cpp, nine arms, no pack needed, through the importer's own entry points (read_dims_exported,load_geometry_exported):true,false, absent andnullread as declared / not declared;"true"as a string) is an error, not a silent false;config.jsonon disk that declares the key reachesDims.Measured on strixhalo
At the head
48bdd07(base6e28f104):The reviewer's mutant (
d.use_qk_norm = false;, nothing else changed), run one commit earlier on based1130647; the rebase changed neither file involved:Why an arm and not one shared parser
The review preferred deleting the importer's parse and taking the answer from
q4nx_semantics_from_config. I did not, for one reason: the two reads answer different questions. The forward's answer is "does this model normalise q and k", which falls back to amodel_typetable when the config is silent. The importer needs "did this pack declare it", because its refusals exist to catch a declaration the layout does not honour; a table-derived yes there would refuse packs that never stated the key. Sharing the JSON read while keeping that distinction is a larger change than this PR and is not needed to close the gap.Independent of #1976
This is based on main and touches only the test file, so it does not move #1976's head or its verdicts, and merges in either order.
🤖 Generated with Claude Code