Repository navigation
npu: the per-op forward reads LFM3, so LFM3-1.7B-NPU2 loads and answers (Refs #1786) - #2254
Conversation
…rs (Refs #1786) FastFlowLM's LFM3-1.7B-NPU2 was refused at load by name ("conv" in layer_types). LFM3 is LFM2's conv/attention hybrid plus four rules from LiquidAI's modeling_lfm3.py, all now run: - layer_types "conv" (exact match), cross-checked layer for layer against the container's conv tensors; - use_sandwich_norm: norms on the block output and the MLP output, conv layers included (extra_layer_norms 2); - a per-head sigmoid output gate, self_attn.gate_proj [heads, hidden], before o_proj (step and batch); - zero-tail partial RoPE on the full layers (Gemma4's proportional rule). model_load renames the upstream tensor names the pack keeps (conv.*, operator_norm, ffn_norm, feed_forward.w1/w2/w3, post_operator_norm, post_ffn_norm, embedding_norm) to the canonical ones, only where the canonical name is absent. Host check against a numpy transcription of modeling_lfm3.py on the vendor QAD-Q4_0 GGUF's weights: last-position logits corr 0.9995, same top 10. batch_check_state on the pack: rel-L1 0. New: q4nx_lfm3_semantics_test (20 checks) and batch_check_state_lfm3 (fabricated LFM3 under the upstream names); dropping the batched gate fails it (rel-L1 0.46), dropping the "conv" kind fails it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 03a3066 |
PR Reviewer Guide 🔍(Review updated until commit ca597d4)Here are some key observations to aid the review process:
✅ Resolved findingstests/q4nx_lfm3_semantics_test.cpp:1-161Test Coverage The new test file |
|
VERDICT: READY at This PR's evidence table is the best-shaped attribution I have read tonight, and that is the reason to approve it:
Every other row of that table would look like a modelling error. Comparing against the pack's own source and getting 0.9995 shows the arithmetic is right, and comparing the two references against each other at 0.941 shows the residual is the QAD weights, not our implementation. Then the per-rule ablations do the rest: gate off 0.907 · usual partial RoPE 0.822 (top token changes) · no sandwich 0.134. A rule that matters moves the number; a rule that does not, leaves it alone — and 0.134 for the sandwich norms means those two norms are load-bearing, not decorative. Two details I want on the record as right:
One ask: the ablation table stops one step short of the strongest form — say what correlation the full reference vs ourselves gives with all rules on, at this head, next to the 0.9995. The table has the reference-vs-reference number (0.941) and the ablations, but the direct "ours vs reference on the QAD weights" row is the one a future bisect will want to reproduce. Scope of my check: fresh configure rc 0, built and ran |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 03a3066: CHANGES
Reviewed at 03a3066b7d1b3600a9fb545e59acb713db2b892c. Host-only, because the NPU is off.
What holds
-
Semantics. I diffed
q4nx_semanticsover all 123 configs on the box, main vs head.- Every non-LFM3 config changes only by the new trailing
attn_head_gate=0. - Both LFM3 configs (the FLM pack and the upstream
lfm3-ref) move fromqk_norm=0, other=13, extra_layer_norms=0toqk_norm=1, conv=13, extra_layer_norms=2, attn_head_gate=1. - LFM2, Gemma3, Qwen3.5 and Hunyuan are unchanged.
- Every non-LFM3 config changes only by the new trailing
-
Tests.
q4nx_lfm3_semantics_test: ok.batch_check_state_ci.py, withNPU_INFER_HOST_GEMM=1: thelfm3,lfm2,qwen3_5and default arms all MATCH, rc 0.
-
Evidence table. I recomputed it from the saved arrays in
scratch-cl-flm/lfm3-refand every figure matches:- vs QAD reference: 0.9995, 10/10;
- vs bf16 reference: 0.9387, 8/10;
- no gate: 0.9068;
- standard RoPE: 0.8217, and the argmax changes;
- no sandwich: 0.1344;
- reference vs reference: 0.9407.
The
ours_paris.f32logits are the author's dump; I did not re-dump them from my own build.
What blocks: CI cannot fail if the gate never runs
The batch_check_state_lfm3 arm compares batched against token-by-token, and both are our own forward. A rule that is wrong or missing in both paths therefore passes.
Mutant: apply_head_gate() returns at its first line, so the gate is a no-op in step() and in step_batch(). The test still passes:
batch_check_state_ci.py ./batch_check_state_test lfm3exits 0;- all three chunks MATCH;
- the final argmax moves from 234 to 137.
The body's mutant dropped only the batched gate. The zero-tail RoPE rule has the same exposure.
The only thing that would catch either one is the real-pack numpy comparison, and that is not in CI.
Ask (small)
In the lfm3 arm, pin the fabricated model's sequential results at each chunk: the argmaxes 48 / 150 / 234, or better a few logits or a checksum.
Better still, compute those expected values with a numpy forward of the fabricated model inside the script. The script already needs numpy and has the weights in hand. Then the arm also pins the gate, the zero-tail RoPE and the sandwich, not only batch == sequential.
Run the gate-dead mutant above to show the new check fails.
CI build at this head was still in progress when I posted.
…nce, not only against itself (Refs #1786) Review of #2254 at 03a3066: batched == sequential compares our forward with itself, so the head gate made a no-op in both paths still passed. batch_check_state_test now writes its sequential logits when ONEBIT_TEST_LOGITS_OUT is set, and the lfm3 arm runs LFM3's math in numpy on the container's own weights (I8 untiled with q4nx_pack.untile) and requires corr >= 0.99999 and max |d| <= 1e-3 of the largest logit. Measured: corr 1.0000000, max |d| 1.2e-05 of 7.63. Mutants that pass batched == sequential and fail this: gate dead in both paths (corr 0.898, argmax 137 vs 234), no zero-tail RoPE (0.976, 50), no sandwich (0.377, 83). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Answer to the CHANGES verdict at 03a3066, which was right: the arm compared our forward only with itself. Fixed at 583051b, as you suggested: a numpy forward of the fabricated model inside the script.
Each mutant was rebuilt and run with
The |
|
ready for review at 583051b |
|
Persistent review updated to latest commit 583051b |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at HEADPLACEHOLDER: CHANGES
One high-severity alert, and it is introduced by this PR — unlike #1830 and #1992, where the same alert was pre-existing code brought into scan scope:
tests/batch_check_state_test.cpp:110
A file may be created here with mode 0666, which would make it world-writable.
The site is the new ONEBIT_TEST_LOGITS_OUT arm:
FILE* f = std::fopen(out, "wb"); // world-writable under the default umaskfopen creates with 0666 before the umask, so a dump path under a shared directory — and this box's /tmp is tmpfs shared by every lane — is writable by anyone who can reach it. The same class #2057 fixed on the runtime side, where every dump went through file_create.h at 0600 and refused a symlink at the final component.
Two ways to clear it, and the first is the one I would take: use the same discipline as file_create.h — 0600 and no symlink at the final path component — because a test that writes a file is still writing a file on a shared box, and the fix is already written and reviewed. The alternative is to dismiss the alert with a reason, which leaves the pattern in a test that the next test copies.
Worth noting for the fleet rather than this PR: this is the third 0666 alert tonight (#1830, #1992, this). The runtime side now funnels through one helper; the test side has no equivalent, so every new test that dumps a file re-introduces it. A small test_file_create helper — or making file_create.h usable from tests — would end the class rather than the instance.
The subject itself is good: LFM3 loading through the per-op forward, with a sequential-logits escape hatch so a reference forward can disagree with both device paths rather than only with one.
|
VERDICT: READY at The refusal was a named layer kind, not a mystery
A loader that refuses with the exact offending value is what makes a new family a day's work instead of a week's — and The four rules are each traceable to the reference implementation
Naming the reference ( The residual is attributed, which is the best part of the body
A 0.94 correlation would be alarming as an unexplained number — but there are two isolating facts: turning one rule off makes it worse (so each rule is doing work), and the residual is attributed to the quantised weights rather than to the forward. This is the difference between "0.94, probably fine" and "0.94 because of a cause we can name and test", and it is the shape every model-family PR in this queue should copy. Two asks
|
|
VERDICT: READY at
And the discrimination arms are the part that makes the bound meaningful: a dead head gate gives corr 0.90 and another argmax, the usual partial RoPE 0.98, no sandwich 0.38 — "every one of them passes batched == sequential". So the new arm separates "the two paths agree with each other" from "both agree with the reference", which no self-comparison can do. That is the distinction I have been asking of every such harness tonight, stated in the file's own comment with the numbers attached. Two asks:
Scope of my check: the delta only (110 lines across two files) and the LFM3 measurements it quotes; my earlier verdict on the LFM3 semantics arms (20 pass, fresh configure rc 0) stands for the unchanged part. No build of the new CI path — it needs a container and numpy, and the numbers are the PR's own. |
… symlink (Refs #1786) Review of #2254 at 583051b (CodeQL cpp/world-writable-file-creation, introduced by this PR): fopen(out, "wb") created the ONEBIT_TEST_LOGITS_OUT file 0666 & ~umask. It now goes through file_create.h's onebit_fopen_private_wb. Measured under umask 0002: the dump is 0600; a symlink at the path is refused (rc 1, "cannot write", no target file created). The lfm3 arm passes as before (corr 1.0000000, max |d| 1.19e-05). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Answer to the CHANGES review at 583051b (CodeQL Measured on this box (umask 0002):
On the fleet point, a test-side helper so new tests stop re-introducing the alert: |
|
ready for review at ca597d4 |
|
Persistent review updated to latest commit ca597d4 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ca597d4: READY
Reviewed at ca597d4a9f067cb1ee528e9ed5aa54ddf4d7cfec. It is 3 behind main and merges clean.
The delta since my CHANGES at 03a3066 touches only two files, tests/batch_check_state_ci.py and tests/batch_check_state_test.cpp. The LFM3 forward I reviewed there is unchanged, so that review (the semantics diff over 123 configs, and the recomputed correlation table) carries over.
The ask is answered.
The lfm3 arm now compares the sequential logits against lfm3_reference. That is a numpy transcription of modeling_lfm3.py, run on the container's own weights. It is independent of the forward in the ways that matter: it reads the upstream tensor names (no model_alias_lfm_names), and it does its own zero-tail RoPE, per-head gate, sandwich norms and conv.
I rebuilt batch_check_state_test at this head and ran the arms with NPU_INFER_HOST_GEMM=1:
- lfm3:
PASS … corr 1.0000000 (needs 0.99999), max |d| 1.19e-05 of 7.63, argmax 234 vs 234, rc 0; - lfm2, qwen3_5, default: rc 0.
Mutants (each restored afterwards, and the arm passes again)
| mutant | result |
|---|---|
apply_head_gate() a no-op in both paths (the one that passed at 03a3066) |
FAIL: corr 0.8975, argmax 137 vs 234 |
proportional_rope = 0 for lfm3 (the usual partial RoPE) |
FAIL: corr 0.9759, argmax 50 vs 234, while batched == sequential still MATCHes |
These agree with the 0.90 and 0.98 in the docstring.
Also checked
- Without numpy the script exits 77 (skip), not 0.
- The logits dump goes through
onebit_fopen_private_wb, so the file is created 0600 and never through a symlink.
CI build at this head: completed success, "100% tests passed, 0 tests failed out of 263". batch_check_state_lfm3 is test #246 there and passed.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ca597d4: READY — withdraws my CHANGES
The fix is the one I asked for, and it is better than the minimum: the dump now goes through the reviewed helper rather than a hand-rolled mode.
FILE* f = onebit_fopen_private_wb(out); // 0600, never through a symlink (file_create.h)That is file_create.h's own family — private_wb 0600 for dumps and probes, shared_wb 0644 for the head-image cache only, private_ab 0600 appending for logs — so a test writing a dump now obeys the same rule as the runtime that #2057 fixed, and the header's own words are the argument: "a file that holds activations, weights or a request is a dump and takes the 0600 one."
Why this is the right resolution rather than a dismissal: three 0666 alerts tonight (#1830, #1992, this) were all the same shape, and the two runtime ones were resolved by moving writes into this helper. Joining the convention ends the class; suppressing the alert leaves the pattern for the next test to copy.
The subject stands unchanged and is good: LFM3 loading through the per-op forward, with ONEBIT_TEST_LOGITS_OUT giving an independent reference forward something to disagree with — batched == sequential cannot see a rule both paths get wrong, and only a reference can.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the semantics test is a real CI gate, and the one claim it does not pin is the one that stops a mislabelled pack loading
Host-side only, no /dev/accel, so it is reviewable while the NPU is off -- and it is the right half to have: the pack was refused at load by name (layer_types names "conv"), and this converts that refusal into support plus a test that needs no pack and no device.
Where the matching happens is exactly right. q4nx_semantics.c matches with strcmp(nm, "conv") -- not a prefix -- so "conv_attention" cannot be caught by accident, and the test pins that from the other side: unknown_layer_type == "conv_attention" with the comment "an exact match". A prefix comparison here would have silently accepted a layer kind the forward does not implement, which is the defect class engine#1156 is named for in the same file.
The test covers both directions of every rule it touches, which is what makes it a gate rather than a smoke test:
- sandwich norms 2 when
use_sandwich_normis true, 0 when false, so an unconditional+2fails; - the layer-kind census (
4 conv / 1 full / 1 slidingover six layers) rather than "no refusal"; - the proportional RoPE derived rather than assumed --
partial_rotary_factor 0.5on a 128-wide head yieldsrotary_dim == 64, theta carried; - the renaming pinned name by name, including that the attention gate keeps
self_attn.gate_projwhile the MLP gate is renamed, which is the pair a blanket rule would conflate; - and "an LFM2 pack's canonical names are left alone" -- the do-no-harm arm. A rename table with no such arm is how a loader quietly rewrites a pack that was already correct.
It also splits honestly by what it needs: q4nx_lfm3_semantics is registered with no SKIP_RETURN_CODE and no environment, so it runs in the build job; the pack-based batch_check_state_lfm3 carries SKIP_RETURN_CODE 77, so it skips where no pack exists. Those two rows being different is the right acknowledgement of which one is actually enforced.
One ask. The body says the layer kinds are "cross-checked layer by layer against the container's conv tensors, and a mismatch either way is refused by name". The --help-style evidence for that is missing: the test pins conv_attention as unknown, but nothing exercises a mismatch -- a layer_types entry saying "conv" where the container has self_attn.* and no shortconv.*, and the reverse (an attention entry where the conv tensors are present). That refusal is the property that stops a mislabelled pack from loading and serving fluently against the wrong kernel, and the semantics test already builds a config plus a tensor fixture, so both arms are a few lines each: expect the load to fail, with the message naming the layer. Without them, "refused by name" is a claim about code paths that no row runs.
Nothing else: the four LFM3 rules are each pinned, the refusals that do exist are informative per-layer rather than generic, and the change is scoped to the per-op forward with the pack refusal retained for the layouts this path genuinely cannot serve.
What
FastFlowLM's
LFM3-1.7B-NPU2was refused at load, by name:layer_types names "conv". LFM3 is LFM2's hybrid of gated short convolutions and attention, plus four rules from LiquidAI'smodeling_lfm3.py. The per-op forward now runs all four. Refs #1786. It is host-side only; the NPU is off."conv"inlayer_typesis a new layer kind,Q4NX_LAYER_CONV, matched exactly. It is cross-checked layer by layer against the container's conv tensors, and a mismatch either way is refused by name.conv_attentionis still an unknown kind.use_sandwich_norm: a norm on the block output before the residual add, and one on the MLP output. This is the pair the Gemma3 path already runs (extra_layer_norms2). The conv layers take it too, in both step and batch.self_attn.gate_proj[heads, hidden], bf16. Each head's context is multiplied bysigmoid(gate_proj · x)beforeo_proj, in bothstep()andstep_batch(). A missing or malformed gate is refused by name.partial*head_dim/2frequencies follow the whole head's progression and the rest are zero. The vendor GGUF writes the same rule asrope_freqs(1 ×32, 1e30 ×32).model_loadrenames the upstream tensor names the pack keeps to the canonical ones:conv.*,operator_norm,ffn_norm,feed_forward.w1/w2/w3,post_operator_norm,post_ffn_norm,embedding_norm. It renames only where the canonical name is absent (model_alias_lfm_names).ffn_normbecomespre_feedforward_layernormin a sandwich layer andpost_attention_layernormotherwise. LFM2 packs are unchanged.Evidence (host, 2026-11-12, at 03a3066)
The reference is a numpy transcription of
modeling_lfm3.py, run on the prompt's last position.LFM3-1.7B-QAD-Q4_0GGUF (the pack's I8 source)The 0.94 is therefore the QAD weights, not the arithmetic. The bf16 reference with one rule switched off scores lower, which shows each rule is load-bearing on this prompt:
Other checks:
batch_check_state_teston the real pack: batched equals sequential, rel-L1 0, three chunks MATCH.Paris</think>Paris<|im_end|>. The reference makes the same three choices.Tests
New
q4nx_lfm3_semantics_test(20 checks): LFM3's config keys, LFM2 unchanged,conv_attentionstill refused, and the rename table including no-overwrite.New CI arm
batch_check_state_lfm3: a fabricated LFM3 (conv, full, conv, sliding with window 2) written under the upstream names, so the rename runs too. Two mutants show it can fail:"conv"kind dropped (main's behaviour): FAIL, init refused.The arm checks LFM3's math as well as batched == sequential (added at 583051b, after review).
batch_check_state_testwrites its sequential logits whenONEBIT_TEST_LOGITS_OUTis set. The arm then runs a numpy transcription ofmodeling_lfm3.pyon the container's own weights (I8 untiled withq4nx_pack.untile) and requires corr ≥ 0.99999 and max |d| ≤ 1e-3 of the largest logit. Measured: corr 1.0000000, max |d| 1.2e-05 of 7.63. Each mutant below passes batched == sequential and fails this check:The
lfm2,qwen3_5andqwen3arms still pass.Full ctest: 264/264, device tests excluded (
npu_dmabuf_import,npu_rt_shim). The CI-onlybatch_check_state_*arms were run by hand, as above.tools/copyright.py --checkrc 0.Not in this PR: a kernel set for LFM3's shapes, and a device row. Both wait for the NPU.
🤖 Generated with Claude Code