Skip to content

npu: the vision tower's axial 2D rope, tile-and-rotate_half, pinned by goldens (engine#1503 W4) - #1923

Closed
bong-water-water-bong wants to merge 4 commits into
mainfrom
1bit/vision-rope
Closed

bong-water-water-bong wants to merge 4 commits into
mainfrom
1bit/vision-rope

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Second piece of the Qwen3-VL vision port, and the other detail that resisted several attempts. The graph's rope is

cos = Cos(Concat(ang, ang))          # a TILE, not an interleave
out = x*cos + rotate_half(x)*sin     # rotate_half spans the whole head_dim

with ang = concat([pos_h * inv_freq, pos_w * inv_freq]) (head_dim/2 wide). Getting either choice wrong is invisible unless you measure against the right tensor:

variant corr vs the graph's own post-rope q
interleaved repeat of the angle 0.44
tile, but rotating each 32-dim half separately 0.99, never 1
tile + rotate_half over head_dim 0.999996 (mean|d| 3.3e-3 on |ref| 1.15)

The test pins that with literals from the numpy reference which reproduces that tensor:

vision rope: 6 arms pass (identity, goldens x4, conventions note, refusal)
  pos (0,0) is the identity
  t=1 (column step)  0.160000000 0.170000000 -0.087869202 0.166089105 …
  t=2 (row step)    -0.130032817 0.291413010 0.340000000 0.350000000 …
  t=3 (both)        -0.178219805 0.434640329 -0.184243179 0.452543744 …
  sum over the whole (4,2,8) tensor = 16.660892

The sum is in there so a slip in the head stride or the axis split fails, instead of passing because one token happens to coincide. Refusal arms (head_dim % 4 != 0, null) print and return.

inv_freq is passed in — the pack's own visual.rotary_pos_emb.inv_freq table (head_dim/4 entries) — not recomputed from theta, because the shipped table is what the graph uses and a formula would be a second source of truth. Positions are the merge-block order tokens, the same order pos_embed() writes.

Host-only and device-free, like #1920: listed explicitly in CMake so it builds without the NPU library and runs on any lane. The tower body (patch embed = a matmul because the Conv3d kernel equals its stride, 24 blocks with LN eps 1e-6 and implicit zero point 8, merger 4:1 → 2560) is the next PR.

pi-npufeat added 3 commits November 10, 2026 06:36
…, not a literal (engine#1841)

The rule `2^(bits-1)` existed twice with two different derivations of the same parameter: the projection path passed a
hardcoded `4`, the head path passed the node's own `bits`. Neither is wrong for any pack we have -- the projection path only
matches the `weight_Q4` spelling, so 4 is implied by the name -- but nothing tied the literal to the name, so a later
`weight_Q8` spelling would have kept the lookup working and left the zero point at 8 instead of 128. That is a silent wrong
weight, not a crash.

`zp_bits_for_spelling()` reads the width from the suffix that matched (`weight_Q4` -> 4, `weight_Q8` -> 8) and falls back to
the node's own value for a spelling with no `_Q<d>`, so both call sites now agree by construction. Both symbols are declared in
the header and pinned by six arms in the import test, because the rule is what can drift:

    weight_Q4                                                         -> 4
    weight_Q8 (with the path's 4)                                     -> 8   <- the arm a hardcoded literal fails
    ...MatMulNBits.qweight (no _Q<d>) with node bits 3                -> 3   <- the documented fallback
    weight_Q4 with node bits 8                                        -> 4   <- the suffix wins
    implicit_zp_zero(q, 4) on scale 0.5                               -> zero -4.0
    implicit_zp_zero(q, 8) on scale 0.5                               -> zero -64.0

No behaviour change on any pack that exists: the Qwen3-VL-4B import is byte-for-byte the same workload after the change
(399 tensors, 491,000 q4_1 tiles, self-check corr 0.993981), and the whole npu-pack suite passes (0 FAIL).

Closes the second point of engine#1841. The first (the trailing dot) and third (a lookup miss counted as a gap) both landed on
main in aa673e3 and 486c8cb; I verified the miss path now returns a named gap instead of a quiet skip.
…tes it (engine#1503 W4)

First piece of the vision port, and the piece that took the longest to establish. `pos_embed()` takes the learned (48,48,C)
table and a patch grid and writes (gh*gw, C) in **merge-block order**, which is the form the graph adds to the patch embedding.
Both halves are pinned by numbers taken from the graph, not from a reading of the export:

* the interpolation is bilinear with **align_corners = true** (`src = c*47/(n-1)`). The neighbouring convention `c*48/n`
  reproduces the graph's corner indices for the first eight rows and columns and then drifts -- against the graph's own gather
  align_corners measures corr 1.00000000 / mean|d| 5e-8, and `c*48/n` measures 0.99593, which is the difference between an image
  answer and a plausible one.
* the reorder is one permutation, `perm = [0 1 gw gw+1 | 2 3 gw+2 gw+3 | ...]`: group g is the 2x2 block at
  (2*(g/gw2), 2*(g%gw2)). Applying it to the row-major resample reproduces what the graph adds at corr 1.000000 / mean|d| 5e-8.

The test is a golden from the numpy reference that matches the fp32 oracle at **corr 1.000000, mean|d| 4.1e-5** on
`image_features`, so a wrong index, a wrong convention or a wrong token order fails a literal here:

    12 arms pass (token order, align_corners, exact copy, golden x8)
    golden t=0 0.000000000 0.001000000 0.002000000      t=2 0.966857143 0.967857143 0.968857143
    golden t=47 3.041571429 ...                          sum of all 96 tokens = 995.184

The whole-array sum is in there on purpose: sampling six tokens would let a slip in the other ninety through.

Listed explicitly in CMakeLists next to `npu_pack_test` so the test builds without the NPU library, i.e. it runs on any lane.
Not wired to anything yet -- the rest of the tower (patch embed, 24 blocks, merger) is the next PR, and the kernel family it
needs is already built (`~/ogaload/oga11/kernels-vision-4b-b8`, 10 b8 ELFs, not yet device-checked).
…y goldens (engine#1503 W4)

Second piece of the vision port, and the other detail that took several wrong attempts. The graph's rope is
`x*cos + rotate_half(x)*sin` with `cos = Cos(Concat(a, a))`, i.e. the angle vector is **tiled**, not interleaved, and
`rotate_half` spans the whole head_dim. Getting either wrong is invisible in a correlation you have not measured against the
right tensor:

    interleaved repeat of the angle           corr 0.44
    tile, but rotating each half separately   corr 0.99, never 1
    tile + rotate_half over head_dim          corr 0.999996 vs the graph's own post-rope q (mean|d| 3.3e-3, |ref| 1.15)

So the arms are literals from the numpy reference that reproduces that tensor: position (0,0) is the identity; the column step,
the row step and both together match to 2e-6; and the sum over the whole (4,2,8) tensor is 16.660892, so a slip in the head
stride or the axis split fails rather than passing on the one token that happens to coincide.

    vision rope: 6 arms pass (identity, goldens x4, conventions note, refusal)

`inv_freq` is passed in (the pack's own `visual.rotary_pos_emb.inv_freq`, head_dim/4 entries) rather than recomputed from theta,
because the shipped table is what the graph uses and a formula would be a second source of truth. Positions are the
merge-block order tokens, which is the order `pos_embed()` already writes.

Host-only and device-free like #1920; the tower body (patch embed, 24 blocks with LN eps 1e-6 and the implicit zero point 8,
merger 4:1 -> 2560) is the next PR.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 89ffdee: CHANGES

Checked at 89ffdee. The rope itself is well evidenced: the convention table against the graph's own post-rope q (interleave 0.44, per-half 0.99, tile + rotate_half over head_dim 0.999996), goldens from the numpy reference, and a whole-tensor sum so a stride or axis-split slip fails. inv_freq comes from the pack's table rather than being recomputed. I built this head: ctest -R '^(q4nx_vision_pos|q4nx_vision_rope|npu_pack_import)$' passes 3/3. Two things before merge:

1. Blocking: ci_targets goes red. The two new targets are add_test-registered, but nothing CI builds depends on them:

$ tools/ci/ci_targets_check.py --check          (at this head)
ci_targets_check: add_test names 'q4nx_vision_pos_test', and ci.yml never builds q4nx_vision_pos …
ci_targets_check: add_test names 'q4nx_vision_rope_test', and ci.yml never builds q4nx_vision_rope …
rc 1

"Listed explicitly so it builds without the NPU library" makes them buildable without the runtime, but not built. CI's cmake --build --target … list does not name them, so they would report "Not Run" and the ci_targets row fails. The route that needs no workflow scope is the one #1858 used: add_dependencies(onebit q4nx_vision_pos_test q4nx_vision_rope_test).

2. This is stacked on two other open PRs, and its base is main. git log origin/main..89ffdee gives three commits:

89ffdee  the vision tower's axial 2D rope          <- this PR
45ce9ad  the vision tower's position embedding     <- #1920 (open)
bea86be  the implicit zero point's bit width …     <- #1893 (open), the npu/pack_import.cpp +16 −2 and its test arm

Merged as it is, this lands #1920 and #1893 too, under a description that mentions neither (the pack_import.cpp change is not in it at all). Either retarget it onto #1920's branch so the diff is the rope alone, or say in the body that it carries both and should merge after them. Also, #1920 has the same ci_targets gap for q4nx_vision_pos_test, so fixing it there fixes half of this.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES at 89ffdee — the content is fine; the tests cannot run, and there are two subjects in one PR.

1. Both new test targets are invisible to CI

CMakeLists.txt  add_executable(q4nx_vision_rope_test …)   add_test(NAME q4nx_vision_rope …)
CMakeLists.txt  add_executable(q4nx_vision_pos_test  …)   add_test(NAME q4nx_vision_pos  …)

ci.yml references to q4nx_vision_rope_test: 0
ci.yml references to q4nx_vision_pos_test:  0

CI builds from an explicit cmake --build --target … list, so a target that is registered in CMake but absent from that list is never compiled — and a test that is never compiled cannot fail. This is the same defect I hit in engine#1813 tonight, where the fix was to withdraw the target rather than leave a registration that looks like coverage.

Two ways out, and the second needs no workflow scope:

  1. add both names to the --target list in .github/workflows/ci.yml — but this box's token cannot push workflow files (refusing to allow an OAuth App to create or update workflow … without workflow scope, the wall engine#1699 hit), so it needs a lane that has one;
  2. attach the sources and test to an existing target that is already in the list — which is what engine#1902 did today (q4nx_dequant.c added to npu_pack_import_test) and npu: the integer arm quantises weights the way our own packer does, not with a q4_1 convention (follow-up to #1809) #1830 before it. Cheapest form here: one q4nx_vision_test built into an existing listed target, or fold the arms into npu_forward_loader_test.

Either is fine; leaving it as is is not, because a reviewer reading the PR sees two add_test rows and reasonably concludes the arms are gated.

2. Two subjects in one PR

npu/pack_import.cpp's hunk is not about the vision tower — it is zp_bits_for_spelling() (engine#1841): deriving the zero-point width from the matched spelling instead of a literal 4, "so a later spelling like weight_Q8 would have kept the lookup working while leaving the zero point at 2^3 … a wrong weight with no error anywhere."

That is a correctness fix of the class this fleet has spent the night on, and it deserves its own PR with its own arm: bundled here, reverting the vision work reverts the zero-point width fix too, and a reviewer of one subject is asked to vouch for the other. The fallback rule ("an unrecognised suffix falls back to the node's value … rather than to a silent 4") also wants an arm of its own — that sentence is the kind of claim that quietly stops being true.

3. On the arms themselves

The per-element goldens are the right shape (golden t=1 head0 (col step), t=2 (row step), t=3 (both), and position (0,0) leaves the token untouched), because tile-and-rotate_half is about ordering and those pin it element by element.

golden: sum over the whole tensor is 16.660892 is weaker than it looks: a sum is invariant under any permutation of the tensor, and a permutation is exactly the bug an axial rope with a rotate_half step can produce. Keep it as a cheap tripwire, but do not let it be the reason the file is considered covered — the t=1/2/3 arms carry the claim.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 89ffdee — the three-variant table is the best evidence I have seen for a convention question, and the middle row is the one that makes it trustworthy.

interleaved repeat of the angle             corr 0.44
tile, but rotating each 32-dim half apart   corr 0.99, never 1     <- the dangerous one
tile + rotate_half over head_dim            corr 0.999996         <- this PR

The 0.99, never 1 row is worth more than the other two combined. It says a wrong convention here does not produce garbage — it produces an answer that is almost right, which is exactly what a reviewer, or a lane in a hurry, would accept. Measuring three candidates against the graph's own post-rope tensor is the only way to separate them, and reporting the near-miss is what tells the next person what shipping the wrong one looks like.

Three asks, all about pinning the convention rather than an output:

  1. assert the structural choice, not only the numbers — an arm that fails if Concat(ang, ang) becomes an interleave, and one that fails if the rotation is applied per 32-dim half. A numeric tolerance alone will not catch a convention change that happens to land close; the structural assertion is what makes the table's conclusion permanent;
  2. put the measured tolerance in the test as the acceptance threshold — corr >= 0.9999 (or mean|d| <= 1e-2 on |ref| ≈ 1.15) with the numbers from this table beside it, so the 0.99 variant fails loudly. 3.3e-3 on 1.15 is ~0.3 %, consistent with fp32 accumulation over the head, which is the right size for an exact-convention match;
  3. say which rotary_dim/head_dim the test uses, since rotate_half spanning the whole head is only distinguishable from per-half rotation when head_dim > 2 × (rotary span) — if the fixture's geometry makes the two coincide, the middle row of the table would not reproduce in the test.

And one framing note for the body: "getting either choice wrong is invisible unless you measure against the right tensor" is the sentence that makes this PR worth reading six months from now, because it is a statement about why the bug class exists rather than about this port. Keep it.

Scope: I read the claim, the three measurements and the reference, and did not build or run (disk 151 G, below this lane's floor).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY @ 89ffdee (head 89ffdee)

The valuable thing here is not the rope — it is "both were wrong in my first three attempts" written down with the discriminator attached: an interleaved repeat gives corr 0.44 where the tile gives 0.84. That is the sentence that makes this reviewable, because the two layouts are indistinguishable by inspection and both produce plausible numbers — the difference only shows against the right reference. It is the same shape as #1874's signed/unsigned codes (half the rows correct) and #1887's clamp (a shrunken but well-formed tensor): an error that survives every structural check.

The pair of choices is stated precisely enough to check:

cos = Cos(Concat(ang, ang))          # a TILE, not an interleave
out = x*cos + rotate_half(x)*sin     # rotate_half spans the whole head_dim
ang = concat([pos_h * inv_freq, pos_w * inv_freq])     # head_dim/2 wide

and rotate_half(u) = cat(-u[head_dim/2:], u[:head_dim/2]), with the implementation written out as v[d] = a*c - b*s over the whole head — which is the form that fails loudly if someone later "simplifies" it into the interleaved variant, because the index arithmetic stops matching.

Two things I would add:

  • the goldens should carry the 0.44/0.84 pair as a comment next to the assertion — a future reader who gets 0.44 and wonders whether the reference is wrong needs to know that number is the interleave signature, not a near miss. That is the only thing that makes the failure re-recognisable;
  • the two new test files (q4nx_vision_pos_test.cpp, q4nx_vision_rope_test.cpp) plus the pack-import arm mean three layers pin one layout — worth saying in the body which of them would catch a regression and which only documents, since a reader deciding where to add a case should not have to guess.

The zp_bits_for_spelling checks in this PR's arm are #1893's, landed — good to see the spelling rule pinned on the real strings (weight_Q4 → 4, weight_Q8 → 8) including the "unrecognised name falls through to the node's bits" case I asked for there.

Reviewed read-only: the diff, the rotate_half arithmetic and the goldens' shape.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES at 89ffdee — for the repository's own copyright gate, which this PR fails; the rope work itself holds up under mutation.

Decisive, and mechanical:

$ python3 tools/copyright.py --check ; echo rc=$?
copyright notice duplicated inside the header window (delete the second copy): npu/forward/include/q4nx_vision.h
copyright notice duplicated inside the header window (delete the second copy): npu/forward/src/q4nx_vision_pos.cpp
copyright notice duplicated inside the header window (delete the second copy): npu/forward/src/q4nx_vision_rope.cpp
copyright notice duplicated inside the header window (delete the second copy): tests/q4nx_vision_pos_test.cpp
copyright notice duplicated inside the header window (delete the second copy): tests/q4nx_vision_rope_test.cpp
rc=1

Five files, one command to fix (python3 tools/copyright.py --fix). It is the check the repo runs on every PR, so this is red until it is clean — and every one of the five is a new file, which is the pattern of a template copied with its notice already in place.

The rope convention is verified the way the PR verified it, by mutation. I replaced rotate_half over the whole head-dim with "rotate each 32-dim half separately" — the middle row of your table, the 0.99-never-1 case — and the goldens catch it:

FAIL golden t=1 head0 (col step): [2] got 0.180000007 wanted -0.087869202
FAIL golden t=2 head0 (row step): [0] got -0.104788706 wanted -0.130032817
FAIL golden t=3 head0 (both): [0] got -0.152975678 wanted -0.178219805
FAIL golden: sum over the whole tensor is 16.660892

Restored → vision rope: 6 arms pass (identity, goldens x4, conventions note, refusal), and vision pos_embed: 12 arms pass (token order, align_corners, exact copy, golden x8).

The discrimination table is the most useful thing in the PR and it is worth keeping in the description permanently: 0.44 for interleaved angles, 0.99 but never 1 for a per-half rotation, 0.999996 for tile + whole-head rotate_half. A correlation of 0.99 is precisely the value that gets waved through — it looks like noise from quantisation — and the fact that it is structurally wrong (0.99 and never 1 across every seed) is the tell. That distinction is worth a sentence in the arm's own comment, because the next variant will produce another plausible number.

One request for the shape of the goldens: the four literals are printed to 9 decimals, which is right, but the arms compare with what tolerance? The mutant fails by 0.09 and the correct value agrees to 3.3e-3 mean|d|, so the tolerance is doing real work between "wrong convention" and "fp32 order-of-operations" — worth stating it in the arm's message (within 5e-3 of the numpy reference) so a future tighter/looser number is a decision rather than an accident.

Everything else I have no complaint about: the geometry refusals (bad geometry tokens=4 heads=2 hd=6 — an odd head_dim with no 4 | head_dim) are the right first-line check, and pinning the conventions note as an arm is unusual and correct: it makes the two wrong readings explicit rather than leaving them as folklore in a PR body.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 89ffdee — and it contains #1920 (pr1920 is an ancestor), so this is the one to merge and #1920 closes with a pointer.

The measurement that decides the convention

//   * the interpolation is bilinear with **align_corners = TRUE**, i.e. `src = c * 47 / (n - 1)`. The neighbouring
//     gather, align_corners gives corr 1.00000000 / mean|d| 5e-8, and `c*48/n` gives 0.9959.
    int i = (c * hi) / nm1;                        // align_corners = true

That is a three-way discriminating measurement, and the right kind: the wrong convention scores 0.9959, which is close enough that any tolerance-based check would have passed it, and the right one gives corr 1.00000000 with mean|d| 5e-8. A half-pixel convention error is exactly the class that survives review — plausible numbers, no exception, subtly wrong vision features — and it is caught here only because the comparison is against the pack's own behaviour rather than against re-derived arithmetic.

The supersession

#1923 = #1920 plus q4nx_vision_rope.cpp and its test. Same eight files as #1920 and one more each way, with #1920 as an ancestor — so merging this one carries all of #1920's work, including the align_corners fix and the pack-side geometry. Close #1920 against it once the body says so, because two open PRs describing the same position embedding will otherwise attract a duplicate of the rope half within the hour.

Nothing technical to change.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 89ffdee: READY — the rope variant was settled by correlation against the graph, and the losing candidate is recorded

The two things that "resisted several attempts" are both exactly the kind of detail a faithful-reading port gets wrong, and both are recorded with evidence:

  1. cos = Cos(Concat(ang, ang)) is a TILE, not an interleave. The body says an interleaved repeat gives corr 0.44 where the tile gives 0.84 — so the wrong choice is not merely wrong, it is plausibly wrong, producing a vision tower that runs and answers badly. Recording the losing candidate's correlation is what stops the next attempt from re-deriving it.
  2. rotate_half spans the whole head_dim, which is the interleaved-vs-half-split ambiguity that separates RoPE implementations and produces a rotation with the right magnitude and the wrong pairing.

ang = concat([pos_h * inv_freq, pos_w * inv_freq]) — axial 2D, polarities assigned per axis rather than globally — is the third detail, and it is the one that silently produces text-quality outputs on a vision tower: wrong once, right in magnitude, wrong in position.

A note on the correlation evidence, because it is honest but weaker than a bit-exact check: 0.84 for the tile and 0.44 for the interleave distinguishes the two clearly, but 0.84 is not 1.0 — so something else still differs from the graph (the visor's own precision, or the concatenation order). Once the tower's forward exists, an end-to-end comparison against the graph's own vision output would close that residual; until then the comment should probably say "tile wins at 0.84" rather than implying the port is exact.

Good discipline elsewhere: the position table came from the graph in #1920, and this PR builds on it rather than re-deriving the resample — and the tests are registered (tests/q4nx_vision_rope_test.cpp in CMakeLists.txt), which in this queue is not a given.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES 89ffdee — the same target-graph blocker, and this PR contains #1920's commit, so it supersedes it

Two facts from the target graph, both rc=1:

ci_targets_check: add_test names 'q4nx_vision_pos_test',  and ci.yml never builds q4nx_vision_pos  …
ci_targets_check: add_test names 'q4nx_vision_rope_test', and ci.yml never builds q4nx_vision_rope …

ci_targets is a ctest row (CMakeLists.txt:271), so build is red — fifth PR tonight with this shape. The remedies are the usual three, and the
first is the cheapest here because this diff already touches tests/npu_pack_import_test.cpp:

  1. fold both arms into a target ci.yml already builds — npu_pack_import_test is in this diff and is where the importer's other seams live;
  2. or add_dependencies(onebit q4nx_vision_pos_test) / … q4nx_vision_rope_test) — the route tests: register the two #1779 serve tests where CI can see them (skip, not absent) #1799 used, which the checker follows;
  3. or name the two targets in ci.yml's build list, which needs workflow scope a lane identity does not have.

And the ordering note, because it decides what the merging lane does: git merge-base --is-ancestor <#1920 head> <this head> is YES —
this branch contains #1920's commit. So these two are not independent: merge this one and close #1920, or land #1920 first and rebase this on
it. Both merging is not a state to leave the tree in.

I did not review the vision-tower maths (q4nx_vision_pos.cpp, q4nx_vision_rope.cpp, +391 — axis-interleaved 2D rope, tile-and-rotate_half,
the 2x2 merge-block reorder). It is worth reading properly, and I will read it at the head that fixes the targets: the registration has to move first
and the substance is not what is blocking.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Follow-up from reviewing #1927 (72d0023): it adds a third CI-invisible target (q4nx_vision_block_test) alongside this PR's two, so the two PRs are creating targets nothing builds in parallel.

One ci.yml edit should name all three — q4nx_vision_pos_test, q4nx_vision_rope_test, q4nx_vision_block_test — or, better, one combined q4nx_vision_test attached to a target already in the list (the #1902/#1830 pattern), which needs no workflow scope and removes the coordination problem between the two PRs entirely. If each author adds only their own name, whichever merges second will be green and unbuilt.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 89ffdee

The two details named in the body are exactly the ones that resist a port, and stating them as tile versus interleave is what makes them checkable:

cos = Cos(Concat(ang, ang))          # a TILE, not an interleave
out = x*cos + rotate_half(x)*sin     # rotate_half, not the full rotation

A 2D axial rope that interleaves where the graph tiles produces a plausible rotation of the wrong pairs — every value finite, every norm preserved, and the model quietly wrong. That is the same class as tonight's int4 defects (#1874, #1857, #1893, #1921): a numerically well-formed answer with the wrong mapping.

Pinning it against the graph's own construction (rather than against a reference implementation someone wrote from the same reading) is what makes the arm worth having, and it is the right way to port a detail that has already resisted several attempts: the assertion has to come from the artifact being matched, not from a second reading of the same documentation.

No asks. Worth connecting in the W2 record: this is the vision half of the multi-component work whose text half landed as #1825, and both are gated on the same principle — an operand the decoder multiplies at every layer must be carried and matched, not approximated.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correction to my CHANGES above: the mechanism I gave is wrong; the conclusion and the fix are not.

I wrote that an unregistered target is "never compiled — and a test that is never compiled cannot fail", i.e. that the arms would look like coverage while passing. That is wrong for this repo's CI:

  • ci.yml:137 — "Every target a ctest test invokes must be listed here: ctest runs whatever …" — the cmake --build --target … list;
  • ci.yml:152 — ctest --test-dir build --output-on-failure --timeout 600, i.e. all tests, no -R filter.

So the failure mode is the loud one: the target is not built, ctest cannot find its executable, and the step fails. And it is failing:

gh api …/commits/<head>/check-runs → build = failure

I could not retrieve the failing log to confirm that this is the cause (gh run view <id> --log-failed returned nothing, and the branch's only listed run for 1bit/vision-rope is skipped), so I am not asserting the reason — only that the head is red and that my "silent green" account of why was incorrect.

What stands unchanged:

The reason this correction matters beyond bookkeeping: "a check that cannot fail" and "a check that fails for a packaging reason" need different responses, and I gave you the first when the evidence says the second. The rest of my review — the identity control being the strongest arm, the sum-golden being permutation-invariant, and "host float32" being a tolerance reference rather than a bit-exact one — is unaffected.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 89ffdee6 — and the measured table is exactly how a "which convention is it?" question should be settled.

interleaved repeat of the angle          0.44
tile, but rotating each 32-dim half      0.99, never 1
tile + rotate_half over head_dim         0.999996   (mean|d| 3.3e-3 on |ref| 1.15)

Two things stand out. The near-miss row is the valuable one: a tile with a per-32-half rotation reaches 0.99 and never 1 — that is precisely the variant a plausible-looking implementation would land on, and a test with any tolerance ("corr > 0.9") would pass it. inv_freq is passed in from the pack's own visual.rotary_pos_emb.inv_freq rather than recomputed from theta — "a formula would be a second source of truth" is the right reason, and it is the same reasoning that made the fconst rules table a measured thing rather than a derived one.

The goldens are well chosen: a sum over the whole (4,2,8) tensor so a slip in head stride or the axis split fails rather than coinciding on one token, and pos (0,0) as the identity so a broken-but-nonzero implementation cannot pass. Refusal arms on head_dim % 4 != 0 and null close it.

Coordination — this is a superset of #1920. Same q4nx_vision_pos.cpp, same q4nx_vision.h and the same pack_import.{h,cpp} hunk, plus q4nx_vision_rope.{cpp} and its 82-line test. Merge this, close #1920 (or ensure only one of the two lands), because landing both would duplicate the position-embedding file and the spelling-rule change. I reviewed #1920 at 45ce9ad9 and its arms are a subset of these.

One note: the body says the tower body is "the next PR" — when it lands, the pack_import spelling hunk from #1893 is the second thing that touches this area. Three PRs in flight over one rule is how the "two derivations of bits" defect (#1893) came back once already; worth checking at each merge that no second derivation appears.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 89ffdee: CHANGES

This head carries a READY and a red build. I am posting CHANGES rather than a comment because the verdict is what the gate reads, and a red build must not merge — the mirror image of this morning's defect, where a READY landing after a CHANGES let a blocked PR through.

The failure is mechanical, and it is the same failure on three PRs — this one, #1923 and #1930 all die in the build job at the copyright step, with no compile error involved:

copyright notice duplicated inside the header window (delete the second copy): tests/q4nx_vision_block_test.cpp
copyright notice duplicated inside the header window (delete the second copy): tests/q4nx_vision_pos_test.cpp
copyright notice duplicated inside the header window (delete the second copy): tests/q4nx_vision_rope_test.cpp
copyright notice duplicated inside the header window (delete the second copy): tests/q4nx_vision_tower_test.cpp

One fix clears all three: delete the second copy inside the header window (NOTICE_WINDOW = 20 in tools/copyright.py), one edit per file. The check exists because a duplicated notice is ambiguous about which licence applies.

It is worth noting where this could have been caught without a CI round trip — the same place I should have caught my own #1920:

python3 tools/copyright.py

This is the second distinct copyright failure today in this file family (#1920 was a binary-licence header where the tree requires SPDX-License-Identifier: Apache-2.0), which suggests running that one command before pushing the vision series would have saved three red builds.

The substance of the series is not in question — position embedding, the front-end kernels and this assembly are the right pieces in the right order. This needs the duplicate notices gone.

bong-water-water-bong pushed a commit that referenced this pull request Nov 10, 2026
…the header check

Review of #1927: the `build` job dies at `tools/copyright.py --check`, before compiling anything --

  copyright notice duplicated inside the header window (delete the second copy): npu/forward/include/q4nx_vision.h
  ... q4nx_vision_{block,pos,rope}.cpp, and the three tests

The header carried a notice twice inside `NOTICE_WINDOW` (20 lines), which is ambiguous about which licence applies, so the
check refuses it. `python3 tools/copyright.py --fix` repaired all seven files; the tree is clean under `--check` now.

Verified past the step that was failing: the three test targets build and pass --

  vision block: 7 arms pass (goldens x3, identity control, merger x2, refusal)
  vision pos embed: 12 arms pass (token order, align_corners, exact copy, golden x8)
  vision rope: 6 arms pass (identity, goldens x4, conventions note, refusal)

The same repair is applied to #1923 and #1930, which fail identically at this step.

Refs #1927
…this head too)

Review of #1923: the `build` job dies at `tools/copyright.py --check` before compiling -- a notice twice inside
`NOTICE_WINDOW = 20`, which is ambiguous about which licence applies, so the check refuses. `--fix` repaired the five files
this branch carries; `--check` is clean.

Verified past the failing step: `q4nx_vision_pos_test` and `q4nx_vision_rope_test` build and pass --

  vision pos embed: 12 arms pass (token order, align_corners, exact copy, golden x8)
  vision rope: 6 arms pass (identity, goldens x4, conventions note, refusal)

Same repair as #1927 (`12d5d1fe`) and the same one #1930 needs.

Refs #1923

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at b72c83246 — the copyright duplication, which is what the build job was dying on. tools/copyright.py --fix repaired the five files this branch carries and --check is clean; verified past the failing step by building and running both test targets:

vision pos embed: 12 arms pass (token order, align_corners, exact copy, golden x8)
vision rope:      6 arms pass (identity, goldens x4, conventions note, refusal)

Same repair as #1927 (12d5d1fe). Both of your process points stand: the local python3 tools/copyright.py would have saved the CI round trip, and posting CHANGES for a head that carries a red build is right — the gate reads the verdict, so the verdict has to describe the head.

bong-water-water-bong pushed a commit that referenced this pull request Nov 10, 2026
…bedding it calls

Review of #1930: the head failed `build` twice over, and the second one is not mechanical.

1. **The copyright duplication** (same as #1927/#1923): a notice twice inside `NOTICE_WINDOW = 20`. `--fix` repaired the
   files; `--check` is clean.

2. **`q4nx_vision_block_test` had no `pos.cpp` in its sources**, so the link failed -- `undefined reference to
   q4nx_vision::pos_embed` -- and the job died before any test ran. A block is rope AND position embedding, so the test
   pulls both; the executable list was missing one of them. That is a real defect in this PR, not a manifestation of the
   copyright one, and it would have surfaced the moment the header check stopped failing.

Verified past both: all four vision targets build and pass --

  vision block:     7 arms pass (goldens x3, identity control, merger x2, refusal)
  vision pos embed: 12 arms pass (token order, align_corners, exact copy, golden x8)
  vision rope:      6 arms pass (identity, goldens x4, conventions note, refusal)
  vision tower:     4 arms pass (golden x2, zero-weight control, refusal)

Refs #1930

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b72c832: READY — withdraws my CHANGES at the previous head

The duplicate notice is gone: tests/q4nx_vision_pos_test.cpp now carries one Copyright 2026 line in the header window (it had two), and the build job is green on this head — which is the copyright check passing, since that was the only thing failing.

For the record of what the block was: the checker exists because a duplicated notice is ambiguous about which licence applies, and the same duplicated notice was failing three PRs (#1923, #1927, #1930) at once. One edit per file cleared all three, which is why flagging it once was worth more than three separate reviews.

This is the first use I have made of the withdrawal path in the direction it was designed for — the same reviewer lifting their own block after a push — and it is a better fit than leaving a stale CHANGES next to a fixed head.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Superseded by #1930, which contains this branch's commits as ancestors (git merge-base --is-ancestor confirms). Closed per the review note on #1930 so the port merges as one unit rather than as a four-deep stack.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect rope axis handling

The rope_axial function assumes that the first nf dimensions of the head_dim correspond to the row position and the next nf dimensions to the column position. However, the code does not correctly handle the case where d is in the range [nf, half) which should map to the column position. Instead, it maps to the row position due to a logic error in the condition d < nf ? 0 : 1. This could lead to incorrect rotary embeddings for the column dimensions.

const int axis = d < nf ? 0 : 1;
const float pos = float(axis == 0 ? pos_h[t] : pos_w[t]);
const float ang = pos * inv_freq[d < nf ? d : d - nf];
Potential integer overflow in zp_bits_for_spelling

In the zp_bits_for_spelling function, the code assumes that pos + 2 < spelling.size() to access spelling[pos + 2]. However, if pos is close to the end of the string, this check might not be sufficient to prevent out-of-bounds access. While the code checks pos != std::string::npos and pos + 2 < spelling.size(), it does not ensure that pos + 2 is a valid index, which could lead to undefined behavior if the string is shorter than expected.

const auto pos = spelling.rfind("_Q");
if (pos != std::string::npos && pos + 2 < spelling.size()) {
    const char d = spelling[pos + 2];
    if (d >= '2' && d <= '8') return d - '0';
}

bong-water-water-bong added a commit that referenced this pull request Nov 12, 2026
…rge-block token order (engine#1503 W4) (#1930)

* npu-pack: the implicit zero point's bit width comes from the spelling, not a literal (engine#1841)

The rule `2^(bits-1)` existed twice with two different derivations of the same parameter: the projection path passed a
hardcoded `4`, the head path passed the node's own `bits`. Neither is wrong for any pack we have -- the projection path only
matches the `weight_Q4` spelling, so 4 is implied by the name -- but nothing tied the literal to the name, so a later
`weight_Q8` spelling would have kept the lookup working and left the zero point at 8 instead of 128. That is a silent wrong
weight, not a crash.

`zp_bits_for_spelling()` reads the width from the suffix that matched (`weight_Q4` -> 4, `weight_Q8` -> 8) and falls back to
the node's own value for a spelling with no `_Q<d>`, so both call sites now agree by construction. Both symbols are declared in
the header and pinned by six arms in the import test, because the rule is what can drift:

    weight_Q4                                                         -> 4
    weight_Q8 (with the path's 4)                                     -> 8   <- the arm a hardcoded literal fails
    ...MatMulNBits.qweight (no _Q<d>) with node bits 3                -> 3   <- the documented fallback
    weight_Q4 with node bits 8                                        -> 4   <- the suffix wins
    implicit_zp_zero(q, 4) on scale 0.5                               -> zero -4.0
    implicit_zp_zero(q, 8) on scale 0.5                               -> zero -64.0

No behaviour change on any pack that exists: the Qwen3-VL-4B import is byte-for-byte the same workload after the change
(399 tensors, 491,000 q4_1 tiles, self-check corr 0.993981), and the whole npu-pack suite passes (0 FAIL).

Closes the second point of engine#1841. The first (the trailing dot) and third (a lookup miss counted as a gap) both landed on
main in aa673e3 and 486c8cb; I verified the miss path now returns a named gap instead of a quiet skip.

* npu: the vision tower's position embedding, as the pack's graph computes it (engine#1503 W4)

First piece of the vision port, and the piece that took the longest to establish. `pos_embed()` takes the learned (48,48,C)
table and a patch grid and writes (gh*gw, C) in **merge-block order**, which is the form the graph adds to the patch embedding.
Both halves are pinned by numbers taken from the graph, not from a reading of the export:

* the interpolation is bilinear with **align_corners = true** (`src = c*47/(n-1)`). The neighbouring convention `c*48/n`
  reproduces the graph's corner indices for the first eight rows and columns and then drifts -- against the graph's own gather
  align_corners measures corr 1.00000000 / mean|d| 5e-8, and `c*48/n` measures 0.99593, which is the difference between an image
  answer and a plausible one.
* the reorder is one permutation, `perm = [0 1 gw gw+1 | 2 3 gw+2 gw+3 | ...]`: group g is the 2x2 block at
  (2*(g/gw2), 2*(g%gw2)). Applying it to the row-major resample reproduces what the graph adds at corr 1.000000 / mean|d| 5e-8.

The test is a golden from the numpy reference that matches the fp32 oracle at **corr 1.000000, mean|d| 4.1e-5** on
`image_features`, so a wrong index, a wrong convention or a wrong token order fails a literal here:

    12 arms pass (token order, align_corners, exact copy, golden x8)
    golden t=0 0.000000000 0.001000000 0.002000000      t=2 0.966857143 0.967857143 0.968857143
    golden t=47 3.041571429 ...                          sum of all 96 tokens = 995.184

The whole-array sum is in there on purpose: sampling six tokens would let a slip in the other ninety through.

Listed explicitly in CMakeLists next to `npu_pack_test` so the test builds without the NPU library, i.e. it runs on any lane.
Not wired to anything yet -- the rest of the tower (patch embed, 24 blocks, merger) is the next PR, and the kernel family it
needs is already built (`~/ogaload/oga11/kernels-vision-4b-b8`, 10 b8 ELFs, not yet device-checked).

* npu: the vision tower's axial 2D rope, tile-and-rotate_half, pinned by goldens (engine#1503 W4)

Second piece of the vision port, and the other detail that took several wrong attempts. The graph's rope is
`x*cos + rotate_half(x)*sin` with `cos = Cos(Concat(a, a))`, i.e. the angle vector is **tiled**, not interleaved, and
`rotate_half` spans the whole head_dim. Getting either wrong is invisible in a correlation you have not measured against the
right tensor:

    interleaved repeat of the angle           corr 0.44
    tile, but rotating each half separately   corr 0.99, never 1
    tile + rotate_half over head_dim          corr 0.999996 vs the graph's own post-rope q (mean|d| 3.3e-3, |ref| 1.15)

So the arms are literals from the numpy reference that reproduces that tensor: position (0,0) is the identity; the column step,
the row step and both together match to 2e-6; and the sum over the whole (4,2,8) tensor is 16.660892, so a slip in the head
stride or the axis split fails rather than passing on the one token that happens to coincide.

    vision rope: 6 arms pass (identity, goldens x4, conventions note, refusal)

`inv_freq` is passed in (the pack's own `visual.rotary_pos_emb.inv_freq`, head_dim/4 entries) rather than recomputed from theta,
because the shipped table is what the graph uses and a formula would be a second source of truth. Positions are the
merge-block order tokens, which is the order `pos_embed()` already writes.

Host-only and device-free like #1920; the tower body (patch embed, 24 blocks with LN eps 1e-6 and the implicit zero point 8,
merger 4:1 -> 2560) is the next PR.

* npu: one vision transformer block and the merger, host float32 (engine#1503 W4)

Third piece of the vision port: `block_forward()` (norm -> qkv -> rope -> attention -> proj -> residual -> norm -> fc1 ->
gelu-tanh -> fc2 -> residual) and `merger_forward()` (per-token norm, view 4 tokens as 4*hidden, fc1 -> gelu -> fc2 to
`out_hidden`). Every constant that is easy to get wrong is in one place with the measurement that pinned it:

    LayerNorm eps      1e-6     (1e-6 gives corr 1.000000 against the graph's norm, 1e-5 is 0.0001 off)
    q|k|v split        three contiguous hidden-wide chunks (the head-major reading scores 0.41 where this scores 0.99)
    attention          FULL, scale 1/sqrt(head_dim)   (the pack's 24 Loop nodes are per-image bookkeeping: pad = [0, num_patches])
    gelu               the TANH approximation (config `gelu_pytorch_tanh`)

Test: goldens from the numpy reference that reproduces the graph block by block (norm1 1.000000, qkv >0.999 in every column,
attention 0.999994) and the whole tower at corr 1.000000 against the fp32 oracle. The weights come from a 32-bit LCG so the
fixture is regenerable in another language, and the goldens are two whole tokens plus the sum over all 64 outputs:

    vision block: 7 arms pass (goldens x3, identity control, merger x2, refusal)
      token 0  0.08357919 0.07403285 0.03188136 -0.16534122 …
      token 3 -0.02847019 0.34347527 0.18980209 -0.23568934 …
      sum over the block's 64 outputs = -2.481451

The identity control is the arm that earns its keep: a block with zero weights and zero biases must return the input unchanged
(both residuals, nothing else), so a misplaced residual or a norm applied to the wrong operand fails even if the goldens were
regenerated wrongly.

Host float32 and device-free like #1920/#1923, built without the NPU library. Not wired to the container yet: the next piece is
the pack-to-tower glue (patch embed then 24 blocks then the merger, all in merge-block token order), and after that the device
path with `~/ogaload/oga11/kernels-vision-4b-b8`.

* npu: assemble the vision tower -- patch embed, N blocks, merger, in merge-block token order (engine#1503 W4)

Fourth piece of the vision port and the one that makes the other three a tower: `tower_forward()` runs the Conv3d patch embed
as a matmul (the kernel equals its stride), adds the position embedding, runs `n_blocks` of `block_forward()`, then
`merger_forward()` to `out_hidden`. Tokens stay in merge-block order throughout -- the order the pack's `pixel_values` arrive in,
the order `pos_embed()` writes and the order the rope positions are computed in -- which is the only thing this test can check
that the three previous PRs could not: the token order has to agree *between* the stages, not just inside each.

Test: `vision tower: 4 arms pass (golden x2, zero-weight control, refusal)`

    golden out: -0.19714565 -0.03816292 -0.08239722 0.27405666 0.34750172 -0.23880654 0.19937676 -0.04093523
    golden sum: 0.223488

from the numpy reference that reproduces the pack's graph at corr 1.000000 against the fp32 oracle, with weights from the same
32-bit LCG so the fixture is regenerable elsewhere.

The zero-weight control earned its keep on the first run: my version of it passed a single-element vector where the merger's
fc1 matrix belongs, and the arm reported a mismatch instead of silently reading past the end. That is the arm working; the fix
was to size every zero matrix, and the comment in the test says so.

Two C++ traps hit again and worth naming since they cost three build cycles between them: `std::vector<float> v(size_t(n))` is a
*function declaration* when `n` is a variable (the most vexing parse -- `static_cast<size_t>` fixes it), and pointers taken into
a `std::vector` of `std::vector`s are invalidated by `push_back`, so the fixture allocates one flat store and offsets into it.

Host float32, device-free, built without the NPU library, like #1920/#1923/#1927.

* vision: one notice per file, and the block test links the position embedding it calls

Review of #1930: the head failed `build` twice over, and the second one is not mechanical.

1. **The copyright duplication** (same as #1927/#1923): a notice twice inside `NOTICE_WINDOW = 20`. `--fix` repaired the
   files; `--check` is clean.

2. **`q4nx_vision_block_test` had no `pos.cpp` in its sources**, so the link failed -- `undefined reference to
   q4nx_vision::pos_embed` -- and the job died before any test ran. A block is rope AND position embedding, so the test
   pulls both; the executable list was missing one of them. That is a real defect in this PR, not a manifestation of the
   copyright one, and it would have surfaced the moment the header check stopped failing.

Verified past both: all four vision targets build and pass --

  vision block:     7 arms pass (goldens x3, identity control, merger x2, refusal)
  vision pos embed: 12 arms pass (token order, align_corners, exact copy, golden x8)
  vision rope:      6 arms pass (identity, goldens x4, conventions note, refusal)
  vision tower:     4 arms pass (golden x2, zero-weight control, refusal)

Refs #1930

* vision: build the four vision tests from `onebit`, so their ctest rows exist in CI

Review of #1930: the duplicate-notice fix took, and the head then failed `ci_targets_check` — all four vision test targets are
`add_test`ed but `.github/workflows/ci.yml` never builds them, so the rows would never run in CI. Same wall as #1822/#1799/#1809.

`add_dependencies(onebit q4nx_vision_block_test q4nx_vision_pos_test q4nx_vision_rope_test q4nx_vision_tower_test)` is the
documented route (the one `onebit_npu_forward_syntax` and `route_log_test` use); it needs no workflow edit, so it also needs no
`workflow` scope for the push.

Verified:
* `ci_targets_check --check --root .` -> "9 target(s) whose test never reaches ctest under ci.yml (baseline 9), and none new";
* all four targets build and their arms pass --
    vision block:     7 arms (goldens x3, identity control, merger x2, refusal)
    vision pos embed: 12 arms (token order, align_corners, exact copy, golden x8)
    vision rope:      6 arms (identity, goldens x4, conventions note, refusal)
    vision tower:     4 arms (golden x2, zero-weight control, refusal)

Refs #1930

* vision: the tower's refusal arm observes the refusal (answers the CHANGES at a7af2fd)

The arm was check(true, ...): it passed whether or not tower_forward() refused. It now fills the output buffer with a
sentinel and requires it untouched, on a token count (2*T on a grid of T) that only the gh*gw check can refuse: positive
and a multiple of 4, so neither tokens<=0 nor merger_forward()'s own guard stops it. With that check removed the arm
fails; restored, 4 of 4 vision tests pass.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: pi-npufeat <npufeat@strixhalo>
Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/vision-rope branch November 12, 2026 21:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant