Skip to content

npu forward: cut the lm-head context out of init_impl, unchanged, so moving it is a copy (A2b, prep) - #1458

Merged
bong-water-water-bong merged 1 commit into
mainfrom
1bit/npu-a2b-head
Nov 8, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
1bit/npu-a2b-head

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What — and what it deliberately is not

A2b's remaining half is the M=16 decode family and the lm-head context. Both are long init_impl() blocks that read a dozen of the forward's fields, which is why moving them straight to the target is surgery rather than a copy — and this lane's evidence is that surgery on this path is exactly where the two defects that survived review lived (a null M=16 context, and the lost host-fallback activation pad).

So this step changes no logic at all:

  • the lm-head block (pb16i ternary images → the int8 ladder → the q4_1 gemv head) becomes Q4nxNpuForward::open_lm_head_context(dev, model_dir, kernels_dir), verbatim;
  • init_impl() calls it where the block was;
  • the only edit inside the body is that the three locals it used (dev, mdir, dir) are now parameters.

Same messages, same order, same early returns, same xd_->head_ctx() state — so "builds clean plus the existing suites" is the whole claim, because the device behaviour is literally the same code in a different frame. No device row is claimed or needed for a body that did not change; the next PR moves this method behind XdnaTarget and deletes make_head_ctx from the forward's side, which is then a copy of one body instead of an extraction.

Rows

row verdict
onebit build (Release, ONEBIT_NPU=ON, runtime from ONEBIT_NPU_RT_DIR, thermal-run.sh) clean, build-npu/1bit relinked
ctest -R "npu_plan|designs_parity|design_grid|op_reference|npu_target" 7/7 — note the first run reported 5 of 7 as Not Run/failed because I had built only onebit; the quoted 7/7 is after building the test targets, the same "a test row is a claim about its binaries" rule this lane keeps meeting
tools/copyright.py --check rc 0

Refs #803 (A2b), #799.

…moving it is a copy

A2b's remaining half is the M=16 decode family and the lm-head context. Both are long inline blocks inside
init_impl() that read a dozen of the forward's fields, which is why moving them straight to the target is
surgery rather than a copy -- and this lane's evidence (A1) is that surgery on this path is where the two
defects that survived review lived.

So this step changes no logic at all: the lm-head block (pb16i ternary images, then the int8 ladder, then
the q4_1 gemv head) becomes Q4nxNpuForward::open_lm_head_context(dev, model_dir, kernels_dir), verbatim, and
init_impl() calls it. The only edit inside the body is that the three locals it used are now parameters.

Nothing else moves: same messages, same order, same early returns, and the same xd_->head_ctx() state, so
'build clean' plus the existing suites is the whole claim -- the device behaviour is literally the same code
in a different frame. The next step moves this method behind XdnaTarget and deletes make_head_ctx from the
forward's side, which is now a copy of one body rather than an extraction.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

#ready for review — a refactor that asserts nothing new about behaviour, on purpose.

The claim is narrow and checkable: the lm-head block's body is unchanged, only its frame. The one edit inside it is that dev/mdir/dir became parameters. So please read the diff for exactly that: if anything other than the signature and the three former locals moved, the claim is wrong and I want to know.

Why cut it out before moving it: moving that block straight into XdnaTarget is surgery on a path where A1's two surviving defects lived (a null M=16 context; the lost host-fallback pad). As one named method, the next PR is a copy of a single body.

One row worth flagging honestly: my first ctest reported 5 of 7 as Not Run/failed because I had built only onebit in that tree — the quoted 7/7 is after building the test targets. Same rule as #808's "a test row is a claim about its test binaries".

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at f45f5cef88b6732af6469628f22ac4235d8de569

A pure extraction, and I checked that it is one rather than taking "verbatim" on trust.

Mechanically: I compared the 133 removed and 143 added lines of q4nx_forward.cpp, with indentation stripped. The body is identical line for line. The only differences are:

  • the replaced section comment;
  • the call site if (!open_lm_head_context(dev, mdir, dir)) return false;;
  • the new signature, its comment, and the closing return true;.

Semantically, the two ways an extraction like this changes behaviour:

  • Early exits. Every return inside the moved block is return false (the three cannot create … context refusals and two plain ones), and those still end init_impl through the call site. There's no early return true, which would have meant "stop init_impl here" and would now mean "carry on".
  • Locals. The block set no init_impl local that later code reads, which the compile confirms: a moved local would no longer be in scope.

At this head (NPU build, thermal-run.sh):

  • onebit_npu_forward plus the plan test targets build clean, with no new warning in q4nx_forward.cpp;
  • ctest -R "npu_plan|designs_parity|design_grid|op_reference|npu_target": 7/7;
  • tools/copyright.py --check: rc 0.

Slicing A2b this way (extract unchanged, then move) is the right response to where this path's earlier defects hid.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at f45f5cef.

The refactor is complete, and I checked that rather than the diff's shape — the count is misleading at a glance (make_head_ctx() still appears
3×, open_lm_head_context( twice), but the three are the three branches now living inside the one function:

:3504  bool Q4nxNpuForward::open_lm_head_context(dev, mdir, dir)   definition
:3535    make_head_ctx()   branch 1: the pb16i ternary image, when the converter produced one
:3576    make_head_ctx()   branch 2: the int8 family fallback
:3626    make_head_ctx()   branch 3: the host-GEMM / any-gemv path
:1323  open_lm_head_context(dev, mdir, dir)                        the single call site (in init)
head_ctx_ref().reset() inside the function: 3                     one cleanup per branch

So the "one place" property holds, and — the risk in this class of change — no branch lost its cleanup: three branches, three resets. Each branch
keeps its own distinct logic (ternary images first because output.weight is PTQ1_0 in the folded GGUF and the image is exact rather than a
requantisation, then the int8 family, then the gemv path), which is what makes the consolidation a refactor rather than a simplification that quietly
picks one behaviour for all three.

The value is the usual one: "make the head context and reset it on failure" was written three times, and three copies is exactly where one of them
eventually forgets the reset and leaves a dangling context. Now there is one.

Not run: needs a device and a pack (npb/host-GEMM both). My assessment is the structural equivalence above — that every branch survived the move
with its own semantics and its own failure path — not a before/after output comparison.

@bong-water-water-bong
bong-water-water-bong merged commit 9d6646d into main Nov 8, 2026
8 of 9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/npu-a2b-head branch November 8, 2026 12:55
bong-water-water-bong pushed a commit that referenced this pull request Nov 8, 2026
…as two items left, not one

Review at cfa7829 is right and this is the correction: the paragraph said the setup side is the target's and
that q4nx_forward.cpp creates no I8Ctx at all. The prefill ELF set (#1380) and the lm-head context (#1458,
#1478) did move - the forward calls open_head() at :1357 and then asks head_ctx() for it - but the M=16 decode
family is still built here: the target allocates the memory (XdnaTarget::make_ctx16) while the forward decides
which decode contexts exist and configures each one field by field (MD, KD, ND, blocked_b, pb16i, t2s, planes)
at q4nx_forward.cpp:1306-1330, and docs/npu-p4-target-split.md section 3 says exactly that ('still in the
forward, and therefore still owed: the M=16 decode family').

So A2 has two items left: A2c moves that decode-family setup behind the target, and load_block stays a named
no-op whose explicit block step is an optimisation with a tokens/s row as its evidence, not a structural move.
The paragraph now says that, cites the lines, and no longer claims the forward creates no I8Ctx.

Docs only; copyright --check rc 0.
bong-water-water-bong added a commit that referenced this pull request Nov 8, 2026
…w stops saying the derivation is all there (#1529)

* docs/npu-plan: the schema table learns about designs[], and the P4 row stops saying the derivation is all there is

Three statements on this page were true when written and are not now, all of them about fields the loader reads:

  * the schema table had no row for designs[] -- the per-role kernel set the plan may name (#1330, #1333), with its
    both-directions agreement check, its bare-file-name artifact, the generator writing it only where it can be complete,
    and Plan::check_artifacts() refusing a missing artifact at load (and npu_plan.py --check asking the same question at
    import time);
  * the layers[] row said the forward 'derives designs from the container', which is now only the no-section case;
  * the 'Not yet' P4 entry stopped at A2b part 1 (#1380). A2b is complete: the M=128 ELF set in open_prefill (#1380) and
    the lm-head context in open_head (#1458, #1478), so q4nx_forward.cpp creates no I8Ctx at all. What is genuinely still
    open is the residency step -- load_block is a named no-op, the cache fills lazily inside gemv -- and that is an
    optimisation whose evidence is a tokens/s row, not a structural move.

Docs only; no code change.

* docs/npu-plan: the M=16 decode family is still the forward's, so A2 has two items left, not one

Review at cfa7829 is right and this is the correction: the paragraph said the setup side is the target's and
that q4nx_forward.cpp creates no I8Ctx at all. The prefill ELF set (#1380) and the lm-head context (#1458,
#1478) did move - the forward calls open_head() at :1357 and then asks head_ctx() for it - but the M=16 decode
family is still built here: the target allocates the memory (XdnaTarget::make_ctx16) while the forward decides
which decode contexts exist and configures each one field by field (MD, KD, ND, blocked_b, pb16i, t2s, planes)
at q4nx_forward.cpp:1306-1330, and docs/npu-p4-target-split.md section 3 says exactly that ('still in the
forward, and therefore still owed: the M=16 decode family').

So A2 has two items left: A2c moves that decode-family setup behind the target, and load_block stays a named
no-op whose explicit block step is an optimisation with a tokens/s row as its evidence, not a structural move.
The paragraph now says that, cites the lines, and no longer claims the forward creates no I8Ctx.

Docs only; copyright --check rc 0.

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@users.noreply.github.com>
Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant