Repository navigation
npu forward: cut the lm-head context out of init_impl, unchanged, so moving it is a copy (A2b, prep) - #1458
Conversation
…moving it is a copy A2b's remaining half is the M=16 decode family and the lm-head context. Both are long inline blocks inside init_impl() that read a dozen of the forward's fields, which is why moving them straight to the target is surgery rather than a copy -- and this lane's evidence (A1) is that surgery on this path is where the two defects that survived review lived. So this step changes no logic at all: the lm-head block (pb16i ternary images, then the int8 ladder, then the q4_1 gemv head) becomes Q4nxNpuForward::open_lm_head_context(dev, model_dir, kernels_dir), verbatim, and init_impl() calls it. The only edit inside the body is that the three locals it used are now parameters. Nothing else moves: same messages, same order, same early returns, and the same xd_->head_ctx() state, so 'build clean' plus the existing suites is the whole claim -- the device behaviour is literally the same code in a different frame. The next step moves this method behind XdnaTarget and deletes make_head_ctx from the forward's side, which is now a copy of one body rather than an extraction.
|
#ready for review — a refactor that asserts nothing new about behaviour, on purpose. The claim is narrow and checkable: the lm-head block's body is unchanged, only its frame. The one edit inside it is that Why cut it out before moving it: moving that block straight into One row worth flagging honestly: my first |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at f45f5cef88b6732af6469628f22ac4235d8de569
A pure extraction, and I checked that it is one rather than taking "verbatim" on trust.
Mechanically: I compared the 133 removed and 143 added lines of q4nx_forward.cpp, with indentation stripped. The body is identical line for line. The only differences are:
- the replaced section comment;
- the call site
if (!open_lm_head_context(dev, mdir, dir)) return false;; - the new signature, its comment, and the closing
return true;.
Semantically, the two ways an extraction like this changes behaviour:
- Early exits. Every return inside the moved block is
return false(the threecannot create … contextrefusals and two plain ones), and those still endinit_implthrough the call site. There's no earlyreturn true, which would have meant "stopinit_implhere" and would now mean "carry on". - Locals. The block set no
init_impllocal that later code reads, which the compile confirms: a moved local would no longer be in scope.
At this head (NPU build, thermal-run.sh):
onebit_npu_forwardplus the plan test targets build clean, with no new warning inq4nx_forward.cpp;ctest -R "npu_plan|designs_parity|design_grid|op_reference|npu_target": 7/7;tools/copyright.py --check: rc 0.
Slicing A2b this way (extract unchanged, then move) is the right response to where this path's earlier defects hid.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at f45f5cef.
The refactor is complete, and I checked that rather than the diff's shape — the count is misleading at a glance (make_head_ctx() still appears
3×, open_lm_head_context( twice), but the three are the three branches now living inside the one function:
:3504 bool Q4nxNpuForward::open_lm_head_context(dev, mdir, dir) definition
:3535 make_head_ctx() branch 1: the pb16i ternary image, when the converter produced one
:3576 make_head_ctx() branch 2: the int8 family fallback
:3626 make_head_ctx() branch 3: the host-GEMM / any-gemv path
:1323 open_lm_head_context(dev, mdir, dir) the single call site (in init)
head_ctx_ref().reset() inside the function: 3 one cleanup per branch
So the "one place" property holds, and — the risk in this class of change — no branch lost its cleanup: three branches, three resets. Each branch
keeps its own distinct logic (ternary images first because output.weight is PTQ1_0 in the folded GGUF and the image is exact rather than a
requantisation, then the int8 family, then the gemv path), which is what makes the consolidation a refactor rather than a simplification that quietly
picks one behaviour for all three.
The value is the usual one: "make the head context and reset it on failure" was written three times, and three copies is exactly where one of them
eventually forgets the reset and leaves a dangling context. Now there is one.
Not run: needs a device and a pack (npb/host-GEMM both). My assessment is the structural equivalence above — that every branch survived the move
with its own semantics and its own failure path — not a before/after output comparison.
…as two items left, not one Review at cfa7829 is right and this is the correction: the paragraph said the setup side is the target's and that q4nx_forward.cpp creates no I8Ctx at all. The prefill ELF set (#1380) and the lm-head context (#1458, #1478) did move - the forward calls open_head() at :1357 and then asks head_ctx() for it - but the M=16 decode family is still built here: the target allocates the memory (XdnaTarget::make_ctx16) while the forward decides which decode contexts exist and configures each one field by field (MD, KD, ND, blocked_b, pb16i, t2s, planes) at q4nx_forward.cpp:1306-1330, and docs/npu-p4-target-split.md section 3 says exactly that ('still in the forward, and therefore still owed: the M=16 decode family'). So A2 has two items left: A2c moves that decode-family setup behind the target, and load_block stays a named no-op whose explicit block step is an optimisation with a tokens/s row as its evidence, not a structural move. The paragraph now says that, cites the lines, and no longer claims the forward creates no I8Ctx. Docs only; copyright --check rc 0.
…w stops saying the derivation is all there (#1529) * docs/npu-plan: the schema table learns about designs[], and the P4 row stops saying the derivation is all there is Three statements on this page were true when written and are not now, all of them about fields the loader reads: * the schema table had no row for designs[] -- the per-role kernel set the plan may name (#1330, #1333), with its both-directions agreement check, its bare-file-name artifact, the generator writing it only where it can be complete, and Plan::check_artifacts() refusing a missing artifact at load (and npu_plan.py --check asking the same question at import time); * the layers[] row said the forward 'derives designs from the container', which is now only the no-section case; * the 'Not yet' P4 entry stopped at A2b part 1 (#1380). A2b is complete: the M=128 ELF set in open_prefill (#1380) and the lm-head context in open_head (#1458, #1478), so q4nx_forward.cpp creates no I8Ctx at all. What is genuinely still open is the residency step -- load_block is a named no-op, the cache fills lazily inside gemv -- and that is an optimisation whose evidence is a tokens/s row, not a structural move. Docs only; no code change. * docs/npu-plan: the M=16 decode family is still the forward's, so A2 has two items left, not one Review at cfa7829 is right and this is the correction: the paragraph said the setup side is the target's and that q4nx_forward.cpp creates no I8Ctx at all. The prefill ELF set (#1380) and the lm-head context (#1458, #1478) did move - the forward calls open_head() at :1357 and then asks head_ctx() for it - but the M=16 decode family is still built here: the target allocates the memory (XdnaTarget::make_ctx16) while the forward decides which decode contexts exist and configures each one field by field (MD, KD, ND, blocked_b, pb16i, t2s, planes) at q4nx_forward.cpp:1306-1330, and docs/npu-p4-target-split.md section 3 says exactly that ('still in the forward, and therefore still owed: the M=16 decode family'). So A2 has two items left: A2c moves that decode-family setup behind the target, and load_block stays a named no-op whose explicit block step is an optimisation with a tokens/s row as its evidence, not a structural move. The paragraph now says that, cites the lines, and no longer claims the forward creates no I8Ctx. Docs only; copyright --check rc 0. --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@users.noreply.github.com> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
What — and what it deliberately is not
A2b's remaining half is the M=16 decode family and the lm-head context. Both are long
init_impl()blocks that read a dozen of the forward's fields, which is why moving them straight to the target is surgery rather than a copy — and this lane's evidence is that surgery on this path is exactly where the two defects that survived review lived (a null M=16 context, and the lost host-fallback activation pad).So this step changes no logic at all:
Q4nxNpuForward::open_lm_head_context(dev, model_dir, kernels_dir), verbatim;init_impl()calls it where the block was;dev,mdir,dir) are now parameters.Same messages, same order, same early returns, same
xd_->head_ctx()state — so "builds clean plus the existing suites" is the whole claim, because the device behaviour is literally the same code in a different frame. No device row is claimed or needed for a body that did not change; the next PR moves this method behindXdnaTargetand deletesmake_head_ctxfrom the forward's side, which is then a copy of one body instead of an extraction.Rows
onebitbuild (Release,ONEBIT_NPU=ON, runtime fromONEBIT_NPU_RT_DIR,thermal-run.sh)build-npu/1bitrelinkedctest -R "npu_plan|designs_parity|design_grid|op_reference|npu_target"Not Run/failed because I had built onlyonebit; the quoted 7/7 is after building the test targets, the same "a test row is a claim about its binaries" rule this lane keeps meetingtools/copyright.py --checkRefs #803 (A2b), #799.