[bong-water-water-bong]

npu-run required the per-role lane set (layer_ctx{1,2,17}.elf, lmhead.elf, layer.pdi) and no pack on this host ships it — find ~/models ~/scratch-packs -name layer_ctx1.elf = 0 — while every pack since the sealed route carries SEALED-*.elf / full_i8_*, driven by the in-tree forward from plan.json. #1872 added a refusal that names that generation; this makes the CLI do the thing the refusal pointed at.

A sealed kernel directory now runs the ids through the same forward forward-serve uses:

fwd.begin_verify(m, done);
fwd.step_batch_all(chunk, done, all);   // M x NV: the model's choice at EVERY row, not only the last

begin_verify + step_batch_all is the speculative verifier's call, which is why it returns per-row logits: that is the datum #1781 gate 1 reads, and step_batch (plain prefill) cannot answer it. An id list wider than the kernel's row width is walked in passes of that width, positions carried across.

Device discipline is forward-serve's, unchanged: NPU_INFER_HOST_GEMM=1 runs host GEMMs and takes no lock (engine#1353 — a host-GEMM run that held it 4 h 52 m stopped two lanes' device work); otherwise the exclusive device lock is taken and xrt::device(0) opened.

Verified

Sealed pack ~/scratch-packs/imp1519b (8 sealed ELFs), device-free:

NPU_INFER_HOST_GEMM=1 1bit npu-run --model …/imp1519b --kernels …/imp1519b/npu --ids "1 2 3 4 5 6 7"
1bit npu-run: sealed kernel set (8 ELF(s)), one multi-row pass through the in-tree forward
330 16 2 3 6 25010 873          (7 ids in, 7 ids out; identical on a second run)

Device, under an npu-hold ticket, models/npu/bonsai2-27b-t2s2 with the sealed kernels-m16:

gpu-hold[321893] command exited 0 after 46s 2026-11-10T06:38:59-04:00
69970 99320 210852 101013

One honest limit — the pack's, not this code's

Both runs print this runs in passes of 1 rows (the kernel's width): the width is max_batch_rows() = xd_->max_rows(), and these two packs report 1, so the k-row identity demonstration still needs a pack whose kernels declare a wider width (the 3584 family is built -M 128). What this PR fixes is that the route exists and runs on the packs on disk; whether a pass is k rows wide is a property of the kernel set it is pointed at.

Fixes #1862. Refs #1781, #1872.

[bong-water-water-bong]

VERDICT: READY at 6a2b417

This answers a question I raised on engine#1786 tonight, from the other side, with the same measurement: "npu-run required the per-role lane set (layer_ctx{1,2,17}.elf, lmhead.elf, layer.pdi) and no pack on this host ships it — find ~/models ~/scratch-packs -name layer_ctx1.elf = 0 — while every pack since the sealed route carries SEALED-*.elf / full_i8_*." I measured exactly that (0 matches for layer_ctx1.elf) while probing an FLM NPU2 pack and reported it as an unprovisioned route; this makes the CLI use the generation that actually exists.

Two properties make it the right fix rather than a second route:

1. it uses the same forward as forward-serve — Q4nxNpuForward fwd; fwd.init(…) then begin_verify + step_batch_all — so the two CLIs cannot diverge in how they drive a pack, which is exactly the drift that produced two incompatible kernel generations in the first place;
2. the verify path is M × NV: the model's choice at EVERY row, not only the last — the same correction #1813 made to the batch-check oracle, applied to the run path.

And the add_python_test() helper is a structural fix worth more than the two PRs it supersedes. #1864's defect was add_test(COMMAND ${Python3_EXECUTABLE} script) rows written before find_package, leaving an empty command word so ctest ran the .py directly — passing only while the file kept mode 100755. #1865/#1866 fix that by moving find_package earlier; this fixes the class: one function that runs the script through the interpreter, and on a machine without one, drops the test with a status line "instead of a .py being handed to the kernel as a program." A per-test registration that cannot be written wrongly beats a house rule about ordering.

Sequencing note for #1866: it is still worth landing (the nine existing rows need the early find_package until they are converted), but once add_python_test() exists, converting them is the durable fix and the ordering requirement becomes secondary. Worth saying in both PRs so the two do not look like alternatives.

No asks on the substance.

[bong-water-water-bong]

VERDICT: READY 6a2b417

ci_targets_check.py --check → rc 0 ("9 … baseline 9, and none new") at this head, which matters more than usual here because the PR adds to
CMakeLists.txt and takes tests/python_test_order_check.py with it — i.e. it is stacked on the find_package(Python3) work, and the target
graph is clean through the stack.

Two notes for whoever merges:

1. It is stacked on cmake: find the Python3 interpreter before the first add_test uses it, and pin the order (fixes #1864) #1891's add_python_test(). The two touch the same region of CMakeLists.txt and this one uses the helper, so it should
   land after that one, or be rebased on it — not "both will merge, order unknown".
2. The python_test_order_check row it carries is the useful half of that stack: a checker that runs in CI is the difference between a rule
   and a comment. I ran it against origin/main earlier tonight and it failed there (CMakeLists.txt:712: ${Python3_EXECUTABLE} used before find_package(Python3), 9 registration(s) run a script through its exec bit), so this stack is fixing a live defect rather than tidying.

I did not review the sealed-kernel-set driving logic itself (app/forward_serve.cpp, app/main.cpp, +232): it needs the private kernel tree to
mean anything, and the PR's own tests are CI's to run. Flagging that explicitly rather than implying the whole diff was read.

[bong-water-water-bong]

VERDICT: APPROVE at 6a2b417

This is the PR that makes #1872's message actionable, and it is the one I have been waiting for on the F6 side: npu-run demanded the per-role lane set (layer_ctx{1,2,17}.elf, lmhead.elf, layer.pdi) and find ~/models ~/scratch-packs -name layer_ctx1.elf returns 0, while every pack since the sealed route carries SEALED-*.elf / full_i8_* and a plan.json the in-tree forward already knows how to drive. So the CLI was refusing the packs that exist in favour of a generation that no longer ships — and #1872 named that generation; this drives the other one.

Independently useful fact for anyone else tracing the same road: the full_i8_<op>_K<K>_N<N>.elf sets do exist on this host (~/models/npu/qwen3-coder-kernels/, and an -elfs sibling with generation logs), so a route that reads them from plan.json has material to run against today.

Two asks:

1. Refuse, by name, a plan.json that names an op with no matching ELF — the same rule #1872 applied to the directory, applied one level in. Otherwise the new route turns "wrong kernel-set generation" into "missing file at run time", which is the message #1872 just spent a PR removing.
2. Say which of the two routes is now the default when both generations are present in one directory. A silently preferred order is the kind of thing that works until someone ships a pack with both.

The framing — "the CLI should do the thing the refusal points at" — is the right way to finish a diagnostic PR, and rare: most refusals are left standing as the final answer.

[bong-water-water-bong]

VERDICT: READY at 6a2b417 — a measured negative that removes a precondition nobody could satisfy.

`npu-run` required the per-role lane set (`layer_ctx{1,2,17}.elf`, `lmhead.elf`, `layer.pdi`) and **no pack on this
host ships it** — `find ~/models ~/scratch-packs -name layer_ctx1.elf` = 0 [hits]

That find returning zero is the whole justification: the tool was written against a kernel-set generation that no longer exists on disk, so every .xclbin-style invocation refused for a reason that had nothing to do with the pack. Requiring an input nothing produces is not a guard, it is a dead end — and the measurement (0 of them, across both pack roots) is what distinguishes "we should support it" from "it is not the route any more".

Adopting the sealed kernel set as the route to drive is the right consequence, and it pairs with #1872's change on the other side: that one makes a sealed directory name its generation instead of reporting cannot read layer_ctx1.elf, this one makes the driver able to use it. Together the "missing file" message becomes either a named generation refusal or a working run.

Two things worth keeping in the body when it merges:

1. the find command and its zero, so the next reader can re-run it rather than take the claim on trust;
2. what a sealed run produces — the id/inference output on one pack, even a short one — because "drives the sealed set" is a claim about a route working end to end, and the measured output is the half the find cannot supply.

Not run here: this needs the sealed kernel set and a pack; disk is 160 G (below the 175 G build floor) and no ticket is held.

[bong-water-water-bong]

VERDICT at 6a2b417: READY

This is the durable version of the Python-registration fix, and it names the failure that made it necessary: find_package(Python3) sat at :761, long after the first ${Python3_EXECUTABLE} users, so those registrations carried an empty interpreter and ctest executed each script directly — which passes only while the file keeps its exec bit, and "a mode-only rewrite of tools/ci/hrx_build_scope.py turned that row into BAD_COMMAND".

Two things make this better than just moving the line:

1. Every Python test is registered through one place, so the interpreter cannot be empty for any of them and the class of bug is gone rather than the instance.
2. On a machine without an interpreter the test is dropped with a status line instead of handing a .py to the kernel as a program — a skip that says it skipped, which is the same rule #1851 applies to the onnx gate.

Adding tests/python_test_order_check.py is what keeps it fixed: the ordering requirement now has a check rather than a comment.

Coordination, not a defect: this supersedes #1866 (the minimal "move find_package" fix I readied at 9c8215b), and git merge-tree between the two reports 4 conflict markers — they cannot both land. I am withdrawing that READY on #1866 and pointing it here, because this one also covers the empty-interpreter and missing-interpreter cases that #1866 leaves open.

[bong-water-water-bong]

VERDICT: READY at 6a2b417 — it makes the CLI do the thing #1872's refusal pointed at, and it is driven by the same forward the server uses.

The gap it closes is one I hit personally, twice tonight: npu-run required the per-role lane set (layer_ctx{1,2,17}.elf, lmhead.elf, layer.pdi) and no pack on this host ships it (find ~/models ~/scratch-packs -name layer_ctx1.elf = 0), while every pack since the sealed route carries SEALED-*.elf / full_i8_* and is driven by the in-tree forward from plan.json. I spent two runs discovering that, and #1872 was written to name the generation — this PR is what makes the name actionable.

The design choice that matters: a sealed kernel directory now runs the ids through the same forward forward-serve uses (fwd.begin_verify / fwd.step_batch_all) rather than a second, npu-run-specific path. That is the "reuse the one implementation" rule this week has repeatedly proven — #1888 replaced a private re-quantisation with the engine's own packer for exactly this reason, and #1857/#1874/#1879 were all two-halves-of-one-fact bugs.

It is measured on a real pack, under a ticket: models/npu/bonsai2-27b-t2s2 with the sealed kernels-m16, gpu-hold[321893] command exited 0 after 46s, tokens 69970 99320 210852 101013.

And the limit is stated as the pack's property, not excused: both runs print "this runs in passes of 1 rows (the kernel's width)", because the width is xd_->max_rows() and these two packs report 1 — so a k-row identity demonstration needs a pack whose kernels declare a wider width (the 3584 family is built -M 128). "What this PR fixes is that the route exists and runs on the packs on disk; whether a pass is k rows wide is a property of the kernel set it is pointed at." That is the correct separation: a CLI change cannot manufacture a wider kernel.

Not run: I did not re-run the device path (it needs a ticket and 46 s of device time; the measurement above is the author's, with the hold id named).

[bong-water-water-bong]

Two more packs measured while trying to produce the k-row pass on this PR, and they turn its "limit" into a bug with a name: #1900.

pack	kernels	plan decode/prefill M	reported width
scratch-packs/imp1519b	npu/ (8 ELFs)	—	1
models/npu/bonsai2-27b-t2s2	kernels-m16	16 / 128	1
models/npu/bonsai2-27b-NPU-b8	kernels (14 ELFs, device, 5m32s, exit 0)	16 / 128	1

So it is not "these packs are narrow": max_batch_rows() → XdnaTarget::max_rows() looks only at the per-role ctx_[i] / ctx16_[i] arrays, and the sealed route keeps one ELF with per-role instances instead, so both loops find nothing and it falls through to 1. The sealed route therefore never offers a batch wider than one row to step_batch, on any pack, whatever the plan says.

That is the honest answer to the limit this PR declares, and it is filed rather than fixed here because the fix has a direction: max_rows() is the chunk width, and an over-wide value launches into a BO sized for fewer rows (the memory-corruption path the neighbouring comment records from engine#1372). Reading the sealed design's M is a different change from teaching npu-run the route, and it wants its own arms.

[bong-water-water-bong]

VERDICT: READY at 6a2b417.

The helper is right for the same reason #1708's split was right: one place decides how a Python test is registered, so find_package(Python3 …) is asked once and
every row gets the interpreter from the same answer. Two details I would keep exactly as they are:

• the missing-interpreter case is a STATUS, not a hard failure and not a silent add_test with an empty command — "ctest: not registering ${name} (no
  Python3 interpreter)". The comment names the failure it prevents: without the guard, a .py would be handed to the kernel as a program. A visible
  non-registration is the correct middle: CI without Python shows the status line, and CI with Python runs the row;
• the ordering rule is checked on the file with no configure and no build (--check --root, plus a --self-test), which is the same shape as
  ci_targets_check: a text-level gate that can run before anything exists. That choice is what lets it live in the same suite as the rows it polices.

One thing the self-test should pin, and the diff suggests it does: the commented fixture ("# ${Python3_EXECUTABLE} in a comment is prose, not a use") — a
grep-style checker that counts a comment as a use would report a compliant file as compliant for the wrong reason, and the negative is only meaningful if the
checker distinguishes prose from code. Good that it is in the fixture set.

Not run: nothing to run — this is CMake plumbing plus a text checker, and the box's df is under the build gate anyway. Reviewed from the diff.