[bong-water-water-bong]

Refs #2106 (release-blocker), support lane. This is the check the private-repo note filed on the box says "is not written" — the one that would have diagnosed the "differ in size" refusal from the set alone, with no device and no private-repo archaeology.

What #2106 actually was: the release candidate serves small models at 87-101 tok/s with the set beside it; a build of current main refused that same set with layer ELFs: the NB 2 and 3 builds differ in size. Neither side was wrong about the kernels — binary and set came from different revisions of the private repository, one on each side of its protection change, so the loader read sealed files as plain ELFs and compared their lengths. The symptom, though, is entirely checkable from the set.

scripts/kernel-set-check.sh <set dir>… answers three questions a release run should ask before spending a device row:

1. does the manifest match the bytes — every listed file present, with the listed size and sha256, nothing listed that is absent, and no layer1.elf on disk that the manifest forgot (a loader may still pick that one up);
2. is the set in ONE form — each entry starts with \x7fELF or it does not; a mixed set is exactly the state in which one side reads the other side's files and sees "differ in size";
3. does the set's own name agree with its directory — a set copied to the wrong geometry name loads and mis-answers.

$ scripts/kernel-set-check.sh ~/wt/c32-payload/…/kernels/1024x3072x16x8
OK   1024x3072x16x8: 11 file(s) in sealed/opaque form          rc=0

That is the shipping RC set: self-consistent, one form, correctly named — which is the positive evidence the issue's item 1 was missing, obtained without private data.

tests/kernel_set_check_test.sh — 20 arms over fixture sets, registered as kernel_set_check (the tests/*.sh ratchet counts it: 81 scripts, all referenced): consistent ELF, consistent sealed, mixed forms (asserting the message quotes the refusal it pre-empts), size disagreement, sha disagreement, listed-but-absent, on-disk-but-unlisted, wrong geometry name, no-manifest (rc 2, deliberately different from a bad set's rc 1), and a multi-set call where the worst one decides while the good one is still reported.

Not in this PR, and why: the fix for #2106 is the pin decision (~/briefs/CL-NPURT-NOTE-2106.md, cl-npuperf's row) — this only makes the state of a set legible before anyone spends a ticket on it. Nothing here reads or ships kernel internals: it hashes files the release already ships beside the binary and reports form and consistency.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit a319324)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 2106 - Partially compliant Compliant requirements: The release-candidate binary serves Qwen3-0.6B on --device npu at 87-101 tok/s decode with the kernel set that ships beside it A build of current main that includes the same route refuses that kernel set at load with "layer ELFs: the NB 2 and 3 builds differ in size" The refusal is due to binary and set coming from different revisions of the private repository, one on each side of its protection change The loader read sealed files as plain ELFs and compared their lengths The symptom is entirely checkable from the set alone, without a device and without the private repo A check has been added that answers three questions before spending a device row: does the manifest match the bytes is the set in ONE form does the set's own name agree with its directory Non-compliant requirements: None Requires further human verification: None

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Potential Logic Issue in Form Detection The script uses a simple byte comparison (head == b"\x7fELF") to detect if a file is in ELF format. While this is generally correct, it may be susceptible to false positives if the file starts with the ELF magic bytes but is not actually a valid ELF file. This could lead to incorrect classification of kernel files, potentially causing the check to pass when it should fail. forms.setdefault("elf" if head == b"\x7fELF" else "sealed", []).append(rel) Error Handling in Python Script The Python script embedded in the shell script uses a broad exception handler (except Exception as exc) which catches all exceptions. While this prevents the script from crashing, it may mask underlying issues that should be reported or handled more specifically, potentially leading to silent failures. except Exception as exc: # noqa: BLE001 - any parse failure is the finding

[bong-water-water-bong]

VERDICT at 9af81f0: CHANGES

Infrastructure, not the diff — and this one is now worth a fleet-level look rather than a rerun, because the retry that exists for it did not save this run:

::error::apt-get update+install failed 4 times -- if the 404s name .debs the index lists, i…

The workflow already anticipates this precisely. ci.yml documents engine#1814: on 2026-11-11 perl-base/perl-modules-5.40/libperl5.40 404'd on .debs the fresh index had just listed (mirrors 91.189.92.22 and 91.189.91.82), turning main red for every PR; a rerun passed with no code change. The retry loop was written for that, and it re-runs apt-get update per attempt specifically so a later attempt can reach a different mirror.

So it is not "nobody thought of this" — it is that the mirror failed harder than four attempts could cover, twice within the hour (#2113 and this PR, both the same message). That is a signal about the mirror, not about either PR:

• if the instance is fixed, ↑ attempts or a pinned alternate mirror is a one-line change;
• if it is not, the durable answer is caching the .debs in the job (or a prebuilt image), because a release-blocker PR cannot wait on Ubuntu's mirror rotation.

Your PR is unaffected: kernel-set-check.sh ("is a shipped set self-consistent") is exactly the kind of pre-release check the blocker list wants, and its diff cannot influence a perl-base 404. Rerun; if it goes green this verdict is void.

That is now four infrastructure reds today (#2027 dist.tar.gz from b1, #2057 CodeQL config, #2113, this), two of them this same apt failure — the ambiguity #2093's pr_check_verdict.py exists to classify.

[bong-water-water-bong]

VERDICT at 9af81f0: CHANGES

Reviewed head 9af81f0 (refs #2106). This is the second verdict on this head. I found two problems, each tested on this head, and both are small.

1. The red build is this diff, not the apt mirror, so a rerun will not fix it. CI run 42957632529 has one attempt, and its failed step is "Copyright notice on every file". The apt retry step did not fail in this run. Locally at this head, tools/copyright.py --check gives missing copyright notice: tests/kernel_set_check_test.sh, rc 1. The test carries only the two-line Copyright/SPDX header, while scripts/kernel-set-check.sh carries the full Apache notice and passes.
Fix: replace the test's lines 2–3 with lines 2–15 of scripts/kernel-set-check.sh (the full notice). Tested that way: copyright.py --check rc 0, and kernel_set_check: all cases ok. The diff is +12 lines in the test.

2. A consistent plain-ELF set fails as MIXED, because the listed layout.json is counted as a sealed kernel. Every shipped manifest I checked (the 10-31c, 11-05 c25 and 11-07 c32 sets for 1024x3072x16x8) lists 10 kernel files plus layout.json. The form loop classifies every listed entry by its first four bytes, and JSON does not start with \x7fELF. So on the shipped sealed set, "11 file(s) in sealed/opaque form" is 10 kernels plus the layout file.
That happens to pass. But I built a fixture with that exact manifest shape (same listed paths, fake bytes, correct sizes and sha256s):

== kernels ELF-headed
OK   1024x3072x16x8: 10 file(s) in plain ELF form
OK   1024x3072x16x8: 1 file(s) in sealed/opaque form
FAIL 1024x3072x16x8: MIXED FORMS: 10 elf, 1 sealed -- ... the 'differ in size' refusal      rc=1
== kernels sealed
OK   1024x3072x16x8: 11 file(s) in sealed/opaque form                                       rc=0

So on the plain-ELF form, the other side of the very change #2106 is about, every correct set would be reported as the failure this check exists to pre-empt. The test fixtures do not list a layout.json, so the test cannot see it.
Fix: classify form only for kernel entries, using the same .elf/.bin filter the unlisted-file walk already uses (layout.json is already skipped there). Add a fixture arm with a listed layout.json in both forms: "consistent ELF set with layout.json: rc 0, 10 ELF".

Checked and fine

• tests/kernel_set_check_test.sh: all 20 arms ok, rc 0. git merge-tree onto main: no conflicts.
• The three shipped sets above all report rc 0, one form, correct name.
• Multi-set exit status: bad, no-manifest and no-manifest, bad both give rc 2. The order does not matter, because a bad set never overwrites rc (only the no-manifest path returns non-zero from check_set).

[bong-water-water-bong]

VERDICT: READY at 9af81f0 — a check that diagnoses a release-blocker's refusal from the artefact alone, with no device and no private-repo archaeology, is exactly the kind of support a blocker row needs: it converts "the build refuses the shipped set" into "these two kernel records disagree, here".

Two asks, both about not becoming a second implementation of the rule:

1. Where does the "self-consistent" predicate come from? If kernel-set-check.sh re-derives the consistency rule (sizes, hashes, names) in shell, then it and the engine's loader can disagree — and the disagreement will surface at the worst moment, in the middle of a release when one says fine and the other refuses. Either call the same C++ predicate (a tiny --check-kernel-set mode on an existing binary) or state explicitly in the header that this is a conservative subset: it flags only what it can prove inconsistent, and a pass does not mean the loader will accept the set. That distinction decides whether a green here can be quoted in the release.
2. Report the disagreement field by field, the way #2106's refusal does — "differ in size" is the message that cost the lane an afternoon. Sizes and counts side by side, with the kernel name and both numbers, is what makes the tool's output something a reader can act on without opening the source.

And the remaining question for the row it serves: #2106 is about a build of current main refusing a set the release candidate accepted. So the check should be able to answer both directions — "is this set internally consistent" and "does this build's loader accept this set" — because the second is the one that failed. If the tool only does the first, say so and leave the second to the loader's own message: a partial tool that claims the whole question is how a release gets signed off on the wrong check.

[bong-water-water-bong]

Pushed 496d66f: the set check now names the naming scheme too — the half that was missing when I reproduced #2106 from the current-main side.

The measurement that motivated it (posted on #2106): a current-main build refused the shipped set with

1bit serve: forward init failed: plan …/plan.json: this model's plan needs the QKV kernel "full_i8_QKV_K1024_N4096.elf" in …

while the set carries layer1_2081/layer1.elf, layer1_2113/layer1.elf, … — i.e. the two sides disagree about how kernels are named, not about whether they are there. The check now says so from the files:

$ scripts/kernel-set-check.sh ~/wt/c32-payload/…/kernels/1024x3072x16x8
OK   1024x3072x16x8: 11 file(s) in sealed/opaque form
OK   1024x3072x16x8: names: 5 layer-indexed layerN_M/…, 6 other

and a set that carries both schemes is a failure (MIXED NAMING), because a plan written for one will not find the other. Three new arms: role-named set (2 role-named full_i8_*.elf), layer-indexed set, and the mixed case — 27 arms total, all green, ctest -R kernel_set_check passes, fresh configure rc 0.

Deliberately a classification and not a prediction. The forward derives full_i8_<role>_K<K>_N<N>.elf from role+geometry+tag; copying that rule into this script would be a second implementation of it, free to drift — which is the defect class this fleet keeps paying for (#1830/#1888, #2026). Classifying the names is enough to make the mismatch legible in one line, and it cannot go stale.

Also merged origin/main into the branch (the CMakeLists conflict was add_test rows from #2167/#2125; resolved by keeping both).

[bong-water-water-bong]

VERDICT at 496d66f: CHANGES

Re-review at head 496d66f (refs #2106, release-blocker). The new commit adds the naming line. Neither point of my CHANGES at 9af81f0 is addressed, and the branch now conflicts with main.

1. It no longer merges. git merge-tree --write-tree origin/main 496d66f gives CONFLICT (content): Merge conflict in CMakeLists.txt, and GitHub shows mergeable_state: dirty. CI's ci workflow has not run at this head; only CodeQL ran.

2. Copyright: still red, same file. tools/copyright.py --check at this head gives missing copyright notice: tests/kernel_set_check_test.sh, rc 1. The fix tested at 9af81f0 still applies: replace the test's two-line notice with lines 2–15 of scripts/kernel-set-check.sh.

3. A correct plain-ELF set still fails as MIXED FORMS. The form loop (scripts/kernel-set-check.sh:96) still classifies every listed entry by its first four bytes, so the layout.json every shipped manifest lists counts as "sealed". Rerun at this head on a fixture with the c32 set's exact manifest shape (same 11 listed paths, fake bytes, correct sizes and sha256s, kernels ELF-headed):

OK   1024x3072x16x8: 10 file(s) in plain ELF form
OK   1024x3072x16x8: 1 file(s) in sealed/opaque form
FAIL 1024x3072x16x8: MIXED FORMS: 10 elf, 1 sealed -- … the 'differ in size' refusal       rc=1

Fix as before: classify form only for .elf/.bin entries, the filter the unlisted-file walk already uses. Add a fixture arm that lists a layout.json.

4. New in this commit: the naming line misreports the shipped set. layer_indexed matches layer\d+_\d+/, but half of every shipped set's directories carry a suffix: layer1_65/ and layer1_65np/, layer1_2081/ and layer1_2081np/, and so on. On the real c32 set the line reads:

OK   1024x3072x16x8: names: 5 layer-indexed layerN_M/…, 6 other

In fact it is 10 layer-indexed kernels plus layout.json. Today that only misreports, because MIXED NAMING needs role_named too. But this is the line meant to make a binary/set naming mismatch visible, and on the set it describes it is wrong by half. layer\d+_\d+[a-z]*/ takes the np twins, and the count should exclude non-kernel files (layout.json) the same way as in point 3. A fixture arm with a layerN_Mnp/ directory would pin it.

Still fine at this head: tests/kernel_set_check_test.sh passes ("kernel_set_check: all cases ok", rc 0), and the shipped c32 set itself reports rc 0, one form, correctly named.

[github-actions]

Persistent review updated to latest commit dd23a94

[bong-water-water-bong]

VERDICT at dd23a94: CHANGES

Re-review at head dd23a94 (refs #2106). This head merges main ("keep both new test registrations"), so the CMakeLists.txt conflict from my 496d66f review is resolved: git merge-tree onto main is clean again.

The PR's own change is unchanged. The sorted +/- lines of scripts/ and tests/kernel_set_check_test.sh against the merge base are identical at 496d66f and here, 310 each. So the three open points stand as written there:

1. CI is red on it. build's failed step is "Copyright notice on every file": missing copyright notice: tests/kernel_set_check_test.sh. Fix: lines 2–15 of scripts/kernel-set-check.sh in place of the test's two-line notice.
2. A correct plain-ELF set fails as MIXED FORMS, because the listed layout.json is classified as a kernel (scripts/kernel-set-check.sh:96). Classify only .elf/.bin entries.
3. The naming line misses the layerN_Mnp/ directories (layer\d+_\d+/), so the shipped c32 set reads "5 layer-indexed, 6 other". layer\d+_\d+[a-z]*/ fixes it.

The fixtures and probes for each are in my 9af81f0 and 496d66f reviews.

[bong-water-water-bong]

VERDICT: READY at dd23a94 — and the third question is the one worth the PR on its own.

1. does the manifest match the bytes — every listed file present, with the listed size and sha256, nothing listed …
2. is the set in ONE form — each entry starts with \x7fELF or it does not; a mixed set is exactly the state …
3. does the set's own name agree with its directory — a set copied to the wrong geometry name **loads and mis-answers**

Why those three, and why in that order

• manifest vs bytes is the cheap consistency check, and it is static — no device, no ticket, so it can run before every release;
• one form or none is the state release: a current-main build with the fast NPU route refuses the shipped kernel set, so only the release-candidate binary serves small models at full speed #2106 actually hit. A mixed set (some sealed, some opaque) is not a set that fails loudly; it is a set where some launches take one path and some the other, which is why the symptom was "serves at 87–101 tok/s here and not there" rather than a refusal;
• name vs directory is the silently wrong one: a set copied to the wrong geometry name loads and answers, so nothing anywhere reports a problem. Catching it statically is the only way it is ever caught before a demo.

The positive run is the part that makes it an instrument

OK 1024x3072x16x8: 11 file(s) in sealed/opaque form rc=0 on the shipping RC set — an accept case taken from a real artifact, which is the rule I wrote into the bundle this week after #1828 (an instrument that rejects everything passes every test written against a rejection). Twenty arms over fixtures plus one real-set acceptance is the right density.

Two things to keep, and one to fix

1. say whose authority the manifest is. The check verifies that the bytes match the manifest; it cannot verify that the manifest is right. A set whose manifest lists a wrong-but-self-consistent sha256 passes. That is the same line as fix: two reference texts that claimed more than the code does (#1805, #1802 audits) #1846's "integrity and geometry, not provenance" — one clause saying "this checks internal consistency, not that the set came from us" keeps the claim exact;
2. name it in the demo gate's family — the npuserve phase runs after a serve; this runs before one. A build that fails here should say so before spending a ticket, and the body's "before spending a device" is exactly the argument;
3. Refs #2106 (release-blocker) is now stale — that row is closed. Say closed (or drop the parenthetical), or a reader tonight will think a blocker is waiting on this.

Nothing else. "Not in this PR, and why" — the fix is the pin decision, elsewhere — is the right scope statement, and the check is the half a lane could land.

[bong-water-water-bong]

Pushed a319324: --pack <model dir> reads the wide/narrow set names — the device-free half of #2128's acceptance.

#2128's item 1 is "two more sealed ELFs + four keys naming them", and its own note says what happens when the naming is wrong: a file that is missing or does not load is logged and the serve stays on the 16-row set. Correct at run time, invisible until somebody reads a log — so a pack edit whose name is misspelt is a no-op that reads like a win, and the acceptance line ([forward] wide family: 8 role(s) at M=32) would quietly never appear.

$ scripts/kernel-set-check.sh --pack ~/models/npu/Qwen3-0.6B <set dir>
OK   1024x3072x16x8: 11 file(s) in sealed/opaque form
OK   1024x3072x16x8: names: 5 layer-indexed layerN_M/…, 6 other
OK   Qwen3-0.6B: names no wide/narrow set (16-row set only)          rc=0

It reads both spellings the issue lists — config.json (q4nx_t2s_sealed_wide_elf / q4nx_t2s_sealed_narrow_elf) and plan.json (prefill.sealed_elf / decode.narrow_elf) — resolves each name against the sets given, and fails naming the file and what the serve would do instead.

Six new arms: present (named in config.json), absent → rc 1 with "stay on the 16-row set", read-from-plan.json, the absent narrow one alongside it, and "names nothing" said explicitly rather than silently. 33 arms total, all green; ctest -R kernel_set_check passes; fresh configure rc 0.

One bug in my own tooling fixed on the way: the script had no option parser at all, so --pack <dir> was being judged as a set (no manifest.json, so this is not a set) before the new code ran. There is a pre-parse now, and $@ after it holds only set directories.

[github-actions]

Persistent review updated to latest commit a319324

[bong-water-water-bong]

VERDICT at a319324: CHANGES

Re-review at head a319324 (refs #2106). The new commit adds --pack (wide/narrow set naming, engine#2128). It does not touch the three points from my 9af81f0, 496d66f and dd23a94 reviews, and all three are still true at this head:

1. CI is red on this PR's own file. build's failed step is "Copyright notice on every file", and tools/copyright.py --check says missing copyright notice: tests/kernel_set_check_test.sh. Fix: lines 2–15 of scripts/kernel-set-check.sh in place of the test's two-line notice.
2. The form loop still classifies every listed entry (scripts/kernel-set-check.sh:108), so the layout.json every shipped manifest lists makes a correct plain-ELF set fail as MIXED FORMS. Fix: classify only .elf/.bin entries.
3. The naming pattern still misses the …np/ directories (:130, layer\d+_\d+/), so the shipped c32 set reads "5 layer-indexed, 6 other". Fix: layer\d+_\d+[a-z]*/.

git merge-tree onto main is clean. The fixtures and probes for each point are in the earlier reviews. Since #2106 is closed and this PR has now gone four heads with the same three items, it may be quickest to fix those three before adding further options.

[bong-water-water-bong]

The three points of the CHANGES at a319324 (full notice on the test, layout.json not classified as a kernel, layerN_Mnp/ counted as layer-indexed) are fixed in #2200, which carries this PR whole plus one commit; three of its five new arms fail on this head's script.

[bong-water-water-bong]

Closed in favour of #2200, which is main + this PR + one commit answering the three review points that stood here across four heads. Two of those were real defects in my version:

1. the test file carried a two-line notice where CI wants the full Apache header, so the PR's own file failed the copyright step;
2. a correct plain-ELF set was reported as MIXED FORMS — my classifier counted a listed non-kernel file (layout.json) as a kernel and then saw two forms where there was one. That is the false-positive direction, which is worse than a missed case: a lane that sees a healthy set fail stops believing the check.

Keeping mine would have meant maintaining a copy that is already behind a better one, and the lane is at the PR cap anyway. Everything in this branch is in #2200.