Repository navigation
npu-pack: role -> engine slot for the roles the layer type does not change (engine#1788, 3c placement) - #2285
Conversation
…hange (engine#1788, 3c placement) An import has two sides. The classifier says WHAT a tensor is; the engine reads weights by SLOT name, so the writer needs role -> slot. This lands the seventeen roles whose destination does not depend on the layer type, each one a string literal that exists in npu/forward/src (grepped, not inferred from the HF name), and names the nine it does NOT place with the reason: qkv_proj's prefix is the layer type's, q_norm/k_norm have no verified weight slot, gate_up_proj needs a measured split, and the shared-expert family differs per architecture. The arm with teeth is the coverage one: every role the classifier can emit must be either placed or explained, so a new recogniser without a destination decision fails the test instead of landing a tensor in a slot the engine reads with another shape. Removing qkv_proj's explanation fails it: 'no role falls through the table (qkv_proj)'.
PR Reviewer Guide 🔍(Review updated until commit e926c3b)Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- a mapping table with the three invariants such a table needs, and the one claim that is still a manual grep
Role -> slot is the side of this stack where a mistake is silent: a wrong destination does not fail, it loads a tensor into the wrong place and the model answers badly. So the arms are what matter, and this PR has the three that make a table trustworthy:
- bijection:
!seen.count(kv.second)-- "no two roles share a destination". Two roles mapping onto one slot is the failure that a plausible-looking table hides (it has the right number of entries and the right shape of names), and it is pinned. - totality:
placed.size() + no_slot.size() == kRoles-- every role is either placed or explicitly unplaced, so nothing is silently dropped. Without this,no_slotcould be empty because the code forgot a role rather than because it placed them all. - refusal with a reason:
qkv_projis not placed (it needs the layer type) and the refusal must say why in more than 20 characters -- the same discipline as #2270'sspansand #2277'sno_layer, applied to the destination side.
The naming discipline is the other half. Every right-hand side is "a literal that exists in npu/forward/src", grepped from the engine rather than translated from HF names, because linear_attn. versus self_attn. is exactly the distinction that a translation step gets wrong. gate_exps -> mlp.experts.gate_proj.weight next to gate_proj -> mlp.gate_proj.weight is the pair that would collide under a substring rule; having them as separate literals is what keeps #2277's classifier fix meaningful downstream.
One ask, and it is the only claim here that is not executable: "I grepped the engine's own strings" is a manual step, so the literals are correct today and can drift the moment npu/forward/src renames a field -- silently, because the mapping still compiles and the test still passes. The test has the tree available, so the check is cheap: for each placed slot, assert the literal appears in at least one npu/forward/src/** file (a std::string search over the sources, or a generated header). That converts "these names exist" from a one-time grep into a property that fails when the engine moves out from under it -- which is the same reason #2274's "bytes 128..511 unread" wants a non-zero fill rather than a comment.
Scope discipline worth noting: this slice places only the roles whose destination does not depend on the layer type, and refuses the one that does (qkv_proj) rather than guessing -- which is how the layered-key work stays coherent across #2270 (the second key), #2277 (the classifier) and this (the destinations).
|
VERDICT: READY at "a wrong slot does not fail, it loads a tensor into the wrong place"Nine roles the writer refuses to guess, each with its reason, and the reasons are of two different kinds — which is the valuable part:
Distinguishing "cannot" from "not yet measured" inside one table is what keeps a refusal list from becoming a place where work goes to be forgotten. Two asks
And the stack discipline holds"Part F1's stack so far says what a container's tensors are: #2099/#2187 report the roles, and #2277 (merged) fixed the classifier. An import needs the other side — the engine rea[der]" — three PRs named by what each contributed, which is why this can be reviewed in isolation. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at eb2cc2b: READY
The refusal table is the part that makes this safe, and the catch-all is the part that keeps it safe for containers nobody has seen yet.
The allow-table is deliberately scoped to "the roles a layer type does not change" — embed, final_norm, input_norm, post_norm, lm_head, q/k/v/o_proj, self_attn_gate_proj, gate/up/down_proj — which is exactly the right restriction one PR after #2270 established that the second key is (layer-elided name, layer type). Anything whose slot prefix depends on the layer type is left to the next slice instead of being guessed here.
kRoleNoSlot[] then names the roles it does not place, each with the reason that decided it, and the reasons are the real ones:
qkv_proj— "the slot prefix is the LAYER TYPE's (linear_attn.orself_attn.), so it needs the se[cond key]" — the same fact #2270 measured;q_norm/k_norm— "the engine's normaliser slots are per-architecture; none is verified for this role yet";gate_up_proj— "fused in some containers, two slots in the engine; needs a measured split".
And the three-way classification is what I would have asked for:
if (r.slot_of.count(c.first)) r.placed.push_back(c.first);
else if (r.why.count(c.first)) r.no_slot.push_back(c.first);
else { r.no_slot.push_back(c.first);
r.why[c.first] = "a role this table does not know: it needs a decision"; }A new role arriving in a future container takes the third branch: it is reported unplaced with a reason, never defaulted to a slot. That is the exact branch whose absence produced the #1840/#1841/#2277 family — a mapping that succeeds with the wrong answer is far worse than one that declines, and "A number here is honest; a guessed slot is not" states the tradeoff in the place the next person will read it.
The tests pin the specific slots rather than the table's shape (q_proj → self_attn.q_proj.weight, gate_exps → mlp.experts.gate_proj.weight, input_norm → input_layernorm.weight), and the file's own comment gives the premise: better unplaced than "in a pack that loads a tensor into the wrong slot".
…des it the way the engine does (3c, engine#1788) qkv_proj was the role this slice refused first, because its prefix belongs to the layer type. Now it is placed where the ENGINE has a slot for it: grep of npu/forward/src finds 'linear_attn.qkv_proj.weight' and no 'self_attn.qkv_proj' at all, so a fused qkv on a linear layer has a destination and on a full or sliding attention layer it has none. The tempting mapping -- always emit self_attn.qkv_proj.weight -- is the mutant this pins: it fails two arms. container_role_slot(role, layer_type) returns when there is no destination, and the caller cannot read that as 'unchanged', which is why the report carries the reasons and this returns a string.
Second slice: the layer type decides exactly one destination — and a grep of the engine says it is
|
|
Persistent review updated to latest commit 9a20094 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 9a20094 -- the refusal of qkv_proj is now a fact about the engine rather than a policy, which is a better reason than the one I reviewed
The new head makes the interesting change, and it is in the justification rather than the table:
the pack carries
linear_attn.qkv_proj.weightand has NOself_attn.qkv_projslot at all (grep the literals innpu/forward/src), so a full-attention qkv has no destination at all.
That upgrades qkv_proj from "we decline to place it because the layer type might matter" to "there is nowhere to place it". A refusal with a fact behind it cannot be argued away by a later reader; a refusal with a policy behind it can. Same for the arm that keeps no_slot honest -- kv.second.find("does not know") == std::string::npos, so a placeholder cannot be counted as a placement -- and placed.size() >= 17 pinning the count the body states.
The invariants from my first review are all still there and still pinned: bijection (no two roles share a destination), totality (placed + no_slot == kRoles), and a refusal that must explain itself in more than 20 characters.
My one ask is still open, and it is the only non-executable claim in the PR: "every right-hand side is a literal that exists in npu/forward/src" is still a manual grep -- the comment now says it twice, which is a good sign, but nothing fails when the engine renames a field. The test already lives beside the tree, so one arm reading npu/forward/src/*.c* and asserting each placed slot appears as a substring would make the claim durable, and it would have caught the linear_attn. vs self_attn. distinction this PR is built on if the grep had been done wrong.
Not blocking, and the PR is a strict improvement on the head I reviewed.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 9a20094 — and I verified the central claim harder than the PR states it. One correction to where the evidence lives, and one trap worth naming for the writer that will follow this table.
The claim, verified against the engine rather than the diff
"Every right-hand side is a literal that exists in
npu/forward/src— I grepped the engine's own strings rather than translating names."
I grepped npu/forward/src/ for all 17 and found 15 — mlp.experts.up_proj.weight and mlp.experts.down_proj.weight are not there. For a minute that looked like a wrong slot, which would be the worst kind of bug here ("a wrong slot does not fail, it loads a tensor into a field"). It is not: they are in npu/forward/include/moe_container.h, returned by the function that owns the names:
inline MoeNames moe_names(MoeFormat f) {
if (f == MoeFormat::EngineConverter)
return {"mlp.experts.gate_proj.weight", "mlp.experts.up_proj.weight",
"mlp.experts.down_proj.weight", "mlp.gate.weight", "mlp.gate.bias", …};
return {"mlp.gate_exps_proj.weight", "mlp.up_exps_proj.weight",
"mlp.down_exps_proj.weight", "moe_router.weight", …};
}So all 17 exist, and the three expert slots are not merely "a literal that exists" — they are the exact set the forward loads when moe_format() decides the container uses the engine-converter convention. That is the stronger statement, and it is the one the table deserves.
Correction to the claim's medium, because it cost me the false alarm: the three expert slots live in npu/forward/**include**/moe_container.h, not src/. A reviewer who greps src/ — as the sentence invites — finds nothing for up/down and has to decide whether the table is wrong. Say "the engine's own strings (moe_container.h's moe_names() for the engine-converter convention)" and the check lands first time.
The trap for the next slice: moe_format() prefers FLM
inline MoeFormat moe_format(bool has_flm_gate, bool has_engine_gate) {
return (!has_flm_gate && has_engine_gate) ? MoeFormat::EngineConverter : MoeFormat::Flm;
}A container carrying mlp.gate_exps_proj.weight is FLM whatever else it holds. So a writer that emits this table's names and also leaves an *_exps_proj tensor in the same pack flips the reader to the other convention, and every expert slice then decodes on the wrong grid — silently, which is exactly what the engine#1035 comment warns about. The table is safe today because moe_router is among the nine refusals, so nothing writes gate_exps_proj; but the writer slice that places the router should carry that condition with it: the engine-converter naming is chosen by the absence of *_exps_proj, not by the presence of experts.*.
What is right, and worth saying because it is the discipline I have been asking for
- Nine refusals with the reason in the report, each of the right shape:
qkv_projneeds the second key (its prefix is the layer type),q_norm/k_normare config fields, not verified weight slots,gate_up_projis fused in some containers and two slots in the engine,moe_router's weight slot is not verified, and the four shared-expert slots differ per architecture. Refusing a slot is the correct answer when the engine has not verified one — a guessed slot is indistinguishable from a correct one until a model answers wrongly. - The arm with teeth is coverage, not the spot checks, and it can fail: removing
qkv_proj's explanation producesFAIL slots: no role falls through the table (qkv_proj). That is the right invariant for a table that will keep growing — every role the classifier can emit is either placed or explained, so a new role cannot arrive unplaced. - The collision arm (no two roles claim one slot) guards the other silent failure, and
qkv_projstaying unplaced with a marker means the refusal cannot be quietly dropped.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 9a20094: READY
The new head closes the role the previous one refused, and it refuses it correctly in the complementary direction. qkv_proj was in kRoleNoSlot because "the slot prefix is the LAYER TYPE's"; now the layer type is passed in and decides it:
if (role == "qkv_proj")
return layer_type.find("linear") != std::string::npos ? "linear_attn.qkv_proj.weight" : std::string();And the three test arms are the right three, which is why this passes: the positive (qkv_proj + linear_attention → linear_attn.qkv_proj.weight), the negative (qkv_proj + full_attention → empty, not placed — a full-attention layer's qkv genuinely has no engine slot), and the invariant (q_proj + full_attention → self_attn.q_proj.weight, i.e. "the layer type does not move a role it does not decide"). The second one is the arm that matters: without it, a container with a fused qkv on a full-attention layer would be wired into a linear-attention slot.
The empty return is also the safe failure, and that is worth naming: if the layer type does not match, container_role_slot returns an empty string, so the role lands in no_slot with a reason rather than being defaulted — the same three-way classification this PR already established for unknown roles. A missed match degrades to "unplaced", which is honest; it does not degrade to "wrong slot".
One change I would make, and it is the PR's own thesis applied to its own line: find("linear") is a substring test, and the whole family of defects this stack exists to fix — #1840's missing arm, #1841's missing dot, #2277's needle ordering — is substring matching where an identity was meant. The declared vocabulary in the tree today is exactly {"linear_attention", "sliding_attention", "sliding_window", "full_attention"}, so a match against an explicit set (or a "linear_attention" prefix) would say which layer types are linear-attention rather than which contain a word.
It is not a blocker, and I want to be precise about why: with a miss the result is unplaced, so the risk is only a false positive — a future declared value that contains "linear" without being linear attention, which would map a qkv to linear_attn.qkv_proj.weight. That is implausible with today's four values and exactly the shape of thing that stops being implausible later. An exact set also makes the test pin the vocabulary rather than a substring, so adding a new linear-attention family (gdn, mamba-style) becomes a deliberate edit with a test failure rather than something that silently starts matching.
Everything else in the diff is unchanged from the head I readied at eb2cc2b, and the refusal table, the catch-all for unknown roles, and "a number here is honest; a guessed slot is not" all still hold.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 9a20094
Built and ran it: all npu-pack checks passed -- and the arms are the four that a role->slot table fails on, not a list of lookups.
The sentence that makes the whole slice safe is in the header, and it is the reason the nine refusals matter:
"a wrong slot does not fail loudly, it loads a tensor into a field the engine reads with another shape, and the symptom is a wrong number rather than an error."
That is the exact failure mode of every mapping table in this family (#2187's share_gate_exps_proj, #2277's self_attn.gate_proj): the mistake compiles, loads and produces a plausible number. So a table that
places only what it can justify, and names the rest, is strictly better than one that places everything.
The four structural checks:
slots: no two roles share a destination (mlp.experts.gate_proj.weight) <- injectivity
slots: the fused qkv is not placed without the layer type
slots: and the refusal says why <- why.at(...).size() > 20
slots: at least the layer-type-independent ones are placed <- ≥ 17
Injectivity is the one I would have asked for: two roles mapping to one slot is the mistake that loads the right tensor in the wrong place, and it is invisible in any per-role check. And the "refusal says why" arm
with a length floor is a cheap way to assert that the reason is a sentence rather than a placeholder -- "" and "no" both fail it.
And the fallback is right too: a role the table does not know gets its own reason ("a role this table does not know: it needs a decision") rather than being silently joined to the deliberate refusals. So
no_slot has two populations with different meanings -- decided not to place and could not decide -- and both are visible.
container_role_slot(role, layer_type) being the one function where the layer type decides is the right seam: qkv_proj places under linear_attention and refuses under full_attention and
sliding_attention, so the three-way negative is pinned rather than a single case.
One note: check(int64_t(s.placed.size() + s.no_slot.size()) == sizeof(kRoles)/…) is the completeness arm -- every role in the table is accounted for -- and that is the guard against a role that falls through
both branches silently. Worth pairing it with a check that placed ∪ no_slot has no duplicates, since a role counted twice would keep the sum right.
Not run: a real container (the arms use role fixtures), so the engine slot strings are pinned to npu/forward/src's literals rather than to a pack.
…g (review of #2285) find("linear") was a substring test where an identity was meant -- the defect family this stack exists to fix (#1840, #1841, #2277's needle order). Now kLinearAttentionLayerTypes is an exact set, so a declared value that merely CONTAINS 'linear' does not map a fused qkv into linear_attn.qkv_proj.weight, and a new family (gdn, mamba-style) has to be added deliberately, with a test failure if the vocabulary moves without it. Two arms pin the difference: 'nonlinear_attention' and 'linear_sliding_attention' must both stay unplaced. With the substring restored (the mutant) both fail; a missed match still degrades to 'unplaced', never to a wrong slot.
Applied —
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at e926c3b — the new commit is the right fix and it belongs to the family this whole stack exists to repair. Two asks from my last review are still open; neither blocks.
The new commit, checked against the real spellings
static const char* const kLinearAttentionLayerTypes[] = {"linear_attention"};
static bool is_linear_attention(const std::string& layer_type) {
for (const char* t : kLinearAttentionLayerTypes) if (layer_type == t) return true;Replacing layer_type.find("linear") with an exact set is correct precisely because an exact set can only be wrong loudly — an unmatched new family returns false, whereas find("linear") silently matched anything containing it. Its comment names the family it belongs to (#1840, #1841, #2277's needle order), which is the right way to record that this is a class of defect and not a one-off.
And the set matches the real values. Every layer_types list across the packs and plans on this box uses the spellings full_attention, sliding_attention, linear_attention, conv — linear_attention exactly, no variant, so the exact match does not narrow a real case. That was the failure mode worth checking: an exact set that forgets a real spelling is worse than the substring, because it fails where the old code worked.
Still open from my last review, and both are cheap
- The claim's medium. The body says the 17 right-hand sides are "a literal that exists in
npu/forward/src". Fifteen are;mlp.experts.up_proj.weightand…down_proj.weightare innpu/forward/include/moe_container.h, returned bymoe_names(MoeFormat::EngineConverter). A reviewer who grepssrc/, as the sentence invites, finds nothing for those two and has to decide whether the table is wrong — which is what happened to me. Naming the header makes the check land first time. - The trap for whoever writes the router.
moe_format()returnsEngineConverteronly when!has_flm_gate && has_engine_gate— a pack carryingmlp.gate_exps_proj.weightis FLM whatever else it holds. So a writer that emits this table's names and leaves an*_exps_projtensor in the same container flips the reader to the other convention, and every expert slice then decodes on the wrong grid, silently. Safe today (moe_routeris among the nine refusals, so nothing writesgate_exps_proj), but the writer slice that places the router should carry that condition with it: the engine-converter naming is chosen by the absence of*_exps_proj, not by the presence ofexperts.*.
|
Persistent review updated to latest commit e926c3b |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at e926c3b
Re-reviewed at the new head: the substring rule is gone, and the two new arms kill it when it comes back -- I reverted it to check.
$ (layer_type.find("linear") != npos -- the old rule, restored by hand)
failing checks: 2
$ (restored to the exact set) all npu-pack checks passed
Exact set, with the reason stated in the source:
static const char* const kLinearAttentionLayerTypes[] = {"linear_attention"};
static bool is_linear_attention(const std::string& layer_type) { … layer_type == t … }"A value that merely contains "linear" is not linear attention, and a new family (gdn, mamba-style) must be added here on purpose."
linear_sliding_attention and nonlinear_attention are the two arms that matter, and they are the right pair: one is a real layer type whose name contains linear but is not linear attention,
the other is the naked substring case. Between them, find("linear") cannot pass and == "linear_attention" cannot over-refuse.
This is the third instance of the family named in its own comment -- "the defect family this stack exists to fix is substring matching where an identity was meant (#1840, #1841, #2277's needle
order)" -- and the first one where the fix is a closed set rather than a longer needle. A closed set is the right shape here because the alternative (ordering the needles, as #2277 did with
share_gate_exps_proj before gate_exps_proj) works only until someone adds a third overlapping name.
One note, and it is about the closed set's cost: a new linear-attention family now needs an edit here -- which is exactly what the comment says, and it is a real cost if a pack arrives with gdn
tomorrow. The behaviour on an unknown type is std::string() (refused, not guessed), so the failure is a refusal rather than a wrong slot, which is the safe direction this stack has chosen everywhere.
Unchanged from my earlier review: injectivity ("no two roles share a destination"), the why length floor, the completeness sum, and the two populations in no_slot.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at e926c3b: READY
The substring is gone, and the replacement is better than what I suggested because it comes with the two tests that make the difference a fact rather than a judgement call.
static const char* const kLinearAttentionLayerTypes[] = {"linear_attention"};
static bool is_linear_attention(const std::string& layer_type) {
for (const char* t : kLinearAttentionLayerTypes) if (layer_type == t) return true;
return false;
}My note said the risk with find("linear") was only a false positive — a declared layer type that contains "linear" without being linear attention — and that it was implausible with today's vocabulary and exactly the shape that stops being implausible later. The diff does not ask me to keep believing that; it pins both halves:
qkv_proj+nonlinear_attention→ empty, with the arm naming the rule: "a layer type that merely contains 'linear' is not linear attention";qkv_proj+linear_sliding_attention→ empty as well.
So a future container whose layer type is a linear-attention variant is now refused (unplaced, with a reason) instead of being mapped on the strength of a word, and the only way to admit a new family is to add it to kLinearAttentionLayerTypes — a one-line edit that shows up in a diff and fails a test if someone forgets the arm. That is the difference between a rule and a coincidence of vocabulary.
is_linear_attention as a named predicate is the other half of the fix. It puts the question "is this layer type linear attention?" in one greppable place instead of inside a ternary, so the next reader (and the next family: gdn, mamba-style) has an obvious address for it.
Everything else is unchanged from the heads I readied at 9a20094/eb2cc2b, and all of it still holds: the allow-table restricted to roles a layer type does not change, the kRoleNoSlot refusal table with a reason per role, the three-way classification where an unknown role becomes no_slot with "a role this table does not know: it needs a decision" rather than a default slot, and "a number here is honest; a guessed slot is not".
#2291) The review's mutant was right and it is the important kind: the result this PR exists for -- every container having an empty unknown bucket -- was printed and never asserted, so removing {"ssm", ...} left the suite green (rc 0) while the box printed UNKNOWN: ssm. Two changes: * in the real-container leg, collect the roles that hit the "does not know" default and check that the set is empty, naming them in the failure; * a NO-PACK arm for the roles the containers actually carry (ssm, gemma_local, container_extra, the four share_*, q_norm, k_norm, moe_router): each must be explained, none may be defaulted, and no reason may be a bare placeholder. The mutant now fails where CI runs: $ ./npu_pack_import_test # no pack: what ctest runs FAIL slots: the table explains ssm rather than defaulting it rc=1 Also spelled out the last placeholder: k_norm's reason still began "same: ..." (inherited from #2285) and read exactly like the bare "same" this PR removed from the shared-expert entries.
…roles it could not name (engine#1788, 3c) (#2291) * npu-pack: the placement measured on the five containers, and the two roles it could not name (engine#1788, 3c) The placement had no real-container evidence at all: the pack leg gates on model.onnx and a container ships model.q4nx. This adds the leg that reads a container with the shipped reader and reports put/held/unnamed plus the reasons and the unknown roles, then closes what it found on the five containers: ssm -> refused with a reason: one role over dt.bias, conv1d, norm and a, four destinations gemma_local -> refused with a reason: a group of seven stems, three of which DO have engine slots container_extra-> refused with a reason: recognised, and deliberately not written (rope_freqs is a flag, not a weight) Before: the 35B reported 210 tensors under "a role this table does not know: it needs a decision", which is the table saying nothing. After, on every container on this box, that bucket is empty: gemma3-270m put 165, held 72, unnamed 0, UNKNOWN none (237 tensors) gemma4-e2b put 320, held 389, unnamed 0, UNKNOWN none (709) bolt-e4b put 383, held 466, unnamed 0, UNKNOWN none (849) MiniCPM 1B put 183, held 174, unnamed 0, UNKNOWN none (357) MiniCPM 35B put 303, held 430, unnamed 0, UNKNOWN none (733) Also fixed: the refusal table carried the literal string "same" as the reason for three shared-expert roles and it reached the output (35B: "held: 120 x same"). A placeholder is not a reason; each of the four now says what is unverified. held is not a failure count: it is the list of decisions the writer still owes, and every one of them now has a stated reason instead of a shrug. * npu-pack: a group role's destination is in the tensor suffix, where the engine has one (engine#1788, 3c) ssm and gemma_local were refused as groups, which was right, but two of gemma_local's seven stems DO have slots in npu/forward/src -- pre_feedforward_layernorm and post_feedforward_layernorm -- and for those the fourth key is not a mystery: the suffix IS the destination. Placed now, with the other five members still refused. Measured on the containers (put / held, before -> after): gemma3-270m 165/72 -> 201/36 (+36, 18 layers x 2) gemma4-e2b 320/389 -> 390/319 (+70, 35 x 2) bolt-e4b 383/466 -> 467/382 (+84, 42 x 2) MiniCPM 35B 303/430 -> 303/430 (unchanged: no gemma_local in a MiniCPM) held falls by exactly what put gains, which is the accounting this PR asserts. The five members with no engine slot (inp_gate, layer_output_scale, per_layer_projection, post_layernorm, per_layer_model_proj) stay refused with their reason. * tests: the unknown bucket is CHECKED, in CI's own invocation (review of #2291) The review's mutant was right and it is the important kind: the result this PR exists for -- every container having an empty unknown bucket -- was printed and never asserted, so removing {"ssm", ...} left the suite green (rc 0) while the box printed UNKNOWN: ssm. Two changes: * in the real-container leg, collect the roles that hit the "does not know" default and check that the set is empty, naming them in the failure; * a NO-PACK arm for the roles the containers actually carry (ssm, gemma_local, container_extra, the four share_*, q_norm, k_norm, moe_router): each must be explained, none may be defaulted, and no reason may be a bare placeholder. The mutant now fails where CI runs: $ ./npu_pack_import_test # no pack: what ctest runs FAIL slots: the table explains ssm rather than defaulting it rc=1 Also spelled out the last placeholder: k_norm's reason still began "same: ..." (inherited from #2285) and read exactly like the bare "same" this PR removed from the shared-expert entries. * npu-pack: the ssm family, mapped one for one onto the engine slots that exist (engine#1788, 3c) ssm was the biggest refusal and the one that mattered most: 210 tensors on the 35B, the acceptance case. Unlike gemma_local, the ENGINE HAS A SLOT FOR EVERY MEMBER -- grepping npu/forward/src finds the whole family, and its literal names match the classifier's own ssm needles one for one: ssm_alpha_proj -> linear_attn.ssm_alpha_proj.weight ssm_dt -> linear_attn.ssm_dt.bias ssm_beta_proj -> linear_attn.ssm_beta_proj.weight ssm_norm -> linear_attn.ssm_norm.weight ssm_conv1d -> linear_attn.ssm_conv1d.weight ssm_out_proj -> linear_attn.ssm_out_proj.weight ssm_a / .ssm.a -> linear_attn.ssm_a (suffix-only: `ssm_a` is a PREFIX of `ssm_alpha_proj`, so a find() would shadow alpha and beta -- the substring defect in its exact form) Measured, put / held before -> after: MiniCPM 35B 303/430 -> 513/220 MiniCPM 1B 183/174 -> 345/12 gemma3-270m 201/36 -> 201/36 (unchanged: no ssm family) gemma4-e2b 390/319 -> 390/319 bolt-e4b 467/382 -> 467/382 And the ssm refusal's own reason is corrected: it used to say the role "covers four destinations with four shapes", which stopped being true the moment six of them mapped -- an unmatched member is now a name not yet seen, not a destination that does not exist. * npu-pack: the MoE destinations are the CONTAINER spelling, not the EngineConverter one (engine#1788, 3c) moe_container.h documents TWO naming conventions and they disagree on every MoE name. This table was using the EngineConverter spelling, and a container is the FLM shape -- moe_format() says so ("a container with neither is the FLM shape") and the 35B proves it on disk: `gate_exps_proj` occurs, `experts.gate_proj` does not. So the placement was naming slots the engine would not look for in a pack of this genre, which fails silently as zero experts rather than as an error. gate/up/down_exps -> mlp.{gate,up,down}_exps_proj.weight (was mlp.experts.*) moe_router -> moe_router.weight (was refused) share_*_exps -> mlp.share_{gate,up,down}_exps_proj.weight shared_expert_gate -> mlp.share_router.weight Measured, put / held: MiniCPM 35B 513/220 -> 713/20 the 20 held are the q_norm/k_norm pairs, config fields in this engine MiniCPM 1B 345/12 -> 345/12 already only the norm pairs gemma3/4, bolt unchanged no MoE, and gemma_local's unplaceable five Also: my first attempt at this edit removed the five entries I had just added instead of the five refusal rows -- both tables now edited by exact text, and the refusal table no longer contains roles that are placed. * npu-pack: q_norm and k_norm ARE weight slots -- the placement is now complete on the acceptance case I refused these two for two PRs on the strength of a grep that found only the rmsnorm debug label ("q_norm") and concluded they were config fields. model.c names them explicitly, in both container spellings: "model.layer{s}.%d.self_attn.q_norm.weight" -> q_norm_weight "model.layer{s}.%d.self_attn.k_norm.weight" -> k_norm_weight Placed, and the two refusals removed. Measured, put / held / unnamed: MiniCPM 35B 733 / 0 / 0 of 733 <- the acceptance case: every tensor has a destination MiniCPM 1B 357 / 0 / 0 of 357 gemma3-270m 237 / 0 / 0 of 237 gemma4-e2b 460 / 249 / 0 of 709 the 249 are gemma_local's five members with no engine slot bolt-e4b 551 / 298 / 0 of 849 So three of the five containers place completely, and the two that do not are held only by tensors this engine genuinely has no field for -- which is the refusal doing its job rather than a gap in the table. * tests: a role may not be both placed and refused, and the five refusals are pinned (review of #2291) The review named a hazard review caught and code should catch: during one of my edits this file briefly held two role->destination tables with the same keys and different values, and container_role_slot() would have returned whichever came first. The observable invariant now has an arm: no role is in both slot_of and why, and every role is in exactly one bucket. The five refusals are pinned by name, and the mix is the insight: two are about the SECOND KEY (qkv_proj needs the layer type, gate_up_proj a measured split) and three are GROUP roles (ssm, gemma_local, container_extra) that have no single destination at role granularity even though their members are placed per tensor by suffix. Role-level refusal and per-tensor placement are different answers and both are correct. Also pinned: the MoE destinations are the CONTAINER (FLM) spelling -- gate_exps -> mlp.gate_exps_proj.weight and moe_router -> moe_router.weight -- so a quiet revert to the EngineConverter names, which the engine would not look for in a pack of this genre, fails here instead of producing a pack with zero experts. * tests: a placed slot is checked against the rank and dtype the engine documents (3c, engine#1788) A placement can reach the right NAME with the wrong shape, and the forward then reads the tensor as something else. The rank and dtype are architecture-independent even where the extents are not, and model.h documents them per field, so the real-container leg now asserts them on every placed row: moe_router.weight 2-D, BF16 *exps_proj.weight (routed and shared) 3-D, I8 Measured on the 35B, the router lands at [2048,256] BF16 -- bit for bit the shape model.h annotates for that field -- and every expert stack is 3-D I8 with K=4736, the container's own hidden size. bolt-e4b passes the same arms on the slots it places. PlacementRow carries shape and dtype now, so the check is against the tensor that was actually placed rather than against the name alone. * npu-pack: layer-qualify the destination, using the container own prefix (engine#1788, 3c) The collision check -- run before it was written as an arm -- said the placement was not writable: 733 tensors pointed at 26 destinations, because every slot was a role-shaped name with no layer. The engine looks up "model.layer{s}.%d.<slot>", so the destination is the source tensor's own prefix through the layer digits plus the slot. Using the SOURCE's prefix rather than choosing a dialect is the point: model.c probes both "model.layers.%d." and "model.layer.%d." because MiniCPM spells it singular, so a placement that echoes the container cannot disagree with it. model.layer.0.input_layernorm.weight [2048] BF16 model.layer.0.linear_attn.qkv_proj.weight [256,8,4736] I8 model.layer.0.linear_attn.ssm_a [32] F32 MiniCPM 35B and 1B and gemma3 now have a UNIQUE destination per placed tensor. Two containers still do not, and the collision check named exactly which: gemma4/bolt: model.embed_tokens.weight <- model.embed_tokens.weight + model.embed_tokens.weight.scale model.norm.weight <- model.norm.weight + model.per_layer_proj_norm.weight Both are mapping questions, not layer artifacts, and both are the silent kind: a .scale companion is not the embedding, and per_layer_proj_norm has its own slot in the engine. Deliberately left for the next commit with the check that names them, rather than shipping a red arm here. * npu-pack: a .scale companion is not its weight, and per_layer_proj_norm is not the final norm (3c, engine#1788) Both found by the collision check naming two source tensors at one destination on gemma4/bolt: model.embed_tokens.weight <- model.embed_tokens.weight + model.embed_tokens.weight.scale model.norm.weight <- model.norm.weight + model.per_layer_proj_norm.weight The engine reads BOTH .scale names as literals of their own, so a scale companion gets role weight_scale and an IDENTITY destination (its own name), which is what the forward looks up. per_layer_proj_norm gets its own role and its own top-level slot; it was landing on the model final norm only because the general norm rule ran first. The .scale rule is placed FIRST in the classifier, before the needle tables, because that is the same ordering defect as attn_q versus attn_q_norm (engine#2097): the general rule was matching model.embed_tokens.weight.scale by substring before anything could ask whether it was a companion. gemma4, bolt and the 35B now have a unique destination per placed tensor. The 1B does not, and the check named why: it carries PAIRED dtype variants -- linear_attn.ssm_alpha_proj.bf16.weight beside linear_attn.ssm_alpha_proj.weight -- so both reach one slot. That is a source-format question (which of the pair the forward wants), not a role question, and it is the next commit rather than a guess here. * npu-pack: dtype-tagged copies are refused, and every destination is now unique on all five containers (3c) The 1B ships each ssm weight twice, once with a .bf16. marker, and both reached one slot. The engine has no .bf16. literal at all, so that copy has no destination and must not reach the untagged one: it becomes role dtype_variant, refused with that reason, from the same first position in the classifier as .scale (before every needle table). And the collision check, which caught the placement being non-writable and then all three mapping classes, now runs as an arm on every real container: placement: every placed tensor has its own destination in <dir> Measured, put / held / unnamed, and the check passing on all five: MiniCPM 35B 733 / 0 / 0 gemma3-270m 237 / 0 / 0 MiniCPM 1B 321 / 36 / 0 gemma4-e2b 461 / 248 / 0 bolt-e4b 552 / 297 / 0 The 1B held count went UP by 36 and that is the fix: 36 .bf16. copies that were silently overwriting their untagged siblings are now named and refused. The placement is layer-resolved, dialect-correct, rank/dtype-verified and collision-free -- which is the minimum for writing bytes without loading one tensor over another. * tests: arm the .scale rule and its ORDERING, and per_layer_proj_norm (review of #2291) The review's mutant was exact: deleting the .scale branch left every arm green, even though its own comment says the cost is a pack that writes scales over the embedding with no error. Four arms now, and the third is the one that also fails if the branch is refactored INTO the needle loop -- which is the change that would restore the defect silently: 1. container_roles: the .scale companion is weight_scale, not embed 2. its destination is its own name (model.embed_tokens.weight.scale), which q4nx_forward reads 3. a per-layer companion takes its own name, so it cannot land in the down_proj slot 4. per_layer_proj_norm is not the final norm and has its own top-level slot The mutant (delete the branch) now fails four arms: FAIL scale: the .scale companion is NOT filed as embed FAIL scale: its destination is its OWN name, which q4nx_forward reads FAIL scale: and it does not collide with the embedding FAIL scale: a per-layer companion is weight_scale too * tests: pin every layer-independent destination, because CI runs with no pack (self-review of #2291) The reviews of this PR kept finding the same class -- a rule whose deletion is invisible -- and applying that to my own leg found it again: the real-container leg arms the destination table only when a pack is given, and ctest runs this binary with NO pack. So a wrong destination for any single role was invisible in CI. Changing one character of o_proj's slot now fails: FAIL tables: o_proj -> self_attn.o_proj.weight (got: self_attn.o_proj.weight ) All 25 layer-type-independent destinations are pinned exactly, with the value printed on failure. --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…rs in, in info (engine#1788, 3c residue) The residue of the closed #2187, rebuilt fresh on main rather than resolved out of ten conflict blocks. Two measurements the writer needs before it can decide whether a tensor is one the engine can consume: payload_rows : counts the 3-D I8 tensors by their payload width (the K the fconst tile is built for) role_shapes : which shapes each role appears in Both are measurements over the container, not mappings, so they cannot be wrong about a destination -- which is why they survive while #2187's placement table did not (main has that table from #2285/#2291). Measured on the 35B: payload_rows {4736: 370, 8704: 1} the one 8704 is lm_head, as its shape says role_shapes 22 roles; down_exps -> [16384,2,4736] I8 x 40 Carried on the Plan and emitted from manifest(), because manifest() is where the container report is rendered and it has no container in scope -- that is the whole reason the first attempt at this port produced two functions with no reader and was reverted. The trip through the plan is the fix, not a style choice. # Conflicts: # npu/pack_import.cpp (cherry picked from commit d7a423080781fd78c5c7ab88c4d8ddbef0036d4f)
…rs in, in info (engine#1788, 3c residue) The residue of the closed #2187, rebuilt fresh on main rather than resolved out of ten conflict blocks. Two measurements the writer needs before it can decide whether a tensor is one the engine can consume: payload_rows : counts the 3-D I8 tensors by their payload width (the K the fconst tile is built for) role_shapes : which shapes each role appears in Both are measurements over the container, not mappings, so they cannot be wrong about a destination -- which is why they survive while #2187's placement table did not (main has that table from #2285/#2291). Measured on the 35B: payload_rows {4736: 370, 8704: 1} the one 8704 is lm_head, as its shape says role_shapes 22 roles; down_exps -> [16384,2,4736] I8 x 40 Carried on the Plan and emitted from manifest(), because manifest() is where the container report is rendered and it has no container in scope -- that is the whole reason the first attempt at this port produced two functions with no reader and was reverted. The trip through the plan is the fix, not a style choice. # Conflicts: # npu/pack_import.cpp (cherry picked from commit d7a423080781fd78c5c7ab88c4d8ddbef0036d4f) (cherry picked from commit 3b866fe)
… in, in info (engine#1788, 3c residue) (#2304) * npu-pack: a container's payload widths and the shapes each role appears in, in info (engine#1788, 3c residue) The residue of the closed #2187, rebuilt fresh on main rather than resolved out of ten conflict blocks. Two measurements the writer needs before it can decide whether a tensor is one the engine can consume: payload_rows : counts the 3-D I8 tensors by their payload width (the K the fconst tile is built for) role_shapes : which shapes each role appears in Both are measurements over the container, not mappings, so they cannot be wrong about a destination -- which is why they survive while #2187's placement table did not (main has that table from #2285/#2291). Measured on the 35B: payload_rows {4736: 370, 8704: 1} the one 8704 is lm_head, as its shape says role_shapes 22 roles; down_exps -> [16384,2,4736] I8 x 40 Carried on the Plan and emitted from manifest(), because manifest() is where the container report is rendered and it has no container in scope -- that is the whole reason the first attempt at this port produced two functions with no reader and was reverted. The trip through the plan is the fix, not a style choice. # Conflicts: # npu/pack_import.cpp (cherry picked from commit d7a423080781fd78c5c7ab88c4d8ddbef0036d4f) (cherry picked from commit 3b866fe) * npu-pack: payload_rows counted TENSORS; emit both, and let the numbers differ (review of #2304) `out[K]++` counted one per tensor under a field called `payload_rows`, so on the 35B the old field said 370 at K=4736 where the truth is 1003840 rows -- off by 2713x at exactly the width the fconst tile is built for. Both are emitted now, each under what it counts, and the 8704 row count (7755) independently matches the lm_head's own tensor shape [7755,8,8704]. (cherry picked from commit 5d619cd) * tests: arm the two measurements, and that their numbers differ (review of #2304) The arm the review asked for, re-applied by hand after the rebase: at K=4736 the fixture has TWO tensors and 2x4096 = 8192 rows, and the assertion that fails if anyone re-merges the two fields is payload_rows != payload_tensors at that width. --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
3c placement, first slice — the writer's destination side, and the nine roles it refuses to guess
Part F1's stack so far says what a container's tensors are: #2099/#2187 report the roles, and #2277 (merged) fixed the classifier. An import needs the other side — the engine reads weights by slot name, not by role — so this lands role → slot for the roles whose destination does not depend on the layer type.
Every right-hand side is a literal that exists in
npu/forward/src— I grepped the engine's own strings rather than translating HF names, becauselinear_attn.versusself_attn.is exactly the distinction that makes this worth doing carefully.The nine it refuses, with the reason in the report — because a wrong slot does not fail, it loads a tensor into a field the engine reads with another shape, and the symptom is a wrong number:
qkv_projlinear_attn.orself_attn.), so it needs the second keyq_norm,k_normk_normis a config field, not a verified weight slotgate_up_projmoe_routermlp.router.bias; the weight slot is not verifiedshare_gate_exps,share_up_exps,share_down_exps,shared_expert_gateThe arm with teeth is coverage, not the spot checks. Every role the classifier can emit must be either placed or explained, so a new recogniser without a destination decision fails here instead of shipping a wrong slot. Mutation, run:
The rest: no two roles may claim the same slot (a collision silently drops one), and
qkv_projmust stay unplaced with a reason longer than a word.Scope, stated plainly: this is the naming half of 3c. It does not write a pack, and it does not decide the layer-type-dependent nine — that is the next slice, and it needs the
(role, layer type)partition the reporting half already produces. Nothing here touches the NPU, and no container was imported.npu_pack_import_testpasses;tools/copyright.py --checkclean.