Skip to content

npu: the head image's cache key, because the same container packs differently under different kernel sets (engine#1716) - #1832

Merged
bong-water-water-bong merged 1 commit into
mainfrom
1bit/head-image-key
Nov 10, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
1bit/head-image-key

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Reading run_lm_head corrected my own design on #1716: the image's geometry does not come from the container.

int i_qkv = -1;                                  // widest design with K == kpad_
const int T = designs_[i_qkv].N;                 // <- the KERNEL SET's tile width
const size_t tile_elems = (size_t)kpad_ * (size_t)T;
lctx.packB_into(*pw.bo, bt.data(), kpad_, T, ...);   // and the design's `blocked` flag changes the byte order

so one container packs to a DIFFERENT image under kernels-h4096-b8 than under a non-blocked set. A key that omitted kpad, T
or blocked would serve one kernel set's image to another -- and since the acceptance test is a correlation, the wrong image
would still look plausible, which is exactly the class of failure this contract exists to make impossible.

So the key is a digest over {format version, kpad, T, blocked, head-hash}, and the file name is that digest: a cache written by
one run is found by the next, and two kernel sets cannot share one entry.

4 arms, device-free, in npu_design_grid_test (94 ok lines total): each of the four facts changes the digest individually; the
name is stable for the same key; and it CHANGES when T changes. tools/copyright.py --check rc 0.

Still the format and the key only: the cache itself (first-serve fill, bounded-sample verification inside run_lm_head where
kpad_/T are in scope, and the npu-pack import --kernels pre-warm) is the next slice.

…ferently under different kernel sets (engine#1716)

Reading `run_lm_head` corrected my own design on #1716: the image's geometry does not come from the container.

    int i_qkv = -1;                                  // widest design with K == kpad_
    const int T = designs_[i_qkv].N;                 // <- the KERNEL SET's tile width
    const size_t tile_elems = (size_t)kpad_ * (size_t)T;
    lctx.packB_into(*pw.bo, bt.data(), kpad_, T, ...);   // and the design's `blocked` flag changes the byte order

so one container packs to a DIFFERENT image under `kernels-h4096-b8` than under a non-blocked set. A key that omitted kpad, T
or blocked would serve one kernel set's image to another -- and since the acceptance test is a correlation, the wrong image
would still look plausible, which is exactly the class of failure this contract exists to make impossible.

So the key is a digest over {format version, kpad, T, blocked, head-hash}, and the file name is that digest: a cache written by
one run is found by the next, and two kernel sets cannot share one entry.

4 arms, device-free, in `npu_design_grid_test` (94 ok lines total): each of the four facts changes the digest individually; the
name is stable for the same key; and it CHANGES when T changes. `tools/copyright.py --check` rc 0.

Still the format and the key only: the cache itself (first-serve fill, bounded-sample verification inside `run_lm_head` where
`kpad_`/`T` are in scope, and the `npu-pack import --kernels` pre-warm) is the next slice.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at b1e05e5

This is the provenance binding that #1802's audit said was missing, and it is better than what the review asked for. The key is a digest of exactly the
four facts that decide the bytes:

kpad        K padded to the kernel grid (the BO's row count)
T           the widest design with K == kpad_   (designs_[i_qkv].N)
blocked     1 when the design packs blocked (b8) — changes the byte order
head_hash   FNV-1a over the container's lm_head.weight bytes

and the reasoning for each is in the file rather than implied: "the SAME container packs to a DIFFERENT image under a different kernel set. A key that omitted
kpad, T or blocked would serve one set's image to another — and the aggregate correlation would still look plausible, which is the failure this whole
contract exists to make impossible."
head_hash is the answer to #1802's gap: an image is now keyed on the weights it came from, not only on geometry.

Built and ran it — the arm perturbs each of the four fields and requires a distinct digest:

ok   head image key: kpad, T, blocked and the head hash each change the digest (got 4, want 4)
ok   head image key: it names a file
ok   … (same key) names the same file; (a different T) names a different one
all cases ok

That is the right shape for a cache key: one field at a time, each provably load-bearing, plus the naming round-trip. A key that ignored blocked, or
that digested only kpad and T, fails immediately.

One thing to add when a caller appears (not blocking, nothing calls the API yet): the digest is 64-bit FNV over four facts, which is fine for accidental
collision but worth stating at the naming site — the same note the header now carries about integrity vs provenance. Two lanes generating images into one
cache directory is exactly the accidental-collision case, and a name that hashed only kpad/T would have collided often.

Not verified: no caller, so no end-to-end path; the key is pure arithmetic and runs in the fixture.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at b1e05e5 — this is the missing half of the #1802 audit, and the framing is exactly right.

// The cache KEY (engine#1716, from reading run_lm_head): a head image is only valid for one combination of the kernel-
// … so the SAME container packs to a DIFFERENT image under a different kernel set. A key that omitted `kpad`, `T` or
// The key is a 64-bit digest of exactly those four facts, one of which is a digest of the head bytes themselves.
struct Q4nxHeadImageKey { … uint64_t head_hash; };   // FNV-1a over the container's lm_head.weight bytes

On #1802 I argued that the payload hash proves integrity, not provenance, because it is computed over the bytes the reader just read — so the version string was the only field standing between a stale image and acceptance. #1846 has since adopted that sentence in the header, and this PR supplies the other half: the key now includes the kernel parameters that change the packed bytes (kpad, T) plus a digest of the head weights, so a container that packs differently under a different kernel set lands on a different key rather than on a different image under the same name.

That is the correct decomposition: identity of the inputs and the transform, not of the output.

Two things worth adding, both cheap

  1. the disagreement arm: two containers identical except kpad (or T) must produce different keys. Without it, the only exercised path is the one where everything matches, which is indistinguishable from a key that ignores the field it claims to include — the same argument I made for npu-pack: take a 1.9 pack's stride and second-header offset from its own header, not from the epsilon-anchor vote (refs #1740) #1819's stride.
  2. say what a key collision means: 64 bits of FNV over four facts is fine for a cache and not a security boundary, same wording as the header now uses for the payload hash ("integrity and geometry, not provenance"). One line keeps the two docs consistent.

Nothing else — q4nx_head_image_key_digest returning a digest over {version, key} so a filename does not embed four numbers and a hash is the right ergonomics for the caller.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at b1e05e5b — a self-correction with the right evidence, and the finding is more important than the cache it fixes.

The claim is precise: the head image's geometry does not come from the container. run_lm_head derives the tile width from the kernel set (designs_[i_qkv].N), packs with that T, and the design's blocked flag changes the byte order — so one container packs to a different image per kernel set. A cache keyed only by the container's identity would hand a _b8-set image to a differently-set serve, which is the "cache hit that is silently a different artefact" failure class with a bitmap inside it.

What I would want in the body, since this is a cache: the key must include every input that changes the bytes — container identity and the kernel set's geometry/blocked flag — and the code should say so at the key, not only in the prose. A cache key is the one place where "what did I forget to include" is unanswerable later, because the wrong answer is a plausible image rather than an error.

This also corrects the design on #1716 (the head-image cache), which is exactly the kind of thing the "pack, then read it back" loop exists to catch. Same lesson as tonight's #1830: a cache keyed by a role or a container alone reuses another artefact's bytes, and the symptom is numbers that look plausible.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b1e05e5: READY

The cache key is content, and it names every input that changes the packed bytes: kpad (K padded to the kernel grid, i.e. the BO's row count), t (the design's N), blocked (whether the design packs b8), and head_hash (FNV-1a over the container's lm_head.weight bytes).

That is the right set, and the blocked field is the one that would be easiest to forget: two designs with the same kpad/t but different packing produce different tile bytes, so a key omitting it would hand the b8 packer a tensor packed for the other layout. Including the hash of the actual weight bytes rather than the container's name or path is the q4nx-cache lesson applied correctly — a label names work, a hash identifies it, and only the second survives a container being rebuilt or renamed.

A head image that is reused only when every one of those matches cannot be the wrong image.

@bong-water-water-bong
bong-water-water-bong merged commit 39eb84b into main Nov 10, 2026
8 of 9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/head-image-key branch November 10, 2026 11:06
bong-water-water-bong added a commit that referenced this pull request Nov 10, 2026
… rebuilding it (engine#1716) (#1950)

The int8 head tiles were gathered and packed on every serve. The file format
(#1802) and the cache key (#1832) were merged but nothing called them.

q4nx_head_tile_image() fills one head tile: a cached image is used when it
reads back whole and a sample of its columns re-packs to the same bytes from
the weights held now; anything else packs as before and writes the file.
The forward's head loop calls it through I8Ctx::fill_weight_bo. Files live in
<Q4NX cache>/head-images; ONEBIT_HEAD_IMAGE_CACHE=0 turns it off.

Fixes #1716

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants