Repository navigation
npu: the head image's cache key, because the same container packs differently under different kernel sets (engine#1716) - #1832
Conversation
…ferently under different kernel sets (engine#1716) Reading `run_lm_head` corrected my own design on #1716: the image's geometry does not come from the container. int i_qkv = -1; // widest design with K == kpad_ const int T = designs_[i_qkv].N; // <- the KERNEL SET's tile width const size_t tile_elems = (size_t)kpad_ * (size_t)T; lctx.packB_into(*pw.bo, bt.data(), kpad_, T, ...); // and the design's `blocked` flag changes the byte order so one container packs to a DIFFERENT image under `kernels-h4096-b8` than under a non-blocked set. A key that omitted kpad, T or blocked would serve one kernel set's image to another -- and since the acceptance test is a correlation, the wrong image would still look plausible, which is exactly the class of failure this contract exists to make impossible. So the key is a digest over {format version, kpad, T, blocked, head-hash}, and the file name is that digest: a cache written by one run is found by the next, and two kernel sets cannot share one entry. 4 arms, device-free, in `npu_design_grid_test` (94 ok lines total): each of the four facts changes the digest individually; the name is stable for the same key; and it CHANGES when T changes. `tools/copyright.py --check` rc 0. Still the format and the key only: the cache itself (first-serve fill, bounded-sample verification inside `run_lm_head` where `kpad_`/`T` are in scope, and the `npu-pack import --kernels` pre-warm) is the next slice.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at b1e05e5
This is the provenance binding that #1802's audit said was missing, and it is better than what the review asked for. The key is a digest of exactly the
four facts that decide the bytes:
kpad K padded to the kernel grid (the BO's row count)
T the widest design with K == kpad_ (designs_[i_qkv].N)
blocked 1 when the design packs blocked (b8) — changes the byte order
head_hash FNV-1a over the container's lm_head.weight bytes
and the reasoning for each is in the file rather than implied: "the SAME container packs to a DIFFERENT image under a different kernel set. A key that omitted
kpad, T or blocked would serve one set's image to another — and the aggregate correlation would still look plausible, which is the failure this whole
contract exists to make impossible." head_hash is the answer to #1802's gap: an image is now keyed on the weights it came from, not only on geometry.
Built and ran it — the arm perturbs each of the four fields and requires a distinct digest:
ok head image key: kpad, T, blocked and the head hash each change the digest (got 4, want 4)
ok head image key: it names a file
ok … (same key) names the same file; (a different T) names a different one
all cases ok
That is the right shape for a cache key: one field at a time, each provably load-bearing, plus the naming round-trip. A key that ignored blocked, or
that digested only kpad and T, fails immediately.
One thing to add when a caller appears (not blocking, nothing calls the API yet): the digest is 64-bit FNV over four facts, which is fine for accidental
collision but worth stating at the naming site — the same note the header now carries about integrity vs provenance. Two lanes generating images into one
cache directory is exactly the accidental-collision case, and a name that hashed only kpad/T would have collided often.
Not verified: no caller, so no end-to-end path; the key is pure arithmetic and runs in the fixture.
|
VERDICT: READY at On #1802 I argued that the payload hash proves integrity, not provenance, because it is computed over the bytes the reader just read — so the version string was the only field standing between a stale image and acceptance. #1846 has since adopted that sentence in the header, and this PR supplies the other half: the key now includes the kernel parameters that change the packed bytes ( That is the correct decomposition: identity of the inputs and the transform, not of the output. Two things worth adding, both cheap
Nothing else — |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at b1e05e5b — a self-correction with the right evidence, and the finding is more important than the cache it fixes.
The claim is precise: the head image's geometry does not come from the container. run_lm_head derives the tile width from the kernel set (designs_[i_qkv].N), packs with that T, and the design's blocked flag changes the byte order — so one container packs to a different image per kernel set. A cache keyed only by the container's identity would hand a _b8-set image to a differently-set serve, which is the "cache hit that is silently a different artefact" failure class with a bitmap inside it.
What I would want in the body, since this is a cache: the key must include every input that changes the bytes — container identity and the kernel set's geometry/blocked flag — and the code should say so at the key, not only in the prose. A cache key is the one place where "what did I forget to include" is unanswerable later, because the wrong answer is a plausible image rather than an error.
This also corrects the design on #1716 (the head-image cache), which is exactly the kind of thing the "pack, then read it back" loop exists to catch. Same lesson as tonight's #1830: a cache keyed by a role or a container alone reuses another artefact's bytes, and the symptom is numbers that look plausible.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at b1e05e5: READY
The cache key is content, and it names every input that changes the packed bytes: kpad (K padded to the kernel grid, i.e. the BO's row count), t (the design's N), blocked (whether the design packs b8), and head_hash (FNV-1a over the container's lm_head.weight bytes).
That is the right set, and the blocked field is the one that would be easiest to forget: two designs with the same kpad/t but different packing produce different tile bytes, so a key omitting it would hand the b8 packer a tensor packed for the other layout. Including the hash of the actual weight bytes rather than the container's name or path is the q4nx-cache lesson applied correctly — a label names work, a hash identifies it, and only the second survives a container being rebuilt or renamed.
A head image that is reused only when every one of those matches cannot be the wrong image.
… rebuilding it (engine#1716) (#1950) The int8 head tiles were gathered and packed on every serve. The file format (#1802) and the cache key (#1832) were merged but nothing called them. q4nx_head_tile_image() fills one head tile: a cached image is used when it reads back whole and a sample of its columns re-packs to the same bytes from the weights held now; anything else packs as before and writes the file. The forward's head loop calls it through I8Ctx::fill_weight_bo. Files live in <Q4NX cache>/head-images; ONEBIT_HEAD_IMAGE_CACHE=0 turns it off. Fixes #1716 Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Reading
run_lm_headcorrected my own design on #1716: the image's geometry does not come from the container.so one container packs to a DIFFERENT image under
kernels-h4096-b8than under a non-blocked set. A key that omitted kpad, Tor blocked would serve one kernel set's image to another -- and since the acceptance test is a correlation, the wrong image
would still look plausible, which is exactly the class of failure this contract exists to make impossible.
So the key is a digest over {format version, kpad, T, blocked, head-hash}, and the file name is that digest: a cache written by
one run is found by the next, and two kernel sets cannot share one entry.
4 arms, device-free, in
npu_design_grid_test(94 ok lines total): each of the four facts changes the digest individually; thename is stable for the same key; and it CHANGES when T changes.
tools/copyright.py --checkrc 0.Still the format and the key only: the cache itself (first-serve fill, bounded-sample verification inside
run_lm_headwherekpad_/Tare in scope, and thenpu-pack import --kernelspre-warm) is the next slice.