Repository navigation
npu: unblock #1931 — main merged in, the fork-join tested for bit-identity and a throwing unit, honest timing labels (Refs #1783) - #2035
Closed
bong-water-water-bong wants to merge 184 commits into
Closed
bong-water-water-bong wants to merge 184 commits into
bong-water-water-bong wants to merge 184 commits into
Conversation
…a failed HRX init says why (#1768) Moves third_party/llama.cpp 831629fe9a -> 5514ebea44. Two merged commits, and the second one fixes a trap that cost this lane two runs today: 5514ebe hrx: a measured zero host figure refuses again, instead of reading as 'unreadable' 6ea7477 hrx: log the status when the HRX runtime fails to initialise 5514ebe is the review point from #259 that merged without it: both call sites wrote `host_raw == 0 ? kHostUnknown : host_raw`, and host_memory_available_bytes() returns 0 both when /proc/meminfo cannot be read AND when MemAvailable is genuinely 0 - so an exhausted host read as 'unknown' and the host term was dropped, in the case the comment above those lines calls the worst moment to admit a launch. The read now reports whether it read anything and the mapping lives in pool_host_term(); a measured zero refuses through the live check, which the suite pins with five new arms. 6ea7477 covers the other half of the same trap: create_registry_context() swallowed the hrx_gpu_initialize failure and registered no devices, so a build with GGML_HRX=ON answered --list-devices with '(none)' and every run died at 'invalid device: HRX0'. That is what the records show for /home/bcloud/wt/pi-gate and ~/wt/hl-engine-build-c20 until LD_LIBRARY_PATH carried the TheRock _rocm_sdk_core lib. Verified: registry regenerated at the new pin (only the sources sha moves; counts stay 298/9/298/298), registry_build.py --check-pins passes, check_pins.py says '831629fe9 -> 5514ebea4 is ahead'. This bump is also the check I could not run locally: ggml-hrx.cpp needs hrx_runtime.h from the hrx-system submodule include path, so 5514ebe's edit to that file has never been compiled here. hrx_build on this PR configures -DONEBIT_HRX=ON and builds onebit, which is what will prove it. Co-authored-by: pi-hrx-sched <pi-hrx-sched@1bit-MONSTER.local>
… it (#1771) #1737 stated, an hour ago and correctly at the time: "Since 2026-11-09 this job is red on every branch, so a red badge is not a signal". That is no longer true, and a live page saying it sends readers past a badge that works. Measured tonight: gh run list --workflow hrx-build.yml --limit 6 21:00-21:27 8 of 8 failure (five branches: 1bit/{fused-tiles-to-int8, pack-blocked-rowmajor,pack-blocked-extract, i669-tests-gpu-hold-include}, fix/1696-...) 23:32 success 1bit/1740-tail-envelope llama.cpp 60ae4c935 00:21 success bump-hrx/831629fe llama.cpp 831629fe9, hrx-system 7f4236924 00:22 success fix/registry-pin-unmerged llama.cpp 60ae4c935 01:31 success bump-hrx/5514ebea llama.cpp 5514ebea4 The 00:21 run is the one that matters: its pins are EXACTLY main's (`git ls-tree 6b0c820 third_party/` -> llama.cpp 831629fe9, hrx-system 7f4236924), and it passes, four hours after the same pins failed on five branches. So the three errors were not a property of the pins in the tree, whatever the first reading said - and the note must not leave a reader believing main cannot build HRX. The paragraph is now a dated window (21:0x-23:1x), with that evidence in it, ending "read a red badge as meaningful again; if it returns, that run's own log is the evidence, not the earlier window". #1733 itself is untouched - whether the fork has a latent skew that only bites on some runner state is that issue's question, and the answer needs the failing run's checkout, not a doc edit. Second time tonight that the site's own heading guard caught my markdown: a wrapped line beginning "> #1733)." renders as a quoted h1 and the build refused the page ("hw-gate.html has 2 <h1> elements"). Reflowed; guard green. Co-authored-by: bong-water-water-bong <bong-water-water-bong@users.noreply.github.com>
…s / 28 layers (Llama-3.2-3B) (#1767) * npu-pack: measure the 1.9 Token fconst rule for hidden 3072 / 24 heads / 28 layers (Llama-3.2-3B) Measured with tools/fconst19_measure.py against amd/Llama-3.2-3B_rai_1.8.0_npu_4K: 1,090 of 1,756,930,368 bytes differ (6.20e-05), i.e. content in a few spots, not layout. The row is worth its own entry rather than a widening of the Phi-3.5 h3072 row, and the measurement is what shows why: same hidden size, same prefix (10,560 = 64 + 2*hidden + 4,352), but the q/k/v bundle is FOUR tiles of 1,740,800 (6,963,200 B) where the Phi/h3072 row is TWO of 5,222,400 -- a different division of the same payload. Head count and layer count differ too (24/28 against 32/32), so this is a distinct geometry on every axis the table keys on. Verified end to end on the pack it was measured from, not just by the tool's line: npu-pack info -> gaps 0 (hidden 3072, 24 heads, 28 layers, IM 8192) npu-pack import -> attention 3584/3584 tiles and MLP 4928/4928 groups survive decode->encode byte for byte, 112 + 84 projections imported, fp32 scale/zero exact for 100.00% of groups, 3.0 GB container The pack's text artifacts are partial (4 of 8: no vocab.json/merges.txt/added_tokens.json/chat_template.jinja), so a serve needs the tokenizer side checked first -- that is a pack property, not this row's. Unlocks amd/Llama-3.2-3B_rai_1.9.0_npu and amd/Llama-3.2-3B-Instruct_rai_1.9.0_npu (same geometry). * npu-pack: the h3072/24/28 row's fraction is 6.20e-07, not 6.20e-05 -- and the tool now prints both spellings CHANGES at 5c4b547, and it is right: 1090 / 1,756,930,368 = 6.204e-07. I had copied the tool's percentage (6.20e-05 %) into the row comment as if it were a fraction, which reads as a 100x larger layout difference -- the opposite of what the row is arguing. Fixed the comment, and fixed the class rather than the instance: tools/fconst19_measure.py now prints the fraction beside the percentage (6.20e-05 % = 6.20e-07 of the file), because this is the second time tonight the same slip landed on a row (#1645 was the first) and that line is what a row is copied from. Everything else the review checked stands: prefix 64 + 2*3072 + 4,352 = 10,560; bundle 4 x 1,740,800 = 6,963,200 B against the Phi/h3072 row's 2 x 5,222,400 = 10,444,800 at the same hidden size; and the triple (3072, 24, 28) occurs once in the table. --------- Co-authored-by: t <t@e.com>
…e test whose env #1588 dropped (engine#1770) (#1772) * WIP: owner 11-09 snapshot - pending work in zc-1770 (3 paths), committed unmodified so the snapshot relays it * tests: npu_repack_path delivers its environment again, so its arms stop testing nothing (#1588) The helper ran every arm as env REPACK_MARKER=... ONEBIT_Q4NX_CACHE=... "$@" \ # --port 0: ... timeout 60 "$dir/1bit" serve ... and a `\` continuation followed by a comment line does not put the comment "in the middle" of the command: the backslash is consumed, the `#` starts a comment at the start of a word, and the command becomes `env VAR=... # ...`. Every assignment is dropped and `timeout 60 ...` runs on the next line as a separate, bare command. #1588 added that comment when it switched the port to `0`; before it the continuation went straight to `timeout`, which is why the test passed when it landed in #543 and has failed since. Consequence on main: the two arms that name behaviour were red on every run — the packaged `scripts/repack_gguf.py` precedence and the `ONEBIT_Q4NX_REPACK` override — while the arms that only grep 1bit's own message passed, because with no script and no override the not-found path is exactly what a bare binary takes anyway. The test's own log line carried the proof and nobody read it: it printed `(exit SHELL=/bin/bash` and a screen of environment variables, because `$rc` held the output of `env` with no command to run. Fixed by hoisting the comment above the command, and the reason is in the comment so the next edit to this helper does not put it back. Before (two arms fail, and the third arm's detail is `env`'s own dump): FAIL a package's scripts/repack_gguf.py beside 1bit is the one that runs ok (the run stops at the stand-in: exit 1, GGUF repack failed) FAIL ONEBIT_Q4NX_REPACK overrides the path ok with no script anywhere, serve says so (exit SHELL=/bin/bash ok and it did not claim a build-machine path exists After (5/5, and the third arm now reports its real exit code): ok a package's scripts/repack_gguf.py beside 1bit is the one that runs ok (the run stops at the stand-in: exit 1, GGUF repack failed) ok ONEBIT_Q4NX_REPACK overrides the path ok with no script anywhere, serve says so (exit 1) ok and it did not claim a build-machine path exists The implementation is untouched: `packaged_path()` already tried `here/scripts/repack_gguf.py` before the build-tree fallback, so the arm's layout was right and only the delivery was broken. Test-only change. Refs #1770 (found while running the suite for it). --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: t <t@e.com>
…rrects #1771) (#1776) #1771 retracted the "read hrx_build as unavailable" instruction, which was right, but its closing clause says the failure "was not a property of the pins in the tree — which is what the same pins failing four hours earlier had seemed to say". The pins are not the same, and the gitlinks come from the runs themselves: red, 21:06 run 42699357566 (fix/1696) hrx-system 7f4236924 llama.cpp 8f124f906 -> failure green, 00:21 run 42727646749 (bump-hrx/831629f) hrx-system 7f4236924 llama.cpp 831629fe9 -> success hrx-system matches; llama.cpp does not. And 8f124f906 is exactly the pin #1733 root-caused (ggml-hrx.cpp declares device_free_total twice: void from #252's forward declaration at :383, bool from #240's 800 lines later), which #1759 replaced. As merged, the paragraph teaches the opposite lesson: "a window, not the pins" instead of "this pin was broken, a bump fixed it, and a llama.cpp bump must clear hrx_build before it lands" — the rule the hw-gate label exists for, and what #1768 did before merging (hrx_build pass 1m10s). Docs only.
…s 2^(bits-1) (engine#1503) (#1790) Qwen3-VL-4B spells its projections as three-input nodes with no qzeros tensor and names them `<proj>.MatMul.weight_Q4` next to `<proj>.MatMul.weight_scales`: MatMulNBits inputs = (activations, ...attn.q_proj.MatMul.weight_Q4, ...attn.q_proj.MatMul.weight_scales) attrs = K 2560, N 4096, bits 4, block_size 32 weight_Q4 [4096, 80, 16] UINT8 N x (K/32) x 16 B (32 quants, 2 per byte) weight_scales [4096, 80] FLOAT one f32 scale per (row, 32-column block) `walk_graph` required `ins.size() >= 4`, so it recorded **no matmuls at all** for that pack: `npu-pack info` printed `matmulnbits_ops 0`, `plan()` fell through to "the graph carries no weights", and the import wrote the embedding and nothing else -- measured on the real pack as `imported 1 tensors, 0 q4_1 tiles`. Three changes, and the third is the one that decides correctness: 1. accept three inputs (data, weight, scales) and take `zeros` only when the node has it; 2. resolve both spellings -- `<proj>.MatMulNBits.qweight` (the 1.8/phi-4 form) and `<proj>.MatMul.weight_Q4` -- in the projection census and in the weight/scales lookup; 3. a node with **no qzeros** takes the implicit zero point `2^(bits-1)`, applied by `implicit_zp_zero()` after the decode. That is ONNX Runtime's own MatMulNBits rule when `qzeros` is absent, and it is corroborated inside this collection: the *embedding* component of the same family (GatherBlockQuantized, also no zeros) decodes to ORT's output at cosine 1.00000000 / max|d| 0 at zero point 8, against 0.317 at zero point 0 (engine#1725). **Measured on the real pack** (`amd/Qwen3-VL-4B-Instruct_rai_1.9.0_npu`, 2.9 GB, on disk under `~/scratch-packs`): imported 147 tensors, 47480 q4_1 tiles into ~/scratch-packs/qwen3vl4b.q4nx self-check: the head reproduces the pack's own embedding (as imported, bf16) corr 0.993981 rel_err 0.092550 controls: nibble-swapped corr -0.0699, no zero-point corr 0.3333 (both must fail) parameters: fp32 scale/zero exact for 100.00% of 32-column groups The head is tied to the embedding after all, so the pack's own reference grades the decode -- at 0.993981, with the nibble-swap control failing as it must and the **no-zero-point control at 0.3333**, which is the direct measurement that the implicit 2^(bits-1) is the right convention for this family. Test: a crafted three-input node with `[N, K/32, 16]` u8 + `[N, K/32]` f32 asserts that the walker records it (1 of 1), counts it as carrying no qzeros, and takes K/N/block from the node's attributes -- 3-input MatMulNBits: 1 matmul(s), 1 without qzeros, K 64 N 32 block 32 all npu-pack checks passed Not in this PR: the RoPE question. The same pack ships `cos_cache`/`sin_cache` that no plain theta reproduces (best 0.5918 at 3.2e6 against a declared 10000) while the forward computes RoPE from theta -- its own issue, not a reader symptom. Co-authored-by: pi-npufeat <npufeat@strixhalo>
…t read as if it were the head (#1789) engine#1740/#1773: the tail is [envelope][rows x tile][PADDING], and 12+ of the 42 1.9 packs end in a multi-hundred-MB all-zero run (110,208,000 B on the hidden-1536 family). This tool computed the head as 'fconst_bytes - ntiles*tile', i.e. the no-padding assumption, so on those packs it read the padding and reported corr ~ 0 -- which is the *symptom* engine#1693 started from, reproduced on a pack where nothing is wrong. That is the one failure mode a measurement tool must not have: a plausible negative. trailing_zeros() already existed here and was only *reported*; it now also places the grid (head_offset(fconst_bytes, ntiles, tile, zeros)), so the count decides where the head ends and the padding is not decoded at all. The docstring says plainly that this is the zero-run-implied candidate and not a proof of placement -- the printed correlation is what decides -- and points at the importer's plan_tail_head(), which walks the neighbouring tile-aligned candidates for the cases where the head does not end exactly at the run. The test that pinned the old behaviour is rewritten rather than deleted: it built a padded synthetic pack and asserted the read came out non-finite ('a read shifted by them is visible'). It now asserts the opposite contract -- the grid is placed before the padding, so a padded pack decodes finitely -- and the two new arithmetic checks pin zeros=0 unchanged and zeros>0 shifting by exactly its length. Verified on the real pack (no padding, zeros=0): corr +1.0000, rel_err 0.0007, unchanged. 16/16 checks. refs #1740 Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…ead of materialising it (engine#1784) (#1795) The container is already mapped read-only (`Model::Model` -> `map_file()`, and every `Tensor::data` points into that mapping), so the token embedding table was being **copied out of a mapping we already had**: bf16 tables via `memcpy` of the whole `NV*H` table, and plain int8 tables via `q4nx_dequant_plain_i8()` over the whole table at 4 bytes per element. The second one is the expensive half: `bonsai2-27b-NPU`'s I8 embedding is 0.74 GiB in the container and became **2.96 GiB resident**, for a table that is read one row per token. Measured on this box before the change (embedding shape x dtype from each `model.q4nx` header): bonsai2-27b-NPU / -b8 I8 0.74 GiB on disk -> 2.96 GiB resident qwen3-coder-30b-a3b BF16 0.58 GiB -> 0.58 GiB ZR1-1.5B-npu-bf16 BF16 0.43 GiB -> 0.43 GiB d3-lite-bf16 BF16 0.47 GiB -> 0.47 GiB Qwen3-0.6B BF16 0.29 GiB -> 0.29 GiB What changes: * bf16 tables: `embed_bf16_` is the pointer into the mapping; `embed_at()` converts the one element asked for. * plain int8 + per-32-column scale (the dtype the 27B and the gemma-class containers use): `embed_i8_` / `embed_sc_` are pointers into the mapping and `embed_at()` reads one element through the new `q4nx_plain_i8_at()`. * `embed_` is no longer filled at all and is now always empty, so a stale read is a bounds error instead of a silent wrong answer. `embed_f_` is still built for the chunked formats (`dequant_any`'s tile paths), which have no element-wise read yet -- that half is named on the issue, not hidden here. * Pad columns (`k >= H`) still read 0.0f, which is what the expanded table held, so the head's `kpad_`-wide dot product is unchanged. `lm_w_` is untouched. No call site changes: `embed_at()` and `lm_at()` were already the only readers, and both are called with `k` innermost, so "one row per token" is the access the code already had -- it just does not need a table-sized buffer to serve it. `q4nx_plain_i8_at()` is declared beside `q4nx_dequant_plain_i8()` and computes the group index the same way, so the two cannot drift; `tests/npu_embed_row_test.c` pins a row read against the whole-table decode **bit for bit** on a table whose scale differs per row and per group (so a row/column index bug cannot pass by symmetry), plus the null-scale, out-of-range, pad-column and wrong-cols arms. Host-only, no model, no device. Gate: full CPU suite **168/168**. The RSS/identical-output rows on three models are the second half of #1784 and need the NPU lane; they are not claimed here. Also: `.github/workflows/ci.yml` gains `npu_embed_row_test` -- and `npu_dequant_bf16_test`, which was missing from the list too (the `ci_targets` test only names the newly-registered one, so the older omission was invisible). Refs #1784. Co-authored-by: t <t@e.com>
… stops it (engine#1371) (#1611) * tools+docs: the OGA-pack route's drafter arm, probed to the gate that stops it (engine#1371) #1371 asks for a drafter on the OGA-pack route with acceptance per pass against a bar. Two arms are already measured and published (PARD over GGUF: HRX 1.012x at 3.095 tokens/round and 0.524 acceptance per offered draft; CPU 0.508x, structural). This lands the third arm's answer, which is that the route cannot carry a measurement yet -- and says exactly where it stops. tools/oga_pack_probe.sh runs the two gates a pack must pass before a drafter means anything and names the one that bit, inside ONE npu-hold ticket (the server it starts is killed by a trap on every exit path): 1. npu-pack import -> container + the tensor count it could not import 2. forward-serve -> HEALTH 200, or the exact init failure Measured here (kernel 7.4.0-rc1-1bit): * Qwen2.5-0.5B rai_1.9.0_npu import: 242 tensors incomplete, "fconst: layer blocks not located" (the 1.9 .fconst rename, engine#1410) -- gate 2 not reached * Qwen2.5-0.5B rai_1.8.0_npu_4K import: 73 incomplete, 48/48 RMSNorm anchors, 768/768 .fconst tiles byte-exact; forward init: "embedding dtype q4_1 but dequant failed" * Llama-3.2-1B rai_1.8.0_npu_4K forward init: "missing full ELF /full_i8_QKV_K2048_N3072.elf" * Qwen3-1.7B rai_1.8.0_npu_4K forward init: "missing full ELF /full_i8_QKV_K2048_N4096.elf" The 1.8.0 0.5B case is diagnosed to the line: the container parses (dims H=896 NL=24 NH=14 NKV=2 IM=4864 NV=151936 = Qwen2.5-0.5B), then npu/forward/src/q4nx_forward.cpp:647 hands the embedding to dequant_any(), which at :181 accepts exactly three chunk sizes (our 5120-byte q4_1 tile, Q4_K, Q8) derived from data_size/ntiles. The OGA import writes the embedding as q4_1 in the VENDOR layout, whose per-tile byte count is none of the three, so it falls through and returns non-zero. Fixable on either side -- the import (where the vendor layout is known) or dequant_any -- and only then does a drafter measurement on this route mean anything. Verified: gate 1+2 run end to end on this box under an npu-hold ticket (exit 4 with the verdict line, token released, nothing left running); bash -n clean; copyright --check rc 0 with the tree staged; git add <paths> only. refs #1371, refs #1410 (the 1.9 fconst rename), refs #1110 (q4_1 layout family). * tools+docs: the OGA probe takes no builder paths, no /tmp scratch, and refuses gate 2 without a ticket (review of #1611) All four points from the CHANGES: 1. NO BUILDER PATHS OR USERNAMES. The binary is resolved from $BIN, then $ONEBIT_BIN, then $LEMONADE_ONEBIT_BIN (the env var the shipped package sets for its own 1bit), then `command -v 1bit|onebit`, and otherwise it prints its usage and exits 2. The two /home/<user> paths of the previous version were class 1 of policies/build-path-leak-classes.md (#1550) and only worked on one box. 2. NO /tmp SCRATCH, NO FIXED LOG NAME. The imported container goes to ${XDG_CACHE_HOME:-$HOME/.cache}/oga-probe/<pack> (disk-backed, because this box's tmpfs hit 0 MB free on 11-08 and broke every lane's tool output), the server log is $OUT/serve.log so two probes cannot collide, the script checks for ~2G free before importing, and it removes $OUT on every exit path (INT/TERM/HUP included) unless KEEP=1. 3. NO STALE LINE NUMBERS. The doc and the script now name `dequant_any()` and the embedding branch that raises "embedding dtype … but dequant failed", instead of :647/:181. The page says why: it had already been wrong once by quoting a line that moved, and a function name survives the next edit (`git log -S dequant_any` finds it). 4. GATE 2 REFUSES TO RUN WITHOUT A TICKET -- with the limitation stated. The walk reads argv[0] of each ancestor (never the whole command line). Two measured facts shaped the final contract: `npu-hold` EXECs its command, so the hold is not in the ancestor chain by the time this runs; and the first version matched the full cmdline, which made a shell whose *text* mentioned npu-hold (a patch script, an editor, a grep) wave an unticketed run through -- a guard that passes on its own search string. So: no ticket detected -> loud warning naming the exact npu-hold command, gate 1 runs, gate 2 does NOT, exit 3; REQUIRE_TICKET=1 refuses outright with exit 2 before any import. Measured after the change: default, no ticket -> exit 3, warning printed, "stopping after gate 1", no NPU handle touched REQUIRE_TICKET=1, no ticket -> exit 2, gate 1 never ran inside an npu-hold ticket -> both gates (unchanged behaviour where the walk can see the hold) shellcheck -S warning -> 0; bash -n clean; copyright --check rc 0 with the tree staged refs #1611, #1371, #1410. * tools+tests: the OGA probe's ticket guard reads the token pid (the argv walk could never match), and the binary check is back Fixes pi-verify's CHANGES on #1611. All three points, and the first one was a real hole rather than a preference. 1. THE TICKET WALK COULD NEVER MATCH. `npu-hold` and `gpu-hold` are bash scripts, so a running one's argv[0] is the INTERPRETER and the script path is argv[1] -- matching argv[0] (or, worse, the whole command line, which my first version did and which matched its own search string) means the documented `npu-hold -t 4 -- tools/oga_pack_probe.sh` never ran gate 2. The guard now asks the question the token file answers: "is an ancestor of mine the pid recorded in ~/NPU-TOKEN or ~/GPU-TOKEN?", i.e. `gpu-hold <label> pid <PID> <date>`. Overridable with TOKEN_FILES / NPU_TOKEN_FILE / GPU_TOKEN_FILE so it is testable. `tests/oga_pack_probe_test.py` (registered as ctest `oga_pack_probe`) covers it with four arms: no token -> gate 1 only, exit 3, warning naming the correct invocation; REQUIRE_TICKET=1 -> exit 2 before any import; **a fake token naming the pid of the shell that runs the probe -> ticket detected, gate 2 attempted** (the positive case both argv versions failed); and a pack path whose basename is "." -> refused. 2. THE BINARY CHECK CAME BACK WITH THE PATHS. `BIN=${BIN:-${ONEBIT_BIN:-$(command -v 1bit || command -v onebit)}}`, and if that resolves to nothing the script now refuses with exit 2 and its usage instead of running `npu-pack` as a command and blaming the pack for "import produced no container". 3. THE `.`/`..` BASENAME IS REFUSED. `OUT` defaults to `$SCRATCH/$(basename "$PACK")` and `rm -rf "$OUT"` runs before gate 1, so a pack path of "." would have wiped the whole scratch dir (every other probe's container with it). Verified: `bash -n` clean, `tests/oga_pack_probe_test.py` -> all checks passed (10 assertions), `tools/copyright.py --check` rc 0 with the tree staged, `git add <paths>` only. The guard's two earlier failures are written into the test's docstring, because "a check that passes on its own search string" is the lesson worth keeping. * tools+tests: the probe refuses a missing binary by name, and a GPU ticket no longer authorises NPU work (engine#1371 / #1611) Both are the re-review's points at `00c4573d`, and both were holes rather than preferences. **1. The binary check was not actually back.** `BIN` defaulted to `command -v 1bit || command -v onebit`, and nothing refused when that resolved to nothing. With neither on `PATH` and `BIN` unset the probe reached gate 1, the shell answered rc 127 ("command not found"), and the verdict read == verdict: import produced no container (rc=127) -- gate 1 is where this pack stops i.e. the *pack* was blamed for the tool not being installed. The refusal now happens before anything is touched, exit 2, naming the variable to set. **2. The guard took a GPU ticket for NPU work.** `TOKEN_FILES` defaulted to `NPU-TOKEN:GPU-TOKEN` and `ticketed()` accepted a match in either, while gate 2 is `forward-serve` on the NPU and the script's own messages say "gate 2 touches the NPU and needs an npu-hold ticket". So `gpu-hold -- tools/oga_pack_probe.sh <pack>` passed the guard and ran NPU work without the NPU token, alongside whichever lane held it -- exactly the collision the guard exists to prevent. The default is now the NPU token alone; `TOKEN_FILES` still widens it explicitly if a GPU-only gate is ever added. Tests (`tests/oga_pack_probe_test.py`, ctest `oga_pack_probe`, 16 checks, all passing): * **no binary anywhere** (`BIN`/`ONEBIT_BIN` empty, `PATH` pointing at an empty dir) -> exit 2, "no 1bit binary" on stderr, and neither "gate 1" nor "no container" in stdout, so the pack cannot be blamed for the tool. The interpreter is invoked absolutely, because the arm empties `PATH`. * **a GPU ticket and no NPU token** -> "GATE 2 WILL NOT RUN" and rc 3: only gate 1 runs, so no NPU work happens under a GPU ticket. Checked the other direction too: widening `TOKEN_FILES` to that GPU token explicitly does detect it, so the arm is pinning the *default*, not disabling the guard. Verified: `python3 tests/oga_pack_probe_test.py` -> all 16 checks pass; `bash -n` clean; `tools/copyright.py --check` rc 0 with the tree staged; `git add <paths>` only. No device touched: every arm runs with a stub binary and a token file that names nobody. * tools: the probe proves the 200 came from its own server, and asks the kernel for a free port The engine#1611 review's single blocker, at its third head: the health loop believed a 200 from anything on the port (`kill -0 $pid` only broke the loop early, nothing checked it before believing the answer), so a foreign listener on the fixed default 18160 produced "HEALTH 200 -- the pack serves" and then received the POST. The tool could lie about the one thing it exists to measure. Both closures the review offered, because they answer different halves: * `PORT` defaults to a port the KERNEL picks (`bind((,0))`) instead of 18160, so a 200 cannot predate this probe; * a 200 is believed only when the server this probe started is still alive (`kill -0 "$pid"`), both in the loop and again before the POST -- and when it is not, the verdict says so by name and exits 5 rather than calling the pack served. That covers the caller who names a port explicitly, which the first change cannot. The verdict line now also names the pid that answered, so a log reader can tell which process produced the 200. refs #1611 * tools+tests: the probe's OUT= is contained in both directions, so it cannot delete the operator's pack (review of #1611) The review's point, and it was the data-loss path it named the first time: the `case` guard protects `pack_base` — the name DERIVED from the pack — while `OUT` is operator-settable and this script's own text tells people to set it ("set OUT= or SCRATCH="). Since `rm -rf "$OUT"` runs before gate 1, `OUT=<the pack>` replaced the operator's model with the probe's container, silently. Containment now runs immediately after the pack-dir check and before anything is created or removed: outward: OUT must not be the pack, inside the pack, or a parent of the pack inward: OUT must live under $SCRATCH, unless the caller sets OGAPROBE_ALLOW_ANY_OUT=1 The third check is what makes the documented `OUT=` affordance safe rather than merely documented: an override still exists, it just has to be deliberate. `realpath -m` resolves both sides, so a symlinked or non-existent OUT is compared as a path, not as a string prefix. Four arms, all with the operator's own file inside the directory: OUT = the pack itself -> rc 2, refusing OUT=… OUT = a parent of the pack -> rc 2, refusing OUT=… OUT = a directory outside the scratch root -> rc 2, refusing OUT=… OUT = <scratch>/scratchy (prefix, not inside) -> rc 2, refusing OUT=… and: the operator's keep-me.txt is still there after all four and: OGAPROBE_ALLOW_ANY_OUT=1 lets an outside OUT through (the override is real, not decorative) Mutation-checked: removing the containment fails all four arms (two of them by proceeding to gate 1/2, which is exactly the deletion path). Verified: `bash -n` clean; `python3 tests/oga_pack_probe_test.py` -> all checks pass (20 arms); `tools/copyright.py --check` rc 0 with the tree staged; `git add <paths>` only. No device touched: every arm uses the stub binary and a token file that names nobody. * tests: the stub's mode is 0700, not 0755 — CodeQL's py/overly-permissive-file flagged it on #1611 The CodeQL check failed this PR with "1 new alert including 1 high severity": `py/overly-permissive-file` at `tests/oga_pack_probe_test.py:69`, "Overly permissive mask in chmod sets file to world readable". That line is the fake `1bit` stub the probe test executes (`write_stub()`), and 0o755 was copied from habit rather than need: the file is run by this test and by nothing else, so it needs no group or other bits. 0o700 is the whole fix; the arms still pass (20 checks, including the four OUT containment arms and the deliberate OGAPROBE_ALLOW_ANY_OUT override). The other red check on this PR, `hrx_build` at 1b02be9, is **not this branch**: the job never started. Its own log says mem-wait: waiting (15 of 15 min): MemAvailable 1 GB (floor 24 GB) … mem-wait: REFUSED after 15 min: … The box is short of memory, so this job did not start its build; re-run it once memory frees (engine#1504). so it needs a re-run when the box has room, not a code change here. --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: t <t@e.com>
… the context limit (engine#1779) (#1791) * tests: a client that hangs up mid-response must not stop the listener (engine#1779 fault 1) F10 from ~/briefs/TNPU-ALL.md 9: FastFlowLM 1.0.12 lists four server fixes, and the item is to test each against `1bit serve`. This is the first of the four, the one stated as 'a server that stopped accepting connections after clients hung up'. The test asserts the LISTENER, not the interrupted request: a client that disconnects mid response is allowed to get nothing, and what must not happen is that the NEXT client goes unserved. Sixteen streaming requests are cut off after 32 bytes (head -c closes the pipe, curl gets EPIPE) plus two non-streaming ones cut by a hard client timeout, then a fresh client must be answered. Device-free by construction — it runs `--device cpu`, which is the issue's rule for these four ('server-path behaviours, so the tests must run without a device where possible'). Two things found while writing it: * serve_port_wait() takes a pid, not a file, and prints the port; my first version passed a path and the wait bailed on its first poll. Use is now the same shape as serve_e2e.sh. * a build that predates engine #1485 reports `ready on http://127.0.0.1:0` when asked for port 0, so the harness falls back to the backend's own 'listening on' line. Current main does not need the fallback; it is there so the test can be pointed at an older payload. Verified here against the only 1bit on this box that is not behind the 175G build gate (~/wt/hl-engine-build-c20, 2026-11-01): 18 interrupted requests, fresh client HTTP 200. That establishes the harness, NOT that today's main is clean — the before/after the issue asks for needs a current build, and disk is at 152G. * tests: fault 2 (context limit) too, and the port fallback moves into the shared helper Fault 2 of #1779, as the issue prescribes: a prompt that FILLS the context is served, then one more turn that cannot fit - and the answer must be a refusal or a truncated-but-clean reply, never one character repeated. The test asserts the SHAPE of the answer, not its wording: 4xx/5xx with a body is a refusal, 200 with ordinary text is clean, an empty 200 with finish_reason length is clean, and a 200 whose content is one character repeated eight or more times fails - '//////' being the reported instance. Both faults are device-free (--device cpu), which is why they are one PR: the issue groups them ('fault 1 and 2 are pure server logic and testable with no device at all, so they come first'). The port-readback fallback moved from the hangup test into tests/serve_port.sh as serve_port_wait_any(): two tests now need it, and a copy in each would drift. Measured here against ~/wt/hl-engine-build-c20 (2026-11-01, the only 1bit not behind the 175G build gate), ctx 512: fill turn: HTTP 200, finish_reason length (served, truncated clean) over-limit turn: HTTP 400 exceed_context_size_error: request (755 tokens) exceeds the available context size (512 tokens) verdict: refusal -> PASS so fault 2 is NOT present in that build - which is the result the issue asks for on the faults we do not have ('a passing test is the evidence that we do not have it'). Today's main still needs the run; disk is 152G, under the gate. --------- Co-authored-by: pi-hrx-sched <pi-hrx-sched@1bit-MONSTER.local>
…s / 36 layers (Qwen2.5_3B_Instruct) (#1774) * npu-pack: measure the 1.9 Token fconst rule for hidden 2048 / 16 heads / 36 layers (Qwen2.5_3B_Instruct) Measured with tools/fconst19_measure.py against amd/Qwen2.5_3B_Instruct_rai_1.8.0_npu_4K: 1,917 of 1,706,629,440 bytes differ (1.12e-06 of the file, printed as 1.12e-04 %). The row is a reminder that hidden size does not decide the layout. Main already has a 2048/16 row for Qwen3-1.7B ({2048, 16, 28, 8512, 3, 1, 1183744}); this pack shares the hidden size and the head count and differs on everything the table keys on beyond those: prefix 4,160 against 1.7B's 8,512 (64 + 2*hidden + 4,352 predicts 8,512 here and is wrong by 4,352) bundle 9 tiles of 295,936 (2,663,424 B) against 3 tiles of 1,183,744 layers 36 against 28 So a row derived from the hidden size would have rotated this pack's layers into the wrong projections and still parsed. Verified end to end on the pack it was measured from, not just by the tool's line: npu-pack info -> gaps 0 (hidden 2048, 16 heads, 36 layers, IM 11008, qwen2) npu-pack import -> MLP 7344/7344 groups byte-exact through decode->encode, 108 projections (plus 72/72 RMSNorm anchors), fp32 scale/zero 100.00% of groups, 2.8 GB container Unlocks amd/Qwen2.5_3B_Instruct_rai_1.9.0_npu (TNPU-ALL W2 row, next shape after h3584). Refs #1503, refs #1740. * npu-pack: pin every rule row in the table, not just this one (W2's third gate clause) Both CHANGES asked for the same thing and both were right: TNPU-ALL W2's gate is "the 1.8 twin measured, residual 0, and a test that pins the row", and this PR shipped the row with +7/-0 and no test. A second reviewer went further and said none of the earlier rows is pinned either, and that one table-driven arm would cover all of them. That is what this does. * npu/pack_import.h/.cpp: fconst19_rules(int* count) exposes the measured table and its row count. * tests/npu_pack_import_test.cpp, same arm as the existing 896/14/24 check: 4. every row of the real table selects itself back with its own prefix/n_tiles/rot/tile, and the table still has >= 8 rows. The row count matters as much as the values: the h4096 row was dropped twice by merges that resolved the table hunk by picking one side, and each time a whole model class became un-measurable with nothing failing. 5. the specific pair this row creates: 2048/16h/28L is still Qwen3-1.7B's (8,512 / 3 tiles of 1,183,744), 2048/16h/36L is this row (4,160 / 9 tiles of 295,936), and 2048/16h/32L still refuses with "has not been measured". If anyone relaxes the key to (hidden, heads) or merges rows, these three separate again loudly. The arm discriminates, which I checked rather than assumed: with q3->prefix changed to 4,161 the test reports "FAIL fconst19 rule: 2048/16h/36L is Qwen2.5_3B's row ..." and "1 check(s) failed"; restored, the suite passes. Not addressed, deliberately: the reviewer's note that a real Qwen2.5-3B pack is not on this box. The row's residual comes from the measurement with its denominator given, and a device row for this pack still needs the 2048/16 multi-head family plus the tail reader (#1740) -- that is W5's re-run, not this PR. --------- Co-authored-by: t <t@e.com>
…#1759 (#1796) * hw-gate: the hrx_build window is 20:01–21:52Z and #1736 closed it, not #1759 The paragraph in main got the window and the credit wrong, and both are checkable against the runs: * **window.** "Between about 21:0x and 23:1x" is #1771's figure. `gh run list --workflow hrx-build.yml --limit 60` shows failures from **20:01** (`1bit/803-op-fallback`) through **21:52** (`1bit/fused-tiles-to-int8`) and none after; the first green on the fixed pin is 21:41, and the branches that had failed come back at 22:04. * **credit.** "`#1759` replaced that pin with `831629fe` and the job has been green since" is wrong: the bump that compiles is `5b784d8b` "bump llama.cpp to 60ae4c935: the pin compiles again", and `gh api .../commits/5b784d8b/pulls` says **#1736**. `5f76f402` → `831629fe` is **#1759**, which came later (green at 00:21) and cannot be what ended a window that closed at 21:41. `60ae4c935` carries the fork fix, llama.cpp #260 `4e570337a`. Also added the one fact the table could not show and a reader will otherwise trip over: the single red run after the window — `2026-11-10 02:23` on `1b02be9`, pin already `5514ebea` — failed in **"Wait for memory (or refuse)" with exit 75**, a memory gate on the box (`gh run view 42742403894 --log-failed`), not a compile failure. Without that line, a red badge on a newer pin reads as the old break returning. Supersedes the closed #1775 and corrects what #1776 merged; the `hrx-system`-vs-`llama.cpp` framing it landed is kept, because that part is right and is what makes the pin the obvious suspect. Refs #1776, #1775, #1733, #1736, #1759 * hw-gate: fifteen failing runs, not ten -- and two of them are not this failure Review of #1796, both points reproduced here before changing anything: * `gh run list --workflow hrx-build.yml --limit 80`, filtered to 20:00–21:55Z and `conclusion==failure`, returns **15** timestamps (20:01 20:13 20:17 20:32 20:59 21:00 21:01 21:02 21:06 21:15 21:24 21:25 21:27 21:42 21:52), not ten. The table says 13 of 15 on `8f124f906`. * the other two are on the **older** pin: `git ls-tree 650f328 third_party/llama.cpp` and `ls-tree 7ccbf74` both give `b970d85ba`, with `hrx-system` `7f4236924` unchanged -- so the 21:00 and 21:01 failures cannot be the `device_free_total` redeclaration, which is `8f124f906`'s error. They now have their own row and a paragraph saying they are unexplained: the reviewer measured that the same pair was green at 19:42/19:45, and I could not re-verify that because those commits are no longer fetchable, and the failing run's log (`gh run view 42698418833 --log-failed`) returns nothing. So the window claim now distinguishes what it can prove from what it cannot: thirteen runs have a named cause, two do not, and neither is evidence for the other. That distinction is the reason to keep the table at all. Refs #1796, #1733 --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…ight-layout reaches the container (#1793) Found on phi-4: importing with `--weight-layout b8` printed "declares q4nx_weight_layout b8" and then wrote a config.json WITHOUT the key -- the report line comes from the shared code path, but import_external never copied opt.weight_layout into the config. So a caller who built the `_b8` family got a container that asks the forward for the UNTAGGED ELFs, and the failure reads as "missing full ELF" for a set that exists on disk. Two lines: the key is written (as the .fconst path already does), and the layout is validated with the same weight_layout_refusal() the other path uses, so `--weight-layout pb16i` still refuses by name instead of writing a container nothing can open. q4nx_source records the layout too, so a container read months later states which family it needs. Verified on the real 9.74 GB pack: with the flag, config.json carries "q4nx_weight_layout": "b8" and q4nx_source.weight_layout "b8"; without it, the key stays absent as before. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…ce at blocked=0 (#1778) * test(head tile): assert the blocked packer matches the pinned reference at blocked=0 The merged #1769 pins q4nx_pack_B_int8, but the forward packs the head through packB_into -> packB_blocked -> q4nx_pack_B_int8_blocked, a separate row-major loop. Nothing asserted the two agree, which is what a head-image cache (#1716) stands in for. Measured red: col_scales am/127 -> am/128 in q4nx_pack_blocked.cpp fails 64 of 64 columns. Measured green on origin/main 408b1c8: identical codes, scales and scale sum. Refs #1716 * test(head tile): make the fixture's columns differ, so the identity arm can fail The review at d014a74 was right: the plain ramp reaches -0.5 in every column, so all T scales are equal and a per-column mistake cannot show. Measured: 1 distinct absmax of 64. - fixture: scale column n by (1 + 0.05n) -> 64 distinct absmax of 64 - arm (b): reference now uses the packer's own arithmetic (absmax, 1e-12 floor, roundf(v * (127/amax))), not v/scales[n], which disagrees on 26 of 6144 codes once the columns differ - new arm: the fixture has T distinct column absmax, so it cannot go flat again Red: q4nx_pack_blocked.cpp:78 tis[j] -> tis[0] now fails 'the blocked packer ... is byte-identical (got 5901, want 0)' (the number the review predicted) and nothing else notices; restored, all pass. * build: link q4nx_pack_blocked.cpp into npu_design_grid_test The head-tile identity arm calls q4nx_pack_B_int8_blocked, which lives in q4nx_pack_blocked.cpp; the target listed only q4nx_pack.c and q4nx_dequant.c, so the CI build failed at link (undefined reference to q4nx_pack_B_int8_blocked) while my standalone compile passed because I added that TU by hand. Found by reading the failed CI job at 6732d39 rather than calling it environmental: the previous head's run failed on apt fetch, this one on the link. --------- Co-authored-by: agent <agent@local>
…ks nobody imports again (10.67 GiB of tmpfs) (#1797) * npu-pack: the scratch sweep reaches the pre-version names and the packs nobody imports again Follow-up to #1770/#1772. #1772 evicts sibling copies of the geometry being written, which is right, but it matches on `ident_prefix` = `1bit-fconst19v<N>-<tag>-<h>-<heads>-<layers>-`, and the generation before the source identity was in the name is spelled `1bit-fconst19-<tag>-<h>-<heads>-<layers>.fconst` -- no `v<N>-`, no size, no mtime. So those files were invisible to it and stayed resident. Measured on this box while writing this: /tmp/1bit-fconst19-*.fconst 3 files, 10.67 GiB (1bit-fconst19-20133236-4096-32-32.fconst and two siblings) /tmp/1bit-fconst19v1-*.fconst 96 files, 11.57 GiB /tmp/1bit-fconst19* 99 files <- tmpfs, i.e. RAM, 22 GiB of it Two additions, both to the same sweep: 1. **the pre-version name for the same pack and geometry**, matched exactly (`legacy_name`), because it carries no identity to compare and there is at most one per geometry. Another geometry's legacy name is kept -- it is not this run's to remove, which is the rule #1772 already applies to current-generation copies; 2. **an age rule**: any `1bit-fconst19*` in the directory older than `max_age_seconds` (6 h; 0 disables) goes. A write-triggered sweep can only reach the geometry being written, so a pack nobody imports again is otherwise resident for the life of the box -- that is the tail #1770 measured (699 files, 8.7 GiB, 691 of them 0-byte) and it needs the clock, not the write, to reach it. 6 h is far above the longest import measured (minutes), so nothing in flight is touched; the concurrent-writer hazard #1772's comment does not cover is bounded by the same number. Verified standalone (the function is `#include`d from the source, not retyped -- the NPU/vendor stack this box does not build is not needed for it): 9 arms, all pass -- keep, same-pack-other-mtime, legacy-for-this-geometry (removed), legacy-for-another-geometry (kept), abandoned-older-than-age (removed), fresh-other-geometry (kept), `max_age = 0` disables the age rule so the existing three-argument callers are unchanged, and the two degenerate arguments still remove nothing. The engine's own test gains the same four arms (16 total in `test_evict_stale_canonical_copies`), with the mtime set 7 h back through `fs::last_write_time`. * npu-pack: reuse touches the canonical copy, so the eviction age means "last use" Review of #1797: the age rule sweeps any `1bit-fconst19*` whose `last_write_time` is older than `max_age_seconds` ("nobody has touched", 6 h), but the reuse path only READS the copy -- `canonical_copy_reusable(work, expected)` then `return work` -- so a copy in continuous use ages past the window and gets swept, and the next run rewrites a multi-hundred-MB file. That is the same cost the whole eviction family exists to remove, arriving by the rule meant to bound it. The reuse path now bumps the mtime before returning, so the timestamp means what the rule assumes. Errors are deliberately ignored: failing to bump an mtime must not fail an import that already has a good copy. Arm: a copy of a DIFFERENT geometry aged 10 h is swept by a 6 h window; one with a fresh mtime survives; the copy being written survives. Different geometry matters -- the name rule would decide for a same-geometry file, and my first version of this arm made exactly that mistake and failed on it, which is why the arm says so where the names are chosen. Verified by building and running `npu_pack_import_test` at this head: `all npu-pack checks passed`, 16 eviction arms. The arm pins the RULE (fresh mtime survives an age sweep); the touch that produces a fresh mtime is the one-line change above, and it is not exercised end to end here because that needs a real pack through `canonical_token_fconst`. Refs #1797 --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…g it to end the file (engine#1740 W1a) (#1773) * npu-pack: locate the 1.9 head in the .fconst tail instead of requiring it to end the file (engine#1740 W1a) The tail is [envelope][rows x tile][padding] and the importer required `have == env + rows*tile` exactly. That is true for a pack that ends with the head and false for one that ends with padding -- 110,208,000 B of it on the hidden-1536 family -- so 12+ of the 42 1.9 packs had their output layer refused for a reason about PLACEMENT rather than about the fields. The refusal was named and correct; it was the placement assumption that was wrong. The placement is now measured: * the trailing all-zero run at EOF is counted, whatever its alignment; * `plan_tail_head()` (exposed in the header) turns (bytes, tail_off, envelope, rows, tile, zeros) into the earliest and latest tile-aligned candidate, or a refusal carrying both lengths. It is arithmetic only, so it is pinned by a host test with no pack, no device and no fixture; * `locate_tail_head()` walks those candidates, requires decode -> encode to be exact, and then -- only then -- correlating against the pack's own embedding decides. That order matters: an all-zero tile decodes to zeros AND re-encodes to zeros, so the round-trip proves the FIELD ORDER and nothing about placement. Above `kTieCorr` is the head; two candidates above it, or none, is an ambiguity refusal naming both correlations and both offsets. Tied and untied are reported rather than assumed. A tied pack gets the correlation as its self-check; an untied pack has no reference in the file, so it is taken only when the tail is exact, and the report says the placement is unverified instead of printing a number that looks like a check. `tie_word_embeddings` is still read from the pack, never inferred. Gate 1, `Llama-3.1-8B_rai_1.9.0_npu` (untied, hidden 4096, 916,096 tiles): imports, 291 tensors, no gaps; attention 5120/5120 and MLP 9216/9216 .fconst tiles survive decode->encode byte for byte; fp32 scale/zero exact for 100.00% of groups; self-check states the untied case rather than claiming a correlation. Regression, `Qwen2.5-0.5B-Instruct 1.9` (tied, hidden 896): unchanged -- 291 tensors / 67,088 tiles, corr 0.999999, rel_err 0.000700, row-shifted control 0.0716. Block-32 heads (the 1.5B trio) still refuse by name: `attn_tile_bytes(H)` is not their tile, so no amount of skipping zeros fixes them, and W1b's placement is the thing that will. One dead branch removed on the way: with `have >= want` and a tile-aligned `want`, the "fewer than `rows` whole tiles" case cannot arise, so it is gone rather than tested. * npu-pack: report WHICH tail placement the head was taken from "imported" cannot distinguish a tail that ends with the head from one that ends in padding, and that difference is the whole point of the search. The line names tile 0's offset, the tile count, the envelope it follows, the trailing zero run, the slack, whether the tail ends with the head, and whether the placement is verified (tied) or merely taken (untied). Measured through it: Qwen2.5-0.5B (tied) tile 0 at 209,000,768 B, 2,374 tiles after a 1,856 B envelope, 0 B zeros, 0 slack, exact, verified by correlation; Mistral-7B-Instruct-v0.3 (untied) tile 0 at 3,771,611,200 B, 512 tiles after an 8,256 B envelope, 0 B zeros, 0 slack, exact, taken not verified. * npu: apply the dynamic-NTK rope base rule instead of refusing it (engine#1740 W2, Hunyuan-0.5B) The 1.9 Hunyuan packs declare `rope_scaling: {type: "dynamic", factor: 1.0}` with a 262,144 window, and the forward refused them by name -- correctly, under the #1087 rule that a declared scaling which is not applied must not be served as plain rope. This is the rule, so the refusal can go. Dynamic NTK is a BASE rule, not a per-dim one, so it is a pure function of (theta, factor, window, position, dim): below the window: base = theta -- bit for bit the same as a model with no scaling past it: base = theta * (factor * pos / window - (factor - 1)) ^ (dim / (dim - 2)) which is HF's `_compute_dynamic_ntk_parameters` and what ggml applies through rope_yarn. Because the base moves with the position it is computed once per position rather than once per dim pair, and because the rule is a pure function it is pinned by the existing rope test with no device, no pack and no fixture: * below the window the base IS theta (position 0, and one position short of the window), and Hunyuan's factor 1.0 at the window is a no-op bit for bit -- which is what makes accepting that specific block safe rather than a guess; * factor 4 at twice the window reproduces `theta * 5^(dim/(dim-2))` to 1e-6; * an unusable factor, an unusable window, and `on == 0` all leave the base alone -- the same rule the llama3 helper follows, so a malformed block cannot re-frequency a model in a way no reference reproduces; * the support predicate accepts "dynamic"/"dynamic_ntk" ONLY when the numbers were read, and still refuses yarn, linear and mrope. `all llama3 rope checks passed`, including the new arms. What this does NOT yet claim: that Hunyuan-0.5B serves. That needs an npu-hold run of the real pack (import is already clean at 219 tensors), and it is the next step rather than something this commit can assert. * npu-pack: pin the tail-head SEARCH, and fix the refusal that left an offset behind Review of #1773 was right on all three counts; this answers the two that are code. **The search is now tested, and writing the test found a real bug.** `tail_head_selftest()` builds a synthetic tied head through `encode_tile` (an embedding table, quantised, four tiles), then asserts: it is located at slack 0 and verified; placed behind one junk tile with three zero tiles after it, it is located at slack 1 with the runner-up below `kTieCorr`; as an untied pack with padding it is REFUSED; as untied and exact it is taken and marked unverified. No pack, no device, no fixture. The untied arm failed on its first run -- not because the test was wrong: `locate_tail_head()` set `r.off` before deciding, so every refusal path left a positive offset behind, and the CALLER decides on `off < 0`. An untied pack whose tail ends in padding would have been read anyway, at a placement the function had just refused. `off` is now -1 until a placement is accepted. That is the failure mode the review predicted the search could have silently, found by the arm it asked for. **The geometry in the arithmetic test is corrected**: the hidden-896 head tile is `group_bytes(896, 64) = 34,816 B`, not 122,880, and the tail is `1,856 + 2,374 x 34,816 = 82,655,040 B` (the 0.5B pack's measured span: tile 0 at 209,000,768 B in a 291,653,952 B file). The `896 * 0 +` noise is gone. **On the gates being untied**: agreed, and the PR body now says so -- "imports; head unverified" rather than "gate passed", exactly as the review put it. What the pairs of gates do still show is that placement is measured and reported, not assumed. A named-reference correlation for the untied case needs the 1.8 twin's `MatMulNBits` head or the upstream checkpoint, i.e. a 7-9 GB download per model; I would rather do that as its own step with the number in the import's self-check line than half-do it here. One datapoint the third pack gave, which is the zero-run logic on a real file: `Hunyuan-0.5B` has **28 B** of trailing zeros -- not a whole tile -- and it still lands at slack 0, verified, because the candidate window clamps at the envelope. That is a sub-tile-aligned zero run in the wild, which the arithmetic test now covers and no fixture had. `all npu-pack checks passed`; Qwen2.5-0.5B 291 tensors (corr 0.999999, control 0.0716), Coder 339 via the block-32 synthesis path, Hunyuan 219. * npu-pack: the tail-head search always tries the envelope's end, and an arm that proves it Review of #1773. The candidate window is anchored on the trailing zero run and walks BACKWARDS from it, so when a pack ends in non-zero junk after the head the window slides forward past the head's true start (`base = tail_off + env`). Arm (c2) builds that shape and asserts `first > env` -- it blessed the span without noticing the one placement that matters was outside it. `base` is known independently of the zero run, so it is now always candidate 0 and the slack window is kept for the rest. Cost: one extra candidate (a decode of `rows` tiles) on packs whose head is followed by junk. Arm (e) in `tail_head_selftest`, where it can bite: 18 junk tiles AFTER the head then 3 zero tiles puts `last = base + 18*tile`, `first = base + 10*tile`, so the window contains none of the true placement. Mutant check -- revert the fix to the window-only loop and the self-test fails with a head followed by junk and then zeros was not located: the tail holds 9 tile-aligned placement(s) and none of them survives decode->encode, so the head's offsets are not established and with the fix restored it passes. Verified by building `npu_pack_import_test` at this head and running it: `all npu-pack checks passed`. Arm (b) cannot see this defect -- with one junk tile BEFORE the head the window is clamped to `base` and the head is inside it -- which is exactly why the review's mutant m2 (loop restricted to `k0`) passed the old self-test. Refs #1773 --------- Co-authored-by: bong-water-water-bong <bong@users.noreply.github.com> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
… directory (#1801) The canonical copy is a cache -- reused when size, mtime and the rules version all match -- but its size is the pack's own .fconst, and it was written to fs::temp_directory_path() first. On strixhalo /tmp is tmpfs, so that is RAM, per pack, held until reboot: three h4096 imports from one afternoon left 11.4 GB of tmpfs behind (/tmp/1bit-fconst19-20133236-4096-32-32.fconst and two siblings), which is a large part of why the box was down to a few GiB available twice that evening. Order is now: 1. ONEBIT_FCONST_SCRATCH, when set (a lane's own scratch dir, or a test's temporary one); 2. $XDG_CACHE_HOME/1bit/fconst19, else $HOME/.cache/1bit/fconst19 -- created if missing, which is the part the first version got wrong: without create_directories() the new first choice never wins on a fresh box, because it does not exist and the loop walked straight past it back to /tmp. Caught by testing the default path as well as the override; 3. the temp directory, then /tmp, then the current directory -- the existing fallbacks, kept for a box with no HOME. Nothing about the cache key changes, so an existing copy in the old location is simply not found and one is written in the new place; the note that already says where the copy went still prints. Measured, both arms: default -> ~/.cache/1bit/fconst19/1bit-fconst19v1-...-1536-12-28-995712640-....fconst ONEBIT_FCONST_SCRATCH=/tmp/scratchtest -> /tmp/scratchtest/1bit-fconst19v1-... (same name) npu_pack_import_test: all npu-pack checks passed, 0 FAIL Co-authored-by: t <t@e.com>
…ipped only as lm_head (refs #1782) (#1803) A 1.0.12 Decision pack (`FastFlowLM/D3-Edge-NPU2`, Apache-2.0) would not load: `model has no tensor model.embed_tokens.weight`. Two defects in one line of `npu/model.cpp:77`: dims_.vocab = int(tensor("model.embed_tokens.weight").shape.at(0)); 1. **It required a tensor name that a tied pack need not ship.** The pack declares `tie_word_embeddings: true` and stores the [248320, 1024] table ONLY as `lm_head.weight`, so the name lookup threw before anything else ran. 2. **It read the wrong dimension.** A tiled entry's shape is [tile_rows, tile_cols, tile_bytes]: that table is `[7760, 4, 8704]`, so `shape[0]` is a TILE count and the loader reported `vocab 7760, hidden 4` -- while `config.json` stated `vocab_size: 248320` all along. Now: `vocab_size` from config.json when present (it is, in every pack that declares it); otherwise the embedding table, falling back to `lm_head.weight` when the model declares `tie_word_embeddings`; and a 3-D tile grid converts to rows (32 per tile, `npu/q4nx.h`). With neither, the refusal names what is missing instead of "model has no tensor …". Measured on this box: the pack then loads and serves on our route with the kernel set we already had (dims H=1024 NL=12 NH=8 NKV=2 HD=256 IM=3584 NV=248320, tied embedding dequantized to 254,279,680 floats, `d3 decision head: 255 options`, warm-up 3.6 s, ready). Three device-free arms in `tests/npu_model_bounds_test.cpp` (the fixture writer now takes extra config fields): config vocabulary with a `lm_head.weight`-only table; no `vocab_size` and a 2x4x8704 tile grid -> 64 rows, not 2; and neither -> refused by name. Co-authored-by: t <t@e.com>
…esign it loaded was packed row-major and measured wrong (#1798) `I8Ctx::blocked_b` defaults to false and only two callers ever set it — the forward (`q4nx_forward.cpp:1303`, from the container's `q4nx_weight_layout`) and the xdna target (`xdna_target.cpp:164`). The EP/probe path (`onebit_npu_gemm.cpp`, the code behind `onebit_npu_gemm_probe` and `ONEBIT_EP_NPU_ELF(S)`) creates its `I8Ctx` and calls `init_elf` without touching it, so a `_b8` ELF got row-major-packed B on every call. That is not a wrong answer you can see as wrong. Measured on the device, one npu-hold ticket each: SHIPPED, device-proven 4B set full_i8_O_K4096_N2560_b8_m16 max_abs=5.763e+01 max_rel=4.195e+04 FAIL newly built 3584 family full_i8_QKV_K3584_N4608_b8_m16 max_abs=5.879e+01 max_rel=4.622e+04 FAIL full_i8_O_K3584_N3584_b8_m16 max_abs=6.336e+01 max_rel=5.560e+04 FAIL A set that already serves NPU rows (the 4B family) fails the same way as a set built minutes earlier, at the same magnitude — which is what exposed this: the harness, not the kernels. `fit got~scale*ref` came out 0.0002–0.0078, i.e. no scale relationship at all, which is the layout end rather than a quantisation end. The fix reads the layout off the **file name**, which is already the family contract (the forward derives the same `_b8` from the container key), with an explicit `ONEBIT_EP_NPU_BLOCKED_B` winning over it so a mis-named file stays testable: kernel->ctx->blocked_b = elf_declares_b8(elf); // before init_elf, at both creation sites Verified: the translation unit compiles clean (`g++ -fsyntax-only -std=c++20 -DONEBIT_NPU_RUNTIME_OWN=1` with the private runtime headers). **The device re-run is not done, and I will not claim it**: every NPU-enabled build tree on this box is either 121 commits stale (`wt/tprs-engine-s`) or points at deleted sources (`lb/onebp-ort-npu`), so an after-fix PASS needs a fresh configure. Whoever has one can settle it in one ticket: the shipped 4B `O` shape should go from `max_abs 5.76e1` to within the 2 % bound. Why this matters beyond my row: **any `_b8` family "verified" through this probe since the b8 layout landed was verified by a harness that could not speak it.** A `READY` on a kernel PR citing `onebit_npu_gemm_probe` for a `_b8` design should be re-read. Side observation from the same ticket: `~/ogaload/oga13/kernels-05b-hyb-b8/full_i8_O_K1024_N1024_b8_m16.elf` fails to *load* at all ("npu runtime: invalid program file (code 401)") while the 4B set loads — two generations of ELF in the same cache directory, and the older one is not readable by the current runtime. Co-authored-by: t <t@e.com>
…he 27B stops materialising 4.74 GiB (engine#1784) (#1807) The bf16 and plain-int8 tables already read in place (#1795). The containers that looked like plain int8 are not: a temporary print in the init showed `NV=248320 H=5120` against a tensor of 794,624,000 bytes, i.e. the file holds 155,200 tiles of 5,120 B = **0.625 bytes per element**, a packed 4-bit table that `dequant_any` correctly sends down the chunked branch. Decoded whole, it is 1,271,398,400 floats = **4.74 GiB of anonymous memory for a 0.74 GiB file**, on a table that is read one row per token. [F4DEBUG] etd=model.embed_tokens.weight dtype=I8 data_size=794624000 en=1271398400 NV=248320 H=5120 kpad_=5120 equal=0 `q4nx_dequant_tile_row()` (public tree, `npu/forward/src/q4nx_dequant.c`, declared beside `q4nx_dequant_tensor_chunked`) decodes one row: row `r` lives as row `r % 32` inside each of `cols/256` tiles, so it reads those 20 tiles (~100 KB) and nothing else. Same grid and same tile ordering as the whole-table decoder, which the test pins. `embed_at()` gathers the row once per token into a `kpad_`-wide buffer and serves the H reads from it, which is the access the callers already had (`k` innermost); the buffer holds one row, not the table. **Measured on the device route** (`npu-hold`, the pack's own `full_i8_*.elf` kernels, one 9-minute ticket per arm): | | RssAnon after load | after one prompt | logits | |---|---|---|---| | before (materialised) | 9773 MiB | 9947 MiB | `c93ff216…` | | after (row per token) | **2658 MiB** | **2678 MiB** | `c93ff216…` | **7.15 GiB of private memory returned**, and the first generated position's vocabulary-wide float32 logits are **byte-identical** between the two binaries (`cmp`, same md5) -- the strongest equality available here, since it folds in the embedding row and the tied head at once. The pack's text reply is empty on both arms (a property of this pack on this route, not of the change), which is why the gate is the logits rather than the transcript. Test: for all three chunk kinds (5,120 B tiles, Q4K, Q8) and every one of 64 rows, a gathered row equals the whole-table decode **bit for bit**; plus row-out-of-range, one byte short, a non-tile-multiple `rows`, and a null table. Host only, no device. Suite 168/168. Refs #1784. Supersedes nothing: #1795 stays as the bf16/plain-int8 half. Co-authored-by: t <t@e.com>
…t slice) (#1802) The forward packs the lm head per tile on every serve: memset, `q4nx_pack_B_int8_blocked`, a BO sync. At the h4096 shapes that is the 10-17 s quiet / minutes contended figure #1716 measured. Nothing device-side enters that image, so it can be produced on the host once and copied on later serves -- but only if a stale or damaged image cannot be mistaken for the right one, which is what this adds. `q4nx_head_image.c` writes header + codes + scales, where the header carries a magic, a FORMAT VERSION, the geometry and an FNV-1a hash over the payload. The reader recomputes the hash and refuses on: a wrong version, a geometry that does not match the container's, a short payload, trailing bytes, or a hash that does not reproduce. Writes go to a temporary name and are renamed into place, so a truncated image can never be read as a valid one -- a partial image copied into a weight BO would answer wrongly with no message at all. Absence is a refusal with a reason, not an error, so the caller can treat "no image" as "pack as today". That keeps the packer and the layout contract (#1769, #1778) the only source of truth; this file is an accelerator, never a definition. The arm (device-free, in `npu_design_grid_test`) packs a head-shaped tile, writes the image, reads it back and asserts the codes and the scales are BYTE-IDENTICAL to a fresh pack -- the bar, not a correlation -- then that a single corrupted payload byte, a mismatched column count, an absent file and a truncated file are each refused with a reason. 16 arms, all green; `tools/copyright.py --check` rc 0. Next slice, not in this PR: the importer writes the image at import time, and the forward's `head_packed` branch copies it and verifies a bounded sample against the tiles it holds. Co-authored-by: bong-water-water-bong <bong@users.noreply.github.com>
…erving) (#1812) The prefix cache merged for #650 has been reachable only through forward-serve: --prefix-slots, --prefix-cache-mb and --prefix-min-gain are parsed in app/forward_serve.cpp:808-810 and nowhere else, so a --device npu serve ran at the defaults (8 slots / 4096 MB / min-gain 128) and could neither be sized nor gated for the flash profile F5 has to build. This only carries the operator's choice through, in the same style as --npu-target: a sentinel of -1 means 'unset' and the forward's own default stays the single source of truth for that. No behaviour change without the flags. Verified on hardware, bonsai2-t2s2-perrole876 + kernels-m16, ctx 1024, under npu-hold, one server, two identical requests, reading the server's own usage.flm_prefix_cache: default gate, 347-token prompt : cached_tokens 0 -> 355 of 356, reason 'hit' default gate, 33-token prompt : refused, 'gain below prefix-min-gain' this change, 33-token prompt with --prefix-min-gain 16 : cached_tokens 0 -> 32 of 33, reason 'hit' So the flag is parsed in serve, forwarded into the forward's argv, and changes the cache's decision on a prompt the default gate refuses. Built with -DONEBIT_NPU=ON (ONEBIT_NPU_RT_DIR from the box's working build); the non-NPU configuration also compiles it. Co-authored-by: bcloud <bcloud@local>
…est row (engine#1800) (#1806) Of the eight tests/*.sh #1800 lists as referenced by nothing, the seven gpu_hold_* scripts are registered: they live in cmake/tests-gpu-hold.cmake (moved there by #1699), which the issue's grep of CMakeLists.txt did not read. Two really had no runner on main: * tests/base_modes_test.sh pins the 1bit OS base layer's mode normaliser (scripts/pack-elf.sh, engine#1474): it only NARROWS modes (a 0600 secret stays 0600) and the keep-list keeps /tmp at 1777. Registered as `base_modes`, next to release_check_selftest and in the same shape: no build, no device, no target. * tests/anchor_rerun.sh is an operator measurement, not a test: real GGUFs, an output directory, a 30-minute health window per model, PASS/FAIL printed per model and exit 0 always. Registered, it would skip forever or go green over a FAIL line, so its header now says that instead. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
main's ci went red twice on the same mirror fault (11-10 04:33Z at b37397b, 05:45Z at 54a259f): apt-get update succeeds, then the install 404s on perl 5.40.1-7ubuntu0.4 from security.ubuntu.com. A re-run of the first one passed with no code change. The step now retries update and install as one unit (3 tries, 30 s then 60 s back-off) and asks apt for per-fetch retries, so a retry re-reads the index and can land on another mirror. A third failure still fails the job. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ure (#1777) * release-check: the hrx-device row repeats the runtime's own init failure A runtime that LOADS is not a runtime that initialises. `ldd` resolves NEEDED libraries and never symbols, so a libhsa that is present but lacks `hsa_amd_queue_create` (measured on a distro ROCr, engine#1766) passed the row's loadability pre-check, listed no devices, and the row reported "no HRX0 listed" -- which reads as a GPU fault when the GPU is fine and the runtime never came up. The backend now logs `<file>:<line>: hrx_gpu_initialize(0) failed: <reason>` (the llama.cpp pin in #1768). This row reads that line and repeats the reason instead of the generic message, and says plainly that the library loaded, so the cause is a symbol or config and not a missing library. Three arms checked against the helper: the real backend line extracts `NOT_FOUND: symbol 'hsa_amd_queue_create' not found in library`; a healthy "(none)" device list stays empty so the existing message is unchanged; and a decoy `hrx_gpu_device_count failed:` line is not mistaken for the init reason. * tests: pin the hrx-device row's init-reason extraction Four arms on hrx_init_reason(), following the tests/gpu_hold_*_test.sh shape (takes the script as $1: these shell tests are run by hand, not registered in ci.yml). The arm that matters is the negative one: a healthy "(none)" device list must yield nothing, or a row that always claimed an init failure would look like an improvement over "no HRX0 listed" while sending the reader after a symbol that is not missing. Also pinned: the file:line prefix is stripped, a later hrx_gpu_device_count failure is not mistaken for the init reason, and the line is found when it is not first. Against the unpatched release-check.sh it fails with "no hrx_init_reason()", so the test bites. * tests: the copyright notice the build requires on the new arm The build job's first step is 'python3 tools/copyright.py --check', and it failed the whole job on this one file: 'missing copyright notice: tests/release_check_hrx_init_reason_test.sh'. Added with tools/copyright.py --fix; --check now passes and the test itself still runs its three arms green. --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: pi-mosaic <pi-mosaic@1bit.local>
…-exact (engine#1808) (#1817) `codes.astype(np.int32).T @ a8s[e]` was a line inside the `gate` subcommand, so the only int8 oracle in the tree could not be pointed at a per-op role: `a8s` comes from a pack's activations. Extracted as `int32_accum(codes, a8)` and `quantise_i8_row(a)` with their call sites updated; no behaviour change. Why it matters: #1808 records that the only host arm the engine offers is f32 (`NPU_INFER_HOST_GEMM=1`) against an int8 x int4 device, so every family gate produces a cosine where TNPU-ALL par.4 asks for bit-exactness. The prerequisite for that gate is a host arm that agrees with this oracle exactly -- which needs three things the tests pin, each of which fails silently otherwise: * GROUP = 128 is the weight scale's divisor (a host using another container's 32 gets a different scale); * the activation scale is max|a|/127 computed in float32 -- dividing in float64 differs in the last bits, and for a `max|d| == 0` gate that difference is decisive; * int32 stays exact because |sum| <= 8*127*K: 2,080,768 at K=2048 and 11,184,128 at K=11008, headroom x1032 and x192. A K beyond ~2.1e6 would break the bound and the accumulator would have to widen. 14 checks, verified in both directions: GROUP 128 -> 32 makes the grouping arm fail, and restoring it passes. One arm the first draft got wrong is instructive and is recorded in the test: "codes stay in [-8, 7]" cannot fail, because |g|/scale is bounded by 7 by construction, so the reachable check is the round-trip error staying inside half a group step (11.5148 against 11.5339 on a saturating fixture). The one-line ci.yml registration cannot come from this lane: the push was rejected with "refusing to allow an OAuth App to create or update workflow .github/workflows/ci.yml without `workflow` scope". The line to add, in the numpy/gguf python-test step, is: python3 tests/moe_expert_chunks_accum_test.py # the int8 host-reference contract (engine#1808) Until then the test is run by hand, like the shell tests in tests/ that ci.yml does not list. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…ine#1740 W2, Hunyuan-0.5B) (#1792) * npu-pack: locate the 1.9 head in the .fconst tail instead of requiring it to end the file (engine#1740 W1a) The tail is [envelope][rows x tile][padding] and the importer required `have == env + rows*tile` exactly. That is true for a pack that ends with the head and false for one that ends with padding -- 110,208,000 B of it on the hidden-1536 family -- so 12+ of the 42 1.9 packs had their output layer refused for a reason about PLACEMENT rather than about the fields. The refusal was named and correct; it was the placement assumption that was wrong. The placement is now measured: * the trailing all-zero run at EOF is counted, whatever its alignment; * `plan_tail_head()` (exposed in the header) turns (bytes, tail_off, envelope, rows, tile, zeros) into the earliest and latest tile-aligned candidate, or a refusal carrying both lengths. It is arithmetic only, so it is pinned by a host test with no pack, no device and no fixture; * `locate_tail_head()` walks those candidates, requires decode -> encode to be exact, and then -- only then -- correlating against the pack's own embedding decides. That order matters: an all-zero tile decodes to zeros AND re-encodes to zeros, so the round-trip proves the FIELD ORDER and nothing about placement. Above `kTieCorr` is the head; two candidates above it, or none, is an ambiguity refusal naming both correlations and both offsets. Tied and untied are reported rather than assumed. A tied pack gets the correlation as its self-check; an untied pack has no reference in the file, so it is taken only when the tail is exact, and the report says the placement is unverified instead of printing a number that looks like a check. `tie_word_embeddings` is still read from the pack, never inferred. Gate 1, `Llama-3.1-8B_rai_1.9.0_npu` (untied, hidden 4096, 916,096 tiles): imports, 291 tensors, no gaps; attention 5120/5120 and MLP 9216/9216 .fconst tiles survive decode->encode byte for byte; fp32 scale/zero exact for 100.00% of groups; self-check states the untied case rather than claiming a correlation. Regression, `Qwen2.5-0.5B-Instruct 1.9` (tied, hidden 896): unchanged -- 291 tensors / 67,088 tiles, corr 0.999999, rel_err 0.000700, row-shifted control 0.0716. Block-32 heads (the 1.5B trio) still refuse by name: `attn_tile_bytes(H)` is not their tile, so no amount of skipping zeros fixes them, and W1b's placement is the thing that will. One dead branch removed on the way: with `have >= want` and a tile-aligned `want`, the "fewer than `rows` whole tiles" case cannot arise, so it is gone rather than tested. * npu-pack: report WHICH tail placement the head was taken from "imported" cannot distinguish a tail that ends with the head from one that ends in padding, and that difference is the whole point of the search. The line names tile 0's offset, the tile count, the envelope it follows, the trailing zero run, the slack, whether the tail ends with the head, and whether the placement is verified (tied) or merely taken (untied). Measured through it: Qwen2.5-0.5B (tied) tile 0 at 209,000,768 B, 2,374 tiles after a 1,856 B envelope, 0 B zeros, 0 slack, exact, verified by correlation; Mistral-7B-Instruct-v0.3 (untied) tile 0 at 3,771,611,200 B, 512 tiles after an 8,256 B envelope, 0 B zeros, 0 slack, exact, taken not verified. * npu: apply the dynamic-NTK rope base rule instead of refusing it (engine#1740 W2, Hunyuan-0.5B) The 1.9 Hunyuan packs declare `rope_scaling: {type: "dynamic", factor: 1.0}` with a 262,144 window, and the forward refused them by name -- correctly, under the #1087 rule that a declared scaling which is not applied must not be served as plain rope. This is the rule, so the refusal can go. Dynamic NTK is a BASE rule, not a per-dim one, so it is a pure function of (theta, factor, window, position, dim): below the window: base = theta -- bit for bit the same as a model with no scaling past it: base = theta * (factor * pos / window - (factor - 1)) ^ (dim / (dim - 2)) which is HF's `_compute_dynamic_ntk_parameters` and what ggml applies through rope_yarn. Because the base moves with the position it is computed once per position rather than once per dim pair, and because the rule is a pure function it is pinned by the existing rope test with no device, no pack and no fixture: * below the window the base IS theta (position 0, and one position short of the window), and Hunyuan's factor 1.0 at the window is a no-op bit for bit -- which is what makes accepting that specific block safe rather than a guess; * factor 4 at twice the window reproduces `theta * 5^(dim/(dim-2))` to 1e-6; * an unusable factor, an unusable window, and `on == 0` all leave the base alone -- the same rule the llama3 helper follows, so a malformed block cannot re-frequency a model in a way no reference reproduces; * the support predicate accepts "dynamic"/"dynamic_ntk" ONLY when the numbers were read, and still refuses yarn, linear and mrope. `all llama3 rope checks passed`, including the new arms. What this does NOT yet claim: that Hunyuan-0.5B serves. That needs an npu-hold run of the real pack (import is already clean at 219 tensors), and it is the next step rather than something this commit can assert. * npu: pin the declared-type half of the dynamic-NTK path, and expose the reader Serving Hunyuan-0.5B to test the rule end to end stopped one layer earlier: the container loads, then serve reports that no kernel set resolves for it, so the rope path is never reached. That is a kernel-family gap, not this rule's, and it is handed over with the shapes (`GU K1024 N7168`, `D K3584 N1024` are in no family). What this adds is the evidence that does not need a device: the declared-type extraction now has an arm proving it SEES Hunyuan's block ({"type": "dynamic", "factor": 1.0}, max_position_embeddings 262144) and that a config with no rope_scaling block reports nothing -- the second half is what keeps every other model on plain rope. read_rope_dynamic() is exposed so the numbers it parses can be pinned the same way once that arm has a C++ home. Measured on the way: the import of the pack is clean (219 tensors, head located at 240,891,456 B, 1,888 tiles, 28 B of trailing zeros, slack 0, verified against the pack's own embedding). * npu rope: the dynamic-NTK exponent uses the rotary width, not the head dim Review of #1792. HF's `_compute_dynamic_ntk_parameters` computes `dim = int(head_dim * partial_rotary_factor)` and applies `dim / (dim - 2)`; this forward has exactly that width in `rot` (`(rotary_dim > 0 && rotary_dim <= HD) ? rotary_dim : HD`), and the dynamic base was being called with `HD`. For any model that declares `rope_scaling: dynamic` together with `partial_rotary_factor < 1` the exponent would be computed from the wrong width -- a wrong constant inside the formula the PR exists to get right. Unreachable in today's catalog (no pack on this box declares both; `grep partial_rotary ~/models/oga/*/config.json` is empty), so this changes no served bytes today and is a one-word correction to a constant rather than a behaviour change. Verified: full build of the tree, `npu_rope_llama3_test` -> "all llama3 rope checks passed", `npu_pack_import_test` -> "all npu-pack checks passed". Refs #1792 * npu rope: pin the dynamic-NTK exponent against a known value, and prove the arm can tell the two widths apart The review of this PR found that every arm asserted a PROPERTY -- below the window the base is theta, past it not below theta -- and every one of them is satisfied by any exponent. That is how `00d9a94`'s bug (the caller passing head_dim instead of the rotary width) survived a green suite: I re-introduced it and the suite still said "all checks passed". With factor 1.0 the rule collapses to `theta * (pos/original)^(dim/(dim-2))`, so one printed constant pins the exponent and the width it is computed over: pos 524288, original 262144 -> ratio 2 dim 64 -> 1e6 * 2^(64/62) = 2.0435e6 <- asserted to 1e-6 relative dim 128 -> 1e6 * 2^(128/126) = 2.0219e6 and the third arm asserts the two differ by more than 1e3, so the arm cannot pass if the widths are confused. Verified the way the review verified its claim: substituting 128 for the 64 call fails BOTH of those arms, and restoring it passes `all llama3 rope checks passed`. --------- Co-authored-by: bong-water-water-bong <bong@users.noreply.github.com> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: agent <agent@local>
…own header, not from the epsilon-anchor vote (refs #1740) (#1819) * npu-pack: take a 1.9 pack's stride and second-header offset from its own header, not from the epsilon-anchor vote (refs #1740) The first import of an h3584 pack (Qwen2-7B, 8.95 GB, geometry exactly the family's) refused: fconst: layer blocks not located: the epsilon headers do not repeat with a single period while `tools/fconst19_measure.py` reported the pack's structure exactly -- `header_off 9,094,208`, `stride9 126,098,736`, byte-identical to the two packs the h3584 rule was measured on. The pack was fine; the locator was not. **Why the vote loses.** `locate_fconst()` derives the stride by voting on `pos[i+P]-pos[i]` over epsilon anchors (bf16 1e-6 + ten zeros) and demanding a 66 % mode. In 1 GiB of these weights, eps-SHAPED byte pairs are common: this pack yields 21 positions for 28 layers and the right stride (`126,098,624` = `126,098,736 - 28*4`) wins only **52.6 %** of lag-2 pairs. Under the bar -> refusal. **Why re-deriving it at all was the mistake.** `canonical_token_fconst()` has already computed the same stride from the pack's 112-byte per-head header (`fconst19_detect`, then `stride8 = stride9 - hdr_len`) to build the very copy the locator then scans. A 112-byte exact structure repeated 28 times is a far stronger witness than a 12-byte coincidence, so the header now wins and the vote is the fallback -- which is what the 1.8 / R1-Distill line still uses, where it works. The second field has to come with it: `h1` (the second eps header inside a block) decides whether the payload reader sees 2 headers per block or 4, and leaving it 0 made every MLP projection refuse. With the stride known, the anchors' residues `pos mod stride` repeat across all 28 blocks, so their mode (excluding 0) is `h1` -- no dominance test needed. Measured on `amd/Qwen2-7B_rai_1.9.0_npu`, same command before and after: before: fconst: layer blocks not located … incomplete: 280 container 1.1 GB after: fconst: stride 126098624, second header 16161856, 56/56 RMSNorm anchors MLP: 9856/9856 .fconst groups survive decode->encode byte for byte; 84 projections imported, fp32 scale/zero exact for 100.00 % of groups incomplete: 196 (attention only) container 5.4 GB Verified: `npu_pack_import_test tests/golden` -> all checks pass; with the reference 1.9 pack -> `all npu-pack checks passed (including the pack leg)`; `tools/copyright.py --check` rc 0 with the tree staged; `git add <paths>` only. Not fixed here, stated so the next step is not a surprise: the **attention** payload still refuses on this pack with "the layer block has 1 epsilon headers, and this importer reads blocks of 2 … or 4", while MLP imports byte-exact. That is a block-shape question (this pack's payload between `prefix` and `h1` is 16 tile-slots; the h3584 rule records an 8-tile bundle), not a locator question, and it needs the same pair-measurement treatment the rule row got -- which is why it is a separate change. * npu-pack: the header-derived stride must also populate the header LIST, or every attention projection refuses (refs #1740) Follow-up in the same change: with `stride` and `h1` set but `hdr`/`period` left empty, the payload reader classified the pack as "1 epsilon header" (`hdr.size() + 1`) and refused all 196 attention tensors while the MLP imported byte-exact. `hdr` is the in-block header list and its SIZE is how a pack is classified as a 2-header (q/k/v/o in one payload) or 4-header (Qwen3: q/k/v, two per-head QK norms, o_proj) block, so the header path has to fill it. Measured on `amd/Qwen2-7B_rai_1.9.0_npu`, same command: incomplete 280 -> 196 (stride + h1) -> 0 container 1.1 -> 5.4 -> 6.0 GB attention: 3584/3584 .fconst tiles survive decode->encode byte for byte; 112 projections + 84 bias tensors imported, fp32 scale/zero exact for 100.00 % of groups MLP: 9856/9856 groups byte-exact; 84 projections imported fconst: stride 126098624, second header 16161856, 56/56 RMSNorm anchors found incomplete: 0 tensors The threshold is "at least 3 blocks agree" rather than `layers/2`: the anchor scan window is the first GiB, about 8 blocks at this geometry, so the right residue repeats ~8 times against 1-2 for a payload coincidence -- requiring half the layers silently dropped the real one and reverted this to 280. Verified: `npu_pack_import_test tests/golden` clean; with the 0.5B 1.9 pack `all npu-pack checks passed (including the pack leg)`; `tools/copyright.py --check` rc 0 with the tree staged. --------- Co-authored-by: t <t@e.com>
… number (engine#1783 step 1) (#1816) First slice of engine#1783 (W8/F8). The bar is FastFlowLM v1.0.12's public benchmark -- encoder 1,889 -> 606 ms per 30 s window on a Ryzen AI MAX+ PRO 395, both encoder and decoder on the NPU -- so the harness measures OUR side on the execution unit we actually have (the AMD pack's `*_encoder.onnx` on an ORT provider) and names the unit in the output. Per 30 s window: ffmpeg decode -> Whisper's log-mel front end (n_fft 400, hop 160, 80 mels, 3000 frames, centred) -> `--iters` encoder runs after one warm-up -> median/min/max, plus peak RSS because FLM's 3x encoder speedup cost them 1.3 GB of host memory. The mel is timed separately and excluded: folding the front end into the encoder number would compare two different things. `--self-test` pins what can be wrong without a model: the window arithmetic (exactly 30 s is ONE pass, 30.001 s is TWO, the benchmark's 308 s recording is 11 windows not 10, the trailing window is partial rather than dropped) and the input shape/normalisation invariant. Measured here, tiny encoder, CPU EP, on a BUSY box (load 50-76): encoder per 30 s window: median 140.8 ms (min 123.1, max 218.4, 5 iters), peak RSS 346.5 MB Three runs of the same command gave medians 121.7 / 70.3 / 140.8 ms with a 682 ms outlier, so **those numbers are not quotable** -- which is why the report now carries `load_avg` and the human output prints it beside the figure. Quote the timing with its load or not at all (engine#1423's rule, applied to a harness instead of a benchmark). Not done, and stated: large-v3 (`amd/whisper-large-v3-onnx-npu`, 6.95 GB) is not on this box, so the model the issue actually cares about is unmeasured here; and the packs + onnxruntime are absent from strixhalo (ORT 1.27 with VitisAI is in `~/ryzenai-1.8/venv`, so the harness runs, but only for a pack someone downloads). Refs #1783 Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
… and edge cases (TNPU-ALL W3) (#1805) * tools: the host reference for the per-op kernel gate, with its inputs and edge cases (TNPU-ALL W3) W3's gate asks for a family to be bit-exact against a HOST reference on the device, on 3 seeds plus an edge case. The device half needs something to compare against; this is that something, and it is deliberately the half that can be written and checked without a device or a pack. * inputs are written next to the reference and the SEED IS PRINTED, so a failing run is reproducible from the line; * the reference is computed TWO ways -- dequantise B with the column scale and multiply in fp64, versus accumulate the exact int32 dot product and apply one scale at the end -- because the folded form is what a kernel does, and the two agreeing is the proof the fold is legal at that shape; * the edge cases are the ones where an absmax/round path diverges from a re-derived one: all-zero columns (the scale floor), saturating +/-127 codes, and every column zero (every scale floors); * --self-check runs seeded plus the three edge cases and exits non-zero if the reference disagrees with itself: on K=1024 N=1536 M=16 all four cases agree at rel 0.000e+00. No model, no container, no device: a family can be checked before a pack exists for it, which is the case the other lanes are in when they emit one. * tools/npu_elf_ref: carry the project's licence block (the copyright gate is what went red) The build job failed in 'Copyright notice on every file', not in a compile: the file had the copyright line and the SPDX identifier but not the Apache-2.0 notice block every other file in tools/ carries. Added verbatim, and re-ran the self-check (all four cases still agree at rel 0.000e+00) plus tools/copyright.py --check (rc 0). * tools/npu_elf_ref: apply the edge cases to the WEIGHTS, quantise both operands, and make the self-check able to fail Review round on #1805, all three points correct and all three fixed: 1. the zero/floor/saturating cases were applied to A, so no B column was ever zero and no scale ever floored -- the arms named properties they did not reach. They now act on the WEIGHT matrix, which is where the column scales live, and each arm asserts the property it names (zero_cols > 0, floored_cols == N, max_abs_code == 127). 2. the scale was absmax/127 of the int8 CODES (about 1.0 everywhere) with no activation scale. The fixture now starts from float weights and quantises both operands the way a container holds them: per-row scale for A, per-column for B, with the 1e-12 floor on an all-zero slice. a_scales.f32 is written beside scales.f32 so the device half compares the same pair. 3. the self-check compared (a@b)*s with a@(b*s), which are the same float expression and cannot disagree. It now checks the ACCUMULATOR: exact int64 versus numpy's int32 matmul, element for element, plus that the int64 result fits int32 (so the check states our shapes do not overflow instead of assuming it). is kept as a second, independent arm on the fold. Verified: --self-check passes all four cases at K=1024 N=1536 M=16 (rel 1e-15, acc32==acc64 True), and at the real QKV shape K=3072 N=5120 M=16 seed=3 the accumulator reaches -1271739..1279718 -- 2.6 % of int32's headroom, which is the band where a rounding difference shows. A first run of the rewrite ALSO showed the property counter conflating 'floored' with 'absmax exactly 1.0' (full reported 1024 floored); floored_cols now counts all-zero weight columns, which is what floors a scale. --------- Co-authored-by: pi-inferzo <pi-inferzo@1bit.gg>
… so (refs #1946) (#2095) * ci: a skipped test is a test that did not run, and now something says so (refs #1946) Five of the defects this session kept finding are one defect: **green because nothing ran.** * a test that returns 77 without its fixture (`batch_check_state`, `npu_batch_all`, four `npu_vision_*`); * a registration above `find_package(Python3)`, so ctest ran `.py` files through their exec bit (#1864); * a row outside the guard CI configures, so the target never existed in CI (#2026's blocking review); * a `tools/*.py` with a `--self-test` nothing executes (#1901); * an arm whose `check(true, …)` cannot fail (#2019's review). `ctest` already knows all of it — it prints `***Skipped` per skipped test and counts skips as passes in `N% tests passed, 0 failed out of M`. Nothing read it. `tools/ci/ctest_skips.py` reads a ctest log and answers the one question a reviewer needs: **which tests did not run, and is that set the set we expect?** * `***Skipped` (the test ran and returned 77) → against `tools/ci/ctest_skips.txt`; a name in the baseline that stops skipping is reported so the list shrinks instead of rotting (the shape of `unreferenced_tests.txt`, #1824); * **`(Not Run)` — the binary is missing — is never baselineable.** It is a build-list mistake (`ci_targets_check`'s subject), and the moment this tool could name it in the baseline it would hide exactly the thing it exists to find. `--write` prints those separately and refuses to put them in the file. ## Measured on the CI-shaped tree (`ONEBIT_NPU=OFF`, 192 tests) ``` $ ctest --output-on-failure --timeout 300 > log $ tools/ci/ctest_skips.py --log log 7 tests ***Skipped -> baselined, one line each, with the reason 53 tests （Not Run) -> reported as missing binaries, not baselined 56 failed -> the summary line, unchanged ``` The 53 are this developer tree's unbuilt targets, not CI's — in CI the Build step lists what it builds and `ci_targets_check` keeps that list honest. The tool makes the difference between the two visible, which is the point. ## Self-test Eight arms on synthetic logs: no skips, both skip shapes, a baselined skip accepted, an unbaselined skip refused, a baseline entry that stopped skipping reported, a missing binary refused with no way to baseline it, the two shapes yielding different verdicts, and a log without a summary being "cannot tell" (rc 2). `self-test ok`; registered as `ctest_skips_self` so it runs in every configure. ## What this PR does not do It does not wire itself into `ci.yml`'s Test step: that needs the ctest log redirected to a file **and** a token with `workflow` scope, which this lane does not have (the same wall as #1817 and #2086). The wiring is one line — `ctest … | tee build/ctest.log` then `tools/ci/ctest_skips.py --log build/ctest.log` — and it belongs to whoever can push a workflow change. Refs #1946, #1864, #2026, #1901, #2019, #1824. * ci: the skip baseline carries its notice, and the copyright gate now knows the name (CI found this on #2095) CI was right and the check did its job: ``` build Copyright notice on every file ##[error]Process completed with exit code 1 no comment style known (add one to LINE_STYLES/BLOCK_STYLES or EXEMPT): tools/ci/ctest_skips.txt ``` `.txt` has no comment style of its own, so `tools/copyright.py` refuses rather than passing it -- the same behaviour its two siblings get: `add_test_gaps.txt` (#1595) and `unreferenced_tests.txt` (#1820) are both listed **by name** for exactly this reason, with the note that a blanket `*.txt` would demand a notice in files that are data rather than code. So this does what those two do, and nothing more: * `ctest_skips.txt` joins the by-name list in `tools/copyright.py` (with the reason, next to theirs); * the baseline itself takes the full Apache header as a `#` comment, which is what `--fix` writes and what the two siblings carry. Verified: `tools/copyright.py --check` rc 0; `ctest_skips.py --self-test` ok; `ctest -R ctest_skips` 1/1 passed. Refs #2095 (this is the fix for its failing build), #1946, #1595, #1820. --------- Co-authored-by: t <t@e.com>
…er accuracy has an after (engine#1313 item 3) (#2112) Co-authored-by: t <t@e.com>
…orts cleanly and answers nonsense (engine#1503) (#2101) * npu-pack: a tied head must correlate AND not be mis-scaled, or it imports cleanly and answers nonsense (engine#1503) Two gemma-2-2b hybrid packs measured on this box import with every check green and serve garbage: pack corr rel_err serves Qwen-2.5_1.5B_Instruct_rai_1.9.0_npu 0.996901 0.000000 'The capital of France is Paris.' x3 gemma-2-2b-onnx-ryzenai-hybrid 0.991539 0.136140 'myself myself ...' x3 gemma-2-2b-awq-...-g32-fp16-onnx-hybrid 0.996053 0.094874 'myselfGetBytes...' x3 A head that is a SCALED copy of its own embedding still correlates, so the correlation-only check passes it -- and the import printed the number that distinguishes them (rel_err) as an advisory field next to the correlation it gated on. Both packs cost a device ticket to discover. - one predicate, selfcheck_head_ok(corr, rel), so the four sites that set selfcheck_tied cannot drift; - kSelfcheckRelMax = 0.02, the measured gap: 0.000000 on the pack that serves, 0.095/0.136 on the two that do not; - Report::selfcheck_scaled, set at all four sites next to selfcheck_tied; - main() refuses on it and NAMES both numbers and the threshold. --allow-incomplete does not override: that flag means 'write the tensors I could not represent', and this is 'the tensors I wrote do not decode'. Verified: both translation units pass g++ -fsyntax-only (the json header from a neighbour build), and the predicate's six arms pass against the three measured triples plus a control, a no-correlation case and the inclusive boundary (~ /wt/relcheck.cpp, run). NOT verified here: the full npu_pack_import_test binary and any container import -- that needs a configured build, which the disk floor did not allow at the time; the two test arms are written for CI. * npu-pack: name the partial output in the mis-scaled-head refusal (engine#1503) Demonstrated on the real pack, not argued from the fixture: importing ~/scratch-packs/gemma22b (gemma-2-2b-onnx-ryzenai-hybrid, the pack that served 'myself myself ...' for 16 tokens) now stops at refusing to import ...: the head correlates with the pack's own embedding (0.991539) but its relative error is 0.136140, which is a scaled or partially decoded head rather than a rounding difference (the threshold is 0.020). A pack in this state imports cleanly and answers garbage on the device; --allow-incomplete does not override this. That run also showed the check happens after import_pack() has already copied the text artifacts and written model.q4nx, so a refusal left five files behind. An importer must not delete a directory the caller named, so the message now says the partial output exists and is not servable -- the operator deletes it before retrying. Verified by rebuilding onebit and re-running the same import, and the test target still passes its six arms. --------- Co-authored-by: pi-mosaic <pi-mosaic@1bit.local>
…ames the decision route (Refs #1786) (#2121) A pack with non-causal attention has no decode to run. /v1/chat/completions and /v1/completions tried anyway and failed inside the forward with HTTP 500 and its internal wording (measured on FastFlowLM's D3-Nano-NPU2). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
… the fast route at speed (refs #2106, #2102) (#2119) * tools: npu_serve_smoke, the release gate's check that npu serve is on the fast route at speed (refs #2106, #2102) A serve that falls back to the per-op forward still answers, so a check that asks 'did it answer' passes on the wrong route. This starts 1bit serve --device npu, sends three prompts with thinking off, and asserts the route's own log line, a decode floor (60 tok/s), the first token (600 ms) and non-empty, non-degenerate content. Measured, Qwen3-0.6B, one npu-hold ticket each: the 2026-11-07 release candidate PASSes (median 84.4 tok/s, 277 ms); a main build without the fast route FAILs route, rate (1.9 tok/s) and first-token while its answers are fine. --self-test drives the judging on 15 shapes (the silent fallback, thinking left on, a degenerate reply, the median, missing timings); registered in ctest as npu_serve_smoke_self. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * tools: npu_serve_smoke's self-test pins the median (a mean mutant passed all 15 cases) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * tools: npu_serve_smoke takes the two route texts as options, so it can follow the serve's wording Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
…t (Refs #1786) (#2118) 'Answer with one word' came back as "\n\nParis", and a model that thinks nothing sent a reasoning chunk of two newlines. The whitespace leading the reasoning, and the whitespace leading content that follows a block, are dropped: the rule app/think_split.h applies on the in-process route. Two existing arms pinned the newline after the opening tag as part of the reasoning; they are updated. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
… takes a ticket (#2108) engine#803's P4 gate is "window11 text identity plus **logits KLD 0** on 8 prompts". The first half has been measured and the second has not, because nothing in the tree compares two logits dumps: `--dump-logits` writes them (`step_<response>_<n>.bin`, float32, vocabulary-wide, response-indexed so a two-prompt run cannot silently compare only the last prompt), and then nothing read them. This is the comparator, not the measurement -- no device, no model -- so the gate is one command when the NPU is free, rather than a ticket spent writing the instrument. Thresholds are the HRX gate's own shape (`tools/hw_gate_manifest.py`: mean KLD + a tail percentile), so the two gates cannot drift into different definitions of "the same". tools/logits_kld.py BASELINE_DIR CANDIDATE_DIR [--max-mean-kld 0.01] [--max-kld-p99 0.2] tools/logits_kld.py --self-test Six self-test arms, each one a case this arithmetic gets wrong: - identical dumps give **exactly 0.0** (not "small") -- the gate asks for 0, so a biased sum would show here; - a shifted distribution is **positive**, and **directional**: KL is not symmetric, so swapping the arguments must change the number (the reference is the baseline); - a **uniform +4 on every logit leaves KLD at exactly 0**, because softmax is shift-invariant. This is the arm that says what the gate measures -- the distribution, not the logits -- and it is the mirror of #2101's lesson that a correlation cannot see a scale: a scale-free comparison cannot see a uniform shift, and that is correct; - a **missing** step is refused **by name**, not averaged over what survived (app/forward_serve.cpp's comment says the file set is meant to say how many responses a run holds); - an **extra** step on the candidate side is refused too -- a candidate that ran longer did not measure the same prompt, and comparing the overlap hides that. Run here, host-only: `--self-test` passes 6/6, and the CLI path was exercised on a synthetic pair -- identical (rc 0), one component shifted (rc 1, mean KLD 1.277), uniform shift (rc 0), missing directory (rc 2). **Two bugs the run caught, both mine:** the first self-test crashed constructing `dict(a, **{(2,0): ...})` (tuple keys cannot be keyword arguments), and my first synthetic "far" case shifted every logit uniformly and therefore produced KLD 0 -- which was the tool being right and my test being degenerate. That case is now an arm. Not claimed: any number for the gate itself. This reads dumps; it has never seen one from a device. Co-authored-by: t <t@e.com>
…fill.sealed_elf, decode.narrow_elf) (Refs #1990, #2073, #799) (#2080) Plan::load reads prefill.sealed_elf / prefill.sealed_rows (17..128) and decode.narrow_elf / decode.narrow_rows (1..15), all optional, and refuses the inconsistent forms (no sealed decode set, a path instead of a bare name, rows missing or out of range, rows without a file). A loaded plan wins over config.json; NPU_WIDE_SEALED_ELF and NPU_NARROW_SEALED_ELF still override both. tools/npu_plan.py copies the config keys into the plan and writes nothing new for a pack without them. Re-applied on main after #2042 and #2074 merged; the change is the one reviewed at 4315c23 / 2c3ad10. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
…ize it was measured on (Refs #1785) (#2075) FastFlowLM's Qwen3.5-2B (IM 6144, H 2048, 144 MiB tail) sat under the 256 MB floor, so every decode token re-dequantised the dense FFN tail. A B B A in one npu-hold ticket: 362-367 ms per token with the cache on, 485-520 ms with it off, identical text in 12 of 12 runs. The size rule moves to npu/forward/include/ffn_cache_policy.h with a test. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
…mp-encoder dumps, with before/after on the game (Refs #1313) (#2117) Item 3 of engine #1313, the only engine item left (audit on the issue): a head-only fine-tune of Laya's routing head (type_emb, two head layers, scorer; encoder frozen) on the CPU, over the dumps `1bit route --dump-encoder` already writes. The output is an ordinary --laya-model dir: only the head tensors' bytes change, the temperature is refit on dev (kept when the refit hits the grid's edge), finetune.json records the run. metrics.py reads --per-case files with laya/eval.cpp's definitions. Measured on the game's logged decisions, episodes held out whole (base / fine-tuned): test 33/33 of 39, log loss 0.543/1.395; test2 107/106 of 110, 0.478/0.294; test3 55/55 of 56, 0.475/0.188. A failed-reflex outcome label taught nothing (0/10 held-out rows moved) and is documented as such. tests/laya_finetune_test.py (ctest laya_finetune, skip 77 without torch): synthetic checkpoint + dump; dev log loss 0.676 -> 0.002, accuracy 0.688 -> 1.000; every non-head byte identical; refuses an existing --out; metrics.py against a hand-computed per-case file. 14/14. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
#2102) (#2113) * npu serve: say it at ready when no 16-row decode kernel was found (engine#2102) * npu serve: the #2102 ready line stays silent off the missing-kernel case, and names the fused route Review fixes on #2113, each reproduced at 692ae19 and re-run here: - the host-GEMM arm loads no decode family; design_roles() is 0 there, so it no longer reads "0 of 4" - NPU_DECODE_M16=0 gets its own sentence naming the switch instead of "missing" kernels - a ternary (pb16i/t2s) container with some roles on M=128 is its geometry's (a missing kernel already fails the load), so only 0 of N warns there - the way out names the fused route: all four _m16 twins measured ~2.7 tok/s vs ~1.7 without and 88-101 fused - tests/npu_decode_note_test.cpp carries the full Apache notice (the build's copyright step was red) - docs/npu.md says what the line is and when it is silent Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <pi@1bit.local> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
One conflict, CMakeLists.txt's onebit_npu source list: main added npu/vision_image.cpp, this branch has npu/audio_exec.cpp. Both kept. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Collaborator
Author
|
(cl-flm) This PR has waited seven hours on a base nothing merges into, and main moved under it four times. I have opened #2133: the same branch with |
|
Persistent review updated to latest commit 7e10fcc |
bong-water-water-bong
left a comment
bong-water-water-bong
left a comment
Collaborator
Author
There was a problem hiding this comment.
VERDICT at 7e10fcc: READY
Reviewed head 7e10fcc (refs #1783). #2035 and #2133 are the same branch at the same head; #2035's base is main-audioops and #2133's is main. I reviewed it against main, which is the merge that can land, and post the same verdict on both.
Merge and scope
git merge-treeonto current main: no conflicts. The diff against main is the audio work only, 14 files, +2111/−1.- CI at this head is green:
build,hrx_build, CodeQL, the hrx-system pin checks. Theunstablestate was the pr_agent job still running. - The
onebit_npusource-list conflict kept both sides: main'snpu/vision_image.cpp(CMakeLists.txt:737) and this branch'snpu/audio_exec.cpp(:738).
The one change to shared code is safe for existing packs. The protobuf reader in npu/pack_import.cpp now keeps a wire-type-5 (fixed32) field's bytes in v; before, every float attribute read as 0.0. I checked every read of a field's v in main's pack_import.cpp (:73, :191, :192, :222, :316). All of them are varint fields: TensorProto data_type/dims, AttributeProto.i, a boolean flag. No existing caller reads a fixed32 field. The LLM-side epsilon logic reads bf16 bytes from .fconst, not a protobuf attribute. So the change is additive for existing packs.
Tests, from an NPU build of this head (ONEBIT_NPU=ON, host path; whisper-small from pi-demo/whisper-small):
audio_ops_test: "conv1d … layer_norm … gelu … softmax … all pass", rc 0.audio_mel_test: "front-end matches the reference extractor on tones, silences on the clamp", rc 0.audio_parallel_test: "audio_parallel: ok", rc 0.onnx_graph_test <encoder>: "topological order, op census (12 layers …) and layer-norm epsilon all hold", rc 0.audio_exec_test <encoder> <decoder>, rc 0: "ran 447 nodes -> dims 1 1500 768 … whole encoder on the host matches onnxruntime" and "decoder ran 786 nodes -> dims 1 128 51865 … decoder graph runs on the host and matches onnxruntime". The body lists the decoder arm as not run at this head; it passes here.
Mutants of npu/audio_parallel.h, each rebuilt and run:
| mutant | result |
|---|---|
a worker skips the last unit (u + 1 < units) |
rc 1: "MatMul at ONEBIT_AUDIO_THREADS=2 is bit-identical to the serial result" |
| exceptions joined but never rethrown | rc 1: "a unit that throws on 2 thread(s) is an exception the caller catches" |
| one unit run twice | rc 1: the same bit-identity arm |
I restored the file (git status clean); the rebuilt test is ok.
Not run by me: whisper_transcribe end to end (built, not run). As the body says, nothing here runs on the NPU; it is the host reference the device work is measured against.
When #2133 merges, #2035 and #1931 can be closed, as its body says.
bong-water-water-bong
left a comment
bong-water-water-bong
left a comment
Collaborator
Author
There was a problem hiding this comment.
VERDICT: READY at 7e10fcc — re-issued for the new head, and this time I checked the file class that decides whether a merge PR of this size can even land.
The workflow question, resolved properly
At cfa2312 I said "zero workflow files". That was measured against origin/main, which is not the comparison that matters for a merge onto #1931's branch. Against #1931's head there are three:
.github/workflows/ci.yml identical-to-main
.github/workflows/pr-agent.yml identical-to-main
.github/workflows/hrx-build.yml identical-to-main
Three files differ from 61426f5 and are byte-identical to main — i.e. the merge carried main's workflow files, it did not edit them. So the workflow-scope wall cannot bite: there is nothing here for a lane's token to be refused for. (git diff --stat origin/main HEAD -- .github/workflows/ is empty, which is the check that separates "the merge brought them" from "the branch changed them".)
The delta is the Whisper slice, and the parts I could run, I ran
14 files changed, 2111 insertions(+), 1 deletion(-) ← vs origin/main
tests/audio_parallel_test.cpp +198
tests/onnx_graph_test.cpp +132
tools/whisper_transcribe.cpp +203
audio_ops_test (built and run, device-free):
audio ops: conv1d (strided + padded + edge), layer_norm (biased var, null affine),
gelu (erf), softmax (stable, shift-invariant) -- all pass
and audio_parallel_test.cpp compiles clean against its own include dir (its link needs the pack library, which is the CI target's job, not mine).
The two answers I asked about at cfa2312 are still there and still the right shape
memcmp(got.data(), serial.data(), …) 2 occurrences ← bit-identity, not a tolerance
test_a_throwing_unit(…) 2 occurrences ← the exception path, asserted by surviving it
Scope, and why this should land next
It answers five CHANGES blocking #1931, moves #1931's head, and needs nothing from a workflow. #1931's own arithmetic is the thing the Whisper work will be built on, and it is the last blocker I can see in that family before the NPU route work (#2106, a release-blocker row) has the audio front-end it needs.
One thing worth stating in the body: merging this moves #1931's head, so #1931 needs a verdict at the new head — which will include the audio arms verified here. If the merge author does that in the same comment (rather than a third lane re-deriving it), the record stays in one place.
This was referenced Nov 11, 2026
bong-water-water-bong
added a commit
that referenced
this pull request
Nov 11, 2026
…ges teach the instrument (engine#2143) (#2162) #2143 measured it: with `GGML_HRX_LOG_DISPATCH=1` set, an 18,782-token request *completes* where the same binary, model and request fails without it, and NaN-logit incidents go from 2 to 0. It also carries a published rate that is a 2.3x casualty of this: one plain 18.6k prefill measured 271.75 t/s with the flag unset and 118.86 t/s with it set. Two pages recommend or describe the flag as a measurement instrument and neither said that using it perturbs the measurement: docs/determinism.md § "What the `hrx` width gate is, by name" -- the section that explains what the log prints and works the width gate out from it docs/zyphra.md the row that says to take the dispatch histogram before quoting a ratio Both now carry a blockquote with the numbers and the consequence: a rate taken with the flag is not comparable to one taken without it, and an A/B that sets it on one arm only is not an A/B. docs/hrx.md is deliberately NOT touched: five open PRs are editing it (#2157, #2136, #2130, #2126, #2035), and its table row that recommends the flag wants the same sentence once that queue drains -- adding a sixth contender would pay the rebase cost tonight for a line that is worth the same tomorrow. Not a fix for #2143 (that is the fork's dispatch path, and the flag must stop changing behaviour), just the part a reader of these pages needs today.
Collaborator
Author
|
VERDICT at Verified rather than taken from the comment: this head is Why that matters and is not just bookkeeping: two open PRs with the same reachable content on different bases is exactly how a merge gate ends up with a merge that reverts half a day of And the one ask from my READY at |
bong-water-water-bong
added a commit
that referenced
this pull request
Nov 12, 2026
…to main — supersedes #1931 and #2035 (Refs #1783) (#2133) * npu-pack: read a component's per-layer embedding table, beside its token table (WIP, engine#1725) The gemma-4 1.9 components carry, beside `embed_tokens.weight`, a PER-LAYER table and a projection norm: embedding.embed_tokens_per_layer.weight (262144, 8960) f32, external (= 35 layers x 256 head_dim) embedding.per_layer_projection_norm.weight (256,) f32 The decoder consumes them as `per_layer_inputs`, so a container with only the token table gives the decoder less than it asks for -- which is why the previous commit refused such a component by name. This adds the reader, so the next slice can write the container section the forward path will load. EmbedPerLayer { table [V][layers*head_dim] row-major, norm [width], V, width } + `ok()`; read_embedding_per_layer(pack_dir, why) resolves the component exactly as the token half does -- genai_config.json's model.embedding.filename (relative only, no ".."), external data beside the GRAPH rather than at the pack root -- reads f32, refuses any other dtype by name rather than mis-decoding it, and says so when the projection norm is missing (the table is still usable, but a consumer expecting a projected row would silently take raw ones). WIP: no container section and no forward consumer yet; that is the next slice, with its test. Builds clean (`npu_pack_import_test` rc 0). No vendor code or layout: only the pack's own ONNX names and dims are read. * npu-pack: write a component's per-layer embedding into the container (WIP, engine#1725) Second half of the reader in the previous commit: `import_pack()` now writes the per-layer table and its projection norm beside the token table -- model.embed_tokens_per_layer.weight BF16 {V, layers*head_dim} (from the component's f32) model.per_layer_projection_norm.weight BF16 {layers*head_dim} -- and the slice-1 refusal in `read_embedding_component()` is relaxed to match: a component that carries `embed_tokens_per_layer` is no longer refused by that reader, because the mechanism it was waiting for now exists. Measured on the real pack with this change (`1bit npu-pack info ~/scratch-packs/pi-demo/e2b`, loader built from main): before: the pack has no model.embed_tokens.weight, and the component carries embedding.embed_tokens_per_layer.weight, i.e. a PER-LAYER embedding table ... the per-layer embedding mechanism is not implemented here yet after: rc 0, a full plan -- dims hidden 1536 / 35L / 8 heads / 1 kv / head_dim 256 / IM 6144 / vocab 262144, 423 tensors, and 350 `incomplete` entries that all say the same NEXT thing: "layer blocks not located: the epsilon headers do not repeat with a single period" So the embedding half is no longer what blocks a gemma-4 pack: the next wall is the Token .fconst block layout, whose rule row cannot be measured for gemma-4 (no 1.8 twin exists) -- the `txn_bins` preformat route designed on engine#1503. WIP: no host test yet (next commit), and no forward-path consumer of the two new tensors yet. * npu-pack: pin the per-layer embedding read, with the fixture mistakes that made it fail first (engine#1725) Test for the two WIP commits before it: `read_embedding_per_layer()` reads the component's `embed_tokens_per_layer` (V x layers*head_dim) and `per_layer_projection_norm` (width) as f32, resolving both through genai_config.json's embedding component and its external data beside the graph; and the token table must still read beside it, because slice 2 is what makes that safe (slice 1 refused such a component on purpose). Two fixture mistakes are worth the comment they now carry, because both produce a fixture that looks right: * `std::vector<float> norm(size_t(width))` is a FUNCTION DECLARATION (most vexing parse); * `std::vector<float> norm{size_t(width)}` is an initializer_list of ONE element = width, so the table is 1 value wide and the reader correctly reports "declares more bytes than its data carries" for the norm. The two-arg form is the only one that does what it looks like. Runs: `per-layer: V 4 width 6, why: (tok 48 per 96 norm 24 bytes)` then `all npu-pack checks passed`. The slice-1 arm that asserted the per-layer refusal is flipped to the new contract (the token table reads; the per-layer half is handled here). * npu-pack/forward: the per-layer gather the decoder will call, pinned by three numbers (engine#1725) The band the per-layer embedding feeds is `out[k] = table[id][k] * norm[k]` over the layer's own slice of a layers*head_dim-wide row; that is the whole arithmetic, so it lives in a header-only function with no container and no device (`npu/forward/include/per_layer_embed.h`) and a test that can be read as the spec: row0 {1,2,3,4} * norm {2,2,0.5,0.25} -> {2,4,1.5,1} row1 {-1,0.5,0,8} * same norm -> {-2,1,0,2} (a different id picks a different row) null norm -> the row passes through, because a MISSING projection is not a ZERO one id out of range, or no table -> output untouched (the loader can hand a short container) Runs: `per-layer gather: row0 -> 2.00 4.00 1.50 1.00` then all checks pass. Also in this commit: the test target gains `npu/forward/include` on its include path (it had only the engine's own), and a comment records the trap I hit twice in this file: `std::vector<T> v(size_t(n))` is a FUNCTION DECLARATION, and `std::vector<T> v{size_t(n)}` is an initializer_list of one element — only the two-arg form does what it looks like. * npu-pack: dequantise the per-layer CONTEXT projection, and split ok()/complete() (engine#1725, WIP) Tracing the component's graph end to end (not its node list) showed the per-layer input is ( rmsnorm((inputs_embeds @ per_layer_model_projection.weight_t_Q4) * H**-0.5, per_layer_projection_norm) + 16*table[id] ) / sqrt(2) so the table and its norm are only TWO of the three tensors: the context projection is the third, and without it the per-layer input is `16*table/sqrt(2)` -- plausible, and wrong at every layer. The component carries it quantised (`per_layer_model_projection.weight_t_Q4` [8960,48,16], `..._t_scales`, packed `..._t_zero_point`), and the loader wants it dequantised as `model.per_layer_model_proj.weight_layer{l}` [P][H] BF16 per layer. * `read_embedding_per_layer(pack_dir, hidden, why)` now dequantises that projection through the file's own `decode_matmulnbits()` (same decoder the rest of the importer uses) and holds it as [NL][P][H] f32; `hidden` is a parameter because it comes from the pack's geometry, not from the component. * `ok()` is what the reader guarantees (table + norm); `complete()` additionally requires the projection. A component with the first but not the second imports the table and is refused by the LOADER on `per_layer_model_proj.weight_layer0`, which is the honest place to refuse -- the importer does not know whether the decoder declares per_layer_inputs. Measured on `amd/gemma-4-e2b-it-mobius-int4_rai_1.9.0_npu`'s component: table 262144x8960 f32, norm 256 (shared by all 35 layers -- the graph reshapes to [.., 35, 256] around the norm), projection K=1536 N=8960 block 32 int4. WIP: the container still writes the two old names; the rename to the loader's names and the projection emission are the next commit, together with the forward's BF16 band path for the table. * npu-pack: write the per-layer PLE tensors under the names the forward loads (engine#1725, WIP) The forward's `build_ple_inputs()`/`load_ple_block()` already implement this family's arithmetic -- I traced the component's graph and it is line for line the same: `context = rmsnorm((emb @ W_l) * H**-0.5, norm)`, `identity = 16 * table[id]`, `per_layer_input = (context + identity)/sqrt(2)` -- but it reads different names than the importer wrote. This commit closes that gap: model.per_layer_token_embd.weight BF16 {V, NL*P} (was model.embed_tokens_per_layer.weight) model.per_layer_proj_norm.weight BF16 {P} (was model.per_layer_projection_norm.weight) model.per_layer_model_proj.weight_layer{l} BF16 {P, H}, l = 0..NL-1 (was absent) The token table is **streamed**, never materialised: `read_per_layer_table_band()` maps the component's external data and converts 32 rows at a time from f32 to BF16 straight into the container blob, because the full table is V x NL*P f32 = 9.4 GiB on gemma-4-e2b and a converting copy of that would be the largest allocation in the import. Two bugs of my own found while writing it, both worth the comments they carry: * the first version of the band reader did `memcpy(dst, src, n * 4)` into a `uint16_t*` of n elements -- 4 bytes per element into a 2-byte-per-element buffer, i.e. a 2x overflow. Removed before it ran; * the projection matrices were being dequantised but never written; `pl.complete()` (table + norm + projection) is what now gates the per-layer emission, and a component with `ok()` but not `complete()` still writes the table, leaving the refusal to the LOADER (`missing model.per_layer_model_proj.weight_layer0`), which is the honest place. All existing checks pass; copyright rc 0. WIP: the forward still lacks a RAW-BF16 band path in `load_ple_block` (it handles plain int8 + scale, or the chunked container formats), so this container is one loader branch away from being readable. * forward: read the per-layer token table as raw BF16 bands (engine#1725) `load_ple_block()` handled two shapes: plain int8 + a per-32-column `.scale`, and the chunked container formats. The multi-component packs state this table as f32 and the importer converts it to BF16 when it writes the container -- a converting copy of the whole V x NL*P table would be 9.4 GiB on gemma-4-e2b -- so the container's table is raw BF16 and neither branch matched it. The band is the same shape the int8 branch handles (32 rows x cols), with a 2-byte element instead of a code, so this is a loop of `bf16_to_f32` over the band, chosen by data_size: a multiple of the BF16 band and NOT the plain-int8 size. That ordering matters -- `plain_bytes` (one byte per element) and a BF16 band differ by a factor of two, and testing int8 first would take a BF16 table whose size happens to equal `plain_bytes` for another geometry. With this, a container written by the importer (#1725's `83c323b`) is readable end to end on the embedding side: table (banded), norm (256), and the per-layer context projections. * npu-pack: retire the table-times-norm primitive -- it is not this model's arithmetic (engine#1725) The previous commit on this branch added `per_layer_gather()` computing `table[id] * norm`, from a reading of the component's node LIST. Tracing the component's graph end to end showed something else: per_layer_inputs = ( rmsnorm((inputs_embeds @ per_layer_model_projection.weight_t_Q4) * H**-0.5, per_layer_projection_norm) + 16*table[id] ) / sqrt(2) -- the norm applies to the projected CONTEXT, the table row is ADDED (times 16 = sqrt(256)), and the sum is scaled by 1/sqrt(2). That is exactly what the forward's `build_ple_inputs()` already computes, so the primitive was not merely unused, it was wrong, and a pure function with a passing test is exactly the wrong thing to leave in a tree for someone to call later. Deleted: `npu/forward/include/per_layer_embed.h`, its test, and the test target's include line it needed. The container and loader work that DOES match the graph stays (earlier commits on this branch): the three PLE tensors written under the names the forward reads, the streamed BF16 table, and `load_ple_block()`'s raw-BF16 band path. Deleting beats keeping a wrong primitive as documentation: the retraction is recorded on engine#1725 with the graph, which is where someone changing this will look. * npu-pack: tell a variable-stride 1.9 fconst from "not a 1.9 fconst" (engine#1503) `fconst19_detect()` requires ONE stride for the whole file. The mobius line does not have one -- measured on gemma-4-e2b: 35 blocks, each opening with 8 anchors (8 heads x 4 B = 32-byte header), sizes 21.87 MiB to 288.06 MiB -- so the detect fails and the pack was reported the same way as a file that is simply not a 1.9 fconst at all. * `fconst19_block_starts()` is the run scan on its own (exactly `layers` runs of `heads*4` anchor bytes), so a caller can tell the two cases apart before the stride test; * when the runs are there but the strides differ, the refusal now NAMES the structure: how many blocks, where the first one starts, and the size range in KiB, plus why the rule table cannot be the answer ("one stride per file" is the table's model) and what the answer is (the per-layer block table; the anchors give the starts, and each block's size follows that layer's own projection shapes). On the real pack: 1bit npu-pack: … this IS a Ryzen AI 1.9 Token .fconst (35 blocks of 32 B headers at 2564864 B) but its blocks are NOT uniform: 22395 KiB to 294973 KiB, i.e. a variable-stride container. The rule table describes one stride per file, so no row can fit it; this family (mobius/MatFormer) needs the per-layer block table instead … Crafted arm: four blocks with different body sizes are refused by name (`variable stride: refused by name (variable-stride container)`), so the distinction is pinned without a 1.3 GiB pack. * forward: decide the per-layer table's layout by its DECLARED dtype, not by arithmetic on the size Review of #1825 caught a real collision class in my BF16-band branch: it separated the two layouts with data_size >= (block+1)*bf16_bytes && data_size % bf16_bytes == 0 && data_size != plain_bytes and every multiple of the band satisfies the first two, so a plain table whose byte count happens to divide would be read as 32-row BF16 bands -- "plausible and wrong", the failure my own comment warns about. The tree already decides this way elsewhere (`model.c` does `strcmp(dtype, "I8")` on the same field) and `q4nx_dequant.h`'s rule is that an ambiguous tile count is refused rather than resolved by which test happens to pass. * BF16 is taken only when the container SAYS BF16, and then its size must be whole 32-row bands (else refused); * plain I8 (or Q4) keeps the existing int8 + .scale path; * any other DECLARED dtype is refused by name, and so is an EMPTY one -- a container written by this importer always states a dtype (`w.add()` writes it for every tensor), so "no dtype" means a foreign or truncated container, not a layout this loader may guess. `model.c` populates `dtype` for every tensor entry that carries the key, which is what makes the branch sound rather than merely tidier; that is stated in the comment next to the test. * npu-pack: a container that carries the PLE tensors now also says how to switch the block on (engine#1725) The per-layer tensors are useless without the two numbers the forward's semantics reader gates the whole block on (`hidden_size_per_layer_input > 0`, q4nx_semantics.c). gemma-4 states them in model_config.json's text_config (256 and 262144); the container's own config.json did not carry them, so an import would write the table, the norm and the per-layer projections and then the forward would skip the block silently -- the worst failure shape, because the container looks complete. * when the component is `complete()` (table + norm + context projection), the container's config.json gains `hidden_size_per_layer_input` = P and `vocab_size_per_layer_input` = V, taken from the COMPONENT (where they are measured) rather than from the pack's config, which may not state them at all; * the three Gemma-4 layer-shape switches the same reader consults -- `num_kv_shared_layers`, `use_double_wide_mlp`, `extra_layer_norms` -- are carried through when model_config.json's text_config states them, and left absent otherwise so the reader's defaults apply. The read happens at the config-writing site, which lives in a different writer from the one that emits the tensors: one metadata parse (names, dims, the quantised projection header -- the 9.4 GiB table is NOT read there) rather than plumbing a value across two functions. * npu: the Whisper front-end four f32 host kernels, known-answer tested (engine#1783) * npu: read an audio pack graph whole -- and the two facts that change the executor (engine#1931) * npu: run an audio pack encoder on the host, verified against onnxruntime (engine#1931) * npu: run the Whisper decoder too -- 1233 nodes, both graphs, verified against onnxruntime (engine#1931) * npu: the log-mel front-end, checked against the reference extractor (engine#1931) * npu: whisper_transcribe -- wav to token ids on the host, matching the reference decode (engine#1931) * npu: whisper_transcribe reports where the host time goes (engine#1931) * npu: split independent MatMul rows across threads, bit-identical, measured at last (engine#1931) * npu: the audio fork-join is tested for bit-identity and survives a throwing unit; honest timing labels; notices (engine#1931) Answers the three open reviews on #1931 at 61426f5: - parallel_units moves to npu/audio_parallel.h; a worker that throws is caught, every worker is joined, the lowest worker's exception is rethrown. - tests/audio_parallel_test.cpp: the executor's MatMul compared with memcmp at 1/2/3/7/8/64 threads and unset, on an in-memory graph (no pack); every unit runs exactly once; a throwing unit is a caught exception. - whisper_transcribe reports wav, mel, load, encoder, decode separately: 'mel+load' did not include the wav read, and 'encoder' included both graph parses. - The audio tests are build dependencies of onebit, so CI builds them without a workflow edit; full Apache notice on the ten audio files. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: pi-demo <pi-demo@1.local> Co-authored-by: bong-water-water-bong <pi@1bit.local> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A PR onto
main-audioops(the branch of #1931, Refs #1783), not onto main. #1931's head61426f5has not moved since 17:56Z; it conflicts with main and carries three unanswered CHANGES. This clears all of it. Merging this PR moves #1931's head; it then needs verdicts at that new head.1. Conflicts with main (4 files)
The branch carried eleven pre-squash commits of #1725, which main has since merged and moved on from; those are what conflicted, not the audio work. Resolution: main's version of
npu/pack_import.{h,cpp},tests/npu_pack_import_test.cppandCMakeLists.txt, with only the audio commits' delta (111ba732..61426f5c) reapplied:parse_onnx_graph()and the audio targets. After it, the diff against main is the audio work and nothing else (14 files,npu/pack_import.cpp+87), andgit merge-tree origin/main HEADreports no conflict.2. "Bit-identical" had no assertion (CHANGES at
61426f5, two reviewers)New
tests/audio_parallel_test.cpp, no pack and no device: it builds a one-MatMulgraph in memory and runs it throughrun_audio_graph()itself.ONEBIT_AUDIO_THREADS= 2, 3, 7, 8, 64 and unset:memcmpequal to serial;0,-4,many, empty: one thread, same bytes;3. No exception path in the pool (CHANGES at
61426f5c)parallel_unitsmoves tonpu/audio_parallel.h. Each worker catches, every worker is joined, and the lowest worker's exception is rethrown on the caller. The test throws from unit 5 on 1, 2, 4 and 8 threads, checks the caller catches it and that the other workers finished first, and checks that with two throwing workers the lower one's is the one seen.Both claims fail on a mutant (header copied aside, tree untouched):
4. Timing labels in
whisper_transcribe(CHANGES carried from56c35aa3)mel+loadstarted after the wav was read and held no load;encoderheld both graph parses. Now each span is named for what it contains, andwallstarts before the file is opened:(Run on a loaded box under
thermal-run; the line is shown for its labels, not as a speed.)5. My own CHANGES at
4bc1d6etools/copyright.py --check: rc 0 (the ten audio files had the short header only; they now carry the full notice).tools/ci/ci_targets_check.py --check: rc 0, "9 target(s) … (baseline 9), and none new". The five audio tests are build dependencies ofonebit, so CI builds them with no workflow edit.Run on this box at
49b27b4Not run: the decoder arm of
audio_exec_test(it skips without a second argument, and I did not pass one), and nothing on the NPU; this is host code.🤖 Generated with Claude Code