Repository navigation
feat(MODEL-TEXT-FLINT): the keep-quant device arm — the expert tier resident, the dispatch wired and mutation-proven - #3514
Merged
Conversation
lu-zero
force-pushed
the
row/FLINT-KEEPQUANT
branch
from
October 28, 2026 11:11
1d57651 to
70cfc30
Compare
The Flint TT device arm expands every GGUF k-quant to bf16 and streams a 70.5 GiB routed-expert tier off the host; this row wires the proven Qwen3.8 keep-quant machinery (admission predicate, int8-dot/grouped TT decodes) into the Flint loader and dispatch so the tier stages on the P150 as quantized blocks, plus the red-first loader test, the device-vs-CPU distance gate, the before/after resident-byte evidence, and the kolibri discipline check. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM [maki]
…s verbatim on device, the expert tier 21.3 GiB resident instead of 70.5 streamed
The loader arm keeps ADMITTED block encodings verbatim on device
(Q4_K/Q5_K/Q6_K/IQ* per gguf_keep_quant.cpp DeviceKeepQuantSupported,
kTENSTORRENT): OwnProjectionKeepQuant + the [E, out, in] tower slicer
with per-slice geometry checks; float tensors and inadmissible encodings
take the bf16 arm or refuse BY NAME (encoding + tensor). The staging
plan generalizes to raw block bytes: the Q4_K_M census reads resident
1.557 GiB + expert tier 21.311 GiB, streamed 0 B — the bf16 arm
streamed 70.5 GiB. The arm is OPT-IN: VT_FLINT_KEEP_QUANT=1 (documented
in docs/ENVIRONMENT.md); unset restores the bf16 arm byte-for-byte
(LoadFlintKeepQuantFromGguf self-gates: unset delegates to
Impl(keep_quant=false), identical to LoadFlintFromGguf).
NAMED GAP (the reviewer's reachability mutation proved it): nothing in
production reaches this loader yet — the registry (flint.cpp:705) always
calls LoadFlintFromGguf, and flint_tt's staging cannot consume block
weights (StageBf16 reinterprets; vt::Matmul requires float). The
registry wiring and the device block-staging/dispatch are the row's
next increment; the tests gate the loader, the census, and the budget
only.
The fixture's expert towers are 3-D ([E, out, in], gguf dims {in, out,
E}) matching the real container; the dead predecessor's flat 2-D fixture
is repaired. RED-FIRST: the refusal tests fired on the pre-arm tree.
4/4 on device.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]
…h rel_rms drift recorded, the chain leg records instead of pins The keepquant dispatch gate's bf16 regression leg failed the morning-calibrated 0.25 fence at 2.683e-1, and the decisive A/B (the known-good tt-matmul-w3 binary against the keepquant binary, one reset, back to back) shows BOTH reading 2.683e-1 bit-identically: the keepquant branch did not change bf16 numerics — the ABSOLUTE distance drifts ~20 percent across device epochs while staying bit-deterministic within one. The fence returns to the drift-tolerant 0.5/64 (it fences hard regressions; a tighter absolute band measures the epoch), and the chain leg records the per-leg chain instead of pinning a stale one — the bf16 and keep-quant arms are bit-identical to each other within an epoch, so the geometry assertions and the dispatch counter pin the arm and config. Both modes GREEN on device: keep-quant 1.557+21.311 GiB resident / 0 streamed; bf16 3.606+70.5 streamed; identical distances. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
…sured the bf16 arm twice The cross-epoch addendum claimed "both arms bit-identical within an epoch"; the reviewer's reachability mutation proved the W0 device case never loads keep-quant weights (ModelRegistry::Load always takes the bf16 loader; flint_tt's staging cannot consume block weights), so that reading was one arm measured twice. The addendum names the registry routing and the flint_tt block staging/dispatch as the row's remaining increment. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
…ward consumes block weights The loader arm was reachable from nothing: the registry's GGUF case always called LoadFlintFromGguf, and flint_tt staged every weight through StageBf16, reinterpreting verbatim Q4_K blocks as bf16, while vt::Matmul hard-requires float weights (ops.cpp:149) — the reviewer's reachability mutation proved nothing consumed blocks. This wires the four pieces: the routing reads VT_FLINT_KEEP_QUANT=1 at the production call site and takes LoadFlintKeepQuantFromGguf; StageWeight stages a block-carrying weight's VERBATIM bytes through ResidentWeight (raw byte length, [N,K] nk, the MatmulBTQuant orientation) and residents the whole 256-expert tier once at build; MatmulWeight routes block weights to vt::MatmulBT (whose IsBlockQuant hand-off reaches kMatmulBTQuant — the Qwen3.8 kernel set) and float weights to vt::Matmul unchanged; the CPU reference arm refuses a block-carrying tree by name, so the W0 harness's keep-quant CPU reference is the bf16 arm over the same container (the CPU dequantizes, the device keeps blocks). The gate is the reachability mutation, and it now bites: the mutated tree's keep-quant leg is RED (CHECK_NOTHROW threw at the first block-carrying tensor, quant_dispatch_calls 0), and the restored tree re-gates bit-identically green — 5138 block dispatches, first-token rel_rms 2.198e-1 / max_abs 3.335e+1 against the CPU reference on flint-Q4_K_M, expert tier resident at 21.311 GiB raw block bytes with zero streamed bytes. The bf16 regression leg (env unset) is byte-identical to the committed arm: 3.606 GiB resident + 70.5 GiB streamed, rel_rms 2.683e-1, chain 2/8, quant_dispatch_calls 0. The distance band stays the committed drift-tolerant 0.5/64: cross-epoch rel_rms drift is ~20 percent and this session measured both arms inside it. Evidence: docs/bench-evidence/flint-keepquant-dispatch-20261027/. No shared keepquant code changed, so the kolibri leg is not owed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
lu-zero
force-pushed
the
row/FLINT-KEEPQUANT
branch
from
October 28, 2026 13:09
70cfc30 to
d386da8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(MODEL-TEXT-FLINT): the keep-quant device arm — the expert tier resident, the dispatch wired and mutation-proven
The Flint keep-quant device-arm row: the GGUF loader keeps ADMITTED block encodings (Q4_K/Q5_K/Q6_K/IQ* per gguf_keep_quant.cpp DeviceKeepQuantSupported, kTENSTORRENT) verbatim on device instead of expanding to the 70.5 GiB bf16 streaming tier, and the device forward consumes them through the Qwen3.8 keepquant kernel set (MatmulBTQuant int8-dot/grouped). The arm is OPT-IN: VT_FLINT_KEEP_QUANT=1 (documented); unset restores the bf16 arm byte-for-byte (LoadFlintKeepQuantFromGguf self-gates via Impl(false) — identical to LoadFlintFromGguf).
CENSUS (Q4_K_M): resident 1.557 GiB + expert tier 21.311 GiB raw block bytes RESIDENT, streamed 0 B — the bf16 arm's 70.5 GiB streamed tier is the kolibri 384-expert wall this row removes. Refusals name encoding + tensor; float tensors keep the bf16 path.
THE DISPATCH GATE, REACHABILITY-PROVEN: the prior increment's gate stayed green under the reachability mutation (the reviewer's MAJOR — nothing production loaded keep-quant weights). This increment wires the registry (flint.cpp reads VT_FLINT_KEEP_QUANT explicitly at the GGUF case), block staging (ResidentWeight, verbatim bytes, whole tier resident at build), and per-dtype forward dispatch (MatmulWeight routes IsBlockQuant to vt::MatmulBT; FlintTTQuantDispatchCalls counts 5138 block dispatches on Q4_K_M). The reviewer's mutations now go RED: routing forced to bf16 → RED (calls 0, distances collapse); throw on the first block tensor → RED. Restored → GREEN, 19/19, bit-identical to the implementer's legs.
DISTANCES: keep-quant rel_rms 2.198e-1 / max_abs 3.335e+1 vs the CPU bf16-dequant reference (function-level near-tie by design — int8-dot activation quant vs CPU dequant); bf16 regression byte-identical to the committed arm (2.683e-1, 18/18). The committed drift-tolerant 0.5/64 fence is kept (cross-epoch drift ~20 percent, recorded in docs/bench-evidence/tt-matmul-noise-20261025/). Keep-quant measuring INSIDE the bf16 arm's distance is plausible: the int8-dot class tracks the CPU dequant reference more closely this epoch.
FRESH REVIEW (this increment): all six claims PASS — registry routing, forward consumption, block staging (byte-count cross-check against the census), distances/band, hygiene (all four checkers), attribution. Reachability mutations run independently by the reviewer on device. test_flint_forward 8/8. One recipe minor (run the W0 binary from the source-tree root — kernel CWD resolution) recorded in the report.
NO-REGRESS: kolibri leg not run — no shared keepquant code changed (flint.cpp/flint_tt.cpp + harness only); the bf16-mode regression leg above IS the shared-path check.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]