From 9e733f5d93eabf9bd535ed564515e48b88f58491 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong <277547417+bong-water-water-bong@users.noreply.github.com> Date: Wed, 11 Nov 2026 05:53:27 -0400 Subject: [PATCH] docs: GGML_HRX_LOG_DISPATCH changes behaviour, so say it where the pages teach the instrument (engine#2143) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #2143 measured it: with `GGML_HRX_LOG_DISPATCH=1` set, an 18,782-token request *completes* where the same binary, model and request fails without it, and NaN-logit incidents go from 2 to 0. It also carries a published rate that is a 2.3x casualty of this: one plain 18.6k prefill measured 271.75 t/s with the flag unset and 118.86 t/s with it set. Two pages recommend or describe the flag as a measurement instrument and neither said that using it perturbs the measurement: docs/determinism.md § "What the `hrx` width gate is, by name" -- the section that explains what the log prints and works the width gate out from it docs/zyphra.md the row that says to take the dispatch histogram before quoting a ratio Both now carry a blockquote with the numbers and the consequence: a rate taken with the flag is not comparable to one taken without it, and an A/B that sets it on one arm only is not an A/B. docs/hrx.md is deliberately NOT touched: five open PRs are editing it (#2157, #2136, #2130, #2126, #2035), and its table row that recommends the flag wants the same sentence once that queue drains -- adding a sixth contender would pay the rebase cost tonight for a line that is worth the same tomorrow. Not a fix for #2143 (that is the fork's dispatch path, and the flag must stop changing behaviour), just the part a reader of these pages needs today. --- docs/determinism.md | 7 +++++++ docs/zyphra.md | 8 ++++++++ 2 files changed, 15 insertions(+) diff --git a/docs/determinism.md b/docs/determinism.md index b5d916989..7789b47df 100644 --- a/docs/determinism.md +++ b/docs/determinism.md @@ -244,6 +244,12 @@ comparators and `logprob_agree.py`'s prompts, so it measures what this page docu ### What the `hrx` width gate is, by name +> **It is not a behaviour-neutral instrument (engine#2143).** With this flag set, an 18,782-token request +> *completes* where the same binary, model and request **fails** without it, and NaN-logit incidents went from 2 +> to 0; a plain 18.6k prefill measured **271.75 t/s** unset against **118.86 t/s** set. A 2.3x gap is not +> "logging cost" — it is a different code path. So a rate taken with the flag is not comparable to one taken +> without it, and an A/B that sets it on one arm only is not an A/B. + `GGML_HRX_LOG_DISPATCH=1` prints each dispatch registration the first time it matches (`dispatch_registry/dispatch-registry.cpp`), so the set of registrations a serve logs is the set of kernels its graphs actually ran. Keying that log on the root node's shape as well — a throwaway local @@ -256,6 +262,7 @@ step splits into two disjoint sets: | 1 (solo, and any sequential history) | `hip.qkv_rope`, `kquant.mul_mat.decode_f32`, `hip.dense_kquant_swiglu`, `hip.gemv_add_norm`, `common.mul_mat.f32_f32_decode` | | 2 (concurrent) | `hip.kquant_tokens.mm`, `hip.kquant_tokens.mm_add`, `hip.kquant_tokens.swiglu`, `common.rmsnorm_binary_f32` | + Seventeen more registrations match at both widths, including the attention kernel (`common.flash_attention_decode_split_next_q8`): on `hrx`, unlike on `cpu`, attention is *not* where the width dependence comes from. What changes is the **fusion boundaries**. The residual add and diff --git a/docs/zyphra.md b/docs/zyphra.md index 57fa6f5ff..2995a5cce 100644 --- a/docs/zyphra.md +++ b/docs/zyphra.md @@ -264,6 +264,7 @@ have been measured in three windows and answered differently every time. prefill drop noted below, and it is why a throughput row here is inseparable from *which quantizer built it*. **Measured here on 2026-11-08**: two arms that differ *only* in the adapter tensors' type (F16 ×26 vs Q6_K ×13 + Q8_0 ×13), benched back to back in one ticket and one window — + | shape | F16 pairs | block-quantized pairs | cost | |---|---|---|---| | `pp8` | 114.19 tok/s | 14.03 tok/s | **8.1×** | @@ -279,6 +280,13 @@ have been measured in three windows and answered differently every time. per tensor: asking for `ffn_gate_up_adapter=q6_K` yields `_a` = Q6_K but `_b` = **Q8_0**, because the `[128, 28672]` pair's contraction has `input_size = 128` and HRX's block-format admission needs `>= 256` while the float predicate needs only `>= 4`. So the pair leaves the device for *any* block type, including the substitute — the same "the label is not evidence" rule this page applies to arm names, applied to the tensors inside them. + +> **It is not a behaviour-neutral instrument (engine#2143).** With this flag set, an 18,782-token request +> *completes* where the same binary, model and request **fails** without it, and NaN-logit incidents went from 2 +> to 0; a plain 18.6k prefill measured **271.75 t/s** unset against **118.86 t/s** set. A 2.3x gap is not +> "logging cost" — it is a different code path. So a rate taken with the flag is not comparable to one taken +> without it, and an A/B that sets it on one arm only is not an A/B. + - **A control on 2026-11-08 shows how far a single reading can be off, and it retire the A/B this page would need.** One file whose form cause 2 did **not** touch (`Zamba-7B-v1` `Q6_K`) was re-benched on a current build: `pp512 211.16 (sd 170.84)`, `pp4096 47.14 (sd 12.73)`, `tg128 4.26 (sd 3.04)` against its own published `108.07 / 92.28 / 12.20`. **The same bytes, 2-2.9× apart, with 71 % sd on decode.** So the rows in this