diff --git a/docs/determinism.md b/docs/determinism.md index b5d916989..7789b47df 100644 --- a/docs/determinism.md +++ b/docs/determinism.md @@ -244,6 +244,12 @@ comparators and `logprob_agree.py`'s prompts, so it measures what this page docu ### What the `hrx` width gate is, by name +> **It is not a behaviour-neutral instrument (engine#2143).** With this flag set, an 18,782-token request +> *completes* where the same binary, model and request **fails** without it, and NaN-logit incidents went from 2 +> to 0; a plain 18.6k prefill measured **271.75 t/s** unset against **118.86 t/s** set. A 2.3x gap is not +> "logging cost" — it is a different code path. So a rate taken with the flag is not comparable to one taken +> without it, and an A/B that sets it on one arm only is not an A/B. + `GGML_HRX_LOG_DISPATCH=1` prints each dispatch registration the first time it matches (`dispatch_registry/dispatch-registry.cpp`), so the set of registrations a serve logs is the set of kernels its graphs actually ran. Keying that log on the root node's shape as well — a throwaway local @@ -256,6 +262,7 @@ step splits into two disjoint sets: | 1 (solo, and any sequential history) | `hip.qkv_rope`, `kquant.mul_mat.decode_f32`, `hip.dense_kquant_swiglu`, `hip.gemv_add_norm`, `common.mul_mat.f32_f32_decode` | | 2 (concurrent) | `hip.kquant_tokens.mm`, `hip.kquant_tokens.mm_add`, `hip.kquant_tokens.swiglu`, `common.rmsnorm_binary_f32` | + Seventeen more registrations match at both widths, including the attention kernel (`common.flash_attention_decode_split_next_q8`): on `hrx`, unlike on `cpu`, attention is *not* where the width dependence comes from. What changes is the **fusion boundaries**. The residual add and diff --git a/docs/zyphra.md b/docs/zyphra.md index 57fa6f5ff..2995a5cce 100644 --- a/docs/zyphra.md +++ b/docs/zyphra.md @@ -264,6 +264,7 @@ have been measured in three windows and answered differently every time. prefill drop noted below, and it is why a throughput row here is inseparable from *which quantizer built it*. **Measured here on 2026-11-08**: two arms that differ *only* in the adapter tensors' type (F16 ×26 vs Q6_K ×13 + Q8_0 ×13), benched back to back in one ticket and one window — + | shape | F16 pairs | block-quantized pairs | cost | |---|---|---|---| | `pp8` | 114.19 tok/s | 14.03 tok/s | **8.1×** | @@ -279,6 +280,13 @@ have been measured in three windows and answered differently every time. per tensor: asking for `ffn_gate_up_adapter=q6_K` yields `_a` = Q6_K but `_b` = **Q8_0**, because the `[128, 28672]` pair's contraction has `input_size = 128` and HRX's block-format admission needs `>= 256` while the float predicate needs only `>= 4`. So the pair leaves the device for *any* block type, including the substitute — the same "the label is not evidence" rule this page applies to arm names, applied to the tensors inside them. + +> **It is not a behaviour-neutral instrument (engine#2143).** With this flag set, an 18,782-token request +> *completes* where the same binary, model and request **fails** without it, and NaN-logit incidents went from 2 +> to 0; a plain 18.6k prefill measured **271.75 t/s** unset against **118.86 t/s** set. A 2.3x gap is not +> "logging cost" — it is a different code path. So a rate taken with the flag is not comparable to one taken +> without it, and an A/B that sets it on one arm only is not an A/B. + - **A control on 2026-11-08 shows how far a single reading can be off, and it retire the A/B this page would need.** One file whose form cause 2 did **not** touch (`Zamba-7B-v1` `Q6_K`) was re-benched on a current build: `pp512 211.16 (sd 170.84)`, `pp4096 47.14 (sd 12.73)`, `tg128 4.26 (sd 3.04)` against its own published `108.07 / 92.28 / 12.20`. **The same bytes, 2-2.9× apart, with 71 % sd on decode.** So the rows in this