Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/determinism.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,12 @@ comparators and `logprob_agree.py`'s prompts, so it measures what this page docu

### What the `hrx` width gate is, by name

> **It is not a behaviour-neutral instrument (engine#2143).** With this flag set, an 18,782-token request
> *completes* where the same binary, model and request **fails** without it, and NaN-logit incidents went from 2
> to 0; a plain 18.6k prefill measured **271.75 t/s** unset against **118.86 t/s** set. A 2.3x gap is not
> "logging cost" — it is a different code path. So a rate taken with the flag is not comparable to one taken
> without it, and an A/B that sets it on one arm only is not an A/B.

`GGML_HRX_LOG_DISPATCH=1` prints each dispatch registration the first time it matches
(`dispatch_registry/dispatch-registry.cpp`), so the set of registrations a serve logs is the set of
kernels its graphs actually ran. Keying that log on the root node's shape as well — a throwaway local
Expand All @@ -256,6 +262,7 @@ step splits into two disjoint sets:
| 1 (solo, and any sequential history) | `hip.qkv_rope`, `kquant.mul_mat.decode_f32`, `hip.dense_kquant_swiglu`, `hip.gemv_add_norm`, `common.mul_mat.f32_f32_decode` |
| 2 (concurrent) | `hip.kquant_tokens.mm`, `hip.kquant_tokens.mm_add`, `hip.kquant_tokens.swiglu`, `common.rmsnorm_binary_f32` |


Seventeen more registrations match at both widths, including the attention kernel
(`common.flash_attention_decode_split_next_q8`): on `hrx`, unlike on `cpu`, attention is *not* where
the width dependence comes from. What changes is the **fusion boundaries**. The residual add and
Expand Down
8 changes: 8 additions & 0 deletions docs/zyphra.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,7 @@ have been measured in three windows and answered differently every time.
prefill drop noted below, and it is why a throughput row here is inseparable from *which quantizer built it*. **Measured here on 2026-11-08**: two arms
that differ *only* in the adapter tensors' type (F16 ×26 vs Q6_K ×13 + Q8_0 ×13), benched back to back in one ticket and one window —


| shape | F16 pairs | block-quantized pairs | cost |
|---|---|---|---|
| `pp8` | 114.19 tok/s | 14.03 tok/s | **8.1×** |
Expand All @@ -279,6 +280,13 @@ have been measured in three windows and answered differently every time.
per tensor: asking for `ffn_gate_up_adapter=q6_K` yields `_a` = Q6_K but `_b` = **Q8_0**, because the `[128, 28672]` pair's contraction has
`input_size = 128` and HRX's block-format admission needs `>= 256` while the float predicate needs only `>= 4`. So the pair leaves the device for *any*
block type, including the substitute — the same "the label is not evidence" rule this page applies to arm names, applied to the tensors inside them.

> **It is not a behaviour-neutral instrument (engine#2143).** With this flag set, an 18,782-token request
> *completes* where the same binary, model and request **fails** without it, and NaN-logit incidents went from 2
> to 0; a plain 18.6k prefill measured **271.75 t/s** unset against **118.86 t/s** set. A 2.3x gap is not
> "logging cost" — it is a different code path. So a rate taken with the flag is not comparable to one taken
> without it, and an A/B that sets it on one arm only is not an A/B.

- **A control on 2026-11-08 shows how far a single reading can be off, and it retire the A/B this page would need.** One file whose form cause 2
did **not** touch (`Zamba-7B-v1` `Q6_K`) was re-benched on a current build: `pp512 211.16 (sd 170.84)`, `pp4096 47.14 (sd 12.73)`, `tg128 4.26
(sd 3.04)` against its own published `108.07 / 92.28 / 12.20`. **The same bytes, 2-2.9× apart, with 71 % sd on decode.** So the rows in this
Expand Down
Loading