From 3a064df5e538f8f3c1760595527bd87705ce60fb Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 21:58:03 -0400 Subject: [PATCH 1/2] npu: the per-op forward reads gemma3 text decoders (Refs #1786) FastFlowLM's Medgemma-4B-NPU2 was refused at load (rope scaling "linear"), and four other gemma3 rules were missing: the layer mix from sliding_window_pattern when there is no layer_types list, the sliding layers' rope_local_base_freq (a sliding layer with no base of its own had theta 0), query_pre_attn_scalar, and the sandwich norms without Gemma4's post-PLE norm and layer scalar. Linear scaling divides the frequency on the full layers only when a local base is stated, and on every layer otherwise. The gemma3 containers carry the norm's 1 already (they equal llama.cpp's GGUF norms to bf16), so gemma3 takes the plain form; with (1 + w) the known answer was "*,,,,,,,", now it is "Paris". Co-Authored-By: Claude Opus 5.5 --- CMakeLists.txt | 5 ++ docs/npu.md | 35 ++++++++- npu/forward/include/q4nx_forward.h | 1 + npu/forward/include/q4nx_semantics.h | 10 ++- npu/forward/src/q4nx_forward.cpp | 49 ++++++++++-- npu/forward/src/q4nx_semantics.c | 46 +++++++++-- tests/q4nx_gemma3_semantics_test.c | 113 +++++++++++++++++++++++++++ 7 files changed, 243 insertions(+), 16 deletions(-) create mode 100644 tests/q4nx_gemma3_semantics_test.c diff --git a/CMakeLists.txt b/CMakeLists.txt index a6a5c3dcb..8d5185da8 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1732,6 +1732,11 @@ add_executable(q4nx_qk_norm_test tests/q4nx_qk_norm_test.c npu/forward/src/q4nx_ target_include_directories(q4nx_qk_norm_test PRIVATE npu/forward/include) target_link_libraries(q4nx_qk_norm_test PRIVATE m) add_test(NAME q4nx_qk_norm COMMAND q4nx_qk_norm_test) +add_executable(q4nx_gemma3_semantics_test tests/q4nx_gemma3_semantics_test.c npu/forward/src/q4nx_semantics.c) +target_include_directories(q4nx_gemma3_semantics_test PRIVATE npu/forward/include) +target_link_libraries(q4nx_gemma3_semantics_test PRIVATE m) +add_test(NAME q4nx_gemma3_semantics COMMAND q4nx_gemma3_semantics_test) +add_dependencies(npu_sampling_test q4nx_gemma3_semantics_test) add_dependencies(npu_sampling_test q4nx_qk_norm_test) add_executable(q4nx_layer_kinds_test tests/q4nx_layer_kinds_test.c npu/forward/src/q4nx_semantics.c) target_include_directories(q4nx_layer_kinds_test PRIVATE npu/forward/include) diff --git a/docs/npu.md b/docs/npu.md index 057da210a..43efa0808 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -1949,6 +1949,37 @@ the missing q/k norm above was found: from one teacher-forced pass. Compared with a reference's per-row choices, a low agreement that is flat across positions is an error in the layers; a step at some row is an error at a position or a batch edge. +**Gemma3 text decoders: the forward reads them (engine#1786).** FastFlowLM's `Medgemma-4B-NPU2` (`gemma3_text`, 34 +layers) now answers correctly through the per-op forward on the host (`NPU_INFER_HOST_GEMM=1`, no device). Before this +change it was refused at load: `config.json declares rope scaling "linear", which this forward does not implement`. +Five things were missing. All are read from the config or decided by the family: + +- `sliding_window_pattern`: the layer mix, when there is no `layer_types` list. Every 6th layer is full attention + (HF's `(i + 1) % 6`), so Medgemma has 5 full and 29 sliding layers. Before, all 34 ran as full attention. +- `rope_local_base_freq`: the sliding layers' own RoPE base (10000, against 1e6 for the full layers). A sliding layer + with no base of its own now takes the model's base. Before, it took 0, which was the case for Gemma3-270M's + stated `layer_types`. +- `rope_scaling` `{"rope_type": "linear", "factor": 8}`: the frequency is divided by the factor, on the full layers + only, because Gemma3's sliding layers rotate unscaled. A model without a local base scales every layer, as HF does. + A `linear` block with no usable factor is still refused. +- `query_pre_attn_scalar`: the attention scale is `1/sqrt(query_pre_attn_scalar)`. It equals `head_dim` on Medgemma + but not on every Gemma3. +- The sandwich norms: `post_attention_layernorm` on the attention output, plus `pre_feedforward_layernorm` and + `post_feedforward_layernorm` (`extra_layer_norms = 2`: Gemma4's set without its post-PLE norm and layer scalar). + +The norm form is the one non-obvious rule. HF's Gemma3 RMSNorm is `x/rms * (1 + w)`, but these containers store +`w` with the 1 already added. Every norm kind of the Medgemma pack (input, post-attention, pre- and post-feedforward, +q, final) equals llama.cpp's GGUF of the same model, whose converter adds the 1, to bf16 rounding. With the 1 added +again, the known-answer prompt answered `*,,,,,,,`; with the plain form it answers `Paris`. So gemma3 takes the plain +form, and gemma2 keeps `(1 + w)` (not measured here). `tests/q4nx_gemma3_semantics_test.c` pins each rule with +Medgemma's own keys. + +Not yet a device row: the pack's four shapes are `QKV K2560 N4096`, `O K2048 N2560`, `GU K2560 N20480` and +`D K10240 N2560`, and `QKV K2560 N4096` is not built in any kernel family on the box (see +[arch-gaps.md](arch-gaps.md)). Gemma3 layers also take the per-token path for a prefill (like Gemma4), so a +prefill costs one `step()` per prompt token. Gemma3-270M-NPU2 still fails at load: its tiles are on an unpadded +K = 640 grid that the decoder does not read yet. + **Per-head q/k normalisation is read from the container (engine#1788).** Whether the forward normalises q and k per head used to be decided by `config.json`'s `model_type` alone: on for `qwen3*` and `gemma*`. A pack of the same architecture under another name got it wrong in silence. FastFlowLM's MiniCPM-V 4.7 @@ -2040,7 +2071,7 @@ engine#587 measured the route on it): | `dflash` | `FastFlowLM/Nemotron-3.5-Bolt-E4B-DFlash-NPU2` | hrx | not served: a draft head, not a decoder; "The draft loop, and whether to serve it" below covers what the HRX route does with it | | `gemma4` | `FastFlowLM/Nemotron-3.5-Bolt-E4B-NPU2`, `FastFlowLM/Gemma4-12B-IT-NPU2` | hrx | not served: no gemma4 model type or kernels in the route | | `hunyuan-dense` | `FastFlowLM/Hy-MT2-1.8B-NPU2` | hrx, npu | **served and measured** on the per-op route ("FastFlowLM's own containers" below) | -| `gemma3` | `FastFlowLM/medgemma-1.5-4b-it-NPU2` | hrx | not served | +| `gemma3` | `FastFlowLM/medgemma-1.5-4b-it-NPU2` | hrx | not served on the device. The forward reads the family and answers on the host ("Gemma3 text decoders" above); one kernel shape, `QKV K2560 N4096`, is missing | | `lfm2` | `FastFlowLM/LFM2-2.6B-NPU2`, `FastFlowLM/LFM2-1.2B-NPU2`, `FastFlowLM/LFM2.5-1.2B-NPU2` | hrx, npu | `LFM2.5-1.2B-NPU2` **served and measured** on the per-op route ("FastFlowLM's own containers" below); `LFM2-1.2B-NPU2` serves too (one paired ticket, same section); `LFM2-2.6B-NPU2` was not run | | `gpt-oss` | `FastFlowLM/GPT-OSS-20B-NPU2`, `FastFlowLM/GPT-OSS-Safeguard-20b-NPU2` | hrx | not served | | `hy_v3` | `FastFlowLM/Hy-Memory-30B-A3B-NPU2` | hrx | not served; the first table above records this one | @@ -2068,7 +2099,7 @@ pack declares (a config with no `architectures` declares its `model_type`): | `DFlashDraftModel` | Nemotron-3.5-Bolt-E4B-DFlash | not served: a draft head, not a decoder | engine#1780 | | `Gemma4ForConditionalGeneration`, `Gemma4UnifiedForConditionalGeneration` | Nemotron-3.5-Bolt-E4B, Gemma4-E4B-IT, Gemma4-E2B-IT, Gemma4-12B-IT | not served: `gemma4`, hrx only | engine#1786 | | `HunYuanDenseV1ForCausalLM` | Hy-MT2-1.8B | not served: `hunyuan-dense`, hrx only | engine#1786 | -| `Gemma3ForCausalLM`, `gemma3_text` | Gemma3-270M, Gemma3-1B, Gemma3-4B, Gemma3-4B-Text, medgemma-4b-it, medgemma-1.5-4b-it, Translategemma-4B-Instruct | not served: `gemma3`, hrx only | engine#1786 | +| `Gemma3ForCausalLM`, `gemma3_text` | Gemma3-270M, Gemma3-1B, Gemma3-4B, Gemma3-4B-Text, medgemma-4b-it, medgemma-1.5-4b-it, Translategemma-4B-Instruct | not served: `gemma3`, hrx only (the per-op forward reads the family on the host; the device needs `QKV K2560 N4096`) | engine#1786 | | `Lfm2ForCausalLM` | LFM2-1.2B, LFM2-2.6B, LFM2-1.2B-Tool, LFM2-2.6B-Transcript, LFM2.5-1.2B, LFM2.5-1.2B-Thinking | not in the registry's NPU set: `lfm2`, hrx only (LFM2.5-1.2B on the per-op route is engine#2170, which adds no `NPU_ARCHS` entry) | engine#1786 | | `GptOssForCausalLM` | GPT-OSS-20B, GPT-OSS-Safeguard-20b | not served: `gpt-oss`, hrx only | engine#1786 | | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B | not served for this class: the route serves `qwen3_5_moe` through `Qwen3_5MoeForCausalLM` (GGUF on the NPU); the VLM wrapper class is hrx only, and FastFlowLM's own pack is not measured on the route | engine#1786 | diff --git a/npu/forward/include/q4nx_forward.h b/npu/forward/include/q4nx_forward.h index 3448703d2..227344fb7 100644 --- a/npu/forward/include/q4nx_forward.h +++ b/npu/forward/include/q4nx_forward.h @@ -517,6 +517,7 @@ class Q4nxNpuForward { std::vector rope_scale_; // LongRoPE short_factor (per-half-dim), empty when none Q4nxRopeLlama3 rope_llama3_; // rope_type "llama3" (Llama-3.1/3.2/3.3); on == 0 for everything else Q4nxRopeDynamic rope_dynamic_; // rope_type "dynamic" (NTK base rule, engine#1740 W2); on == 0 elsewhere + double rope_linear_factor_ = 0.0; // rope_type "linear" factor (Gemma3's full layers); 0 elsewhere // MiniCPM4 scales (config "embedding_scale" / "residual_scale"); 1.0 for // every other architecture. MiniCPM4 scales the embedding by 12 and each // residual branch by scale_depth/sqrt(NL) (0.2475 for 32 layers); skipping diff --git a/npu/forward/include/q4nx_semantics.h b/npu/forward/include/q4nx_semantics.h index 1a5af0c9b..e7bb68e1f 100644 --- a/npu/forward/include/q4nx_semantics.h +++ b/npu/forward/include/q4nx_semantics.h @@ -152,8 +152,16 @@ typedef struct { * Source: num_kv_shared_layers. */ int num_kv_shared_layers; /* Gemma4 adds pre/post feedforward norms and a post-PLE norm that the plain - * layer does not have. Source: presence of those tensors / model_type. */ + * layer does not have (1). Gemma3 has the same sandwich around attention and + * MLP but no post-PLE norm and no layer scalar (2). Source: model_type. */ int extra_layer_norms; + /* Gemma3 states the sliding layers' RoPE base on its own, as + * `rope_local_base_freq`, and its `rope_scaling` applies to the full layers + * only. 1 when that key was read into rope_theta_sliding. */ + int rope_local_base; + /* Gemma2/3: attention scores are scaled by 1/sqrt(query_pre_attn_scalar) + * instead of 1/sqrt(head_dim). 0 when absent. Source: query_pre_attn_scalar. */ + float query_pre_attn_scalar; /* Gemma4 use_double_wide_mlp: KV-shared layers run a 2*IM-wide MLP. * Source: use_double_wide_mlp. */ int double_wide_mlp; diff --git a/npu/forward/src/q4nx_forward.cpp b/npu/forward/src/q4nx_forward.cpp index fd8529883..d0b7d2d42 100644 --- a/npu/forward/src/q4nx_forward.cpp +++ b/npu/forward/src/q4nx_forward.cpp @@ -234,6 +234,26 @@ Q4nxRopeDynamic read_rope_dynamic(const std::string& cfg_path) { return d; } +// rope_type "linear": the factor of the config's rope_scaling block when that block declares "linear" with a factor +// above zero, else 0 (no scaling; a "linear" block without a usable factor is still refused by the caller). +static double read_rope_linear(const std::string& cfg_path) { + std::ifstream f(cfg_path); + if (!f) return 0.0; + std::string t((std::istreambuf_iterator(f)), std::istreambuf_iterator()); + char type[64]; + q4nx_rope_declared_type(t.c_str(), type, (int)sizeof type); + if (std::strcmp(type, "linear")) return 0.0; + const size_t rs = t.find("\"rope_scaling\""); + const size_t at = rs == std::string::npos ? rs : t.find("\"factor\"", rs); + // the key must sit inside the block: before the first closing brace after it (the block holds no objects) + if (at == std::string::npos || at > t.find('}', rs)) return 0.0; + const size_t colon = t.find(':', at); + if (colon == std::string::npos) return 0.0; + char* end = nullptr; + const double v = std::strtod(t.c_str() + colon + 1, &end); + return (end != t.c_str() + colon + 1 && v > 0.0) ? v : 0.0; +} + static Q4nxRopeLlama3 read_rope_llama3(const std::string& cfg_path) { Q4nxRopeLlama3 s{}; std::ifstream f(cfg_path); @@ -803,6 +823,7 @@ bool Q4nxNpuForward::init_impl(xrt::device* dev, const char* model_path, rope_scale_ = read_rope_short_factor(cfg_path); // LongRoPE (MiniCPM4) rope_llama3_ = read_rope_llama3(cfg_path); // rope_type "llama3" (engine#1048) rope_dynamic_ = read_rope_dynamic(cfg_path); // rope_type "dynamic" (NTK by base, engine#1740 W2) + rope_linear_factor_ = read_rope_linear(cfg_path); // rope_type "linear" (Gemma3's full layers, engine#1786) { // A declared scaling this forward does not apply is refused BY NAME (engine#1087): serving it // would run plain rope, stay fluent and be wrong past the original window, with nothing in the @@ -811,15 +832,17 @@ bool Q4nxNpuForward::init_impl(xrt::device* dev, const char* model_path, std::string t((std::istreambuf_iterator(f)), std::istreambuf_iterator()); char declared[64]; q4nx_rope_declared_type(t.c_str(), declared, (int)sizeof declared); - if (!q4nx_rope_type_supported(declared, &rope_llama3_, rope_scale_.empty() ? 0 : 1, rope_dynamic_.on)) { + const bool linear_ok = !std::strcmp(declared, "linear") && rope_linear_factor_ > 0.0; + if (!linear_ok && + !q4nx_rope_type_supported(declared, &rope_llama3_, rope_scale_.empty() ? 0 : 1, rope_dynamic_.on)) { if (!std::strcmp(declared, "llama3")) err_ = "config.json declares rope_type \"llama3\" but its numbers are not usable (factor > 0, " "original_max_position_embeddings > 0, high_freq_factor > low_freq_factor), so the " "scaling cannot be applied; serving it would run plain RoPE"; else err_ = std::string("config.json declares rope scaling \"") + declared + - "\", which this forward does not implement (only llama3 and longrope short_factor " - "are), so it would be served plain RoPE and reply wrongly past the model's " + "\", which this forward does not implement (only llama3, linear with a factor and " + "longrope short_factor are), so it would be served plain RoPE and reply wrongly past the model's " "original window (engine#1048, #1087)"; return false; } @@ -1866,8 +1889,12 @@ bool Q4nxNpuForward::load_norms() { pre_ff_norm_[l].data(), H)) return false; if (!rd("model.layers.%d.post_feedforward_layernorm.weight", post_ff_norm_[l].data(), H)) return false; - if (!rd("model.layers.%d.post_layernorm.weight", post_ln_[l].data(), H)) return false; - if (!rd("model.layers.%d.layer_output_scale.weight", &layer_scale_[l], 1)) return false; + // Gemma3 (extra_layer_norms 2) has the two feedforward norms only: no post-PLE norm and no + // layer scalar, so post_ln_ stays unused (ple_dim_ is 0) and layer_scale_ stays 1. + if (sem_.extra_layer_norms == 1) { + if (!rd("model.layers.%d.post_layernorm.weight", post_ln_[l].data(), H)) return false; + if (!rd("model.layers.%d.layer_output_scale.weight", &layer_scale_[l], 1)) return false; + } } if (ple_dim_ > 0) { inp_gate_[l].assign((size_t)ple_dim_ * (size_t)H, 0.0f); @@ -4232,7 +4259,9 @@ bool Q4nxNpuForward::step_batch_impl(const std::vector& tokens, int start_p } { LhScope _t_attn(LH_ATTN); - const float scale = sem_.attn_scaling_one ? 1.0f : 1.0f / std::sqrt((float)l_hd); + const float scale = sem_.attn_scaling_one ? 1.0f + : 1.0f / std::sqrt(sem_.query_pre_attn_scalar > 0 ? sem_.query_pre_attn_scalar + : (float)l_hd); const int nt = ((long)M * (start_pos + M) * l_hd >= 8192) ? omp_heads_threads() : 1; q4nx_attn::rows_context(qall.data(), kcache_[l].data(), vcache_[l].data(), M, c_.NH, l_nkv, l_hd, Tm.data(), t0m.data(), scale, CTX.data(), nt); @@ -5081,6 +5110,10 @@ bool Q4nxNpuForward::attention(const std::vector& qkv, int pos, const int // llama3 (Llama-3.1/3.2/3.3): the wavelength-band rule, on the FREQUENCY, so it costs // nothing per token. No-op unless the config declares rope_type "llama3". inv = q4nx_rope_llama3_inv_freq(inv, &rope_llama3_); + // rope_type "linear": positions divided by the factor, i.e. the frequency is. Gemma3 applies its + // block to the full layers only; its sliding layers have their own base (rope_local_base_freq) and + // no scaling. A model without that split scales every layer, as HF does. + if (rope_linear_factor_ > 0.0 && (window == 0 || !sem_.rope_local_base)) inv /= rope_linear_factor_; } // Interleaved MRoPE: each rotary frequency rotates by its own axis's position (t, h or w). With the // three planes equal this is exactly `pos * inv`, which is why text-only prompts are bit-identical @@ -5121,7 +5154,9 @@ bool Q4nxNpuForward::attention(const std::vector& qkv, int pos, const int const int T = kv_len > 0 ? kv_len : pos + 1; // Gemma4 sets attention scaling to 1.0 (q/k carry an RMSNorm); everyone else // uses the usual 1/sqrt(head_dim). - const float scale = sem_.attn_scaling_one ? 1.0f : 1.0f / std::sqrt((float)HD); + // Gemma2/3 scale by query_pre_attn_scalar, which equals head_dim on most of the family but not all (Gemma3-27B). + const float scale = sem_.attn_scaling_one ? 1.0f + : 1.0f / std::sqrt(sem_.query_pre_attn_scalar > 0 ? sem_.query_pre_attn_scalar : (float)HD); // Sliding-window attention: ignore positions older than the window. const int t0 = (window > 0 && T > window) ? T - window : 0; diff --git a/npu/forward/src/q4nx_semantics.c b/npu/forward/src/q4nx_semantics.c index 56a20bbbf..671069218 100644 --- a/npu/forward/src/q4nx_semantics.c +++ b/npu/forward/src/q4nx_semantics.c @@ -25,7 +25,7 @@ #include static const char* kFields[Q4NX_SEM_COUNT] = { - /* RMSNORM_OFFSET */ "model_type/architectures (gemma* uses the 1+w form)", + /* RMSNORM_OFFSET */ "model_type/architectures (gemma2 uses the 1+w form; gemma3 containers carry the 1)", /* QK_NORM */ "use_qk_norm when stated, else model_type/architectures", /* ROPE_THETA */ "rope_theta", /* ROTARY_DIM */ "partial_rotary_factor * head_dim", @@ -209,12 +209,20 @@ int q4nx_semantics_from_config(const char* json_path, const char* model_dir, * initialised to ones, whereas gemma2/gemma3 do `x/rms * (1 + w)`. */ const int is_gemma4 = strstr(model_type, "gemma4") != NULL || strstr(arch, "Gemma4") != NULL; + /* Gemma3 (gemma3, gemma3_text, FastFlowLM's gemma3_text_only): the sandwich norms, a local RoPE base for the sliding + * layers and the 5:1 sliding pattern. gemma3n is another architecture (AltUp, laurel) and is not this. */ + const int is_gemma3 = (starts_with(model_type, "gemma3") && !starts_with(model_type, "gemma3n")) || + (starts_with(arch, "Gemma3") && !starts_with(arch, "Gemma3n")); const int is_qwen3 = starts_with(model_type, "qwen3") || starts_with(arch, "Qwen3"); const int is_phi = starts_with(model_type, "phi") || starts_with(arch, "Phi"); const int is_qwen35 = starts_with(model_type, "qwen3_5") || starts_with(arch, "Qwen3_5"); - /* RMSNorm convention: gemma2/gemma3 scale by (1 + w); gemma4 by w. */ - s->rmsnorm_offset = (is_gemma && !is_gemma4) ? 1 : 0; + /* RMSNorm convention: gemma2 scales by (1 + w); gemma4 by w. Gemma3 is (1 + w) in HF, but the containers this + * forward reads (FastFlowLM's gemma3 NPU2 packs) store the norm weights with the 1 already added, as llama.cpp's + * GGUF does: on Medgemma-4B-NPU2 every norm kind (input, post-attention, pre/post feedforward, q, final) equals + * the GGUF's to bf16 rounding, and adding 1 again answered the known-answer prompt "*,,,,,," where the plain + * form answers "Paris" (engine#1786). */ + s->rmsnorm_offset = (is_gemma && !is_gemma4 && !is_gemma3) ? 1 : 0; /* qwen3_5 ships both conventions under the same model_type, so the forward resolves it from the * layer-0 input_layernorm weight it loads (a config-only answer would be a guess). */ s->rmsnorm_offset_probe = (is_qwen35 && !is_gemma) ? 1 : 0; @@ -366,6 +374,13 @@ int q4nx_semantics_from_config(const char* json_path, const char* model_dir, } else if (get_num(json, "rope_theta", &d) && d > 0) { s->rope_theta = (float)d; } + /* Gemma3 gives the sliding layers their own base (10000 against the full layers' 1e6). Without it a sliding + * layer was rotated with theta 0, i.e. no usable rotation at all (engine#1786). */ + if (!s->per_layer_rope && get_num(json, "rope_local_base_freq", &d) && d > 0) { + s->rope_theta_sliding = (float)d; + s->rope_local_base = 1; + } + if (get_num(json, "query_pre_attn_scalar", &d) && d > 0) s->query_pre_attn_scalar = (float)d; /* partial rotary: top-level key, or nested in rope_parameters (scalar form), * or nested per layer type (Gemma4's full_attention.partial_rotary_factor). */ @@ -476,6 +491,24 @@ int q4nx_semantics_from_config(const char* json_path, const char* model_dir, else s->sliding_window = 0; + /* Gemma3 configs that predate layer_types state the pattern instead: every `sliding_window_pattern`-th layer is + * full attention, the rest slide (HF: is_sliding = bool((layer_idx + 1) % sliding_window_pattern)). */ + if (s->n_layer_types == 0 && s->sliding_window > 0 && get_num(json, "sliding_window_pattern", &d) && d >= 1) { + double nl = 0; + const int pat = (int)(d + 0.5); + if (get_num(json, "num_hidden_layers", &nl) && nl > 0 && nl <= Q4NX_MAX_LAYERS) { + int hybrid = 0; + for (int i = 0; i < (int)nl; i++) { + s->layer_types[i] = ((i + 1) % pat) ? Q4NX_LAYER_SLIDING : Q4NX_LAYER_FULL; + if (s->layer_types[i] != Q4NX_LAYER_FULL) hybrid++; + } + s->n_layer_types = (int)nl; + s->hybrid_layers = hybrid; + } + } + /* A sliding layer with no base of its own uses the model's (theta 0 is not a base). */ + if (s->rope_theta_sliding <= 0) s->rope_theta_sliding = s->rope_theta; + /* epsilon */ if (get_num(json, "rms_norm_eps", &d) && d > 0) s->eps = (float)d; /* LFM2 spells it norm_eps (and has no rms_norm_eps at all) */ @@ -489,7 +522,7 @@ int q4nx_semantics_from_config(const char* json_path, const char* model_dir, if (get_num(json, "num_kv_shared_layers", &d) && d > 0) s->num_kv_shared_layers = (int)(d + 0.5); /* Gemma4 adds pre/post feedforward norms and a post-PLE norm. */ - s->extra_layer_norms = is_gemma4 ? 1 : 0; + s->extra_layer_norms = is_gemma4 ? 1 : (is_gemma3 ? 2 : 0); /* `use_double_wide_mlp` is a JSON boolean, so read it as "present in text". */ s->double_wide_mlp = (strstr(json, "use_double_wide_mlp\":true") != NULL || strstr(json, "use_double_wide_mlp\": true") != NULL) ? 1 : 0; @@ -602,9 +635,10 @@ void q4nx_semantics_dump(const Q4nxSemantics* s) { fprintf(stderr, " [semantics] layer mix: %s <- %s\n", mix, q4nx_semantics_field(Q4NX_SEM_LAYER_TYPES)); fprintf(stderr, " [semantics] per-type rope: full(theta=%.6g rot=%d) " - "sliding(theta=%.6g rot=%d) rope_type_named=%d\n", + "sliding(theta=%.6g rot=%d%s) rope_type_named=%d query_pre_attn_scalar=%.6g\n", s->rope_theta, s->rotary_dim, s->rope_theta_sliding, - s->rotary_dim_sliding, s->rope_type_named); + s->rotary_dim_sliding, s->rope_local_base ? ", rope_local_base_freq" : "", + s->rope_type_named, s->query_pre_attn_scalar); fprintf(stderr, " [semantics] ple=%d (dim=%d vocab=%d) kv_shared_layers=%d " "extra_layer_norms=%d double_wide_mlp=%d\n", q4nx_semantics_uses_ple(s), s->hidden_size_per_layer_input, diff --git a/tests/q4nx_gemma3_semantics_test.c b/tests/q4nx_gemma3_semantics_test.c new file mode 100644 index 000000000..7eafddcd0 --- /dev/null +++ b/tests/q4nx_gemma3_semantics_test.c @@ -0,0 +1,113 @@ +/* Copyright 2026 bong-water-water-bong + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* q4nx_gemma3_semantics_test -- what the forward reads from a gemma3 config (engine#1786). + * + * Medgemma-4B-NPU2's config.json states its layer mix as `sliding_window_pattern: 6` (no layer_types), its sliding + * layers' RoPE base as `rope_local_base_freq` and its attention scale as `query_pre_attn_scalar`. Before this, none + * of the three was read: the 29 sliding layers ran as full attention with no window, and a sliding layer of a config + * that does list layer_types (Gemma3-270M-NPU2) was rotated with theta 0. The gemma3 containers also carry the norm's + * `1 +` already, so the plain form is the right one for them, where gemma2 keeps (1 + w). + * + * The strings are cut down from the configs on this box to the fields the rules read. No model, no device. */ +#include "q4nx_semantics.h" + +#include +#include +#include +#include +#include + +static int failures = 0; + +static void check(int ok, const char* what) { + printf("%s %s\n", ok ? "ok " : "FAIL", what); + if (!ok) failures++; +} + +static int sem_of(const char* dir, const char* json, Q4nxSemantics* s) { + char path[512]; + snprintf(path, sizeof path, "%s/config.json", dir); + int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0600); + FILE* f = fd >= 0 ? fdopen(fd, "w") : NULL; + if (!f) { printf("FAIL cannot write %s\n", path); failures++; return -1; } + fputs(json, f); + fclose(f); + if (q4nx_semantics_from_config(path, dir, s) != 0) { printf("FAIL cannot read %s\n", path); failures++; return -1; } + return 0; +} + +static const char* kMedgemma = + "{\"head_dim\": 256, \"hidden_size\": 2560, \"model_type\": \"gemma3_text\", \"num_attention_heads\": 8," + " \"num_hidden_layers\": 34, \"num_key_value_heads\": 4, \"query_pre_attn_scalar\": 256," + " \"rope_local_base_freq\": 10000.0, \"rope_scaling\": {\"factor\": 8.0, \"rope_type\": \"linear\"}," + " \"rope_theta\": 1000000.0, \"sliding_window\": 1024, \"sliding_window_pattern\": 6}"; + +int main(void) { + char dir[] = "/tmp/q4nx-gemma3-sem-XXXXXX"; + if (!mkdtemp(dir)) { printf("FAIL mkdtemp\n"); return 1; } + Q4nxSemantics s; + + if (sem_of(dir, kMedgemma, &s) == 0) { + int full = 0, sliding = 0, pattern_ok = 1; + for (int i = 0; i < s.n_layer_types; i++) { + if (s.layer_types[i] == Q4NX_LAYER_FULL) full++; + else if (s.layer_types[i] == Q4NX_LAYER_SLIDING) sliding++; + const int want = ((i + 1) % 6) ? Q4NX_LAYER_SLIDING : Q4NX_LAYER_FULL; + if (s.layer_types[i] != want) pattern_ok = 0; + } + check(s.n_layer_types == 34 && full == 5 && sliding == 29, + "Medgemma: sliding_window_pattern 6 over 34 layers gives 5 full and 29 sliding layers"); + check(pattern_ok, "... every 6th layer (5, 11, ..., 29) is the full one, as HF's (i + 1) % 6 says"); + check(s.hybrid_layers == 29, "... and the hybrid count says 29"); + check(s.rope_theta == 1000000.0f && s.rope_theta_sliding == 10000.0f && s.rope_local_base, + "Medgemma: full layers theta 1e6, sliding layers rope_local_base_freq 10000"); + check(s.query_pre_attn_scalar == 256.0f, "Medgemma: query_pre_attn_scalar is read"); + check(s.extra_layer_norms == 2, "gemma3: the two feedforward norms, without Gemma4's post-PLE norm and scalar"); + check(s.rmsnorm_offset == 0, "gemma3 containers carry the norm's 1: the plain form"); + check(s.sliding_window == 1024, "the window is unchanged"); + } + + /* Gemma3-270M-NPU2 lists layer_types and has no local base key: the list wins, and the sliding layers take the + * model's theta rather than 0. */ + if (sem_of(dir, "{\"model_type\": \"gemma3_text_only\", \"num_hidden_layers\": 3, \"rope_theta\": 1000000.0," + " \"sliding_window\": 512, \"sliding_window_pattern\": 6," + " \"layer_types\": [\"sliding_attention\", \"full_attention\", \"sliding_attention\"]}", + &s) == 0) { + check(s.n_layer_types == 3 && s.layer_types[1] == Q4NX_LAYER_FULL && s.layer_types[0] == Q4NX_LAYER_SLIDING, + "a stated layer_types list is not replaced by the pattern"); + check(s.rope_theta_sliding == 1000000.0f && !s.rope_local_base, + "with no rope_local_base_freq a sliding layer takes the model's theta, not 0"); + check(s.extra_layer_norms == 2 && s.rmsnorm_offset == 0, "gemma3_text_only is gemma3 too"); + } + + /* The neighbours keep what they had. */ + if (sem_of(dir, "{\"model_type\": \"gemma2\", \"hidden_size\": 2304}", &s) == 0) + check(s.rmsnorm_offset == 1 && s.extra_layer_norms == 0, "gemma2 keeps (1 + w) and is not gemma3"); + if (sem_of(dir, "{\"model_type\": \"gemma3n_text\", \"hidden_size\": 2048}", &s) == 0) + check(s.extra_layer_norms == 0, "gemma3n is another architecture, not gemma3"); + if (sem_of(dir, "{\"model_type\": \"qwen3\", \"rope_theta\": 1000000.0, \"sliding_window\": 4096," + " \"num_hidden_layers\": 4}", &s) == 0) + check(s.n_layer_types == 0 && s.query_pre_attn_scalar == 0.0f && !s.rope_local_base, + "a config with a window but no pattern gets no layer list and no gemma keys"); + + char path[512]; + snprintf(path, sizeof path, "%s/config.json", dir); + unlink(path); + rmdir(dir); + printf(failures ? "q4nx_gemma3_semantics: %d FAILED\n" : "q4nx_gemma3_semantics: ok\n", failures); + return failures ? 1 : 0; +} From de3a5d4adf29acc7166f28a6035c42ef6f4d4c5a Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 22:58:33 -0400 Subject: [PATCH 2/2] npu: gemma3 prefills batched, and the serve adds the BOS tokenizer_config.json states step_batch now runs gemma3 (extra_layer_norms 2) with the post-attention, pre- and post-feedforward norms instead of falling back to one step() per token; batch_check_state_test on Medgemma-4B: batched = sequential, rel-L1 0. forward-serve took bos_token_id/add_bos_token from config.json only. Medgemma's states neither, its tokenizer_config.json states both, and the served reply lost its BOS ("The sky is blue because of the way the light is."). What config.json does not state now comes from tokenizer_config.json; the device reply is the host forward's ("...due to a phenomenon called Rayleigh scattering"). Co-Authored-By: Claude Opus 5.5 --- app/forward_serve.cpp | 16 ++++++++++++++ app/npu_stop_tokens.h | 23 +++++++++++++++++--- docs/npu.md | 19 +++++++++++----- npu/forward/src/q4nx_forward.cpp | 37 +++++++++++++++++++++++++------- tests/npu_stop_tokens_test.cpp | 19 ++++++++++++++++ 5 files changed, 98 insertions(+), 16 deletions(-) diff --git a/app/forward_serve.cpp b/app/forward_serve.cpp index 27dede566..209f003ba 100644 --- a/app/forward_serve.cpp +++ b/app/forward_serve.cpp @@ -1115,6 +1115,22 @@ int run_forward_serve(int argc, char** argv) { e.tok = std::make_unique(tok_json); e.bos_id = json_int(cfg, "bos_token_id", -1); e.add_bos = json_int(cfg, "add_bos_token", 0) != 0; + // config.json is the first source; what it does not state, tokenizer_config.json does (the BOS text, looked up + // in the tokenizer, and add_bos_token). Without this a Gemma3 pack was served with no BOS at all. + if (e.bos_id < 0 || json_int(cfg, "add_bos_token", -1) < 0) { + const nlohmann::json tc = npu::read_json_or_null(dir + "/tokenizer_config.json"); + if (e.bos_id < 0) { + const std::string b = npu::special_token_text(tc, "bos_token"); + if (!b.empty()) e.bos_id = e.tok->token_id(b); + } + if (json_int(cfg, "add_bos_token", -1) < 0) { + const int a = npu::add_bos_token_flag(tc); + if (a >= 0) e.add_bos = a != 0; + } + if (e.bos_id >= 0 && e.add_bos) + std::fprintf(stderr, "1bit forward-serve: BOS %d from tokenizer_config.json (config.json does not state it)\n", + e.bos_id); + } e.eos = npu::eos_token_ids_from_config(cfg); const size_t eos_from_config = e.eos.size(); e.endoftext = e.tok->token_id("<|endoftext|>"); diff --git a/app/npu_stop_tokens.h b/app/npu_stop_tokens.h index 281adf045..4f357e514 100644 --- a/app/npu_stop_tokens.h +++ b/app/npu_stop_tokens.h @@ -58,14 +58,31 @@ inline std::vector eos_token_ids(const nlohmann::json& cfg) { /// tokenizer_config.json's `eos_token` as text: a plain string, or the AddedToken object form /// `{"content": "<|im_end|>", ...}`. Empty when the key is absent, null or another shape. -inline std::string eos_token_text(const nlohmann::json& tokenizer_config) { - if (!tokenizer_config.is_object() || !tokenizer_config.contains("eos_token")) return {}; - const nlohmann::json& e = tokenizer_config["eos_token"]; +/// A special token's text from tokenizer_config.json, in either spelling HF writes ("" or {"content": ""}). +inline std::string special_token_text(const nlohmann::json& tokenizer_config, const char* key) { + if (!tokenizer_config.is_object() || !tokenizer_config.contains(key)) return {}; + const nlohmann::json& e = tokenizer_config[key]; if (e.is_string()) return e.get(); if (e.is_object() && e.contains("content") && e["content"].is_string()) return e["content"].get(); return {}; } +inline std::string eos_token_text(const nlohmann::json& tokenizer_config) { + return special_token_text(tokenizer_config, "eos_token"); +} + +/// tokenizer_config.json's add_bos_token: 1 or 0 when it states a boolean (or integer), -1 when it says nothing. +/// A container's config.json often omits bos_token_id and add_bos_token (FastFlowLM's Medgemma-4B-NPU2 has neither) +/// while its tokenizer_config.json states both; HF's tokenizer then adds the BOS, and Gemma without it answers +/// "The sky is blue because of the way the light is" where it should explain Rayleigh scattering (engine#1786). +inline int add_bos_token_flag(const nlohmann::json& tokenizer_config) { + if (!tokenizer_config.is_object() || !tokenizer_config.contains("add_bos_token")) return -1; + const nlohmann::json& a = tokenizer_config["add_bos_token"]; + if (a.is_boolean()) return a.get() ? 1 : 0; + if (a.is_number_integer()) return a.get() != 0 ? 1 : 0; + return -1; +} + /// The marker a chat template closes an assistant turn with, read off the template's own output. `after` is the /// text a template rendered immediately after an assistant message's content; the marker is the `<...>` token it /// starts with (leading whitespace skipped), or empty when it starts with anything else. diff --git a/docs/npu.md b/docs/npu.md index 43efa0808..c3f6d9a68 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -1974,11 +1974,20 @@ again, the known-answer prompt answered `*,,,,,,,`; with the plain form it answe form, and gemma2 keeps `(1 + w)` (not measured here). `tests/q4nx_gemma3_semantics_test.c` pins each rule with Medgemma's own keys. -Not yet a device row: the pack's four shapes are `QKV K2560 N4096`, `O K2048 N2560`, `GU K2560 N20480` and -`D K10240 N2560`, and `QKV K2560 N4096` is not built in any kernel family on the box (see -[arch-gaps.md](arch-gaps.md)). Gemma3 layers also take the per-token path for a prefill (like Gemma4), so a -prefill costs one `step()` per prompt token. Gemma3-270M-NPU2 still fails at load: its tiles are on an unpadded -K = 640 grid that the decoder does not read yet. +A prefill runs batched (`step_batch`) with the same three extra norms; only Gemma4 still falls back to one +`step()` per prompt token. `batch_check_state_test` on the Medgemma pack: batched equals sequential, rel-L1 0. + +The serve also needs the BOS. Medgemma's `config.json` states neither `bos_token_id` nor `add_bos_token`, and its +`tokenizer_config.json` states both (``, true), as HF reads them. forward-serve now takes what `config.json` +does not state from `tokenizer_config.json`. Served without it, the reply to "Explain in three sentences why the sky +is blue." was "The sky is blue because of the way the light is." With it, the reply is "The sky appears blue due to a +phenomenon called Rayleigh scattering", the same as the host forward with the BOS. + +On the device, with a locally built kernel set of the four shapes (`QKV K2560 N4096`, `O K2048 N2560`, +`GU K2560 N20480`, `D K10240 N2560`) and their 16-row twins, in one `npu-hold` ticket, the known answer came back +`Paris` three times, identical. That is not yet a leaderboard row: decode is 0.68 tok/s, and no kernel set for these +shapes ships. Gemma3-270M-NPU2 still fails at load: its tiles are on an unpadded K = 640 grid that the decoder +does not read yet. **Per-head q/k normalisation is read from the container (engine#1788).** Whether the forward normalises q and k per head used to be decided by `config.json`'s `model_type` alone: on for `qwen3*` and `gemma*`. diff --git a/npu/forward/src/q4nx_forward.cpp b/npu/forward/src/q4nx_forward.cpp index d0b7d2d42..0a372ff85 100644 --- a/npu/forward/src/q4nx_forward.cpp +++ b/npu/forward/src/q4nx_forward.cpp @@ -3936,8 +3936,11 @@ bool Q4nxNpuForward::step_batch_impl(const std::vector& tokens, int start_p // recurrence), so lin_kd_ no longer disqualifies a model, and attn_output_gate // is handled in the attention block below. MLA, per-layer embeddings, Gemma4 // and shared-KV stacks still fall back. + // Gemma3 (extra_layer_norms 2) is Gemma4's sandwich without PLE or KV sharing, so the batched body + // below runs it with the three extra norms; Gemma4 itself still falls back. + const bool sandwich = gemma4_layer_ && ple_dim_ == 0 && sem_.extra_layer_norms == 2; const bool simple_dense = - !mla_ && ple_dim_ == 0 && !gemma4_layer_ && kv_self; + !mla_ && ple_dim_ == 0 && (!gemma4_layer_ || sandwich) && kv_self; if (!simple_dense || getenv("NPU_NO_BATCH")) { if (inject_embeds_) { err_ = "step_batch_embeds: this model's batched body falls back to per-token step(), which needs " @@ -4313,14 +4316,23 @@ bool Q4nxNpuForward::step_batch_impl(const std::vector& tokens, int start_p { LhScope _t_res(LH_RES); const int nrow = q4nx_pad_n(H); - for (int m = 0; m < M; m++) - for (int k = 0; k < H; k++) - X[(size_t)m * H + k] += O[(size_t)m * nrow + k] * residual_scale_; + if (sandwich) { + // as step(): the attention output is normalised (post_attention_layernorm) before the add + std::vector on((size_t)H); + for (int m = 0; m < M; m++) { + rmsnorm(O.data() + (size_t)m * nrow, post_norm_[l].data(), H, on.data()); + for (int k = 0; k < H; k++) X[(size_t)m * H + k] += on[(size_t)k]; + } + } else { + for (int m = 0; m < M; m++) + for (int k = 0; k < H; k++) + X[(size_t)m * H + k] += O[(size_t)m * nrow + k] * residual_scale_; + } } // ---- MLP ---- for (int m = 0; m < M; m++) - rmsnorm(X.data() + (size_t)m * H, post_norm_[l].data(), H, + rmsnorm(X.data() + (size_t)m * H, (sandwich ? pre_ff_norm_ : post_norm_)[l].data(), H, XN.data() + (size_t)m * H); if (n_expert_ > 0) { // MoE FFN: the router/top-k pick different experts per row, so this @@ -4409,9 +4421,18 @@ bool Q4nxNpuForward::step_batch_impl(const std::vector& tokens, int start_p { LhScope _t_res(LH_RES); const int nrow = q4nx_pad_n(H); // as above: the design's N, identity when H is on the grid - for (int m = 0; m < M; m++) - for (int k = 0; k < H; k++) - X[(size_t)m * H + k] += D[(size_t)m * nrow + k] * residual_scale_; + if (sandwich) { + // post_feedforward_layernorm on the down projection, then the add (as step()) + std::vector dn((size_t)H); + for (int m = 0; m < M; m++) { + rmsnorm(D.data() + (size_t)m * nrow, post_ff_norm_[l].data(), H, dn.data()); + for (int k = 0; k < H; k++) X[(size_t)m * H + k] += dn[(size_t)k]; + } + } else { + for (int m = 0; m < M; m++) + for (int k = 0; k < H; k++) + X[(size_t)m * H + k] += D[(size_t)m * nrow + k] * residual_scale_; + } } // Same post-residual hidden-state dump as step() emits as [L], one row per diff --git a/tests/npu_stop_tokens_test.cpp b/tests/npu_stop_tokens_test.cpp index 9ebb686f6..6daff22eb 100644 --- a/tests/npu_stop_tokens_test.cpp +++ b/tests/npu_stop_tokens_test.cpp @@ -152,6 +152,25 @@ int main() { "a marker the tokenizer does not have as one token adds nothing"); } + // engine#1786: the BOS a container's tokenizer_config.json states, for a config.json that states none + // (FastFlowLM's Medgemma-4B-NPU2: no bos_token_id, no add_bos_token in config.json). + { + using onebit::npu::add_bos_token_flag; + using onebit::npu::special_token_text; + const json med = json::parse(R"({"bos_token": "", "add_bos_token": true, "eos_token": ""})"); + check(special_token_text(med, "bos_token") == "" && add_bos_token_flag(med) == 1, + "Medgemma's tokenizer_config.json: BOS , added"); + check(special_token_text(json::parse(R"({"bos_token": {"content": "", "lstrip": false}})"), "bos_token") == "", + "bos_token in the object spelling"); + check(add_bos_token_flag(json::parse(R"({"add_bos_token": false})")) == 0 && + add_bos_token_flag(json::parse(R"({"add_bos_token": 1})")) == 1, + "add_bos_token false and 1"); + check(add_bos_token_flag(json::parse("{}")) == -1 && add_bos_token_flag(json::parse(R"({"add_bos_token": null})")) == -1 && + add_bos_token_flag(json(nullptr)) == -1 && special_token_text(json::parse(R"({"bos_token": null})"), "bos_token").empty(), + "an absent or null key says nothing"); + check(onebit::npu::eos_token_text(med) == "", "eos_token_text is unchanged by the shared reader"); + } + if (failures == 0) std::printf("all stop-token checks passed\n"); return failures == 0 ? 0 : 1; }