diff --git a/docs/npu-plan.md b/docs/npu-plan.md index 8c89ad25c..9cef8b26a 100644 --- a/docs/npu-plan.md +++ b/docs/npu-plan.md @@ -68,6 +68,8 @@ tools/npu_plan.py [--kernels DIR] [--out PATH] [--check] [--print] | `designs[]` | **optional**: the per-role kernel set — `role` (`QKV`, `QKVLIN`, `O`, `GU`, `G`, `U`, `D`, the MoE `GB`/`UB`, the Gemma4 second attention type, the MLA and GDN roles), `M`, `K`, `N` and the artifact `elf` (a **bare file name**). When present, `Plan::designs_agree()` requires it to be exactly the set the container derives — same roles, same geometry, same artifact, **both directions** — and refuses by naming the role and both sides; a plan without the section keeps the engine's own derivation, which is what every plan written before the field means. Written by `tools/npu_plan.py` only for the families it can state completely (a partial section would be refused at load, so MLA, GDN mixes and Gemma4's second attention type / per-layer embeddings / double-wide MLP get none). `Plan::check_artifacts()` then refuses at load, before any context is opened, when a named artifact is not present and non-empty in the kernel directory — and `npu_plan.py --check` asks the same question with the same rule at import time | | `rope.kind`, `rope.planes`, `rope.sections` | **optional**: which rotary the decoder applies. `plain` (the meaning of an absent block, and of every plan written before the field) is one position index with one plane. `mrope` is the vision-conditioned form — a **3 x N block of position ids** whose planes (t, h, w) advance differently across an image, so a text row with all three planes equal is bit-identical to plain RoPE while an image row is not. `sections` is the pack's own split of the rotary width over the planes (Qwen3-VL: `[24, 20, 20]`) and is **required** for `mrope`: without it a caller has no declared width per plane and would rotate a different model than the pack was trained as. `planes` defaults to the number of sections and must agree with it. A `plain` rope that carries `sections`, or `planes` other than 1, is refused rather than read as if the planes existed. This is a description like `kv`: a caller that cannot build the planes must refuse `mrope` **by name** — falling back to one index predicts fluently, and text after a 448×448 image resumes at position **19** where a token-count advance would give 201 (engine#799) | | `inputs.kind`, `inputs.vision` | **optional**: what the decoder is fed. `tokens` (the meaning of an absent block) is one id per row, looked up by the decoder. `embeds` is rows × hidden of precomputed embeddings already merged with any vision features — the split AMD's reference makes with a separate `embedding.onnx` feeding `text.onnx`. `inputs.vision.merge` says who merged (`external` or `none`) and `inputs.vision.feature_dim` how wide the merged features are; `external` requires a positive `feature_dim`, `none` requires 0, and a `vision` block on a `tokens` plan is refused, because ignored vision features serve the text half of a vision model and answer confidently about an image it never saw. Descriptive only: a plan that says `embeds` and a caller that feeds ids is the caller's refusal to make | +| `drafter.head`, `drafter.k`, `drafter.tau`, `drafter.verify_rows`, `drafter.tree` | **optional**: the speculator the plan names (engine#799 §1). Absent means no drafter, which is what every plan written before the section means. `head` is a **bare file name** under the model dir (a plan must not be able to aim the loader at a file the command line did not name, the rule `kv.hadamard.signs` follows), `k` is draft tokens per pass (≥ 1), `tau` the acceptance gate (**0 < tau ≤ 1** — it is a gate, not a temperature, and a tau of 1.5 is refused rather than clamped), `verify_rows` the rows checked per pass (≥ 1), `tree` `leaf` or `none`. Naming a head without stating k, verify_rows and tau is refused as a **half-stated contract**: a speculator with different numbers answers different text for the same prompt and raises nothing | +| `state.gdn_commit` | **optional**: `immediate` (the meaning of an absent section, and what the step path does today) or `deferred` — the speculator's contract, where a rejected draft row must not leave its recurrence behind. A target that cannot defer must refuse `deferred` by name: committing early and reporting the accepted tokens gives right-looking text with a state no future token agrees with | | `companion` | **refused by name, with a reason**: a plan describes **one** decoder stream over one token sequence. A second graph (a vision encoder, a projector) is its own model directory that the serving layer merges into `inputs.embeds`; a companion list would put two graphs and their scheduling in a file whose only consumer can neither open nor order them (engine#799) | | `layers[]` | one entry per layer, in order: `index`, `role` (`attn`/`gdn`/`moe`/`mla`), and projections `qkv {K, N, fused}`, `attn_gate {K, N}`, `gate_up {K, N}`, `down {K, N}`. The count must equal `model.layers`. The projection shapes are recorded for P4’s `NpuModel`; the forward reads the count (for its [plan] log line), and takes the design set from `designs[]` when the plan names it, deriving it from the container otherwise | diff --git a/npu/plan/plan.h b/npu/plan/plan.h index 818382542..643d7adf2 100644 --- a/npu/plan/plan.h +++ b/npu/plan/plan.h @@ -162,6 +162,30 @@ struct Plan { std::string vision_merge = "none"; // "external" | "none" int vision_feature_dim = 0; bool takes_embeds() const { return inputs_kind == "embeds"; } + // The drafter the plan may name (engine#799 §1 `drafter {head path, k, tau, verify_rows, tree}`). Absent means + // no drafter, which is what every plan written before this field means. Descriptive, like `rope` and `kv`: the + // speculator (P6) reads it, and a plan that names a drafter this engine cannot drive is refused at the point that + // tries to drive it -- never silently served as plain decode, because a speculator with the wrong tau or k answers + // with different text for the same prompt and no error at all. + // + // head the draft head artifact, a BARE FILE NAME under the model dir (the same rule kv.hadamard.signs + // follows: a plan must not be able to aim the loader at a file the command line did not name) + // k draft tokens per pass, >= 1 + // tau the acceptance gate, 0 < tau <= 1 + // verify_rows rows the verifier checks per pass, >= 1 (4 in P6's design: one draft plus a leaf fill) + // tree "leaf" (P6's tree) or "none" (a plain chain) + std::string drafter_head; + int drafter_k = 0, drafter_verify_rows = 0; + double drafter_tau = 0.0; + std::string drafter_tree = "none"; // "leaf" | "none" + bool has_drafter() const { return !drafter_head.empty(); } + // When a GDN layer's recurrent state is committed (engine#799 §1 `state {gdn commit}`): "immediate" is every plan + // written before this field and is what the step path does today; "deferred" is the speculator's contract, where a + // rejected draft row must not leave its recurrence behind. A target that cannot defer must refuse it by name: + // committing early and reporting the accepted tokens gives the right-looking text with a state that no future token + // agrees with. + std::string gdn_commit = "immediate"; // "immediate" | "deferred" + bool gdn_deferred() const { return gdn_commit == "deferred"; } // P7 (#806) part A: what the KV cache is stored as. `f16` is every plan written before this field // existed, and stays the default for that reason. `int8` stores that side as int8 with one fp32 // scale per (head, token), taken AFTER a per-head Hadamard rotation -- the rotation is what makes one @@ -386,6 +410,58 @@ struct Plan { " but inputs.vision.merge is none (no merge, no features)"); } } + // The drafter (engine#799 §1). Absent means no drafter; every stated field is checked, because a speculator + // with a plausible-but-wrong k/tau/rows answers different text for the same prompt with no error at all. + if (j.contains("drafter")) { + const auto& dr = obj(j, "drafter", path); + p.drafter_head = typed(dr, "head", "drafter", path); + if (!p.drafter_head.empty() && + (p.drafter_head.find('/') != std::string::npos || p.drafter_head.find("..") != std::string::npos)) + throw PlanError("plan: " + path + " drafter.head \"" + p.drafter_head + + "\" must be a bare file name (no directory part)"); + if (dr.contains("k")) { + p.drafter_k = int_field(dr, "k", "drafter", path); + if (p.drafter_k < 1) + throw PlanError("plan: " + path + " drafter.k " + std::to_string(p.drafter_k) + + " must be at least 1"); + } + if (dr.contains("verify_rows")) { + p.drafter_verify_rows = int_field(dr, "verify_rows", "drafter", path); + if (p.drafter_verify_rows < 1) + throw PlanError("plan: " + path + " drafter.verify_rows " + + std::to_string(p.drafter_verify_rows) + " must be at least 1"); + } + if (dr.contains("tau")) { + if (!dr.at("tau").is_number()) + throw PlanError("plan: " + path + " drafter.tau must be a number"); + p.drafter_tau = dr.at("tau").get(); + if (!(p.drafter_tau > 0.0 && p.drafter_tau <= 1.0)) + throw PlanError("plan: " + path + " drafter.tau " + std::to_string(p.drafter_tau) + + " is not in (0, 1] (it is an acceptance gate, not a temperature)"); + } + if (dr.contains("tree")) { + p.drafter_tree = typed(dr, "tree", "drafter", path); + if (p.drafter_tree != "leaf" && p.drafter_tree != "none") + throw PlanError("plan: " + path + " drafter.tree \"" + p.drafter_tree + + "\" is not leaf or none"); + } + // A named drafter with no k/tau/rows is a half-stated contract: the values exist to be checked against what + // the speculator actually does, so a plan that omits them cannot be honoured either. + if (p.has_drafter() && (p.drafter_k == 0 || p.drafter_verify_rows == 0 || p.drafter_tau == 0.0)) + throw PlanError("plan: " + path + + " names drafter.head but not k, verify_rows and tau (a drafter's numbers are part of " + "its contract: the same prompt answers different text at a different tau)"); + } + // When GDN state is committed (engine#799 §1). Absent means immediate, which is what the step path does today. + if (j.contains("state")) { + const auto& st = obj(j, "state", path); + if (st.contains("gdn_commit")) { + p.gdn_commit = typed(st, "gdn_commit", "state", path); + if (p.gdn_commit != "immediate" && p.gdn_commit != "deferred") + throw PlanError("plan: " + path + " state.gdn_commit \"" + p.gdn_commit + + "\" is not immediate or deferred"); + } + } // A second graph (a vision encoder, a projector) is deliberately NOT a plan field: one plan is one // decoder stream over one token sequence. A hybrid is two model directories and a merge above them, // which is the serving layer's business -- a companion list would put two graphs and their scheduling in diff --git a/tests/npu_plan_test.cpp b/tests/npu_plan_test.cpp index ffe3717a8..50001ed94 100644 --- a/tests/npu_plan_test.cpp +++ b/tests/npu_plan_test.cpp @@ -639,9 +639,63 @@ void test_rope_and_inputs() { } } +// engine#799 §1: the two sections the spec names and the loader could not express -- `drafter {head, k, tau, +// verify_rows, tree}` and `state {gdn_commit}`. Both are absent-by-default (no drafter, immediate commit), and a +// half-stated contract is refused by name, because a speculator with different numbers answers different text for the +// same prompt and raises nothing. +void test_drafter_and_state() { + { + const Plan p = Plan::load(write_plan("drafter_absent", base_plan())); + check(!p.has_drafter() && !p.gdn_deferred(), "no drafter and immediate GDN commit are the defaults"); + } + { + auto j = base_plan(); + j["drafter"] = {{"head", "draft_head.q4nx"}, {"k", 4}, {"tau", 0.6}, {"verify_rows", 4}, {"tree", "leaf"}}; + j["state"] = {{"gdn_commit", "deferred"}}; + const Plan p = Plan::load(write_plan("drafter_full", j)); + check(p.has_drafter() && p.drafter_head == "draft_head.q4nx" && p.drafter_k == 4 && + p.drafter_verify_rows == 4 && p.drafter_tau == 0.6 && p.drafter_tree == "leaf", + "a fully stated drafter loads as written"); + check(p.gdn_deferred(), "and a deferred GDN commit is reported"); + } + { + auto j = base_plan(); + j["drafter"] = {{"head", "draft.q4nx"}, {"k", 2}, {"tau", 1.0}, {"verify_rows", 2}}; + const Plan p = Plan::load(write_plan("drafter_edge_tau", j)); + check(p.drafter_tau == 1.0 && p.drafter_tree == "none", "tau 1.0 is allowed and tree defaults to none"); + } + { + auto j = base_plan(); + j["drafter"] = {{"head", "draft.q4nx"}, {"k", 2}, {"tau", 1.5}, {"verify_rows", 2}}; + const std::string why = refusal(write_plan("drafter_tau", j)); + check(has(why, "drafter.tau") && has(why, "(0, 1]"), "a tau outside (0,1] is refused, naming the range: " + why); + } + { + auto j = base_plan(); + j["drafter"] = {{"head", "sub/draft.q4nx"}, {"k", 2}, {"tau", 0.5}, {"verify_rows", 2}}; + const std::string why = refusal(write_plan("drafter_path", j)); + check(has(why, "drafter.head") && has(why, "bare file name"), + "a drafter head with a directory part is refused (a plan cannot aim the loader): " + why); + } + { + auto j = base_plan(); + j["drafter"] = {{"head", "draft.q4nx"}, {"k", 2}, {"verify_rows", 2}}; + const std::string why = refusal(write_plan("drafter_half", j)); + check(has(why, "drafter.head") && has(why, "tau"), + "a drafter with no tau is refused as a half-stated contract: " + why); + } + { + auto j = base_plan(); + j["state"] = {{"gdn_commit", "promptly"}}; + const std::string why = refusal(write_plan("state_bad", j)); + check(has(why, "state.gdn_commit"), "an unknown gdn_commit is refused by name: " + why); + } +} + int main() { test_valid(); test_rope_and_inputs(); + test_drafter_and_state(); test_kv(); test_tile_order(); test_refusals();