From f993a57aff69c598b24c5d3c78e9ce7090bc822b Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 09:12:48 -0400 Subject: [PATCH 1/5] census: an answered FastFlowLM gap stays answered, and the d3 pin watches its paths (Fixes #2171) CI keeps no watcher state, so every scheduled run sees the whole FastFlowLM org as new. Each family docs/npu.md had already answered printed !! NPU FLM / !! NEW FLM again and re-filed the census issue the day after it closed. The 2026-11-11 run alerted on 31 packs, every one answered or tracked (#1786, #1788, #1780, #1783). - registry/flm_answers.json: the ledger's answers, keyed by the architecture a pack declares (by repo id for the config-less smolvla). An answered gap prints ". NPU FLM" / ". FLM" with its tracking issue and does not alert. A family with no entry still alerts. An answer cannot quiet a family the NPU route serves or a derivative drop. - support_pins.json: a pin may name `paths`; it then moves only when a commit touches one. semantic-router reviewed a9219240 -> 15b4fb5a (88 commits): the d3 package/model/decode files have the same blobs, and #6192's API versioning does not reach npu/systemone.h. - docs/npu.md: a row for every answer, including phi3, qwen2vl, gemma-embedding, the Qwen3_5Moe VLM class and Whisper, which had none. - tests: sections 27 and 21b (before/after on the 31 rows of run 43016580831, stale-answer and doc-row guards derived from the files). The alert-rule sections pass no answers explicitly. Co-Authored-By: Claude Opus 5.5 --- docs/npu.md | 28 +++++++ registry/flm_answers.json | 91 ++++++++++++++++++++ registry/support_pins.json | 12 ++- tests/census_watch_test.py | 167 ++++++++++++++++++++++++++++++++++--- tools/census_watch.py | 116 ++++++++++++++++++++++---- 5 files changed, 387 insertions(+), 27 deletions(-) create mode 100644 registry/flm_answers.json diff --git a/docs/npu.md b/docs/npu.md index c17e721f8..18c6ad579 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -1856,6 +1856,34 @@ route loads and a captured `--device npu` run stands behind it; the regenerated GGUF-on-NPU forward), while every D3-* pack in this body declares the dense `qwen3_5` family, so the two are not the same answer and the registry keeps them apart. +The answers are also machine-readable, in `registry/flm_answers.json`, because CI keeps no watcher state: every scheduled +run sees the whole org as new, and before that file each family answered above printed `!! NPU FLM` again and re-filed +the census issue the day after it closed (engine#2171; the 2026-11-11 run alerted on 31 packs, every one already +answered or tracked). A pack whose declared architecture is listed there prints `. NPU FLM` / `. FLM` with the issue +that tracks it and does not alert; a family with no entry still alerts. The test fails on an entry for a family the +registry serves on the NPU route, so a family that starts to load drops its answer. Keyed by the architecture each +pack declares (a config with no `architectures` declares its `model_type`): + +| declared architecture | packs on 2026-11-11 | answer | tracked by | +|---|---|---|---| +| `MiniCPMV4_7ForConditionalGeneration` | MiniCPM-V-4.7-1B, MiniCPM-V-4.7-35B-A3B | not served: a VLM class the pin does not register | engine#1788 | +| `MiniCPMOcrForConditionalGeneration` | MiniCPM-OCR-1.0 | not served: an OCR/vision pack; the route claims text decoders only | engine#1786 | +| `HYV3ForCausalLM` | Hy-Memory-30B-A3B | not served: `hy_v3`, hrx only | engine#1786 | +| `DFlashDraftModel` | Nemotron-3.5-Bolt-E4B-DFlash | not served: a draft head, not a decoder | engine#1780 | +| `Gemma4ForConditionalGeneration`, `Gemma4UnifiedForConditionalGeneration` | Nemotron-3.5-Bolt-E4B, Gemma4-E4B-IT, Gemma4-E2B-IT, Gemma4-12B-IT | not served: `gemma4`, hrx only | engine#1786 | +| `HunYuanDenseV1ForCausalLM` | Hy-MT2-1.8B | not served: `hunyuan-dense`, hrx only | engine#1786 | +| `Gemma3ForCausalLM`, `gemma3_text` | Gemma3-270M, Gemma3-1B, Gemma3-4B, Gemma3-4B-Text, medgemma-4b-it, medgemma-1.5-4b-it, Translategemma-4B-Instruct | not served: `gemma3`, hrx only | engine#1786 | +| `Lfm2ForCausalLM` | LFM2-1.2B, LFM2-2.6B, LFM2-1.2B-Tool, LFM2-2.6B-Transcript, LFM2.5-1.2B, LFM2.5-1.2B-Thinking | not in the registry's NPU set: `lfm2`, hrx only (LFM2.5-1.2B on the per-op route is engine#2170, which adds no `NPU_ARCHS` entry) | engine#1786 | +| `GptOssForCausalLM` | GPT-OSS-20B, GPT-OSS-Safeguard-20b | not served: `gpt-oss`, hrx only | engine#1786 | +| `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B | not served for this class: the route serves `qwen3_5_moe` through `Qwen3_5MoeForCausalLM` (GGUF on the NPU); the VLM wrapper class is hrx only, and FastFlowLM's own pack is not measured on the route | engine#1786 | +| `Qwen2_5_VLForConditionalGeneration` | Qwen2.5-VL-3B-Instruct | not served: `qwen2vl`, hrx only (the route serves the text-only `qwen2` class) | engine#1786 | +| `Phi3ForCausalLM` | Phi4-mini-Instruct | not served: `phi3`, hrx only | engine#1786 | +| `Gemma3TextModel` | Embedding-Gemma-300M | not served: `gemma-embedding`, an embedding model, hrx only | engine#1786 | +| `WhisperForConditionalGeneration` | Whisper-V3-Turbo | not served: a speech encoder-decoder the registry does not map | engine#1783 | + +`FastFlowLM/smolvla-NPU2` is answered by its repo id (`repos` in the same file): it has no config and no `*.q4nx` +file, so there is no architecture to key on and nothing for the route to load. + ### FastFlowLM's own containers, served as published: measured rows (engine#1786) The tables above answer "does the registry map this family". This one answers a different question for the packs diff --git a/registry/flm_answers.json b/registry/flm_answers.json new file mode 100644 index 000000000..e4e50bb92 --- /dev/null +++ b/registry/flm_answers.json @@ -0,0 +1,91 @@ +{ + "about": "Recorded answers for FastFlowLM packs this engine does not serve, read by tools/census_watch.py (engine#2171). CI keeps no watcher state, so without this file every scheduled run re-alerted on families docs/npu.md had already answered. A pack whose declared architecture (or, for a repo with no config, whose id) is listed here prints `. NPU FLM` / `. FLM` with the issue that tracks it and does not alert. Hand-written: add an entry only with a row for it in docs/npu.md, and remove it when the family is served (tests/census_watch_test.py fails on an answer for a family the registry serves on the NPU route).", + "architectures": { + "MiniCPMV4_7ForConditionalGeneration": { + "answer": "not served: a vision-language pack whose class the pin does not register; serving it on the NPU route is engine#1788", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1788 + }, + "MiniCPMOcrForConditionalGeneration": { + "answer": "not served: an OCR/vision pack whose class the pin does not register; the route claims text decoders only", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "HYV3ForCausalLM": { + "answer": "not served: hy_v3 is mapped for hrx only; no hy_v3 model type or kernels in the NPU route", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "DFlashDraftModel": { + "answer": "not served: a draft head, not a decoder; running the DFlash drafts as NPU sources is engine#1780", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1780 + }, + "Gemma4ForConditionalGeneration": { + "answer": "not served: gemma4 is mapped for hrx only; no gemma4 model type or kernels in the NPU route", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "Gemma4UnifiedForConditionalGeneration": { + "answer": "not served: gemma4 is mapped for hrx only; no gemma4 model type or kernels in the NPU route", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "HunYuanDenseV1ForCausalLM": { + "answer": "not served: hunyuan-dense is mapped for hrx only", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "Gemma3ForCausalLM": { + "answer": "not served: gemma3 is mapped for hrx only (medgemma, translategemma and the Gemma3 text packs declare it)", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "gemma3_text": { + "answer": "not served: gemma3 is mapped for hrx only (medgemma, translategemma and the Gemma3 text packs declare it)", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "Lfm2ForCausalLM": { + "answer": "not served by the registry yet: lfm2 is mapped for hrx only; LFM2.5-1.2B on the per-op route is engine#2170, which adds no NPU_ARCHS entry", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "GptOssForCausalLM": { + "answer": "not served: gpt-oss is mapped for hrx only", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "Qwen3_5MoeForConditionalGeneration": { + "answer": "not served for this class: the route serves qwen3_5_moe through Qwen3_5MoeForCausalLM (GGUF on the NPU); the VLM wrapper class FastFlowLM declares is mapped for hrx only, and its own Q4NX pack is not measured on the route", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "Qwen2_5_VLForConditionalGeneration": { + "answer": "not served: qwen2vl is mapped for hrx only; the route serves the text-only qwen2 class", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "Phi3ForCausalLM": { + "answer": "not served: phi3 is mapped for hrx only", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "Gemma3TextModel": { + "answer": "not served: gemma-embedding (an embedding model, not a decoder) is mapped for hrx only", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1786 + }, + "WhisperForConditionalGeneration": { + "answer": "not served: a speech encoder-decoder the registry does not map; the Whisper encoder on our runtime is engine#1783", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve", + "issue": 1783 + } + }, + "repos": { + "FastFlowLM/smolvla-NPU2": { + "answer": "not a pack: a config-less VLA repo with no *.q4nx file, so there is no architecture to map and nothing for the route to load", + "doc": "docs/npu.md#fastflowlms-npu2-drops-the-families-the-route-does-not-serve" + } + } +} diff --git a/registry/support_pins.json b/registry/support_pins.json index 000019e3c..7d8cb1a5c 100644 --- a/registry/support_pins.json +++ b/registry/support_pins.json @@ -7,9 +7,15 @@ "why": "The official gguf Q4_0/Q4_1 -> Q4NX converter with per-architecture configs (configs/*.json, 18 of them: gemma3, gemma4, gpt-oss, lfm2, llama, nanbeige, phi4, qwen2, qwen2vl, qwen3, qwen3.5_{0.8b,2b,4b,9b,a3b,omni}, qwen3vl). docs/npu.md's repack path runs it (scripts/repack_gguf.py, ONEBIT_Q4NX_CONVERTER) and fails its declared-scale guard when the converter or config drops a critical scale. A per-arch config here is what makes a family packable on our side at all. Reviewed 2026-11-08 (d1d5232 -> 93b8398, 2 commits, \"Qwen3.5 / Qwen3.5-MoE / Omni converters on stock llama.cpp GGUFs\"): three changes touch the family this route serves. (a) full-attention layers are now detected from `attn_q.weight` instead of \"every Nth layer\", because the d3 layouts are irregular -- the commit names full attention at 2, 5, 8, 9, 10, 11 for d3:edge, which is the same irregularity our own converter takes from `layer_types`; (b) an MTP block (nextn_predict_layers > 0, counted in block_count) no longer leaks out as a bogus extra decoder layer; (c) the Qwen35 value-head reorder is decided from the v/qk head ratio instead of feed_forward_length <= 6144, and a ratio > 2 raises. Stock Qwen3.5 0.8B/2B/4B (Q4_K, Q4_1) are tested 124/124 in that PR, and the output is byte-identical for GGUFs without an MTP block, so the repack path's declared-scale guard is unaffected; the entry point and configs/ layout that scripts/repack_gguf.py invokes are unchanged." }, "vllm-project/semantic-router": { - "commit": "a9219240e6f9dd4d9e1bea83596565cc8f18989b", - "reviewed": "2026-11-09", - "why": "The Decision 3.0 (d3) source: the router calls d3 for routing, guardrails and tool selection, and FastFlowLM v1.0.11 serves those weights through /v1/systemone with the same HTTP API as the upstream d3 server. Engine #653 is our side of that; a change here is what a new d3 pack would follow. Reviewed 2026-11-08 (632f087 -> b7adcfa, 152 commits, 300 files): **the d3 contract is unchanged** -- `d3_format.py`, `d3_server.py` and `d3_runtime.py` are not in the delta at all, and the only System One files touched are dashboard frontend pages (`SystemOnePlaygroundPage.tsx`, `systemOneTargets.ts`). What moved is router-side: `config/fragments/signal/decision/vela{2,3}-labels-spans.yaml` (their own newer decision models), a `config/recipes/decision-balance/` recipe, and a `canonical_decision_model` CRD. So what this route must speak is unchanged, and the vela2/vela3 names are upstream's packs, not ours. Reviewed 2026-11-09 (b7adcfa -> a921924, 9 commits, engine #1667): the d3 contract is unchanged again -- no d3_* file is in the delta. The one System One file is the router's client, pkg/systemone/backend.go (#5144): a backend marked trusted_gateway is now refused before dispatch (errTrustedGatewayUnsupported), because System One dispatch does not carry the request's gateway credential. The wire protocol string stays vllm-sr/systemone@1, and FastFlowLM's /v1/systemone is not a trusted gateway, so nothing this route speaks or serves moves. The rest is router-side (header_mutation append, the lifecycle client's 5xx retry, contrastive jailbreak embedding validation, adaptive thinking on Responses backends, a repeated [DONE] in protocolcodec) plus a Claude Code plugin and maintainer docs." + "commit": "15b4fb5ad46b5c40a3bd99f35d0feb28997dac68", + "reviewed": "2026-11-11", + "why": "The Decision 3.0 (d3) source: the router calls d3 for routing, guardrails and tool selection, and FastFlowLM v1.0.11 serves those weights through /v1/systemone with the same HTTP API as the upstream d3 server. Engine #653 is our side of that; a change here is what a new d3 pack would follow. Reviewed 2026-11-08 (632f087 -> b7adcfa, 152 commits, 300 files): **the d3 contract is unchanged** -- `d3_format.py`, `d3_server.py` and `d3_runtime.py` are not in the delta at all, and the only System One files touched are dashboard frontend pages (`SystemOnePlaygroundPage.tsx`, `systemOneTargets.ts`). What moved is router-side: `config/fragments/signal/decision/vela{2,3}-labels-spans.yaml` (their own newer decision models), a `config/recipes/decision-balance/` recipe, and a `canonical_decision_model` CRD. So what this route must speak is unchanged, and the vela2/vela3 names are upstream's packs, not ours. Reviewed 2026-11-09 (b7adcfa -> a921924, 9 commits, engine #1667): the d3 contract is unchanged again -- no d3_* file is in the delta. The one System One file is the router's client, pkg/systemone/backend.go (#5144): a backend marked trusted_gateway is now refused before dispatch (errTrustedGatewayUnsupported), because System One dispatch does not carry the request's gateway credential. The wire protocol string stays vllm-sr/systemone@1, and FastFlowLM's /v1/systemone is not a trusted gateway, so nothing this route speaks or serves moves. The rest is router-side (header_mutation append, the lifecycle client's 5xx retry, contrastive jailbreak embedding validation, adaptive thinking on Responses backends, a repeated [DONE] in protocolcodec) plus a Claude Code plugin and maintainer docs. Reviewed 2026-11-11 (a9219240 -> 15b4fb5a, 88 commits; engine#2171): **the d3 contract is unchanged** -- `vela3_d3/package.py`, `model.py` and `decode.py` have the same blob at both commits. Three commits touch the family: #5886 adds router config fragments for the decision model (config/fragments/signal/systemone/*, no runtime code), #6113 records ROCm golden answers for the d3 sizes (a new registry/golden_answers_vela3_d3.json and the test that reads it), and #6192 versions the System One HTTP API (systemone/0.2, vllm-sr/5.0; vllm-sr/4.0 stays a selectable compatible superset, `/v1/models` now reports api_version 5.0.0, a request may send `roles`, and `roles: null` makes an object state the user's content for span questions). Its own description says model input stays byte-identical on every family. Nothing here reaches npu/systemone.h, which copies the pack's d3_format.py/d3_runtime.py (choice/noul/score prompts and answers; no span questions and no api_version field). From this review on the pin watches `paths` only: the d3 family directory, the runtime's systemone.py and api/ (the HTTP contract), and config/schemas/systemone; router-side commits no longer re-file the census issue every day.", + "paths": [ + "src/model-runtime/vllm_srun/families/vela3_d3", + "src/model-runtime/vllm_srun/systemone.py", + "src/model-runtime/vllm_srun/api", + "config/schemas/systemone" + ] } }, "d3_sources": { diff --git a/tests/census_watch_test.py b/tests/census_watch_test.py index 69270cde7..d52f5a8c9 100644 --- a/tests/census_watch_test.py +++ b/tests/census_watch_test.py @@ -46,7 +46,11 @@ "EmptyForCausalLM": {"gguf": "empty", "backends": []}, } SIG = {"Qwen3_5MVLAAbsorbedForCausalLM": {"model_type": "qwen3_5_text", "family": "Qwen3.5 MVLA"}} -FLM_LINE_PREFIXES = (" + FLM FastFlowLM/", " !! NEW FLM ", " !! NPU FLM ", " . FLM ") +FLM_LINE_PREFIXES = (" + FLM FastFlowLM/", " !! NEW FLM ", " !! NPU FLM ", " . FLM ", " . NPU FLM ") +# Sections 17-22 pin the alert rule itself, on a fixture org whose MiniCPM-V pack is answered in the +# committed registry/flm_answers.json; they pass no answers so the rule is tested on an unanswered gap. +# The answers file has its own section (27). +NO_ANSWERS = {"architectures": {}, "repos": {}} failures = [] @@ -514,7 +518,7 @@ def counting_detail(url): _buf = io.StringIO() with contextlib.redirect_stdout(_buf): cw.report_flm(_declined_rows, cw.sweep_releases(fetch=flm_fetch), - [r for r in _declined_rows if r["id"] == "FastFlowLM/LFM3-1.7B-NPU2"], []) + [r for r in _declined_rows if r["id"] == "FastFlowLM/LFM3-1.7B-NPU2"], [], answers=NO_ANSWERS) _declined_listing = _buf.getvalue() # 18. One line per repo, `!! NEW FLM` only for a family we cannot serve, and the exit code @@ -522,7 +526,7 @@ def counting_detail(url): state = flm_state() buf, err = io.StringIO(), io.StringIO() with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(err): - flm_rc, flm_ok = cw.watch_flm(live_archs, state, fetch=flm_fetch, detail=flm_detail) + flm_rc, flm_ok = cw.watch_flm(live_archs, state, fetch=flm_fetch, detail=flm_detail, answers=NO_ANSWERS) first = buf.getvalue() check("flm first run: both watches ran", flm_ok) check("flm first run: an unmapped family alerts (exit 1)", flm_rc == 1) @@ -549,7 +553,7 @@ def counting_detail(url): with contextlib.redirect_stdout(buf): gap_all = cw.sweep_org(live_archs, fetch=lambda url: fixture("org.json") + [HRX_ONLY_REPO]) cw.report_flm(gap_all, cw.sweep_releases(fetch=flm_fetch), - [r for r in gap_all if r["id"] == HRX_ONLY_REPO["id"]], []) + [r for r in gap_all if r["id"] == HRX_ONLY_REPO["id"]], [], answers=NO_ANSWERS) gapfirst = buf.getvalue() check("flm report: an NPU pack for an HRX-only family is marked !! NPU FLM", " !! NPU FLM FastFlowLM/Gemma3-12B-NPU2" in gapfirst) @@ -566,7 +570,7 @@ def counting_detail(url): # 19. The delta: the second run prints nothing, a moved repo prints one line. buf = io.StringIO() with contextlib.redirect_stdout(buf): - quiet_rc, quiet_ok = cw.watch_flm(live_archs, state, fetch=flm_fetch, detail=flm_detail) + quiet_rc, quiet_ok = cw.watch_flm(live_archs, state, fetch=flm_fetch, detail=flm_detail, answers=NO_ANSWERS) second = buf.getvalue() check("flm delta: a second run is quiet and green", quiet_ok and quiet_rc == 0 and "nothing new since the last run" in second) @@ -579,7 +583,7 @@ def counting_detail(url): check("flm delta: known releases are not new", fresh_rel == []) buf = io.StringIO() with contextlib.redirect_stdout(buf): - moved_rc = cw.report_flm(gap_repos2, cw.sweep_releases(fetch=flm_fetch), fresh, fresh_rel) + moved_rc = cw.report_flm(gap_repos2, cw.sweep_releases(fetch=flm_fetch), fresh, fresh_rel, answers=NO_ANSWERS) check("flm delta: a moved NPU-gap pack alerts (an HRX-only family with a Q4NX pack)", moved_rc == 1 and " !! NPU FLM FastFlowLM/Gemma3-12B-NPU2" in buf.getvalue()) # D3-Mini moving used to be that alert; it is the same move on a family the NPU route now serves, @@ -591,14 +595,14 @@ def counting_detail(url): check("flm delta: the moved repo is reported", [r["id"] for r in fresh1] == [moved[0]["id"]]) buf = io.StringIO() with contextlib.redirect_stdout(buf): - served_rc = cw.report_flm(moved_repos, cw.sweep_releases(fetch=flm_fetch), fresh1, []) + served_rc = cw.report_flm(moved_repos, cw.sweep_releases(fetch=flm_fetch), fresh1, [], answers=NO_ANSWERS) check("flm delta: a moved NPU-served pack does not alert", " !! NPU FLM FastFlowLM/D3-Mini-NPU2" not in buf.getvalue()) served = cw.sweep_org(live_archs, fetch=lambda url: fixture("org.json")) fresh2 = [r for r in served if r["id"] == "FastFlowLM/Qwen3-8B-NPU2"] # qwen3: served on the NPU buf = io.StringIO() with contextlib.redirect_stdout(buf): - served_rc = cw.report_flm(served, cw.sweep_releases(fetch=flm_fetch), fresh2, []) + served_rc = cw.report_flm(served, cw.sweep_releases(fetch=flm_fetch), fresh2, [], answers=NO_ANSWERS) check("flm delta: a moved family the NPU route serves stays green", served_rc == 0 and buf.getvalue().count(" + FLM ") == 1 and "!! NPU FLM" not in buf.getvalue()) @@ -606,7 +610,7 @@ def counting_detail(url): # 20. `--full` prints every repo and release on demand. buf = io.StringIO() with contextlib.redirect_stdout(buf): - full_rc = cw.report_flm(moved_repos, cw.sweep_releases(fetch=flm_fetch), [], [], full=True) + full_rc = cw.report_flm(moved_repos, cw.sweep_releases(fetch=flm_fetch), [], [], full=True, answers=NO_ANSWERS) full = buf.getvalue() check("flm --full: every repo listed", sum(full.count(p) for p in FLM_LINE_PREFIXES) == 5) check("flm --full: every release listed", full.count(" + FLM RELEASE ") == 3) @@ -714,7 +718,7 @@ def fail_fetch(url): buf, err = io.StringIO(), io.StringIO() with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(err): - dead_rc, dead_ok = cw.watch_flm(live_archs, flm_state(), fetch=fail_fetch) + dead_rc, dead_ok = cw.watch_flm(live_archs, flm_state(), fetch=fail_fetch, answers=NO_ANSWERS) check("flm failure: reported as a failure, not as nothing new", dead_ok is False and dead_rc == 0 and "FLM WATCH FAILED" in err.getvalue()) check("flm failure: combined exit code is 2, so the daily job notices", @@ -959,6 +963,149 @@ def _fake_urlopen(req, timeout=0): check("engine#1922 class stays out of the runtime registry", _DS not in archs) check("engine#1922 entry records its model_type", sig.get(_DS, {}).get("model_type") == "deepseek_v4_flash_superexpert") + +# 27. engine#2171: an answered gap stays answered. CI keeps no state, so every scheduled run sees the whole +# FastFlowLM org as new; before registry/flm_answers.json each family docs/npu.md had already answered +# printed `!! NPU FLM` / `!! NEW FLM` again and re-filed the census issue the day after it closed. +# The rows are the 2026-11-11 run's alerting packs (run 43016580831: id and the architecture each declares). +RUN_2171 = [ + ("FastFlowLM/MiniCPM-V-4.7-1B-NPU2", "MiniCPMV4_7ForConditionalGeneration"), + ("FastFlowLM/Hy-Memory-30B-A3B-NPU2", "HYV3ForCausalLM"), + ("FastFlowLM/MiniCPM-OCR-1.0-NPU2", "MiniCPMOcrForConditionalGeneration"), + ("FastFlowLM/MiniCPM-V-4.7-35B-A3B-NPU2", "MiniCPMV4_7ForConditionalGeneration"), + ("FastFlowLM/Nemotron-3.5-Bolt-E4B-DFlash-NPU2", "DFlashDraftModel"), + ("FastFlowLM/Nemotron-3.5-Bolt-E4B-NPU2", "Gemma4ForConditionalGeneration"), + ("FastFlowLM/Hy-MT2-1.8B-NPU2", "HunYuanDenseV1ForCausalLM"), + ("FastFlowLM/Gemma4-12B-IT-NPU2", "Gemma4UnifiedForConditionalGeneration"), + ("FastFlowLM/smolvla-NPU2", None), + ("FastFlowLM/medgemma-1.5-4b-it-NPU2", "gemma3_text"), + ("FastFlowLM/LFM2-2.6B-NPU2", "Lfm2ForCausalLM"), + ("FastFlowLM/LFM2-1.2B-NPU2", "Lfm2ForCausalLM"), + ("FastFlowLM/GPT-OSS-Safeguard-20b-NPU2", "GptOssForCausalLM"), + ("FastFlowLM/GPT-OSS-20B-NPU2", "GptOssForCausalLM"), + ("FastFlowLM/Gemma3-4B-NPU2", "gemma3_text"), + ("FastFlowLM/Gemma3-1B-NPU2", "Gemma3ForCausalLM"), + ("FastFlowLM/Gemma4-E4B-IT-NPU2", "Gemma4ForConditionalGeneration"), + ("FastFlowLM/Gemma4-E2B-IT-NPU2", "Gemma4ForConditionalGeneration"), + ("FastFlowLM/Qwen3.6-35B-A3B-NPU2", "Qwen3_5MoeForConditionalGeneration"), + ("FastFlowLM/medgemma-4b-it-NPU2", "gemma3_text"), + ("FastFlowLM/Translategemma-4B-Instruct-NPU2", "gemma3_text"), + ("FastFlowLM/Qwen2.5-VL-3B-Instruct-NPU2", "Qwen2_5_VLForConditionalGeneration"), + ("FastFlowLM/LFM2.5-1.2B-Thinking-NPU2", "Lfm2ForCausalLM"), + ("FastFlowLM/Phi4-mini-Instruct-NPU2", "Phi3ForCausalLM"), + ("FastFlowLM/LFM2-2.6B-Transcript-NPU2", "Lfm2ForCausalLM"), + ("FastFlowLM/LFM2.5-1.2B-NPU2", "Lfm2ForCausalLM"), + ("FastFlowLM/LFM2-1.2B-Tool-NPU2", "Lfm2ForCausalLM"), + ("FastFlowLM/Whisper-V3-Turbo-NPU2", "WhisperForConditionalGeneration"), + ("FastFlowLM/Embedding-Gemma-300M-NPU2", "Gemma3TextModel"), + ("FastFlowLM/Gemma3-270M-NPU2", "Gemma3ForCausalLM"), + ("FastFlowLM/Gemma3-4B-Text-NPU2", "gemma3_text"), +] + + +def run_row(rid, arch, q4nx=True): + rec = {"id": rid, "lastModified": "2026-11-11T00:00:00.000Z", "tags": [], + "siblings": ([{"rfilename": "model.q4nx"}] if q4nx else [{"rfilename": "README.md"}])} + if arch: + rec["config"] = ({"architectures": [arch]} if arch[0].isupper() else {"model_type": arch}) + return cw.flm_repo(rec, live_archs) + + +_rows2171 = [run_row(rid, arch, q4nx=(arch is not None)) for rid, arch in RUN_2171] +for _answers, _tag in ((NO_ANSWERS, "before"), (None, "after")): + _buf = io.StringIO() + with contextlib.redirect_stdout(_buf): + _rc = cw.report_flm(_rows2171, [], _rows2171, [], answers=_answers) + if _tag == "before": + _before_rc, _before = _rc, _buf.getvalue() + else: + _after_rc, _after = _rc, _buf.getvalue() +check("engine#2171 before: the 2026-11-11 rows alert with no answers (the rule is unchanged)", + _before_rc == 1 and _before.count(" !! NPU FLM ") + _before.count(" !! NEW FLM ") == len(RUN_2171)) +check("engine#2171 after: every one of the 31 rows is answered, the run is quiet", + _after_rc == 0 and " !! " not in _after + and _after.count(" . NPU FLM ") + _after.count(" . FLM ") == len(RUN_2171)) +check("engine#2171 after: each answered line names the issue that tracks it", + "FastFlowLM/MiniCPM-V-4.7-1B-NPU2" in _after and "answered: engine#1788" in _after + and "answered: engine#1783" in _after and "answered: engine#1786" in _after) +check("engine#2171 after: the summary says how many were answered, not that a family is unserved", + "31 pack(s) already answered" in _after and "does not serve" not in _after) + +_fresh_family = run_row("FastFlowLM/Brandnew-3B-NPU2", "BrandnewForCausalLM") +_buf = io.StringIO() +with contextlib.redirect_stdout(_buf): + _rc = cw.report_flm(_rows2171 + [_fresh_family], [], _rows2171 + [_fresh_family], []) +check("engine#2171: a family with no answer still alerts beside the answered ones", + _rc == 1 and " !! NEW FLM FastFlowLM/Brandnew-3B-NPU2" in _buf.getvalue()) +_served = run_row("FastFlowLM/Qwen3-8B-NPU2", "Qwen3ForCausalLM") +_fake = {"architectures": {"Qwen3ForCausalLM": {"answer": "x", "issue": 1}}, "repos": {}} +check("engine#2171: an answer cannot touch a family the NPU route serves", + cw.answer_for(_served, _fake) is None and cw.flm_line(_served, answers=_fake).startswith("+ FLM ")) +_deriv = cw.flm_repo({"id": "FastFlowLM/Qwen3-8B-gguf", "tags": ["gguf"], "siblings": []}, live_archs) +check("engine#2171: a derivative drop is never answered (it is not a gap)", + cw.answer_for(_deriv, {"architectures": {}, "repos": {"FastFlowLM/Qwen3-8B-gguf": {"answer": "x"}}}) is None) + +# The committed file: every answer names its doc, is for a family that is still a gap, and has a row in docs/npu.md. +_committed = json.load(open(os.path.join(ROOT, "registry", "flm_answers.json"))) +_npu_md = open(os.path.join(ROOT, "docs", "npu.md")).read() +_arch_answers = _committed.get("architectures") or {} +check("flm answers: the committed file parses and is not empty", len(_arch_answers) >= 10) +check("flm answers: each entry says what the answer is and where it is recorded", + all(e.get("answer") and (e.get("doc") or "").startswith("docs/npu.md") + for e in list(_arch_answers.values()) + list((_committed.get("repos") or {}).values()))) +check("flm answers: a tracking issue is a number", + all(isinstance(e["issue"], int) for e in _arch_answers.values() if "issue" in e)) +_stale = [a for a in _arch_answers + if "npu" in ((live_archs.get(cw.resolve_family(a, live_archs) or "") or {}).get("backends") or [])] +check(f"flm answers: no answer is for a family the registry serves on the NPU route (stale: {_stale})", not _stale) +_undocumented = [a for a in list(_arch_answers) + list(_committed.get("repos") or {}) if f"`{a}`" not in _npu_md] +check(f"flm answers: every answered architecture has a row in docs/npu.md (missing: {_undocumented})", + not _undocumented) + +# 21b. engine#2171: a pin that names its `paths` moves only when a commit touches one of them. +# vllm-project/semantic-router lands dozens of router-side commits a day; the d3 contract it is pinned for +# did not change between a9219240e and 15b4fb5ad, yet the whole-repo head alerted on every run. +PATH_PINS = {"repos": {"vllm-project/semantic-router": { + "commit": "pin000", "reviewed": "2026-11-11", "why": "d3", + "paths": ["src/model-runtime/vllm_srun/families/vela3_d3", "config/schemas/systemone"]}}} + + +def path_fetch(head="head99", touched=()): + """Head `head`; the last commit touching each path is the same at pin and head unless the path is in `touched`.""" + def fetch(url): + if "&path=" in url: + sha = url.split("sha=")[1].split("&")[0] + path = urllib.parse.unquote(url.split("path=")[1].split("&")[0]) + return [{"sha": ("new-" + path) if (sha == head and path in touched) else ("old-" + path)}] + if "/commits" in url: + return [{"sha": head, "commit": {"committer": {"date": "2026-11-11T00:00:00Z"}}}] + return [] + return fetch + + +import urllib.parse # noqa: E402 +buf = io.StringIO() +with contextlib.redirect_stdout(buf): + p_rc, p_ok = cw.watch_support(PATH_PINS, fetch=path_fetch()) +check("support paths: a head that moved without touching a watched path is green", p_ok and p_rc == 0) +check("support paths: ...and the line says why it is green", + "no commit since the pin touches the 2 watched path(s)" in buf.getvalue() + and "!! SUPPORT MOVED" not in buf.getvalue()) +buf = io.StringIO() +with contextlib.redirect_stdout(buf): + p_rc, _ = cw.watch_support(PATH_PINS, fetch=path_fetch(touched=("config/schemas/systemone",))) +check("support paths: a commit touching a watched path alerts and names the path", + p_rc == 1 and "!! SUPPORT MOVED vllm-project/semantic-router" in buf.getvalue() + and "watched paths changed: config/schemas/systemone" in buf.getvalue()) +with contextlib.redirect_stdout(io.StringIO()): + p_rc, _ = cw.watch_support(PATH_PINS, fetch=path_fetch(head="pin000")) +check("support paths: a head equal to the pin reads no paths and is green", p_rc == 0) +check("support paths: a pin without `paths` still moves on any commit (the converter's rule is unchanged)", + cw.watch_support(PINS, fetch=support_fetch(router="zzz999"))[0] == 1) +check("support paths: the committed router pin watches the d3 family and the System One contract", + any("vela3_d3" in x for x in committed["repos"]["vllm-project/semantic-router"].get("paths", [])) + and any("systemone" in x for x in committed["repos"]["vllm-project/semantic-router"].get("paths", []))) + if failures: print(f"\n{len(failures)} check(s) failed") sys.exit(1) diff --git a/tools/census_watch.py b/tools/census_watch.py index f2672fc51..14e5b9c96 100644 --- a/tools/census_watch.py +++ b/tools/census_watch.py @@ -79,6 +79,16 @@ FastFlowLM files, sizes, licenses and tags only — nothing about our NPU kernels or their speed. +A gap that has been answered stays answered (engine#2171). CI keeps no state between runs, so +every scheduled run sees the whole org as new, and each `!! NPU FLM` / `!! NEW FLM` family that +docs/npu.md had already answered alerted again every day, re-filing the census issue as soon as +the last one closed. `registry/flm_answers.json` is the machine-readable form of those answers: +a pack whose declared architecture (or, for a config-less repo, whose id) has an entry prints +`. NPU FLM` / `. FLM` with the issue that tracks it and does not alert. An answer only quiets a +gap: once the registry serves the family on the NPU route the line is `+ FLM` again, and the +test fails on a committed answer for a family that is no longer a gap, so a stale answer is +removed rather than kept. A family with no entry still alerts on its first sighting. + State (the model ids already classified) lives outside the checkout, at $CENSUS_WATCH_STATE (default ~/.1bit/census-watch-state.json), so a run never dirties the tree it reads. A model whose config is missing (a gated repo) is reported the first @@ -118,6 +128,14 @@ # fact, while "a new commit appeared" would be true on every run. SUPPORT_PINS_PATH = os.path.join(ROOT, "registry", "support_pins.json") GH_REPO_HEAD = "https://api.github.com/repos/{repo}/commits?per_page=1" +# A pin may name the paths the NPU route depends on (`paths`). Then "moved" means a commit that +# touches one of them, not any commit: vllm-project/semantic-router lands dozens of router-side +# commits a day, so the whole-repo head moved on every run and re-filed the census issue daily +# while the d3 contract it is pinned for stayed byte-identical (engine#2171). The last commit +# touching a path is read at the pin and at the head; a difference is a move. +GH_PATH_HEAD = "https://api.github.com/repos/{repo}/commits?sha={sha}&path={path}&per_page=1" +# Recorded answers for FastFlowLM packs this engine does not serve (engine#2171). +FLM_ANSWERS_PATH = os.path.join(ROOT, "registry", "flm_answers.json") HF_ORG_URL = "https://huggingface.co/api/models?author={org}&sort=createdAt&direction=-1&limit=100" GH_RELEASES_LIMIT = 20 # Words that mean a release notes body or an asset name talks about the Q4NX container: @@ -553,6 +571,46 @@ def reviewed_report_classes(): return _REVIEWED_CACHE["v"] +_ANSWERS_CACHE = {} + + +def flm_answers(path=None): + """`registry/flm_answers.json` as {"architectures": {...}, "repos": {...}} (empty when absent).""" + path = path or FLM_ANSWERS_PATH + if path not in _ANSWERS_CACHE: + try: + with open(path) as f: + data = json.load(f) + except FileNotFoundError: + data = {} + _ANSWERS_CACHE[path] = {"architectures": data.get("architectures") or {}, + "repos": data.get("repos") or {}} + return _ANSWERS_CACHE[path] + + +def answer_for(repo, answers=None): + """The recorded answer for a repo that would alert, else None (engine#2171). + + Only a gap can be answered: a repo the NPU route serves, or a derivative drop, gets None + whatever the file says, so an answer cannot hide a family that starts to load. The repo id + is looked up first (a config-less repo has no architecture to key on), then the + architecture the repo declares. + """ + if not (npu_gap(repo) or class_unmapped(repo)): + return None + answers = answers if answers is not None else flm_answers() + hit = (answers.get("repos") or {}).get(repo["id"]) + if hit is None and repo["architecture"]: + hit = (answers.get("architectures") or {}).get(repo["architecture"]) + return hit + + +def answer_note(answer): + """The short form printed on the line: the tracking issue, else the doc that records it.""" + issue = answer.get("issue") + return f"answered: engine#{issue}" if issue else f"answered: {answer.get('doc') or 'registry/flm_answers.json'}" + + def class_unmapped(repo): """True when `registry/architectures.json` does not map the repo's architecture class.""" return not repo["derivative"] and not repo["mapped"] @@ -669,7 +727,7 @@ def gib(size): return f"{size / float(1024 ** 3):.2f} GiB" -def flm_line(repo, reviewed=None): +def flm_line(repo, reviewed=None, answers=None): """One repo, one line: files, license, base model, architecture, both mapping answers. `reviewed` is `reviewed_report_classes()`: a class recorded there as not-an-alias is printed as @@ -699,8 +757,15 @@ def flm_line(repo, reviewed=None): parts.append("derivative drop (no config by design)") seen = (reviewed if reviewed is not None else reviewed_report_classes()) decl = class_unmapped(repo) and repo["architecture"] in seen - marker = ("!! NPU FLM" if npu_gap(repo) - else (". FLM" if decl else ("!! NEW FLM" if unmapped(repo) else "+ FLM"))) + answer = answer_for(repo, answers) + if answer is not None: + parts.append(answer_note(answer)) + if npu_gap(repo): + marker = ". NPU FLM" if answer is not None else "!! NPU FLM" + elif decl or (answer is not None and unmapped(repo)): + marker = ". FLM" + else: + marker = "!! NEW FLM" if unmapped(repo) else "+ FLM" return f"{marker} {repo['id']} ({repo['lastModified'][:10]}) " + " ".join(parts) @@ -711,19 +776,22 @@ def release_line(release): + f"assets {len(release['assets'])}") -def report_flm(repos, releases, fresh_repos, fresh_releases, full=False): - """Print the FastFlowLM delta and return its exit code (1 on an unserved family).""" +def report_flm(repos, releases, fresh_repos, fresh_releases, full=False, answers=None): + """Print the FastFlowLM delta and return its exit code (1 on an unserved, unanswered family).""" shown_repos = repos if full else fresh_repos shown_releases = releases if full else fresh_releases reviewed = reviewed_report_classes() + answers = answers if answers is not None else flm_answers() + answered = [repo["id"] for repo in shown_repos if answer_for(repo, answers) is not None] unmapped_repos = [repo["id"] for repo in shown_repos - if class_unmapped(repo) and repo["architecture"] not in reviewed] + if class_unmapped(repo) and repo["architecture"] not in reviewed + and repo["id"] not in answered] # Split the two reasons in the summary: "map the class" and "give the NPU route this # family" are different pieces of work, and a Q4NX pack named for an HRX-only family # is the second one. gaps = {} for repo in shown_repos: - if npu_gap(repo): + if npu_gap(repo) and repo["id"] not in answered: family = repo["registry_gguf"] or repo["registry_class"] or repo["architecture"] gaps.setdefault(family, []).append(repo["id"]) flagged = [r["tag"] for r in shown_releases if r["kernel_format"]] @@ -731,7 +799,7 @@ def report_flm(repos, releases, fresh_repos, fresh_releases, full=False): f"{len(releases)} release(s) checked; {len(fresh_repos)} new/changed repo(s), " f"{len(fresh_releases)} new release(s)" + ("; full listing" if full else "")) for repo in shown_repos: - print(" " + flm_line(repo, reviewed)) + print(" " + flm_line(repo, reviewed, answers)) for release in shown_releases: print(" " + release_line(release)) if not shown_repos and not shown_releases: @@ -745,6 +813,9 @@ def report_flm(repos, releases, fresh_repos, fresh_releases, full=False): f"route does not serve: {detail}") print(" mapped for another backend only, so the census counts the family as covered: " "the NPU route (or a recorded decision that it will not serve it) is what is missing") + if answered: + print(f" -> {len(answered)} pack(s) already answered in registry/flm_answers.json (listed, not " + "alerted; the issue on each line tracks the work)") if flagged: print(f" -> release(s) {', '.join(flagged)} name the Q4NX container format " f"({', '.join(KERNEL_FORMAT_MARKERS)}): read the notes before the pinned " @@ -752,7 +823,7 @@ def report_flm(repos, releases, fresh_repos, fresh_releases, full=False): return 1 if (unmapped_repos or gaps) else 0 -def watch_flm(archs, state, limit=FLM_LIMIT, full=False, fetch=None, detail=None): +def watch_flm(archs, state, limit=FLM_LIMIT, full=False, fetch=None, detail=None, answers=None): """Sweep the org + releases, print the delta, remember it: (exit code, did it run?)""" try: repos = sweep_org(archs, limit, fetch) @@ -763,7 +834,7 @@ def watch_flm(archs, state, limit=FLM_LIMIT, full=False, fetch=None, detail=None fresh_repos, fresh_releases = flm_deltas(state, repos, releases) for repo in (repos if full else fresh_repos): fill_sizes(repo, detail) - rc = report_flm(repos, releases, fresh_repos, fresh_releases, full) + rc = report_flm(repos, releases, fresh_repos, fresh_releases, full, answers) remember_flm(state, repos, releases) return rc, True @@ -827,15 +898,27 @@ def sweep_support(pins, fetch=None): fetch = fetch or get_json heads = {} for repo, entry in sorted((pins.get("repos") or {}).items()): + entry = entry or {} commit = (fetch(GH_REPO_HEAD.format(repo=repo)) or [{}])[0] or {} sha = commit.get("sha") or "" + pinned = entry.get("commit") or "" + paths = list(entry.get("paths") or ()) + moved_paths = [] + if paths and sha and sha != pinned: + for path in paths: + at_pin = (fetch(GH_PATH_HEAD.format(repo=repo, sha=pinned, path=path)) or [{}])[0] or {} + at_head = (fetch(GH_PATH_HEAD.format(repo=repo, sha=sha, path=path)) or [{}])[0] or {} + if (at_pin.get("sha") or "") != (at_head.get("sha") or ""): + moved_paths.append(path) heads[repo] = { "repo": repo, "commit": sha, "date": ((commit.get("commit") or {}).get("committer") or {}).get("date") or "", - "pinned": (entry or {}).get("commit") or "", - "reviewed": (entry or {}).get("reviewed") or "", - "moved": sha != ((entry or {}).get("commit") or ""), + "pinned": pinned, + "reviewed": entry.get("reviewed") or "", + "paths": paths, + "moved_paths": moved_paths, + "moved": (bool(moved_paths) if paths else sha != pinned), } sources = pins.get("d3_sources") or {} org, prefix = sources.get("org") or "", sources.get("prefix") or "" @@ -855,7 +938,12 @@ def report_support(heads, new_d3, org="", prefix=""): for h in sorted(heads.values(), key=lambda x: x["repo"]): if h["moved"]: print(f" !! SUPPORT MOVED {h['repo']} head {h['commit'][:12]} ({h['date'][:10]}) " - f"pinned {h['pinned'][:12]} reviewed {h['reviewed']}") + f"pinned {h['pinned'][:12]} reviewed {h['reviewed']}" + + (f" watched paths changed: {', '.join(h['moved_paths'])}" if h.get("moved_paths") else "")) + elif h.get("paths") and h["commit"] != h["pinned"]: + print(f" + SUPPORT PIN {h['repo']} head {h['commit'][:12]} ({h['date'][:10]}) pinned " + f"{h['pinned'][:12]} reviewed {h['reviewed']}: no commit since the pin touches the " + f"{len(h['paths'])} watched path(s)") else: print(f" + SUPPORT PIN {h['repo']} {h['commit'][:12]} ({h['date'][:10]}) reviewed {h['reviewed']}") if moved: From 8625695ba4aef615f441e394085af660a76af2b0 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 10:21:39 -0400 Subject: [PATCH 2/5] census_watch: a watched path with no commit at the pin alerts (engine#2171) Review on #2189 (f993a57): a `paths` entry that matches nothing (a typo, or a directory renamed before the pin) gives [] at both commits, so that pin could never move and nothing said so. sweep_support now reads each watched path at the pin on every run (also when the head equals the pin), and a path with no commit there marks the pin moved with "watched paths with no commit at the pin (fix `paths`)". Tests 260 -> 262; the mutant that drops the check fails 2. The live run on the committed pins is green. Co-Authored-By: Claude Opus 5.5 --- tests/census_watch_test.py | 16 +++++++++++++--- tools/census_watch.py | 19 ++++++++++++++----- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/tests/census_watch_test.py b/tests/census_watch_test.py index d52f5a8c9..9966c2c6b 100644 --- a/tests/census_watch_test.py +++ b/tests/census_watch_test.py @@ -1070,12 +1070,15 @@ def run_row(rid, arch, q4nx=True): "paths": ["src/model-runtime/vllm_srun/families/vela3_d3", "config/schemas/systemone"]}}} -def path_fetch(head="head99", touched=()): - """Head `head`; the last commit touching each path is the same at pin and head unless the path is in `touched`.""" +def path_fetch(head="head99", touched=(), dead=()): + """Head `head`; the last commit touching each path is the same at pin and head unless the path is in `touched`. + A path in `dead` has no commit at all (the API answers []).""" def fetch(url): if "&path=" in url: sha = url.split("sha=")[1].split("&")[0] path = urllib.parse.unquote(url.split("path=")[1].split("&")[0]) + if path in dead: + return [] return [{"sha": ("new-" + path) if (sha == head and path in touched) else ("old-" + path)}] if "/commits" in url: return [{"sha": head, "commit": {"committer": {"date": "2026-11-11T00:00:00Z"}}}] @@ -1099,7 +1102,14 @@ def fetch(url): and "watched paths changed: config/schemas/systemone" in buf.getvalue()) with contextlib.redirect_stdout(io.StringIO()): p_rc, _ = cw.watch_support(PATH_PINS, fetch=path_fetch(head="pin000")) -check("support paths: a head equal to the pin reads no paths and is green", p_rc == 0) +check("support paths: a head equal to the pin is green", p_rc == 0) +for _head in ("head99", "pin000"): + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + p_rc, _ = cw.watch_support(PATH_PINS, fetch=path_fetch(head=_head, dead=("config/schemas/systemone",))) + check(f"support paths: a watched path with no commit at the pin alerts and names it (head {_head})", + p_rc == 1 and "!! SUPPORT MOVED vllm-project/semantic-router" in buf.getvalue() + and "no commit at the pin (fix `paths`): config/schemas/systemone" in buf.getvalue()) check("support paths: a pin without `paths` still moves on any commit (the converter's rule is unchanged)", cw.watch_support(PINS, fetch=support_fetch(router="zzz999"))[0] == 1) check("support paths: the committed router pin watches the d3 family and the System One contract", diff --git a/tools/census_watch.py b/tools/census_watch.py index 14e5b9c96..5b83ee5ba 100644 --- a/tools/census_watch.py +++ b/tools/census_watch.py @@ -903,12 +903,18 @@ def sweep_support(pins, fetch=None): sha = commit.get("sha") or "" pinned = entry.get("commit") or "" paths = list(entry.get("paths") or ()) - moved_paths = [] - if paths and sha and sha != pinned: + moved_paths, dead_paths = [], [] + if paths and sha: for path in paths: at_pin = (fetch(GH_PATH_HEAD.format(repo=repo, sha=pinned, path=path)) or [{}])[0] or {} + if not at_pin.get("sha"): + # no commit at the pin touches it (a typo, or renamed before the pin): it could never move + dead_paths.append(path) + continue + if sha == pinned: + continue at_head = (fetch(GH_PATH_HEAD.format(repo=repo, sha=sha, path=path)) or [{}])[0] or {} - if (at_pin.get("sha") or "") != (at_head.get("sha") or ""): + if at_pin["sha"] != (at_head.get("sha") or ""): moved_paths.append(path) heads[repo] = { "repo": repo, @@ -918,7 +924,8 @@ def sweep_support(pins, fetch=None): "reviewed": entry.get("reviewed") or "", "paths": paths, "moved_paths": moved_paths, - "moved": (bool(moved_paths) if paths else sha != pinned), + "dead_paths": dead_paths, + "moved": (bool(moved_paths or dead_paths) if paths else sha != pinned), } sources = pins.get("d3_sources") or {} org, prefix = sources.get("org") or "", sources.get("prefix") or "" @@ -939,7 +946,9 @@ def report_support(heads, new_d3, org="", prefix=""): if h["moved"]: print(f" !! SUPPORT MOVED {h['repo']} head {h['commit'][:12]} ({h['date'][:10]}) " f"pinned {h['pinned'][:12]} reviewed {h['reviewed']}" - + (f" watched paths changed: {', '.join(h['moved_paths'])}" if h.get("moved_paths") else "")) + + (f" watched paths changed: {', '.join(h['moved_paths'])}" if h.get("moved_paths") else "") + + (f" watched paths with no commit at the pin (fix `paths`): {', '.join(h['dead_paths'])}" + if h.get("dead_paths") else "")) elif h.get("paths") and h["commit"] != h["pinned"]: print(f" + SUPPORT PIN {h['repo']} head {h['commit'][:12]} ({h['date'][:10]}) pinned " f"{h['pinned'][:12]} reviewed {h['reviewed']}: no commit since the pin touches the " From 7c0f8452a944c1ba195e9e3df8691f6739e7a7e5 Mon Sep 17 00:00:00 2001 From: agent Date: Wed, 11 Nov 2026 12:40:35 -0400 Subject: [PATCH 3/5] census: OpenSci is a reviewed remote-code class (engine#2171, folded in from #2207) Folded into this PR rather than left as a second open PR for the same issue, under the 16:45Z lane cap (at most two open PRs per lane) -- the change itself is unchanged from #2207. The watcher reported OpensciForCausalLM as an uncovered architecture class. It is not a mapping gap: the model's config carries an auto_map pointing at its own modeling_opensci.py / configuration_opensci.py, so the pinned backends cannot accept the architecture without that custom code. Reviewed and deliberately not mapped, which is what registry/significant.json records -- the census then counts those models separately and excludes them from top_unmapped, because a recorded decision is not work remaining. Confirmed from the source, not the class name: architectures ["OpensciForCausalLM"], model_type "opensci", auto_map AutoModelForCausalLM -> modeling_opensci.OpensciForCausalLM. The other spelling the census carries (OpenSciForCausalLM, 7 models against 121) is deliberately NOT waved through here: exact-key matching is a stated design of census_watch.py ("there is no strip_arch either -- architectures.json is keyed by the HF architecture class, which is the key"), so it is a second reviewed entry needing the repo list from a census run, not a case-fold. tools/registry_build.py --check-gaps: none of the 38 reviewed significant classes is mapped without a reason (rc=0). Refs #2171 --- registry/significant.json | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/registry/significant.json b/registry/significant.json index 75ac3b192..f04811af1 100644 --- a/registry/significant.json +++ b/registry/significant.json @@ -172,6 +172,11 @@ "model_type": "deepseek_v4_flash_superexpert", "family": "DeepSeek V4.1 Flash SuperExpert, published as remote code and with AW-SECD-compressed weights (8.5 GB, claimed 0.962 retention); MLA-style geometry (kv_lora_rank 512, q_lora_rank 1536, qk_rope_head_dim 64, v_head_dim 128, 32 heads / 16 KV, IM 4096). Not an alias of deepseek_v41: nothing at the pin decodes AW-SECD weights, so mapping it would name a backend that cannot read it", "watch": "report" + }, + "OpensciForCausalLM": { + "model_type": "opensci", + "family": "OpenSci (1.7B 'nemotron-hq' recipe). A REMOTE-CODE class: its config carries an auto_map pointing at modeling_opensci.py / configuration_opensci.py, so the pinned backends cannot accept the architecture without that custom code. Not an alias of any supported family, and not mappable until the code is vendored or upstreamed.", + "watch": "report" } } } From 8c97bb0ab3706eb31a881fb1f1085c5052ccad6a Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 13:11:41 -0400 Subject: [PATCH 4/5] docs/arch-gaps: record OpensciForCausalLM, so the folded-in #2207 entry passes (engine#2171) 7c0f845 added OpensciForCausalLM to registry/significant.json without a row here. tests/census_watch_test.py requires every reviewed class to be cited in this file, so the head failed 1 check ("every one of the 38 reviewed classes is cited in docs/arch-gaps.md -- missing: OpensciForCausalLM"). The DNS-red CI hid it. The row records the remote-code reason, from the config of open-sci/open-sci-1.7b-nemotron-cc-1T-oh-dcft (auto_map to modeling_opensci), and notes that OpenSciForCausalLM is a separate exact key. Tests: 1 FAIL -> 262/262. Co-Authored-By: Claude Opus 5.5 --- docs/arch-gaps.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/arch-gaps.md b/docs/arch-gaps.md index aa1d2cb0a..40bda945e 100644 --- a/docs/arch-gaps.md +++ b/docs/arch-gaps.md @@ -1087,3 +1087,14 @@ the single undocumented entry out of 37. MLA-style geometry (`kv_lora_rank` 512, `q_lora_rank` 1536, `qk_rope_head_dim` 64, `v_head_dim` 128, 32 heads / 16 KV, IM 4096). **Not an alias of `deepseek_v41`**: nothing at the pin decodes AW-SECD weights, so mapping it to that architecture would name a backend that cannot read the file. Reported, not alerted (`"watch": "report"`). + +## 2026-11-11: `OpensciForCausalLM` — reviewed remote code (engine#2171) + +* `OpensciForCausalLM` (`model_type` `opensci`, 121 models in the 2026-11-11 census) — the open-sci reference models + (for example `open-sci/open-sci-1.7b-nemotron-cc-1T-oh-dcft`: hidden 2048, 24 layers, 32 heads / 32 KV, + `qk_layernorm`, vocab 50304). **Published as remote code**: `config.json` carries an `auto_map` to its own + `configuration_opensci.OpensciConfig` and `modeling_opensci.OpensciForCausalLM`, so no pinned backend accepts the + class without that code. **Not an alias** of any supported family. Reported, not alerted (`"watch": "report"`). + The other spelling, `OpenSciForCausalLM` (7 models, same `model_type` and the same `auto_map`), is a separate exact + key: `census_watch.py` matches architecture classes exactly, so that spelling needs its own entry and is not covered + by this one. From a0d5b0442561e331eec73060a663326da8dead89 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 13:30:43 -0400 Subject: [PATCH 5/5] flm_answers: an answer also goes when its tracking issue closes unserved (engine#2171) Review at 8c97bb0: the staleness rule covered one edge only. The test fails on an answer for a family the registry now serves, but an answer whose issue (#1780/#1783/#1786/#1788) closes without serving the family would keep printing ". NPU FLM" with a link to a closed issue. No test can see that, so `about` now states the human step. Text-only change; tests 262/262. Co-Authored-By: Claude Opus 5.5 --- registry/flm_answers.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/registry/flm_answers.json b/registry/flm_answers.json index e4e50bb92..b1ca96217 100644 --- a/registry/flm_answers.json +++ b/registry/flm_answers.json @@ -1,5 +1,5 @@ { - "about": "Recorded answers for FastFlowLM packs this engine does not serve, read by tools/census_watch.py (engine#2171). CI keeps no watcher state, so without this file every scheduled run re-alerted on families docs/npu.md had already answered. A pack whose declared architecture (or, for a repo with no config, whose id) is listed here prints `. NPU FLM` / `. FLM` with the issue that tracks it and does not alert. Hand-written: add an entry only with a row for it in docs/npu.md, and remove it when the family is served (tests/census_watch_test.py fails on an answer for a family the registry serves on the NPU route).", + "about": "Recorded answers for FastFlowLM packs this engine does not serve, read by tools/census_watch.py (engine#2171). CI keeps no watcher state, so without this file every scheduled run re-alerted on families docs/npu.md had already answered. A pack whose declared architecture (or, for a repo with no config, whose id) is listed here prints `. NPU FLM` / `. FLM` with the issue that tracks it and does not alert. Hand-written: add an entry only with a row for it in docs/npu.md, and remove it when the family is served (tests/census_watch_test.py fails on an answer for a family the registry serves on the NPU route) or when the issue that tracks it closes without serving it (no test can see that: check the `issue` numbers here whenever one of them closes).", "architectures": { "MiniCPMV4_7ForConditionalGeneration": { "answer": "not served: a vision-language pack whose class the pin does not register; serving it on the NPU route is engine#1788",