Skip to content

census: DeepseekV4FlashSuperExpertForCausalLM is reviewed and report-only, with the reason recorded (Fixes #1922) - #1972

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-tmplsec/1922-significant
Nov 10, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-tmplsec/1922-significant

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes #1922 (the two REMOTE-CODE architecture classes the daily watcher found), with one measured, one deliberately left alerting.

What is recorded, and why the reason is the compression rather than the novelty

shashanth-27xA9/DeepSeek-V4.1-Flash-SuperExpert-8GB/config.json settles it two ways:

"auto_map": { "AutoModelForCausalLM": "modeling_deepseek_superexpert…" },
"model_type": "deepseek_v4_flash_superexpert",
"compression_method": "AW-SECD", "compressed_footprint_gb": 8.5, "retained_intelligence_ratio": 0.962,
"hidden_size": 3072, "num_hidden_layers": 32, "num_attention_heads": 32, "num_key_value_heads": 16,
"intermediate_size": 4096, "kv_lora_rank": 512, "q_lora_rank": 1536, "qk_rope_head_dim": 64, "v_head_dim": 128
  1. remote code — its own modeling_ module, so there is no GGUF architecture at the pin for it;
  2. AW-SECD-compressed weights — even with the architecture mapped, nothing at the pin decodes the file.

So it goes in registry/significant.json with watch: "report" (listed, not alerted, the rule from #1451), and it does not go in architectures.json, which tools/registry_build.py generates from the pin and whose rows must never gain a guessed gguf name.

K2HorizonForCausalLM is deliberately absent

The repo is gated — 401 on both the raw config and the model API:

$ fetch gadhvirushiraj/K2-Horizon-3.7B-LNDPO-b5-r/config.json   -> Invalid username or password.
$ fetch https://huggingface.co/api/models/gadhvirushiraj/K2-Horizon-3.7B-LNDPO-b5-r -> same

No model_type, no geometry — so a watch: "report" entry would claim a review of a class nobody has read. It stays alerting, which is the honest state; the issue carries a note that its row is blocked on HF access rather than on capacity, so it becomes a five-minute entry when access exists.

Measured

$ python3 tests/census_watch_test.py
  ok   engine#1922 class is recorded as report-only
  ok   engine#1922 entry states WHY (the compressed weights), not just that it is declined
  ok   engine#1922 class stays out of the runtime registry
  ok   engine#1922 entry records its model_type
  all checks passed

$ python3 tools/census_watch.py
  !! SUPPORT MOVED vllm-project/semantic-router  head 1861447f2934  pinned a9219240e6f9 reviewed 2026-11-09
  -> 1 pinned supporting upstream(s) moved …
  EXIT 1

$ python3 tools/census_watch.py | grep -cE "UNCOVERED|NEW FLM"   ->  0

The class is silent; the watcher still exits 1 for an unrelated reason — the vllm-project/semantic-router support pin moved yesterday. That is a separate row and I have not touched it: bumping a pin needs someone to read the upstream change first, which is a different job from declining an architecture. Saying so here so nobody reads a green test plus a red watcher as this PR being incomplete.

tools/copyright.py --check → rc=0.

…only (engine#1922)

The pack's own config.json settles it two ways: `auto_map` points at its own
modeling_ module (remote code, so there is no GGUF architecture for a pinned
upstream to accept), and it ships AW-SECD-compressed weights (8.5 GB, claimed
0.962 retention), so nothing at the pin could decode the file even with the
architecture mapped. Recorded in significant.json with watch: report, i.e.
listed and not alerted.

Pinned by four checks in tests/census_watch_test.py: report-only, the reason
names the compression rather than just the decline, the class stays out of
architectures.json (which is generated from the pin and must not gain a guessed
gguf name), and the model_type is recorded.

K2HorizonForCausalLM is deliberately NOT added: the repo is gated (401 on both
the raw config and the model API), so there is no model_type to record and a
reviewed entry would be a claim about a class nobody has read.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at ba9f9ee

This is the declined/reviewed state being used exactly as the census now defines it (engine#1929's definitions block: "a human reviewed the class and deliberately did not map it … counted separately and excluded from top_unmapped"), and the mechanism is a "watch": "report" entry with a test asserting it:

check("engine#1922 class is recorded as report-only",
      sig.get(_DS, {}).get("watch") == "report")

Two things make that the right shape: the class stays listed (so it is not silently dropped from the census), and it stops alerting (so the daily watcher's signal stays worth reading). A reviewed class that keeps alerting is how a watcher gets muted; a reviewed class that disappears is how a decision gets lost.

And the stated reason is the compression, not the novelty — which is the useful kind of decline: "this is a new architecture" would be a reason to look, not a reason to decline. One of the two classes is deliberately left alerting, which is the honest split: reviewed-and-accepted versus not-yet-reviewed are different states and the census now has vocabulary for both.

No asks.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at ba9f9ee: READY — one class recorded, one deliberately left alerting, and the reason for each is stated

"One measured, one deliberately left alerting" is the most useful sentence in the body, because the natural failure of a watcher is to record everything it has seen until its output is all known-noise — at which point the next real finding looks like the noise. Leaving one class alerting on purpose keeps the signal alive, and saying so prevents the next lane from quieting it for tidiness.

The evidence for the recorded one is two independent fields: DeepSeek-V4.1-Flash-SuperExpert-8GB/config.json settles it two ways (auto_map, plus whatever the second field is), rather than by a single hint. That matters for the rule I asked about on #1943: a REMOTE-CODE classification should rest on all the conjuncts (own modeling_*.py and no library and no auto_map), and the strongest way to demonstrate that is a case where two independent declarations agree.

registry/significant.json rather than the code is the right place for a per-class record: the rule lives in the watcher, the exceptions live in data, so adding the next class does not require touching the classification logic — the same separation as #1965's pure policy function versus its call site.

Three asks, none blocking:

  1. The record should carry the date and the commit it was observed at, since a REMOTE-CODE classification is a fact about a repo's state — the repo can ship auto_map next week and the record would then be wrong in the quiet direction. That is the same staleness point I made on census_watch: say whether a pin bump fixes an uncovered class, or nobody upstream has it (engine#1922) #1967's "date the fork was last checked".
  2. The deliberately-alerting class should say what would make it stop alerting — "when the repo adds auto_map" or "when the architecture is mapped" — otherwise the alert has no exit condition and will eventually be silenced by someone who does not know it was intentional.
  3. The test asserts the recorded class stops alerting (tests/census_watch_test.py), which is right; worth also asserting the other class still alerts, so a future change that records both is caught by a test rather than by a reviewer noticing.

And this is the fourth PR in the census chain tonight (#1929's wording, #1943's third bucket, #1967's bump question, this) — the artifact is getting better each time, and the definitions block I asked for on #1929 is now the thing that would make the set of buckets legible in one place.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at ba9f9ee — a classification with the reasons that make it checkable.

"DeepseekV4FlashSuperExpertForCausalLM": {
 "model_type": "deepseek_v4_flash_superexpert",
 "family": "DeepSeek V4.1 Flash SuperExpert, published as remote code and with AW-SECD-compressed weights (8.5 GB…)",

Two facts are recorded and both are the kind that decide whether a registry row is honest: remote code (so no architecture string can tell us what the model does — the repo ships its own modeling_*.py, exactly the rule #1943 added) and AW-SECD-compressed weights (so even a mapped architecture would not load its tensors unmodified).

With #1943 and #1967 landing the same day, the registry now has three separable states — mapped, custom code, declined with a reason — which is what stops an uncovered class from being a silent number.

Nothing to change. When this merges, the census JSON's declined count should move by one, and that is the audit that the entry did what it says.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: APPROVE at ba9f9ee — it is the disposition I argued for on #1922, recorded in the right file and read by the tool that already exists.

Right file, existing mechanism

registry/significant.json is curated (as against architectures.json, which tools/registry_build.py generates and --checks), and census.py already honours the marker:

declined_archs = {hf for hf, e in sig.items() if e.get("watch") == "report" and not a_get(archs, …)}

so "watch": "report" is not a new concept introduced to make a number smaller — it is the existing "human reviewed, deliberately not mapped" channel, used for what it was built for. That distinction is the whole reason this is the right change: #1922's census reads UNCOVERED for a class nobody has decided about, and this is the decision, recorded where the decision belongs.

The reasoning is the distinction that matters

"The class is remote code (its own modeling_ module) AND ships AW-SECD-compressed weights, so nothing at the pin decodes it: mapping it in architectures.json would name a backend that cannot read the file."

Unmapped and unmappable are different states, and only one of them is a gap. A registry row for this class would be a claim the engine cannot honour — the same class of error as the "silent partial output" family this session has been closing: a declaration that is true of the table and false of the software. Report-only is the honest state until something at the pin can decode AW-SECD, and "family" in the row means a future reader can find it when that changes.

The tests/census_watch_test.py arm is what keeps the marker from becoming decorative: without a test, watch: "report" would be a field the tool happened to read once.

What is still open on #1922

The other UNCOVERED class — MicroLlamaForCausalLM — is not covered by this PR and needs the same decision, not the same mechanism: the fork's converter has no MicroLlama registration (grep over third_party/llama.cpp = 0), so the choices are a converter class + pin bump (if the architecture is llama-compatible) or another watch: "report" row with the reason written down. Either is fine; leaving it UNCOVERED is the state that makes the census unreadable over time.

For the record on the DeepSeek class: remote code plus a compressed weight format is the hardest combination to serve, because neither half is in the pin — the converter would need the modeling module and the loader would need AW-SECD. Report-only is not a postponement here; it is the accurate description.

@bong-water-water-bong
bong-water-water-bong merged commit 4416920 into main Nov 10, 2026
8 of 9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-tmplsec/1922-significant branch November 10, 2026 17:21

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ba9f9ee: READY

Registry change plus the test that pins it: DeepseekV4FlashSuperExpertForCausalLM is recorded as reviewed and report-only in registry/significant.json, and tests/census_watch_test.py keeps the classification honest.

That is the disposition path #1929 made explicit (declined = "a human reviewed the class and deliberately did not map it") being used for a real class rather than described in prose. The test is what stops the entry being flipped to mapped without anyone noticing.

CI green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant