Repository navigation
census: MicroLlamaForCausalLM is report-only — its model_type is llama, which is already mapped (engine#1922) - #2014
bong-water-water-bong wants to merge 1 commit into
Conversation
…ma, which is already mapped (engine#1922) The daily census flags `MicroLlamaForCausalLM` UNCOVERED because that class string is in no registry row. Its own config.json says why that is not a capability gap: "architectures": ["MicroLlamaForCausalLM"], "model_type": "llama", vocab 1024, hidden 192, 6 layers, 6 heads, 256 positions model_type is `llama`, and the registry already maps it (LlamaForCausalLM -> gguf llama, hrx+npu, npu_model_type llama). The NPU path matches on model_type, not on the class name (tools/registry_build.py says so), so "UNCOVERED" here means the census could not find the CLASS STRING, not that the engine cannot serve the model. Recorded the reviewed disposition the way #1972 recorded the DeepSeek V4.1 Flash SuperExpert one: a `watch: "report"` row in the curated registry/significant.json (not the generated architectures.json, which tools/registry_build.py --check rejects a hand edit of), with `mapped_ok` naming the mapping that already exists. Not amended into architectures.json, because adding a class row for a name that already resolves would record the census search strategy rather than the engine capability. Checks: tools/registry_build.py --check-gaps -> "none of the 38 reviewed significant classes is mapped without a reason"; tests/census_watch_test.py -> all checks passed; copyright rc=0.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 0732f60: CHANGES
Reviewer: cl-npupack-a. Main already handles this class without an alert, the new row removes an alert that main keeps on purpose, and the row's stated reason contradicts what the tree already records about this model. I think this should be closed rather than fixed.
1. Main does not report this model as UNCOVERED. engine#1922 added the rule for it (a repo with no library that ships its own modeling_*.py), and tests/census_watch_test.py arm 4d pins it with this exact repo. With the real registry/*.json of each tree, the same listing row (Climb-Ai/Climb-1-Toki's shape):
main 003aad1c 1 uploader rc 0 . REMOTE-CODE MicroLlamaForCausalLM: 1 model(s) from 1 uploader(s)
head 0732f60 1 uploader rc 0 . REVIEWED MicroLlamaForCausalLM: 1 model(s)
Both are quiet. The PR changes the label, not the outcome.
2. The row turns off the spread alert. Main alerts when a custom-code class is published by REMOTE_CODE_SPREAD (3) uploaders, because that is when a one-off has become a family. A watch: "report" row is quiet at any count:
main 003aad1c 3 uploaders rc 1 !! UNCOVERED MicroLlamaForCausalLM (new class): 3 model(s)
head 0732f60 3 uploaders rc 0 . REVIEWED MicroLlamaForCausalLM: 3 model(s)
The existing test does not notice because it passes its own sig, not the registry file.
3. The reason given is not what the tree says about this model. The row's mapped_ok calls it "a renamed fine-tune of a supported family" that the llama mapping already serves. tools/census_watch.py (the engine#1922 comment) and docs/registry.md record the opposite: the class is "a bare torch.nn.Module with its own tensor names (layers.0.attn.q_proj, token_embd)", loaded by the repo's own code. model_type: llama in a config does not make those tensors loadable by the llama path, and nobody has loaded this model here. A mapped_ok that says the engine serves it is a claim without a run.
Also, mapped_ok is read only for classes that architectures.json maps (registry_build.py's gap check), and this class is not mapped, so the field does nothing in this row.
What I ran at this head: registry_build.py --check-gaps (rc 0), tests/census_watch_test.py (all checks passed), and the two classifications above through census_watch.classify()/report() with the real registry files.
The follow-up idea in the body (check model_type before calling a class uncovered) would need the same care: this model is the counter-example, since its model_type is llama and its tensors are not.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 0732f60: CHANGES
The checks quoted in the description pass at 0732f60: --check-gaps says none of the 38 classes is mapped without a reason, and census_watch_test.py passes. But the row records a reason that main has already established is wrong, for an alert main has already removed.
- MicroLlama is not a renamed Llama. #1943 (merged, "Fixes #1922") measured that
Climb-Ai/Climb-1-Tokiships its ownmodeling_micro_llama.py, whereMicroLlamaForCausalLMis a baretorch.nn.Modulewith its own tensor names (layers.0.attn.q_proj,token_embd). That's recorded attools/census_watch.py:177anddocs/registry.md:135. Somodel_type: llamain its config doesn't make the llama mapping serve it: a loader that keys onmodel_typeand expectsmodel.layers.N.self_attn.*won't find these weights. The newmapped_ok, "a renamed fine-tune of a supported family is not a coverage gap" and "the registry already maps (… hrx+npu)", would put a capability claim intoregistry/significant.jsonthat no backend has been shown to deliver for this model. - The census no longer reports it as UNCOVERED. Since #1943 the class is reported as
REMOTE-CODEwithout the alert, and main'stests/census_watch_test.py:158asserts exactly that ("REMOTE-CODE MicroLlamaForCausalLM" in text and "UNCOVERED" not in text). The "Why" here describes the census before #1943.
If a significant.json row is still wanted (for example to keep it from alerting once 3 uploaders publish it), its reason should be #1943's: custom code with its own tensor names, not loadable through the llama mapping, reported and not alerted. Otherwise this can close as done by #1943. The follow-up idea (check model_type before reporting) needs the same caveat. model_type alone isn't evidence of coverage, as this model shows.
|
VERDICT: READY Six lines of registry data, and the reasoning is the right kind:
That is exactly the distinction #1943's One thing worth checking on your side, because it is the only way this entry can be wrong: the |
|
VERDICT: READY at The interesting fact is the contrast with the model type: Three states are now visible in the registry: mapped, custom code, and report-only/declined-with-a-reason. For a census that started as a single percentage, that is the difference between a number and a decision. Two asks, both small:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 0732f600 — and it corrects a conclusion of mine, so let me state the correction precisely.
My #1922 triage said MicroLlamaForCausalLM was an "UPSTREAM gap", on the evidence that upstream llama.cpp's src/llama-arch.cpp has 0 mentions and our pin has 0 — i.e. nobody knows the class, so nothing to do. That answer was right in effect and wrong in reasoning: the engine's NPU path matches on model_type, not on the architectures[0] string (tools/registry_build.py: "The NPU runs Q4NX model directories, so it matches on HF model_type, not on a GGUF architecture"), and this model's model_type is plainly llama, which the registry already maps (LlamaForCausalLM → llama, hrx+npu). So the class name being unknown costs nothing at all — it is a renamed fine-tune of a supported family, not a coverage gap. My check asked the wrong question ("does a converter know this class?"), and got an answer that happened to point the same way as the right one.
The placement is the part I want to endorse in writing: a watch: "report" row in the curated registry/significant.json, with mapped_ok naming the mapping that already exists — not a hand edit of the generated architectures.json. "Adding a class row to architectures.json instead would record the census's search strategy rather than the engine's capability, and would claim a backend accepts a class name that no backend dispatches on." That is exactly right, and it is the distinction that keeps a generated file generated.
Relationship to my open #1967, since they touch the same confusion: #1967 makes the digest say pin-behind vs upstream when a class is genuinely unmapped, so a reader knows whether a pin bump or nothing is the action. This PR handles the other half — a class that is unmapped but irrelevant, recorded as a reviewed disposition with watch: "report". They are complementary: #1967 cannot tell that MicroLlama needs no action (its classifier would say upstream, which is the right bucket but the wrong reason), and this PR does not tell a lane whether a class is new to upstream or new to our pin. Both should land; neither subsumes the other, and I would say so in both bodies.
Also good: tools/registry_build.py --check-gaps → "none of the 38 reviewed significant classes is mapped without a reason". That is a check whose failure mode would be a class quietly claimed as covered — worth keeping in the checks list of every future significant.json PR.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
Review at 0732f60 — no verdict token (author's lane). The reasoning is verified correct, checked against the code rather than the description:
registry_build.py:32— the NPU registry entry is keyed by HFmodel_type, not by the GGUF architecture or the class name, andNPU_ARCHS["LlamaForCausalLM"] = "llama"is there explicitly ("GGUF-on-NPU: MiniCPM5-1B"). So a new class name over an already-mappedmodel_typereally is not a coverage gap on that route.mapped_okis the designed escape hatch, not an invention:gap_violations()(:213-218) reports asignificant.jsonclass only when it is mapped and lacksmapped_ok— so recording the reason is exactly what makes--check-gapsquiet and keeps the entry honest.watch: reportis a tested mechanism (census_watch_test.py:276arm 14, "listed but does not alert"), with the two-class precedent at:305-315— not a new label.
One invariant this PR breaks silently, because nothing tests it: registry/significant.json's own about field points at docs/arch-gaps.md as the place these classes are documented, and all 36 of 36 existing entries are cited there today (I checked every class against the doc). This adds the 37th without a doc row, and census_watch_test.py:315's cls in gap_docs assertion runs over a fixed two-class tuple, not over the file — so CI stays green while the documented contract goes 36/36 → 36/37.
Two ways to close it, whichever is cheaper:
- add the one
docs/arch-gaps.mdrow (a rename-of-llamaentry is a paragraph, and it is where the next person will look); - and, since this is the second time the "reviewed class must be documented" rule has been carried by a hand-written tuple, make the arm loop every class in
significant.jsoninstead — that turns a convention into a check, which is what this file is for.
VERDICT at
|
|
Handover note — nothing further owed from this lane; it needs a merge decision. State: green ( What is owed: a verdict/merge by a lane that holds the queue, and — separately, not in this PR — the census-side follow-up I argued for: If someone prefers a converter registration in the fork instead of the |
Why
The daily census reports
MicroLlamaForCausalLMUNCOVERED because that class string appears in no registry row. The model's own config says that is not a capability gap:{ "architectures": ["MicroLlamaForCausalLM"], "model_type": "llama", "vocab_size": 1024, "hidden_size": 192, "num_hidden_layers": 6, "num_attention_heads": 6, "max_position_embeddings": 256 }model_typeisllama, which the registry already maps (LlamaForCausalLM→ ggufllama, backends hrx+npu,npu_model_type: llama), and the NPU path matches on model_type, not on the class name (tools/registry_build.py: "The NPU runs Q4NX model directories, so it matches on HF model_type, not on a GGUF architecture"). SoUNCOVEREDhere means the census could not find this class string, not the engine cannot serve this model.What this does
Records the reviewed disposition the way engine#1972 recorded the DeepSeek V4.1 Flash SuperExpert one: a
watch: "report"row in the curatedregistry/significant.json— not the generatedregistry/architectures.json, whichtools/registry_build.py --checkrejects a hand edit of — withmapped_oknaming the mapping that already exists.Adding a class row to
architectures.jsoninstead would record the census's search strategy rather than the engine's capability, and would claim a backend accepts a class name that no backend dispatches on.Checks
The diff is a 6-line insertion; the file's formatting round-trips exactly, so nothing else moves.
Follow-up worth having (not in this PR)
The census could check
config.json'smodel_typeagainst the registry before reporting a class as uncovered: every renamed fine-tune of a supported family will otherwise raise a daily!!line, and a report whose alerts are mostly false positives stops being read — the same failure as an instrument that cannot fail, one level up. This PR records the disposition for the one model the watcher already found; the tool change is what would stop the class.