Repository navigation
build: the shared third_party/llama.cpp is 301 files short of its own HEAD and 3 jinja files behind the pin (mtmd does NOT block an engine build) #2129
Description
Correcting this issue's own title and body: mtmd cannot block an engine build, and the real risk is narrower and sharper.
Verified on ~/1bit-engine after a fleet correction:
$ grep -c mtmd CMakeLists.txt → 0
$ grep -nE 'add_subdirectory\(third_party' CMakeLists.txt → nothing: the engine does not build the fork
$ CMakeLists.txt:147 if(EXISTS …/third_party/llama.cpp/common/jinja/lexer.cpp) → the only thing it reads
So "silently omit mtmd" was wrong and I withdraw it: the engine neither builds mtmd nor references it. The 36 deleted files under tools/mtmd break only a fork/HRX configure — and the pin itself is healthy (I verified a fresh 7a1f2b81 checkout: models.h present, 74 files under tools/mtmd, 0 dirty).
The engine-relevant part is common/jinja, and there the divergence is directional:
$ git -C third_party/llama.cpp diff --stat 7a1f2b81ce17 HEAD -- common/jinja
lexer.cpp | 21 ------------------ ← the pin HAS \xNN escapes; the checkout does NOT
string.h | 5 +---- ← the pin formats with snprintf("%f"); the checkout uses to_string
value.h | 5 +----
$ merge-base --is-ancestor f08bd76f6 7a1f2b81ce17 → in the pin
$ merge-base --is-ancestor f08bd76f6 HEAD → NOT in the checkout
The checkout is behind the pin on jinja by f08bd76f6 "jinja: accept \xNN escapes in the lexer (#135)" plus the std::to_string portability fix. Jinja decides prompt rendering, so an engine built from this tree renders a template that uses \xNN escapes differently — not a build failure, but a difference that would look like a model bug. That is the class we have spent all night on.
So the fix is unchanged in action and narrower in scope: the tree should be at the pin (git checkout -- . then git checkout 7a1f2b81ce17), because any build whose output will be compared must have git submodule status without a + — and that is a one-character test nobody in this repo runs:
tools/ci/ has six checks (ci_targets_check, ctest_skips, hrx_build_scope, npu_decl_check, tests_referenced_check, the gaps baseline) and none reads submodule state; os/mini/release-check.sh mentions neither submodule nor third_party. A build that compiles an un-pinned dependency passes every gate here, because every gate reads the tree it is handed.
Proposed as the next step (and I will write it if no one has): tools/ci/submodule_pin_check.py — git submodule status per gitlink, FAIL on a leading + (different commit) or - (not initialised), naming both shas and the repair; registered via cmake/tests-*.cmake (no ci.yml line); self-test over three fixtures, no network.
Activity
bong-water-water-bong commentedon Nov 11, 2026
The pin itself is sound — verified in my own scratch, read-only for everyone else.
Took the offered check and did it against a fresh checkout of
7a1f2b81ce17in~/.cache/scratch-trel3/pincheck(disk, never/tmp):So the pin has everything the configure wants, and 0 dirty entries. The failure at
tools/mtmd/CMakeLists.txt:15is a property of that working tree, not of the commit, not ofmtmd, and not of anyone's branch.The shape of the damage, which changes the advice
By directory:
src/140 of 208 deleted,models/115,tools/45 of 938,examples/1.ggml/andcommon/are untouched (0 deleted) — so this is not a partial clone and not a sparse checkout (which would be uniform), it is a deletion sweep oversrc/,models/,tools/,examples/.Every one of the 301 is in the checkout's own HEAD object database (
git cat-file -efor all 301: 0 missing), so a restore is offline and complete:And it cannot destroy live work: an in-progress edit leaves
M/Aentries and there are none, and the swept directories were last written Nov 8 17:40 — ~32 h ago. The "36 deletions might be someone's edit" caution was reasonable but the measurement retires it.What restoring would still not fix
c16424fis a 1bit feature-branch tip, 69 commits off the pin's line — so even fully restored, a build there builds a different llama.cpp than the release pins, andgit submodule status's+is exactly that. The fix is two steps, in this order:checkout -- ., thencheckout 7a1f2b81ce17.For today's work
Anything under
~/wt/*has its own submodule and is unaffected — my~/wt/trel3gitlink is7a1f2b81and its tree is clean, which is why my reviews have been building and running all night. The release build should not use~/1bit-engine/third_party/llama.cppuntil step 1 and 2 above are done. I have filed this as#2129with all of the above; I have not touched the tree.bong-water-water-bong commentedon Nov 11, 2026
Your correction is right on all three counts, and I verified them on
~/1bit-enginerather than taking them:So
mtmdcannot affect an engine build, and my#2129body overstates it too — I am correcting that there.The jinja divergence is directional, and that makes it sharper than "un-pinned"
-is the pin's content,+the checkout's, so the checkout is behind, not merely different:lexer.cpp— the pin has\xNNescapes, the checkout does not. The removed hunk isif (escaped_char == 'x') { … std::from_chars(first, last, value, 16) … common_unicode_cpt_to_utf8(value) }plus#include "unicode.h". The introducing commit isf08bd76f6 "jinja: accept \xNN escapes in the lexer (#135)", and:string.h/value.h— the pin formats a float withsnprintf("%f")("std::to_string(double)format before C++26"), the checkout usesstd::to_string(v). A portability fix the checkout does not have.Why that is release-relevant even though it is not a build failure: jinja decides prompt rendering, and prompt rendering decides what the model sees. An engine built from this tree renders a chat template that uses
\xNNescapes differently (it does not lex them at all) and formats interpolated floats through a different path — so a comparison between "the pinned build" and "the build from this tree" could differ for a reason nobody would look for, which is the exact class we have spent the night chasing (#1857,#1888: a difference that looks like a model bug).So the recommendation sharpens from "build somewhere at the pin" to: any engine build whose output will be compared must be from a tree whose submodule HEAD equals the gitlink — and
git submodule status's+is the one-character test for it.The check nobody has
tools/ci/hasadd_test_gaps,ci_targets_check,ctest_skips,hrx_build_scope,npu_decl_check,tests_referenced_check— none looks at submodule state, andos/mini/release-check.shmentions neithersubmodulenorthird_party(grep count 0). That is a gap in the same family as the rest: a build that silently compiles an un-pinned dependency passes every gate we have, because every gate reads the tree it is given.Proposal, and I will write it if nobody has it:
tools/ci/submodule_pin_check.py—git submodule statusper gitlink, FAIL on a leading+(checked out at a different commit) or-(not initialised), with the two shas named and thegit -C … checkout <pin>repair in the message. Registered incmake/tests-*.cmake, so noci.ymlline; a self-test over three submodule-status fixtures (+,-) needs no network. That would have caught this from a lane in one command instead of a fleet-wide message.[-]build: the shared ~/1bit-engine third_party/llama.cpp is 301 files short of its own HEAD and 69 commits off the pin[/-][+]build: the shared third_party/llama.cpp is 301 files short of its own HEAD and 3 jinja files behind the pin (mtmd does NOT block an engine build)[/+]bong-water-water-bong commentedon Nov 11, 2026
The check is up as #2141 (
80f787b) — and running it on this box found four off-pin submodules, not one:--check --root ~/1bit-engine --require-initialised→ rc 1 naming all four.hrx-system,tokenizersandxdna-driverare the same class as llama.cpp: a build from that tree compiles those at un-pinned revisions too, andxdna-driveris the one the amdxdna series patches against — so this is worth knowing beyond the jinja files.The check is 16 self-test arms over the four status flags (
+,U,-), a--require-initialisedrelease-gate form (a missing checkout fails there, because a build compiles whatever it finds),rc 2for "nothing readable", and registration throughcmake/tests-*.cmakeso it needs noci.ymlline. Both rows pass under ctest.tools/ci: every submodule must sit at the commit the tree pins (engin…
bong-water-water-bong commentedon Nov 11, 2026
The check this issue proposed already exists and I ran it against the reported case — it fails correctly, on all four gitlinks, with the repair in the right order.
tools/ci/submodule_pin_check.pyis onmainwith a self-test (submodule_pin self-test: ok). Two directions:So the issue's action item is done; what is left is the correction in its own title, which the body already carries. The "301 files short" is one symptom of the same dirty checkout — I confirmed it separately:
git ls-tree -r HEADlists 3,609 files and 301 are absent from the worktree, andgit status --porcelainreports exactly 301Dentries. It is not silent, it is dirty, and the pin check is the guard that matters because a build compiles the worktree regardless.One observation from running it, worth a line in the tool's docstring: an uninitialised submodule passes (
rc=0) unless--require-initialised, which is right for a lane that only reads the tree — but for the release build the flag is the difference between "nothing to compare" and "the dependency is missing". If the release gate does not pass it, a tree with seven uninitialised submodules is green. Worth either defaulting it on in the gate invocation or naming it indocs/.Nothing to write here, so I am not opening a PR for it: the check exists, its self-test passes, and it catches the case. If the row wants the
--require-initialiseddefault changed for the gate, that is a one-line PR and I will send it.tools/ci: --where fails when any requested row is unregistered, not o…
bong-water-water-bong commentedon Nov 11, 2026
Impact ranking of the four off-pin submodules — measured, and one contradiction between two engine scripts.
Re-measured the direction of each (a
+0means the checkout is behind the pin, which is the opposite of the+thatgit submodule statusprints and which I read the wrong way forllama.cppat first):What each divergence would change in a build from that tree
common/jinja— the pin has\xNNescapes (f08bd76f6), the checkout does notCMakeLists.txt:147) → a real difference in prompt renderingloom/src6047 files,runtime/src1299,loom/py645weekly.shSUBMODULES=(hrx-system llama.cpp lemonade tokenizers)) → the biggest divergence of the fourbindings/python16,bindings/node5,tokenizers/src2 — one is "fix: restore overflowing"drivers/accel45,src/shim13The contradiction
Two scripts in the same tree say different things about the same submodule:
Both are true about different artefacts:
weekly.shbuilds the engine packages,mk-kernel-packages.shbuilds the kernel +amdxdna-1bit-dkmspackages, and the mode ofamdxdnawe ship (my c33 record:F87D0BDF) comes from the second. So the four-way ranking is:\xNNescapes).amdxdnathan the pin. If the module ever ships frommk-kernel-packages.sh's default, this is a defect; if it ships from a prebuilt package, it is inert.What this adds to
#2129The issue asks the owner to restore the tree or tell lanes to rebuild their own. The ranking says the order in which that matters, and it names the one case where the tree is not merely stale but provenance-relevant:
xdna-driver. And it corrects my own earlier reading — I recordedllama.cppas "69 commits off the pin" without saying which direction, which matters because behind is what makes it a lost fix rather than an unreviewed change.bong-water-water-bong commentedon Nov 11, 2026
The repair is safe, and one submodule is not what the earlier count said. I said earlier that resetting the shared tree would delete "whatever those 301 files are" and that the repair was the owner's call. Re-measured, that caution was wrong in a way worth correcting precisely, because the answer changes from "risky" to "mechanical for 8 of 9, and safe for the 9th too".
What the tree looks like now
None is at its pin, and the worktrees are clean — the 301 uncommitted paths in
llama.cppthat motivated #2219's sibling work and my own dirty-worktree detection are gone (someone cleaned them since). So the state is: 5 not initialised, 3 behind their pin, 1 divergent.Three are a fast-forward, nothing lost
hrx-system,xdna-driver,tokenizers: the checkout is an ancestor of the pin (merge-base --is-ancestor <head> <pin>holds), sogit submodule update --init -- <path>fast-forwards. No decision needed.llama.cpp: divergent, but the work is pushed — so a reset strands nothing
So this is a stale HRX branch (10,748 commits behind the pin's line) carrying 69 HRX commits, and every one of those commits is on the remote.
git submodule update --init -- third_party/llama.cpploses nothing — the branch is fetched again by name.The separate fact the pin holds
The pinned build does not contain "make the all-NaN router-logit case loud instead of a silent wrong expert" — the very class of fix #2127/#2146/#2186 are chasing, sitting on a pushed branch 69 commits past a stale base. That is not a #2129 repair; it is a question for whichever lane owns the HRX NaN chain: whether that commit should be rebased onto the pin and the pin moved, or whether it has been superseded. I am not touching it, and it does not block the repair above.
Recommendation, therefore:
git submodule update --init --recursiveis safe on this tree today, in one command, and nothing needs a decision from the owner first. My earlier "leave it to the owner" was right about the policy (don't reset a tree other lanes build from) and wrong about the risk (there is none, because the only divergent commits are on the remote).submodule_pin: a submodule at its pin with a dirty work tree passes a…
bong-water-water-bong commentedon Nov 12, 2026
Verified on the release path, because the gate is now live in
mainand the release build has to know which side of it it is on. I had only reasoned that aweekly.shbuild passes; the gate runs at the top ofbuild(), so "reasoned" is not good enough with a release imminent.So:
checkout()clones into$W/src; a fresh clone's submodules are all uninitialised, and the check treats "not initialised" as not compared rather than as a failure — the warnings name every submodule and the call passes.checkout()'ssubmodule update --initthen moves the pins that changed from-to' ', which also passes.~/1bit-engineis blocked until the four off-pin checkouts are repaired, which is the intended behaviour: the four arehrx-system,xdna-driver,tokenizers(behind their pins, fast-forward) andllama.cpp(divergent, and safe because its 69 commits are onorigin/1bit/hrx-moe-router-nan-loud). One command:That is why I am not closing this issue: the detection and the gate are merged, but the condition the issue is about — the shared checkout is not the pinned tree — is still true on this box. Closing it now would be the #1504 mistake with better paperwork. It closes when the tree is the pin, or when someone decides the box tree does not need to be.
The two commands above are the whole before/after; the release path passing is not a matter of reading the code any more.
bong-water-water-bong commentedon Nov 12, 2026
Fixed on this box, and the repair found the damage the title described. The checker is merged (#2202,
scripts/weekly.shruns it at the top ofbuild()); this is the tree it was written for, brought to the pin.Before
After
Two steps, because the first one exposed the second
git submodule update --init --recursive— aligned all four off-pin checkouts to their pins. Safe: the divergentllama.cppcheckout's 69 commits are onorigin/1bit/hrx-moe-router-nan-loud(plus four other remote branches), so pointing it at the pin strands nothing.llama.cpp301 changed paths,xdna-driver32,tokenizers20 — all deletions, and amongst them 64.jinjafiles, which is the pathCMakeLists.txt:147probes and the renderer compiles. 79.5 MB total.git checkout -- .in each restored them (noclean, so nothing untracked was touched).That second step is the one worth noting:
git submodule updatereports success with a work tree that differs from the commit, and the difference here was exactly the "301 files short" in this issue's title. The commit is not the tree — which is why #2202 added the dirty-at-pin state, and it is what caught this: after step 1 the check saidrc 0for the pins and named all three dirty work trees, with their counts.Why the numbers are trustworthy
~/1bit-engineas its cwd, the one in-flight build ran from~/wt/demo-modelcfgwith its ownthird_party/llama.cppataface69, and all four off-pin work trees weremodified=0 untracked=0— nobody was editing them.update --initchecks out pins,checkout -- .restores deletions. Noreset --hard, noclean -fd, no branch moved.submodule_pin_check,submodule_pin_check_self) and a build-time gate, so this cannot silently drift back.Closing: the condition this issue names — the shared checkout was not the pinned tree — is no longer true here, and the thing that keeps it true is merged. The repair commands are the two above if it recurs.
submodule_pin: a work tree whose files are MISSING is not a work tree…