Skip to content

bump llama.cpp to 50d28600a: the pin on main is five commits BACKWARDS (restores #2033's regression) - #2090

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
fix-1945-pin-restore
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
fix-1945-pin-restore

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes the red predicate under engine#1945, and restores five landed fixes that a pin bump dropped.

The regression

origin/main records gitlink 2293f95ea, and that commit is an ancestor of 50d28600a:

$ gh api repos/1bit-MONSTER/llama.cpp/compare/2293f95ea49d...50d28600a
status=behind   ahead_by=0   behind_by=5

The gitlink history says how:

0f305ed  bump llama.cpp to 50d28600a: Qwen3-VL's rotation runs on the device (fixes #1467) (#2011)
153b7b6  bump llama.cpp to 2293f95ea: prefill attention keeps its rate at long context (fixes #1989) (#2033)

#2033 pinned five commits before #2011's pin, silently dropping:

dropped what it fixed
50d2860 the multi-section rotation of Qwen3-VL runs on the device engine#1467
fa533bd a view into storage the graph does not hold is skipped —
dfbf2b1 keep dense matmul activations inside the f16 range engine#1962 / llama.cpp#285
a5dc597 a shared pool_tally pair, so the #1823 charge has device-free arms engine#1856's ledger arm
c47eae4 read the host term before classifying it, so a clang build opens the device engine#1942

Three of the five closed issues. None of this is a rebase artefact: an ancestor cannot contain its descendants, so the bump chose an older commit rather than a divergent one.

Why this is safe to land

This is not a new route decision. It restores the value tools/check_pins.py already asks for — its own message is "point it at a commit that contains the old one (git submodule update --remote)" — and 50d28600a is exactly the commit #2011 pinned before #2033 overwrote it.

Checks, on this branch

python3 tools/check_pins.py HEAD              -> pass    (FAILS on origin/main: "2293f95ea -> 50d28600a is behind")
python3 tools/registry_build.py --check-pins  -> "registry/architectures.json records the committed pins"
python3 tools/copyright.py --check            -> pass

The registry regenerated to a one-line change — the recorded pin only, 298 architectures unchanged — because those five commits changed behaviour, not the class list. That is also why a --check-pins-only review could not see it.

The durable half, not in this PR

check_pins.py produced the right message on main before this PR and nothing gates a merge on it, so a backwards pin lands green and only surfaces on the next scheduled bump-hrx sync — up to a day later, which is #1945's own title ("failed on every scheduled run … without anyone noticing"). Running it in CI on any PR that touches third_party/llama.cpp converts that day-latency silent failure into a red PR.

Refs #1945, #2033, #2011

…DS (restores #2033's regression)

`origin/main` records gitlink `2293f95ea`, and that commit is an **ancestor** of `50d28600a`:

    $ gh api repos/1bit-MONSTER/llama.cpp/compare/2293f95ea...50d28600a
    status=behind  ahead_by=0  behind_by=5

The gitlink history says how:

    0f305ed  bump llama.cpp to 50d28600a (fixes #1467) (#2011)
    153b7b6  bump llama.cpp to 2293f95ea (fixes #1989) (#2033)

so #2033 pinned five commits *before* #2011's pin, silently dropping fixes that had already landed:

    50d2860  hrx: the multi-section rotation of Qwen3-VL runs on the device   (engine#1467)
    fa533bd  hrx: a view into storage the graph does not hold is skipped
    dfbf2b1  hrx: keep dense matmul activations inside the f16 range          (engine#1962)
    a5dc597  ggml-hrx: a shared pool_tally pair                               (engine#1856's ledger arm)
    c47eae4  hrx: read the host term before classifying it                    (engine#1942)

This restores the pin to `50d28600a`, which is the value `tools/check_pins.py` asks for — it is the check's
own prescription ("point it at a commit that contains the old one"), not a new route decision, and it puts the
five fixes back.

Why this belongs to engine#1945 as well as to the pin: `tools/check_pins.py` already produced the right
message on `main` before this PR, and **nothing gates a merge on it**, so a backwards pin lands green and only
surfaces on the next scheduled `bump-hrx` sync — up to a day later, which is that issue's own title. Running it
in CI on any PR that touches `third_party/llama.cpp` is the durable half and is not in this PR.

Checks, run locally on this branch (`HEAD` = origin/main + these two changes):

    python3 tools/check_pins.py HEAD                 -> pass   (fails on origin/main)
    python3 tools/registry_build.py --check-pins     -> "records the committed pins"
    python3 tools/copyright.py --check               -> pass

The registry regenerated to a **one-line** change (recorded pin only; 298 architectures unchanged), because the
five dropped commits changed behaviour, not the class list — which is also why a `--check-pins`-only review could
not see the regression.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

1467 - Partially compliant

Compliant requirements:

  • The PR restores the commit that fixes the CLIP graph execution on HRX backend
  • The vision model projector should now run on the device instead of CPU
  • The image TTFT should be improved with the projector on HRX0

Non-compliant requirements:

  • The PR does not include actual performance measurements to confirm the 17s TTFT target

Requires further human verification:

  • The actual performance and error behavior after applying the fix needs to be verified on hardware

1989 - Partially compliant

Compliant requirements:

  • The PR restores the commit that includes fixes for prefill attention cost

Non-compliant requirements:

  • The PR does not include a detailed analysis or kernel change to address the slope

Requires further human verification:

  • The attribution of the slope and its fix need to be verified through performance measurements

1823 - Partially compliant

Compliant requirements:

  • The PR restores the commit that includes the fix for host-weight buffer charging

Non-compliant requirements:

  • The PR does not include the actual implementation of the fix for the host-weight buffer charging

Requires further human verification:

  • The fix for host-weight buffer charging needs to be implemented and verified
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Subproject Commit Mismatch

The PR updates the llama.cpp submodule to commit 50d28600a, which is five commits ahead of the previous commit 2293f95ea. This change restores several fixes that were inadvertently dropped due to the previous pin. However, it's important to verify that the new commit does not introduce any regressions or unexpected behavior in the HRX backend or other components.

Subproject commit 50d28600a5f7a38df390709b94c463df02785a54
Architecture Version Update

The architecture version for llama.cpp (hrx) has been updated from 2293f95ea to 50d28600a. This change aligns with the submodule update and ensures that the registry reflects the correct version of the underlying library. It's important to confirm that this version update does not introduce any compatibility issues or unexpected behavior.

"llama.cpp (hrx)": "50d28600a5f7a38df390709b94c463df02785a54"

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 8e89ddf — and I verified the direction of the regression from the API rather than from the table, because "the pin went backwards" is the kind of claim that is easy to state and important to check:

gh api …/compare/2293f95ea49d...50d28600a5f7
  status=behind   ahead_by=0   behind_by=5      # main's gitlink is five commits BEHIND 50d28600a

So pinning 50d28600a moves forward, and the five dropped commits are exactly the kind that a bisect would blame the engine for rather than the pin: Qwen3-VL's multi-section rotation (engine#1467), the f16-range dense matmul (engine#1962 / llama.cpp#285), the graphics-view elision, and the shared pool_tally pair that carries engine#1856's device-free ledger arm. A regression that lands as "the fix stopped working" with nothing in the engine's history to explain it is the worst kind to debug, and this is the second time in two days that a scheduled bump has moved the pin in the wrong direction (#1945 is the first).

One thing worth stating in the body, because it changes what a reader should expect from this pin: 50d28600a is not the fork tip. I checked the tip too:

compare/50d28600a5f7...a5dc597b81f   ahead_by=3   # a5dc597b8 is 3 commits NEWER than what this PR pins
compare/2293f95ea49d...a5dc597b81f   ahead_by=8

So this PR restores the five it lost and lands three commits short of where the fork has since gone — including c47eae4a7 (the host term read before classifying, which is what makes a clang build register the device, engine#1942) and 7fb31383b (the fused Q decode at every head size, engine#1926). That is fine if the intent is "undo the regression, take the newest separately", but the body reads as a restoration and a reader will assume the pin is now current. One sentence — "the fork is three commits ahead; #2011's + #282's + #279's contents are a separate bump" — prevents the next lane filing the same issue again.

Second note: the PR fixes the red predicate under #1945, which is also the issue whose reporting half I sent as #2086 tonight. Those two are complementary and neither blocks the other: this one makes the pin right, that one makes the next failure legible.

Scope of my check: verified both compares against the fork's API, the gitlink change in third_party/llama.cpp and registry/architectures.json, and the dropped-commits table against the compare output. Did not build llama.cpp at either pin, so "the five fixes work again" is the PR's claim; "the five commits are missing today" is what I verified.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8e89ddf: CHANGES

This PR moves the pin backwards. Main's pin 2293f95ea is five commits AHEAD of 50d28600a, and all five "dropped" commits in the table are already in it. Please do not merge; it would remove #1989's fix from main.

The body quotes compare/2293f95ea49d...50d28600a → status=behind, behind_by=5 and reads it as "main is behind". GitHub's compare describes the second commit relative to the first: 50d28600a is behind 2293f95ea by five. Both directions, from the API just now:

$ gh api repos/1bit-MONSTER/llama.cpp/compare/2293f95ea49d...50d28600a5f7
status=behind  ahead_by=0  behind_by=5  merge_base=50d28600a        <- the merge base IS 50d28600a
$ gh api repos/1bit-MONSTER/llama.cpp/compare/50d28600a5f7...2293f95ea49d
status=ahead   ahead_by=5  behind_by=0
  69aac7fa2  tests: known-answer MUL_MAT cases with a resident weight (#287)
  7cc3187b5  ggml: a failed graph compute is reported as one (engine #1997) (#290)
  139ccd741  hrx: prefill attention keeps its rate at long key counts (engine #1989) (#289)
  cf5febf04  hrx: keep activations inside the f16 range in the two small-batch Q6_K kernels (engine #1962) (#288)
  2293f95ea  hrx: the attention kernel derives the transposed pitch from its key count (#291)

So the merge base of the two is 50d28600a itself: it is the ancestor, 2293f95ea the descendant. #2033 moved the pin forward from #2011's, as it said.

The five commits the table says were dropped are each an ancestor of main's pin:

dfbf2b1...2293f95ea : ahead 7, behind 0
a5dc597...2293f95ea : ahead 8, behind 0
c47eae4...2293f95ea : ahead 9, behind 0
fa533bd...2293f95ea : ahead 6, behind 0
50d2860...2293f95ea : ahead 5, behind 0

What merging this would do: take #287, #290, #289, #288 and #291 off main, i.e. reopen engine #1989 (prefill attention at long context), the second part of #1962 and the #1997 coverage.

On the check_pins.py line in the body ("FAILS on origin/main: 2293f95ea -> 50d28600a is behind"): I have not run it. If it does print that on main, it is comparing against something other than the previous gitlink, and that is worth its own look, but it is not evidence that main's pin is old.

To the READY at this head: it re-ran the same one-direction compare and read it the same way; the reverse direction above is the check that decides it.

The forward move is coming separately: I have verified the fork head 7a1f2b81c (seven commits ahead of main's pin) on a full build and am opening that bump now.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The forward bump is #2092 (2293f95ea → 7a1f2b81c, ahead 7 / behind 0). It contains everything this PR's table lists, because main's pin already does. Suggest closing this one.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing as wrong, and the error is mine. 2293f95ea is 5 commits AHEAD of 50d28600a, not behind it — so this PR moves the pin backwards, and CI's Submodule pins only move forward gate was right to refuse it.

Ground truth, run locally after unshallowing the submodule:

$ git -C third_party/llama.cpp merge-base --is-ancestor 50d28600a 2293f95ea  -> YES
$ git -C third_party/llama.cpp merge-base --is-ancestor 2293f95ea 50d28600a  -> NO
$ git -C third_party/llama.cpp rev-list --count 50d28600a..2293f95ea         -> 5
$ git -C third_party/llama.cpp log -1 --format='%h %s' 2293f95ea
  2293f95ea hrx: the attention kernel derives the transposed pitch from its key count … (engine #1989) (#291)

So #2033's bump was correct, all five commits I listed are contained within 2293f95ea, and nothing was dropped. main is also self-consistent: gitlink 2293f95ea, and registry/architectures.json records 2293f95ea49da….

What I did wrong

I read GitHub's compare/{A}...{B} status field as "A is behind B", when it describes B relative to A — status=behind, behind_by=5 there meant B (50d28600a) is 5 behind A (2293f95ea). From that inversion I built a false regression, five "dropped" fixes, and this PR.

Worse: when merge-base --is-ancestor disagreed with my reading, my first explanation was "the clone must be shallow" — and I nearly left it at that. Unshallowing kept the same answer, which is what should have told me the reading was wrong, not the tool.

The lesson I am recording: compare/A...B is B-relative, and git merge-base --is-ancestor is the ground truth for a pin direction. A pin PR should start from that test, not from an API field whose direction has to be remembered.

Nothing else in #1945 was affected by this — the sync's other predicate (AMD-branch containment of 44ee571) stands as verified, and the #2033-related claims in my comment there are retracted separately on that issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant