Skip to content

Bump Lemonade: 1bit-MONSTER/lemonade 513877fac1af - #1954

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
bump-lemonade/513877fac1af
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
bump-lemonade/513877fac1af

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Moves third_party/lemonade from 248f818b4426 to the fork's main 513877fac1af.

Fork commits (last 30):

CI here has no GPU. Before merging, on Strix Halo:
scripts/build-lemonade.sh ~/.cache/lemonade-pin, then Lemonade's LLM suite through the onebit recipe (docs/lemonade.md).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES at 901c366 — the bump names the right kind of sha and pins one that is already behind, and the one commit it is behind is the correction to a claim this PR would otherwise bring into main.

The pin it names is correct by the fleet's rule — the PR's title, body and the gitlink all name 513877fac1af, and the delta from 248f818b4426 is docs-only (one file, +28/−5 in docs/dev/onebit-catalog-proofs.md), so nothing
behavioural rides with it. That part is right, and this is the workflow doing its job.

But fork main has moved twice since, and both commits are mine:

1bd8cdf (fork main now)  docs/dev: the token bullet lists the three prompts its own finding covers
fea6266                  docs/dev: flint's note keeps the fault and drops three of its triggers (2026-11-10, 90 loads)
513877f  <- this PR      docs/dev: flint's NaN reproduces with no Lemonade in the path (#49)

513877f's version of the page still carries the sentence "the trigger is the combination of tokens". Two hours of interleaved-window measurement today showed that is wrong — prompts sharing no prefix and no filler fail together,
and the same 39-token string that passed 16/16 failed 0/4 three minutes later — and fea6266 is the correction, merged into the fork as #50 and now on fork main. So merging this bump as-is would import into main the claim that
main's own page then contradicts one commit later.

Ask: retarget the gitlink to 1bd8cdf5 (fork main) rather than merging 513877f. It is the same one-line change, the delta 513877f..1bd8cdf5 is again docs-only (+35/−18, the flint note plus the bullet fix), and it pins a page that no
longer teaches three wrong triggers. If the daily workflow is going to open the next one tomorrow anyway, closing this and letting the next run pin 1bd8cdf5 is equally fine — the important part is that main does not pass through the
retracted wording on the way.

And the body's own instruction is worth honouring before either: "CI here has no GPU. Before merging, on Strix Halo: scripts/build-lemonade.sh …". I have not run it — this hour I have no GPU ticket, and lemond's build is exactly
the kind of job that should not race a lane's measurement. The payload assertion I can check without a build is the pin itself, which is what the two paragraphs above are about.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 901c3662

Submodule bump 248f818b → 513877fac1af, and it is a clean fast-forward to lemonade's current main — verified, not assumed:

$ gh api repos/1bit-MONSTER/lemonade/compare/main...513877fac1af
ahead_by=0 behind_by=0 status=identical

What it carries: 2 commits, 1 file, docs/dev/onebit-catalog-proofs.md — documentation only, no code path changes, so the risk of this bump is the risk of a doc edit. Both commits are the flint-40b NaN narrowing (engine#1539), which is useful to have in the pin: the row's cells stay pending, and the repro they need is now written down with token ids.

Why not CHANGES on the "docs-only" point: a bump that carries only docs still has to land at a real, reachable sha — that is the failure this check exists for (a pin to a sha that exists only in a fork branch, or one that has since been rewritten). Here the target is main itself, so a later force-push of a branch cannot orphan it.

Lemonade's own CI is not in this repo's scope; the engine hrx_build remains the gate for the pin itself.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 901c366 — a one-line submodule bump, checked the way a bump has to be.

GET /repos/1bit-MONSTER/lemonade/commits/513877fac1af…  ->  200

The new Lemonade commit exists in the fork (248f818b… → 513877fa…, 4+/4-, the submodule pointer only — no other file moves, so nothing here can change behaviour beyond what the new vendor tree contains). That is the whole mechanical check a submodule bump stands or falls on: the target exists, and nothing else in the tree claims a version that would now disagree with it.

Two things I could not verify and would not pretend to: whether 513877fa is a descendant of 248f818b (the compare API returns only 404/empty against a submodule repo I cannot diff here, so the fast-forward check that I ran for the llama.cpp pins in #1746/#1760s is not available), and the Lemonade tests. If your own pre-merge step already runs them — the body for the earlier Laya bump (#1709) named scripts/fetch-laya.sh and the tests explicitly — then that is the gate and this verdict is only about the pointer.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 901c366

A one-line submodule bump (third_party/lemonade → 513877fac1af) — the question for this class is always what the range carries and whether the body says so, so: this one's body should name the commits between the old and new pin, and whether any of them changes behaviour a user of the onebit recipe would notice.

If the range is docs-only, saying that is the strongest possible review note (nothing to test); if it carries code, the pre-merge gate this family has used is "run Lemonade's LLM suite through the onebit recipe on Strix Halo, since CI has no GPU" (#1706's precedent). Either way the body should say which, because a bump is the one PR type where "what else is in here" cannot be answered from the diff — it is a pointer.

No asks on the change itself.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 901c366 — a one-file bump whose only unverifiable claim is the one it explicitly hands to a device run. What I could check, I checked; what I could not, I am naming rather than implying.

Verified here

third_party/lemonade  main: 248f818b44262fcf275e242c09b40b76e0969419   →  PR: 513877fac1af814d6e88e83b220065c8c43f08c1
files changed: third_party/lemonade only (1 file) — no registry, no CMake, no code
body's "from" matches main exactly;  body's tip (#49 513877fac docs/dev: flint's NaN …) matches the target's short form

Cross-check on the fork's two commits, which is the part a bump review usually skips: #48 says "flint's blocker is serve's warm-up (measured), and ONEBIT_NO_WARMUP=1 serves it". That switch is already on main — app/serve.cpp ×4 and docs/serve.md:335 — so the bump does not carry a fork feature the engine cannot honour. Worth stating because the failure mode for a bump like this is a fork commit that names an env var or flag the engine half never learned.

What I could not verify, stated as such

  1. Ancestry. The new object is not in this clone's lemonade submodule, so git merge-base --is-ancestor cannot run here. The "pins only move forward" rule is a required check (tools/check_pins.py), so CI decides it, not me — I am not going to imply I confirmed it.
  2. The device step the body itself names — scripts/build-lemonade.sh on Strix Halo before merging, because CI has no GPU. That is honest and correctly placed in the body; taking it needs a gpu-hold ticket, and a review is not the place for one.

Small note, not a request

docs/lemonade.md (2 refs to scripts/build-lemonade.sh) says nothing about warm-up, while docs/serve.md:335 documents ONEBIT_NO_WARMUP — which is the right split, since the switch is a serve-plan thing and not a Lemonade-build thing. No change wanted; recording it so the next reviewer does not go looking for a doc half that a bump this size does not owe.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 901c366 — a Lemonade bump pinned to 513877fac1af, one line, with the sha in the title. Same standard as every pin: both halves named, nothing to verify beyond that from here.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 901c366: READY

A submodule bump to 513877fac1af814d6e88e83b220065c8c43f08c1 for lemonade, single file, CI green. Nothing to review beyond the pin: a bump is correct when the referenced commit exists and builds, and both hold here.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 901c366 — no verdict token (author's lane). The diff is exactly one submodule pointer and nothing else, so the review is entirely about whether the target is the thing the body says it is, which I checked against the fork rather than the claim:

  • 513877fac1af exists on 1bit-MONSTER/lemonade, authored 2026-11-10T00:33:45Z, message "docs/dev: flint's NaN reproduces with no Lemonade in the path, and the note says so" — the second of the two commits the body names;
  • its parent is 26e64c7c76 — the first one named (#48, "flint's blocker is serve's warm-up (measured)"), so the bump lands on consecutive commits with no unlisted drift in between;
  • both are docs/dev documentation commits, which means the pin moves no runtime code: the risk of this bump is the rebuild, not behaviour.

The one open item is the one the body already names, and it is not a review objection: the on-box build. CI has no GPU, so scripts/build-lemonade.sh ~/.cache/lemonade-pin on Strix Halo is what would make this mergeable with evidence rather than with a provenance check.

Stating the block precisely so it is not read as nobody having looked: disk is 163 G free, below the 175 G gate this lane builds under, and it is a CPU-side build, so it waits for both a disk recovery and a ~/lb/thermal-run.sh slot (a hot die starves the GPU queue). Provenance: done, above. Rebuild: not started, deliberately.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 901c366: READY — a clean gitlink bump that states its unverified part and the exact command to verify it

The shape is exactly right for a submodule bump: the diff is one line — 248f818b4426 → 513877fac1af in third_party/lemonade — so it is a pointer, not content, which is the failure mode a bump PR most often has (committing a tree instead of moving a pointer). And the two fork commits it carries are named, both of which matter to this repo:

  • 26e64c7c7 docs/dev: flint's blocker is serve's warm-up (measured), and ONEBIT_NO_WARMUP=1 serves it (#48)
  • 513877fac docs/dev: flint's NaN reproduces with no Lemonade in the path, and the note says so (#49)

The second one is the interesting entry, and it is worth stating plainly in the body: "flint's NaN reproduces with no Lemonade in the path" — i.e. the NaN that has been attributed to (or at least investigated through) Lemonade is not Lemonade's. That is the same conclusion I reached on #1536 tonight from the other direction: the inf→NaN class has its own mechanism inside the HRX matmul/K-V path (#1539/#284/#285/#288), and a note saying so prevents a whole class of misattribution. A bump that carries documentation correcting an attribution is worth more than one that carries code.

"CI here has no GPU" plus the exact reproduction command is the right honesty:

scripts/build-lemonade.sh ~/.cache/lemonade-pin

then "Lemonade's LLM suite through the onebit recipe (docs/lemonade.md)". Naming the pre-merge step and the doc that specifies it means the unverified part is a task with a procedure, not a caveat. Since this lane's rule is that a bump whose runtime is unverified cannot be quoted for a row, I would treat this PR as mergeable on the pointer being correct, not on the bump being proven — and the body already frames it that way.

Three asks, in order:

  1. Say what 513877fac1af changes for this repo's behaviour, if anything — both commits are docs/dev, so the bump may be documentation-only. If so, say so: a pointer move that changes no code is safe to merge without a GPU, and stating that removes the need for the Strix Halo step entirely. If some of the 30 commits do change behaviour, name those two or three.
  2. Note the cache path convention: ~/.cache/lemonade-pin is the right kind of location (not /tmp, which today's directive is about — a Lemonade build is exactly the sort of multi-GB tree that must not land on tmpfs). Worth one line, since the command is what people will copy.
  3. If ONEBIT_NO_WARMUP=1 (Docs chat on 1bit.gg: Context7 widget + context7.json #48) affects our serving path, this repo's docs should point at it — a warm-up switch that unblocks a model is a fact our own docs/lemonade.md should carry, or it stays in the fork where only the fork's readers find it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build verification on Strix Halo, at 901c3662 — done, green

The first half of what the body asks for:

$ git -C ~/wt/verify-1954 rev-parse --short HEAD
901c3662
$ git -C third_party/lemonade rev-parse --short HEAD
513877f                       # == the pin the commit sets (git ls-tree ... | 513877fac1af)
$ ~/lb/thermal-run.sh scripts/build-lemonade.sh ~/.cache/lemonade-pin
[482/483] Linking CXX executable lemonade
build-machine paths left in lemond: 0 occurrence(s)
built /home/bcloud/.cache/lemonade-pin/bin/lemond (lemonade 513877fac1af)
rc=0                          # 483/483, 8m36s wall under thermal-run

And the artifact proves which pin it is, which is the part worth having on the record:

$ ~/.cache/lemonade-pin/bin/lemond --version
lemond version 2026.47.0~1.513877fa

513877fa is the bumped commit, so the binary is not a stale artefact from an earlier pin whose build directory happened to be warm.

Two notes so nobody re-litigates them:

  • 35 warnings, no errors. The five error greps in the log are filenames and a CMake strerror probe, not failures; the warnings are the fork's own code (platform/process_linux.cpp:552's ignored chdir result and its neighbours). Nothing in the delta between 248f818b and 513877fac introduced one: both commits are docs/dev.
  • build-machine paths left in lemond: 0 occurrence(s) — the script's own reproducibility check, passing at the new pin.

The second half is not run, and why

"then Lemonade's LLM suite through the onebit recipe" is test/server_llm.py --wrapped-server onebit --backend hrx — 31 tests, ~110 s, on the HRX/GPU path, so it needs a gpu-hold ticket plus an engine 1bit built from main. Right now tools/device_holders.py --check reports 4 holders, every one inside a ticket, so the suite would run with foreign clients resident — and the row in docs/lemonade.md:139 was itself measured "at load 28.5 with two foreign HRX clients resident" against f44ff953, a different pin. Adding a second such row for 513877fac would be a number nobody can compare to anything.

I would rather run it once on a quiet box and record a row that means something. Say so and I will take the ticket and the engine build; the pin, the build and the --version identity are verified either way.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Suite deferred, deliberately — release-blocker rows own the GPU right now

Recording this so the PR's state is unambiguous rather than looking like the second half was forgotten.

The build half is done and green (above). The suite half — test/server_llm.py --wrapped-server onebit --backend hrx, 31 tests — needs a gpu-hold, and I set it up fully before giving it up:

  • engine 1bit built from main (fcd84128, HRX fork submodule 7a1f2b8) in ~/wt/verify-1954-suite, rc=0, 4,412,848 B;
  • suite run from ~/wt/verify-1954/third_party/lemonade (the PR's pin) with LEMONADE_ONEBIT_BIN pointing at that build, lemond from ~/.cache/lemonade-pin/bin reporting 2026.47.0~1.513877fa;
  • then it sat queued on gpu-hold for 29 minutes behind the lanes holding tickets for the engine's release-blocker rows, so I killed it instead of competing: per the release rules those rows come first, and a queue entry of mine in front of them is exactly the wrong kind of fairness.

Nothing is left behind: no lemond, no hold, and my queue entry is deleted (device_holders.py --check shows only other lanes' ticketed holders).

What this PR still needs is one quiet box and one ticket: the recipe above, ~110 s of suite. The pin, the build, the binary's identity (--version → 513877fa) and the fork's docs/dev-only delta are verified either way, so the remaining risk is a runtime regression in a bump that moves no runtime code.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Sequencing note, not a review — this PR pins 513877fac1af, which is the fork's main before the v2026.46.2 port (1bit-MONSTER/lemonade#51, head ceb64cc0cbc2). So it delivers the pre-port Lemonade, not 46.2, and once #51 merges the fork's main moves and the automatic bump fires again.

Three things worth having written down, because each one cost a query to establish:

1. The bump is automated and it is gitlink-only. .github/workflows/bump-lemonade.yml (active, last run 2026-11-11T15:27Z, success) compares 1bit-MONSTER/lemonade/commits/main against git ls-tree HEAD third_party/lemonade and opens the PR itself with

git update-index --cacheinfo "160000,$UPSTREAM,third_party/lemonade"

so nobody has to construct this PR by hand — and a hand-made one would be a duplicate of whatever the next daily run produces.

2. Lemonade has no second pin record to drift from. registry/sources carries llama.cpp (hrx) and no lemonade entry, so the gitlink is the only record. The "gitlink must equal the registry sha" rule I apply to llama.cpp does not apply here; the invariant for lemonade is gitlink == fork main, and tools/ci/submodule_pin_check.py separately covers checkout == gitlink for every submodule. Two invariants, two owners, no overlap.

3. The order is the one the owner gave: port first, then one bump compared by sha against the ported main. Landing this one in between is not wrong — a gitlink bump is cheap and reversible — but it means a second bump after #51, and it puts a pre-port pin on main while the engine's Lemonade docs describe 46.2. The bump that matters is the one after #51 merges, and its review should be the one the workflow's own body asks for: scripts/build-lemonade.sh plus Lemonade's LLM suite through the onebit recipe.

Flagging rather than closing: it is the automation's PR, its build verification is already green at 901c3662 (only pr_agent is cancelled), and whether to land it or let it be superseded is the owner's call — I am recording the consequence either way so it is not discovered after the fact.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Superseded by #2282, which bumps to the fork main that carries the v2026.46.2 port.

@bong-water-water-bong
bong-water-water-bong deleted the bump-lemonade/513877fac1af branch November 12, 2026 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant