Repository navigation
Keep 4 calls for local goals on models that reason - #50
Merged
Merged
Conversation
A one-path goal on a model that reasons kept its output reserve of one calibrated response per call, up to 16,384 tokens, but went from 4 to 8 calls. With output settlement the later calls get full limits, yet in eight laptop runs of qwen3.5:4b only 1 of 75 goals that found a passing candidate needed a fifth call, and a failing goal ran for up to 346 s. Keep the 4 calls; the token reserve is unchanged.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
#47 gave a local goal with one editable path, on a model that reasons, 8
model calls and up to 16,384 output tokens, where other models get 4 calls
and 4,096 tokens. The 4,096-token reserve had cut the later calls to 512
output tokens, which the reasoning used up. #49 settles each call's charge to
the output the runtime reports, so later calls now get their full limit, and
the 16,384-token reserve alone gives each of 4 calls a full 4,096-token
response on the laptop profile.
The extra calls did not pay for themselves. Across eight laptop runs of
qwen3.5:4bwith thinking on and 8 calls (126 goals), 75 goals found acandidate that passed their checks:
The one fifth-call pass is state-of-tic-tac-toe in the 2026-11-10 ablation
(
exp-all-c): four repairs failed, the fifth passed, and the goal took 172 s.Failing goals pay for the extra calls. On #49's laptop run, calls 4 to 8 had
a median limit of 3,996 tokens, 14 of them reached a check and none passed,
and the six goals that ran out of budget took 212 to 346 s of their 600 s.
Change
adjust_reviewed_plan_budgetinphonton-cli/src/local_goal_cli.rsnolonger raises a one-path goal on a model that reasons to 8 calls. It keeps
the default 4 calls, reserves one calibrated response of output per call
(up to 16,384 tokens, so 16,384 on the laptop profile, as before), and keeps
enough check launches for a baseline and 4 candidates. With 16,384 tokens
and 4 calls, every call gets the full 4,096-token limit even when each
earlier call writes its whole limit.
docs/local-harness.mdand the changelog describe the 4-call budget. Theunreleased changelog entry for Charge local model calls the output tokens they report #49's settlement described the 8-call goal;
it now gives the default 4,096-token budget's limits (2,048, 1,024, 512 and
512, as Keep a local run going when its restart strategy cannot answer #48's CI smoke run recorded them) and that run's charged and
reported output.
Tests
search_replace_review_budget_covers_best_case_scope_within_capsnowuses a laptop-sized profile that reasons (16,384-token context, 4,096-token
calibrated output): a one-path goal gets 4 calls, 16 check launches and
16,384 output tokens, a one-path goal with 4 checks gets 20 check launches,
and the same profile without reasoning, or with an explicit budget, keeps
4,096 tokens. The earlier case used a 1,024-token allowance, where 4 calls
of 1,024 equal the default 4,096 and the test could not tell whether the
reserve was raised.
assertion fails.
Measured
CI
CI passes on ba52c3f: Rust workspace (Linux), Windows (workspace) and macOS
(local runtime and CLI).
No CI smoke run: the smoke model never takes this branch. In #49's smoke run
all 33 one-path goals had 4 calls and 4,096 tokens, so none had a profile
that reasons.
Laptop rerun
qwen3.5:4bon the same 20 Exercism exercises as #48 and #49, with freshcalibration (the runtime's thinking mode, SearchReplace, a 16,384-token
context and 4,096 output tokens), a Windows build of ba52c3f and the same
runner, which waits for free memory before each exercise and reruns an
attempt that stops on memory pressure (three were rerun this time).
4,096-token limits; three of them used the whole limit. The seven goals that ended
budget_exhaustedused allfour calls, were charged 4,109 to 9,935 tokens and took 142 to 228 s. The
four exercises that ran out of budget on all three builds
(scale-generator, dominoes, transpose and markdown) took 151 to 201 s here,
212 to 346 s on Charge local model calls the output tokens they report #49 and 146 to 290 s on Keep a local run going when its restart strategy cannot answer #48.
state-of-tic-tac-toe, nucleotide-count and satellite passed on their first
call and killer-sudoku-helper on its third (on Charge local model calls the output tokens they report #49 its first check timed
out). pascals-triangle, which passed on its second call on Charge local model calls the output tokens they report #49, failed
here: its first edit failed the tests and its repair reasoned up to the
4,096-token limit. Calls 1 to 3 get the same limits on both builds.
Against Keep a local run going when its restart strategy cannot answer #48, say and pascals-triangle, first-call passes there, failed.
free RAM at the start of each exercise was 3.0 to 4.3 GB here, 3.2 to 4.7
on Charge local model calls the output tokens they report #49 and 2.1 to 4.4 on Keep a local run going when its restart strategy cannot answer #48.
Limits
call, fails with this budget.
on the call budget with 6,449 to 12,275 of its 16,384 output tokens unused.
makes no change leads to a restart, which needs a strategy call and an edit
call (shop-coupon-expiry in the CI smoke run). No laptop goal hit this
here.