Repository navigation
Charge local model calls the output tokens they report - #49
Merged
Merged
Conversation
Each local model call was charged its whole output limit even when the model wrote far less. With half of what is left kept for later calls, calls 5 to 8 of an 8-call goal got 1,024, 512, 256 and 256 tokens, and replies were cut off there. Charge the runtime's reported output count once a call answers; a reply without a count stays charged its limit.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
… line A later edit call with a larger output limit can offer a small symbol as one whole-block choice where the strategy call saw its separate lines, so the strategy's one-line anchor was missing from the edit choices and the goal stopped. The edit is now bound to the smallest choice containing the anchor, and its returned search is checked against that choice.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Each model call of a local goal charged its whole output limit to the goal's
shared output budget before inference, and the charge was never settled to
what the model actually wrote. Each call also keeps half of what is left for
a later call, so the limits halved call by call however short the replies
were.
In the 20-exercise laptop run of #48 (
qwen3.5:4b), the 85 model callsreserved 226,764 output tokens and the runtime reported 102,370. Edits that
finished wrote a median of 1,057 tokens, half of them between about 880 and
1,310, against a first-call limit of 4,096. Three of the four goals that
failed ended with their last calls cut off at 256 to 512 tokens, after
several edits that finished:
The CI smoke run of #48 (
qwen2.5-coder:3b, 36 runs) reserved 120,160output tokens and the runtime reported 29,353.
The fourth failure, transpose, is a different problem: its repairs reasoned
until they ran out at 4,000, 2,000, 1,000 and 500 tokens, and only one of its
edits finished.
A restart anchor inside a whole-symbol choice
Larger restart limits exposed a stop that #48's smoke run already hit. An
edit prompt offers a parsed symbol of at most its output limit, and at most
1,024 bytes, as one whole-symbol choice, and leaves that symbol's lines out
of the choices. The strategy call works at 192 tokens, so it sees the same
symbol as separate lines and can pick one of them as its anchor. When the
edit's choices no longer held that line, the goal stopped with "Strategy
source anchor was omitted from the edit choices" before the edit call.
In the CI smoke run of this branch's first commit:
class Stock {. The 980-byteclass was separate lines at the 832-token restart limit of Keep a local run going when its restart strategy cannot answer #48's run, and
those runs ended
review_ready. At this branch's limit of about 3,000 itwas one choice, and both runs stopped.
function parseAmount(text, line) {,a 214-byte function, and stopped. On Keep a local run going when its restart strategy cannot answer #48 their strategies were rejected
and the edit ran without one.
anchored on
function topProducts(orders, n = 5) {.Change
settle_outputinphonton-worker/src/local_run.rs: once a strategy oredit call answers, the receipt's
generated_tokens_reserveddrops thatcall's limit and adds the output count the runtime reports (
eval_count).A call without a count (a failed request, a timeout, a lost resident model)
stays charged its whole limit, as before. Later limits are computed from
the settled total with the same half-for-later rule.
search::bound_edit_choices: a strategy anchor that is not an edit choiceof its own binds the edit to the smallest choice that contains it.
Anchors and choices are unique in the source, so that choice holds the
anchor's only occurrence. An anchor that no choice contains still ends the
search. With SearchReplace the restart edit must return that choice as its
search, and the candidate's decision reason records the binding. A
UnifiedDiff restart is still checked against the anchor's own lines.
LocalRunReceipt::generated_tokens_reserveddoc,docs/local-harness.md,docs/local-harness-reference.mdand thechangelog describe the settlement rule;
docs/local-harness.mdand thechangelog also describe the anchor binding.
Tests
short_replies_leave_later_calls_their_output_limitruns the wholeloop against a mock Ollama with a 2,048-token goal budget, a 1,024-token
profile limit and 4 calls. Every reply writes 100 tokens, the first three
edits fail the check, and the mock cuts off any call whose limit is under
600 tokens. The limits are now 1,024, 974, 924 and 1,024, the fourth edit
passes, and the receipt charges 400 tokens. Before this change the limits
were 1,024, 512, 256 and 256.
restart_line_anchor_inside_a_symbol_choice_binds_to_that_symbolrunsthe whole loop against a mock Ollama. A 230-byte function is one choice at
the restart edit's 512-token limit and separate lines at the strategy's
192. The first edit is cut off, the strategy anchors on the return line,
and the restart edit replaces the whole function. The run ends
review_readywith candidate 2 after 3 calls, and the decision reasonrecords the binding. Before this change it stopped after 2 calls.
restart_line_anchor_binds_to_the_smallest_block_choice_around_it:a line inside a method binds to the method rather than its class, a line
only in the class binds to the class, an exact choice binds to itself, a
UnifiedDiff restart gets no search constraint, and an anchor no choice
contains is still an error.
charged tokens (the mock reports 50 per reply) where it expected 3,072
(the 2,048- and 1,024-token limits).
fixture_runtimeandvalue_fixturehold the mock Ollama and theone-file Python repository that Keep a local run going when its restart strategy cannot answer #48's test built inline; that test and the
new ones share them.
Measured
CI and red run
CI passes on 3192c85: Rust workspace (Linux), Windows (workspace) and macOS
(local runtime and CLI).
The red run
puts this PR's tests on the old code (branch
tmp/output-settle-red). Exactlythe four tests above fail, on Linux and on Windows; the macOS job runs only
the
phonton-localtests, so it passes.short_replies_leave_later_calls_their_output_limit: limits of 1,024, 512,256 and 256 where 1,024, 974, 924 and 1,024 are expected.
expected.
restart_line_anchor_inside_a_symbol_choice_binds_to_that_symbol: the runstops after 2 calls where 3 calls and
review_readyare expected.restart_line_anchor_binds_to_the_smallest_block_choice_around_it: a lineanchor returns "Strategy source anchor was omitted from the edit choices".
CI smoke run
qwen2.5-coder:3bon the ledger, shop and todo-api fixtures, 12 tasks with 3runs each, the same workflow as #48
(run 44326524768).
A run is accepted when every original and hidden test passes after the apply.
review_readysearch_stoppedbudget_exhaustednone of them was accepted. In all three shop-top-ties runs the restart edit
had a limit of about 3,675 tokens, wrote 306, passed the visible checks and
ended
review_ready, and the hidden tests failed. shop-cancel-restock runs2 and 3 had restart limits of 3,038 and 3,039, wrote 813 and 695 tokens,
failed the visible checks and ended
budget_exhausted.are first-reply variance between runners, not this change.
shop-review-duplicate went from 1 to 3 accepted: all three of its first
replies were the same 92-token edit, which Keep a local run going when its restart strategy cannot answer #48 got in one run of three.
ledger-thousands went from 0 to 1 and todo-sort from 3 to 2, each on a
different first reply.
stop: all three strategies were rejected as "Strategy phrase appears
incomplete", the restart edits ran without one, and each run ended
review_readywith its hidden tests failing, as on Keep a local run going when its restart strategy cannot answer #48.about twice as long, so the runner was slower. shop-cancel-restock and
shop-top-ties take longer because they now continue past the old stop.
Laptop rerun
qwen3.5:4bon the same 20 Exercism exercises as #48, with fresh calibration,a Windows build of 3192c85 and the same runner, which waits for free memory
before each exercise and reruns an attempt that stops on memory pressure.
verified after its third call in this run, in Keep a local run going when its restart strategy cannot answer #48's or in the rerun of Make the measured local loop the default for models that reason #47
on 2026-11-15. Calls 1 to 3 got 4,096-token limits on both builds, and an
edit prompt depends on the budget only through that limit, so the five
exercises that flipped (say, state-of-tic-tac-toe, nucleotide-count,
killer-sudoku-helper and satellite) were decided before this change can
act. First calls passed in 7 exercises on Make the measured local loop the default for models that reason #47's rerun, 12 on Keep a local run going when its restart strategy cannot answer #48's and 8
here.
tokens (median 3,996); 13 ran to their limit and 14 reached a check. On Keep a local run going when its restart strategy cannot answer #48,
23 had limits of 256 to 2,048 (median 512); 21 ran to their limit and 6
reached a check. None produced a passing candidate on either build.
builds (scale-generator, dominoes, transpose and markdown) took 212 to 346 s
here against 146 to 290 s, and reported 9,693 to 16,384 output tokens
against 6,081 to 12,975.
check timed out after 120 s, and a check that is unavailable still ends a
run as
review_unverified.Limits
laptop exercise in three runs was verified after its third call, and every
accepted smoke run, on Keep a local run going when its restart strategy cannot answer #48 and here, made one call.
of ending at about 5,900 to 13,000 reported tokens, so on the laptop it
takes up to twice as long (1.1 to 2.0 times for the four exercises above).
A goal that passes by its third call is unchanged. If failures should end
sooner, the lever is a smaller call or token budget for models like this
one, not unsettled reservations; that needs its own change and run.
eval_countcan exceed the call'snum_predicton a replythat finishes normally: 5 calls in the counted attempts of Keep a local run going when its restart strategy cannot answer #48's laptop run
(markdown call 5 reported 1,585 tokens under a 1,024 limit) and 2 in this
one (dominoes call 6 reported 4,629 under 4,096). Settlement charges what
the runtime reports, so such a call costs more than its limit.
budget_exhaustedafter 3 of its 4 callswith 871 of 4,096 tokens charged. Its third call is a repair that makes no
change, which leads to a restart, and a restart needs a strategy call and
an edit call, so the fourth call goes unused. The single-call fallback in
next_with_budgetcovers only a baseline edit that produced no candidate.limit. Here calls 2 to 5 were cut off at 4,096, 4,096, 3,571 and 1,786
tokens and spent the budget by call 8.