Skip to content

Charge local model calls the output tokens they report - #49

Merged
mattbaconz merged 3 commits into
mainfrom
mbcz/output-settle
Nov 18, 2026
Merged

mattbaconz merged 3 commits into
mainfrom
mbcz/output-settle

Conversation

@mattbaconz

@mattbaconz mattbaconz commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Each model call of a local goal charged its whole output limit to the goal's
shared output budget before inference, and the charge was never settled to
what the model actually wrote. Each call also keeps half of what is left for
a later call, so the limits halved call by call however short the replies
were.

In the 20-exercise laptop run of #48 (qwen3.5:4b), the 85 model calls
reserved 226,764 output tokens and the runtime reported 102,370. Edits that
finished wrote a median of 1,057 tokens, half of them between about 880 and
1,310, against a first-call limit of 4,096. Three of the four goals that
failed ended with their last calls cut off at 256 to 512 tokens, after
several edits that finished:

Goal Edits that finished (output tokens) Last calls, all cut off at their limit
scale-generator 962, 977, 1,369, 2,153, 1,144 512, 256, 256
dominoes 821, 892, 1,034, 939, 1,115, 792 488
simple-linked-list 1,031, 874, 1,079, 841 1,024, 512, 256, 256

The CI smoke run of #48 (qwen2.5-coder:3b, 36 runs) reserved 120,160
output tokens and the runtime reported 29,353.

The fourth failure, transpose, is a different problem: its repairs reasoned
until they ran out at 4,000, 2,000, 1,000 and 500 tokens, and only one of its
edits finished.

A restart anchor inside a whole-symbol choice

Larger restart limits exposed a stop that #48's smoke run already hit. An
edit prompt offers a parsed symbol of at most its output limit, and at most
1,024 bytes, as one whole-symbol choice, and leaves that symbol's lines out
of the choices. The strategy call works at 192 tokens, so it sees the same
symbol as separate lines and can pick one of them as its anchor. When the
edit's choices no longer held that line, the goal stopped with "Strategy
source anchor was omitted from the edit choices" before the edit call.

In the CI smoke run of this branch's first commit:

Change

  • settle_output in phonton-worker/src/local_run.rs: once a strategy or
    edit call answers, the receipt's generated_tokens_reserved drops that
    call's limit and adds the output count the runtime reports (eval_count).
    A call without a count (a failed request, a timeout, a lost resident model)
    stays charged its whole limit, as before. Later limits are computed from
    the settled total with the same half-for-later rule.
  • search::bound_edit_choices: a strategy anchor that is not an edit choice
    of its own binds the edit to the smallest choice that contains it.
    Anchors and choices are unique in the source, so that choice holds the
    anchor's only occurrence. An anchor that no choice contains still ends the
    search. With SearchReplace the restart edit must return that choice as its
    search, and the candidate's decision reason records the binding. A
    UnifiedDiff restart is still checked against the anchor's own lines.
  • The LocalRunReceipt::generated_tokens_reserved doc,
    docs/local-harness.md, docs/local-harness-reference.md and the
    changelog describe the settlement rule; docs/local-harness.md and the
    changelog also describe the anchor binding.

Tests

  • New short_replies_leave_later_calls_their_output_limit runs the whole
    loop against a mock Ollama with a 2,048-token goal budget, a 1,024-token
    profile limit and 4 calls. Every reply writes 100 tokens, the first three
    edits fail the check, and the mock cuts off any call whose limit is under
    600 tokens. The limits are now 1,024, 974, 924 and 1,024, the fourth edit
    passes, and the receipt charges 400 tokens. Before this change the limits
    were 1,024, 512, 256 and 256.
  • New restart_line_anchor_inside_a_symbol_choice_binds_to_that_symbol runs
    the whole loop against a mock Ollama. A 230-byte function is one choice at
    the restart edit's 512-token limit and separate lines at the strategy's
    192. The first edit is cut off, the strategy anchors on the return line,
    and the restart edit replaces the whole function. The run ends
    review_ready with candidate 2 after 3 calls, and the decision reason
    records the binding. Before this change it stopped after 2 calls.
  • New unit test restart_line_anchor_binds_to_the_smallest_block_choice_around_it:
    a line inside a method binds to the method rather than its class, a line
    only in the class binds to the class, an exact choice binds to itself, a
    UnifiedDiff restart gets no search constraint, and an anchor no choice
    contains is still an error.
  • The large-context case of the existing creation test now expects 100
    charged tokens (the mock reports 50 per reply) where it expected 3,072
    (the 2,048- and 1,024-token limits).
  • fixture_runtime and value_fixture hold the mock Ollama and the
    one-file Python repository that Keep a local run going when its restart strategy cannot answer #48's test built inline; that test and the
    new ones share them.

Measured

CI and red run

CI passes on 3192c85: Rust workspace (Linux), Windows (workspace) and macOS
(local runtime and CLI).

The red run
puts this PR's tests on the old code (branch tmp/output-settle-red). Exactly
the four tests above fail, on Linux and on Windows; the macOS job runs only
the phonton-local tests, so it passes.

  • short_replies_leave_later_calls_their_output_limit: limits of 1,024, 512,
    256 and 256 where 1,024, 974, 924 and 1,024 are expected.
  • The large-context creation case: 3,072 charged tokens where 100 are
    expected.
  • restart_line_anchor_inside_a_symbol_choice_binds_to_that_symbol: the run
    stops after 2 calls where 3 calls and review_ready are expected.
  • restart_line_anchor_binds_to_the_smallest_block_choice_around_it: a line
    anchor returns "Strategy source anchor was omitted from the edit choices".

CI smoke run

qwen2.5-coder:3b on the ledger, shop and todo-api fixtures, 12 tasks with 3
runs each, the same workflow as #48
(run 44326524768).
A run is accepted when every original and hidden test passes after the apply.

#48 This PR
Accepted 13 of 36 15 of 36
Ended review_ready 24 27
Ended search_stopped 3 0
Ended budget_exhausted 9 9
Output tokens charged / reported by the runtime 120,160 / 29,353 29,912 / 29,912
Wall time of all 36 runs 48 min 58 min
  • No run stopped on a missing strategy anchor. The binding ran in 5 runs, and
    none of them was accepted. In all three shop-top-ties runs the restart edit
    had a limit of about 3,675 tokens, wrote 306, passed the visible checks and
    ended review_ready, and the hidden tests failed. shop-cancel-restock runs
    2 and 3 had restart limits of 3,038 and 3,039, wrote 813 and 695 tokens,
    failed the visible checks and ended budget_exhausted.
  • The changes in accepted runs come from runs that made one call, so they
    are first-reply variance between runners, not this change.
    shop-review-duplicate went from 1 to 3 accepted: all three of its first
    replies were the same 92-token edit, which Keep a local run going when its restart strategy cannot answer #48 got in one run of three.
    ledger-thousands went from 0 to 1 and todo-sort from 3 to 2, each on a
    different first reply.
  • ledger-parens, which stopped twice on this branch's first commit, did not
    stop: all three strategies were rejected as "Strategy phrase appears
    incomplete", the restart edits ran without one, and each run ended
    review_ready with its hidden tests failing, as on Keep a local run going when its restart strategy cannot answer #48.
  • The todo-api runs made the same calls with the same output counts and took
    about twice as long, so the runner was slower. shop-cancel-restock and
    shop-top-ties take longer because they now continue past the old stop.

Laptop rerun

qwen3.5:4b on the same 20 Exercism exercises as #48, with fresh calibration,
a Windows build of 3192c85 and the same runner, which waits for free memory
before each exercise and reruns an attempt that stops on memory pressure.

#48 This PR
Tests passed 16 11
Claimed 15 10
False claims 0 0
Passed on the first call 12 8
Median time 38 s 94 s
Output tokens reported 61,186 120,749

Limits

  • The longer late calls did not raise the pass rate on either model. No
    laptop exercise in three runs was verified after its third call, and every
    accepted smoke run, on Keep a local run going when its restart strategy cannot answer #48 and here, made one call.
  • A failing goal now spends the generated-token budget it was given instead
    of ending at about 5,900 to 13,000 reported tokens, so on the laptop it
    takes up to twice as long (1.1 to 2.0 times for the four exercises above).
    A goal that passes by its third call is unchanged. If failures should end
    sooner, the lever is a smaller call or token budget for models like this
    one, not unsettled reservations; that needs its own change and run.
  • The runtime's eval_count can exceed the call's num_predict on a reply
    that finishes normally: 5 calls in the counted attempts of Keep a local run going when its restart strategy cannot answer #48's laptop run
    (markdown call 5 reported 1,585 tokens under a 1,024 limit) and 2 in this
    one (dominoes call 6 reported 4,629 under 4,096). Settlement charges what
    the runtime reports, so such a call costs more than its limit.
  • shop-coupon-expiry still ends budget_exhausted after 3 of its 4 calls
    with 871 of 4,096 tokens charged. Its third call is a repair that makes no
    change, which leads to a restart, and a restart needs a strategy call and
    an edit call, so the fourth call goes unused. The single-call fallback in
    next_with_budget covers only a baseline edit that produced no candidate.
  • transpose fails as before: its repairs reason until they reach their
    limit. Here calls 2 to 5 were cut off at 4,096, 4,096, 3,571 and 1,786
    tokens and spent the budget by call 8.

Each local model call was charged its whole output limit even when the
model wrote far less. With half of what is left kept for later calls,
calls 5 to 8 of an 8-call goal got 1,024, 512, 256 and 256 tokens, and
replies were cut off there. Charge the runtime's reported output count
once a call answers; a reply without a count stays charged its limit.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T13:51:46.413371Z 4411efb PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

… line

A later edit call with a larger output limit can offer a small symbol as one
whole-block choice where the strategy call saw its separate lines, so the
strategy's one-line anchor was missing from the edit choices and the goal
stopped. The edit is now bound to the smallest choice containing the anchor,
and its returned search is checked against that choice.
@mattbaconz
mattbaconz merged commit 3cf8d5d into main Nov 18, 2026
7 checks passed
@mattbaconz
mattbaconz deleted the mbcz/output-settle branch November 18, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant