Skip to content

Keep 4 calls for local goals on models that reason - #50

Merged
mattbaconz merged 1 commit into
mainfrom
mbcz/reasoning-four-calls
Nov 19, 2026
Merged

mattbaconz merged 1 commit into
mainfrom
mbcz/reasoning-four-calls

Conversation

@mattbaconz

@mattbaconz mattbaconz commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem

#47 gave a local goal with one editable path, on a model that reasons, 8
model calls and up to 16,384 output tokens, where other models get 4 calls
and 4,096 tokens. The 4,096-token reserve had cut the later calls to 512
output tokens, which the reasoning used up. #49 settles each call's charge to
the output the runtime reports, so later calls now get their full limit, and
the 16,384-token reserve alone gives each of 4 calls a full 4,096-token
response on the laptop profile.

The extra calls did not pay for themselves. Across eight laptop runs of
qwen3.5:4b with thinking on and 8 calls (126 goals), 75 goals found a
candidate that passed their checks:

Call that produced it 1 2 3 4 5 6 to 8
Goals 55 14 5 0 1 0

The one fifth-call pass is state-of-tic-tac-toe in the 2026-11-10 ablation
(exp-all-c): four repairs failed, the fifth passed, and the goal took 172 s.

Failing goals pay for the extra calls. On #49's laptop run, calls 4 to 8 had
a median limit of 3,996 tokens, 14 of them reached a check and none passed,
and the six goals that ran out of budget took 212 to 346 s of their 600 s.

Change

  • adjust_reviewed_plan_budget in phonton-cli/src/local_goal_cli.rs no
    longer raises a one-path goal on a model that reasons to 8 calls. It keeps
    the default 4 calls, reserves one calibrated response of output per call
    (up to 16,384 tokens, so 16,384 on the laptop profile, as before), and keeps
    enough check launches for a baseline and 4 candidates. With 16,384 tokens
    and 4 calls, every call gets the full 4,096-token limit even when each
    earlier call writes its whole limit.
  • Other models, multi-path goals and explicit budgets are unchanged.
  • docs/local-harness.md and the changelog describe the 4-call budget. The
    unreleased changelog entry for Charge local model calls the output tokens they report #49's settlement described the 8-call goal;
    it now gives the default 4,096-token budget's limits (2,048, 1,024, 512 and
    512, as Keep a local run going when its restart strategy cannot answer #48's CI smoke run recorded them) and that run's charged and
    reported output.

Tests

  • search_replace_review_budget_covers_best_case_scope_within_caps now
    uses a laptop-sized profile that reasons (16,384-token context, 4,096-token
    calibrated output): a one-path goal gets 4 calls, 16 check launches and
    16,384 output tokens, a one-path goal with 4 checks gets 20 check launches,
    and the same profile without reasoning, or with an explicit budget, keeps
    4,096 tokens. The earlier case used a 1,024-token allowance, where 4 calls
    of 1,024 equal the default 4,096 and the test could not tell whether the
    reserve was raised.
  • No red run: on the old code the one-path goal has 8 calls, so the first
    assertion fails.

Measured

CI

CI passes on ba52c3f: Rust workspace (Linux), Windows (workspace) and macOS
(local runtime and CLI).

No CI smoke run: the smoke model never takes this branch. In #49's smoke run
all 33 one-path goals had 4 calls and 4,096 tokens, so none had a profile
that reasons.

Laptop rerun

qwen3.5:4b on the same 20 Exercism exercises as #48 and #49, with fresh
calibration (the runtime's thinking mode, SearchReplace, a 16,384-token
context and 4,096 output tokens), a Windows build of ba52c3f and the same
runner, which waits for free memory before each exercise and reruns an
attempt that stops on memory pressure (three were rerun this time).

#48 #49 This PR
Tests passed 16 11 14
Claimed 15 10 13
False claims 0 0 0
Claimed on the first call 12 8 11
Median time 38 s 94 s 41 s
Time of the 20 counted runs 27 min 46 min 30 min
Output tokens reported 61,186 120,749 66,276

Limits

  • A goal like state-of-tic-tac-toe in the ablation, which passed on its fifth
    call, fails with this budget.
  • A failing goal still takes two to four minutes on this laptop. It now ends
    on the call budget with 6,449 to 12,275 of its 16,384 output tokens unused.
  • A 4-call goal can still leave its last call unused: a third call that
    makes no change leads to a restart, which needs a strategy call and an edit
    call (shop-coupon-expiry in the CI smoke run). No laptop goal hit this
    here.

A one-path goal on a model that reasons kept its output reserve of one calibrated response per call, up to 16,384 tokens, but went from 4 to 8 calls. With output settlement the later calls get full limits, yet in eight laptop runs of qwen3.5:4b only 1 of 75 goals that found a passing candidate needed a fifth call, and a failing goal ran for up to 346 s. Keep the 4 calls; the token reserve is unchanged.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T23:17:54.236509Z ba52c3f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@mattbaconz
mattbaconz merged commit dfe76ae into main Nov 19, 2026
3 checks passed
@mattbaconz
mattbaconz deleted the mbcz/reasoning-four-calls branch November 19, 2026 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant