Skip to content

Keep a local run going when its restart strategy cannot answer - #48

Merged
mattbaconz merged 1 commit into
mainfrom
mbcz/strategy-stop
Nov 18, 2026
Merged

mattbaconz merged 1 commit into
mainfrom
mbcz/strategy-stop

Conversation

@mattbaconz

@mattbaconz mattbaconz commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem

On a model that reasons (calibrated in the runtime-default mode, with thinking
output in its probes), every baseline restart first asks for a source-anchored
strategy under a 192-token limit, and the model's reasoning used up that limit
on every such call: 14 of 14 strategy calls across four 20-exercise runs of
qwen3.5:4b on the 16 GB laptop. Each one ended its run search_stopped. In
10 of the 14 the restart came right after a first edit that had also reasoned
past its output limit and returned no content, so 6 of the run's 8 calls went
unused. In the 12-of-20 rerun of main after #47 this was 4 of the 8
failures, and it cost sum-of-multiples, which every other run passed.

With think:false the same model answered both strategy calls of the
thinking-off laptop runs inside the limit (77 and 82 tokens). One was
accepted; the other was rejected as "Strategy phrase appears incomplete",
which also ended its run. In the CI smoke run of main (qwen2.5-coder:3b,
which does not reason) strategies took 70 to 110 tokens, 7 of 10 were
accepted, and the 3 ledger-parens runs ended on an incomplete phrase.

Change

  • ModelProfile::strategy_thinking(): a profile that reasons() and has a
    passing think:false calibration probe asks for strategies with
    think:false. Every other profile keeps its saved mode, and edit calls are
    unchanged.
  • A strategy the model answered but that cannot be used (cut off, outside the
    reviewed spans, malformed, or a repeat of an accepted one) is recorded as
    rejected, and the restart edit runs from the original source without it.
    The candidate's decision reason says so. A strategy call that fails, times
    out or loses the resident model still ends the search.
  • THINK_OFF_PROBE_PREFIX names the probe-name prefix calibration already
    used, so the profile can find those probes.
  • docs/local-harness.md, docs/local-harness-reference.md and the
    changelog describe the new behaviour.

Tests

  • New restart_after_an_output_limit_keeps_a_reasoning_run_going runs the
    whole loop against a mock Ollama. The first edit runs out of output. With a
    passing think:false probe the strategy call sends think:false and its
    answer is accepted; with a failing one the strategy runs out of output too,
    is rejected, and the restart edit runs without it. Both cases end
    review_ready with candidate 2 after 3 calls, and both ended
    search_stopped before this change.

Measured

CI. All 6 checks pass on Linux, macOS and Windows, including the new test.
A windows-latest release build of this commit succeeded and supplied the
laptop binary.

Laptop (qwen3.5:4b, 16 GB laptop with a 6 GB RTX 4050, the same 20
Exercism Python exercises and memory-gated harness as the rerun of main):

Run Tests pass Phonton claimed False claims Median time
main after #47 12 of 20 11 0 109 s
This branch 16 of 20 15 0 38 s
  • Fresh calibration chose the runtime-default mode, the profile reasons, and
    its think:false probes passed, so strategy calls sent think:false.
  • 4 strategy calls, all accepted at 72 to 83 output tokens: 2 in counted
    runs (transpose, dominoes) and 2 in attempts the harness stopped for low
    memory and ran again (scale-generator, transpose). None reached the
    192-token limit. In the rerun of main all 4 strategy calls did, and each
    ended its run. No run on this branch stopped early.
  • The score gain is not evidence for this change. The four exercises that
    now pass (say, sum-of-multiples, nucleotide-count, killer-sudoku-helper)
    passed on their first call or a call-2 repair and never asked for a
    strategy. The two counted runs that took the new path (transpose,
    dominoes) used all 8 calls and still failed. One run of 20 cannot
    separate first-reply variance, so the time and score differences are not
    attributed to this change.

CI smoke (local-smoke.yml, qwen2.5-coder:3b on a CPU runner, 12 tasks
by 3 runs, "accepted" means the hidden acceptance tests pass):

Run Accepted review_ready but not accepted search_stopped
main 14 of 36 7 6
This branch 13 of 36 11 3
  • ledger-parens: all 3 runs stopped on an incomplete strategy phrase on
    main. On this branch all 3 end review_ready and fail the hidden test
    (parenthesized negatives). Runs 2 and 3 got there through the new path;
    run 1 passed the visible checks on its first reply, without a restart.
  • ledger-thousands run 1: accepted on main, a first-reply candidate that
    fails the hidden test here. Neither run restarted, so this is first-reply
    variance; the prompt includes the run folder's name.
  • Every other task has the same status and outcome in every run.
  • This run calibrated the shop and todo-api fixtures in the runtime-default
    mode because those runners generated at 20 tokens per second or faster
    (the Make the measured local loop the default for models that reason #47 rule), while main's run calibrated all three fixtures with
    think:false. The model does not reason, so strategy calls keep the saved
    mode; both fixtures' outcomes match main exactly, and shop-top-ties uses
    the same output tokens per call. ledger, where every difference is, was
    calibrated with think:false in both runs.

Limits

  • Profiles saved by 0.27.1 or earlier keep think:false and do not take the
    reasoning path until phonton models calibrate MODEL runs again, as with
    Make the measured local loop the default for models that reason #47.
  • Continuing after a rejected strategy spends one more edit call where the
    run used to stop. In the smoke run that turned 2 stops into candidates that
    pass the visible checks and fail the hidden ones; review still has to catch
    those.
  • An accepted strategy whose anchor is missing from the edit call's search
    choices still ends the run ("Strategy source anchor was omitted from the
    edit choices"). shop-top-ties stops this way after 3 calls in all 3 runs,
    on main and on this branch. That is a separate fix.

A reasoning model's first edit can use up its output limit, which leaves
nothing to repair, so the search restarts with a strategy call capped at
192 tokens. The model's reasoning used up that cap on all 14 strategy
calls in four 20-exercise laptop runs, and each of those runs ended
search_stopped with the rest of its calls unused.

- Ask for the strategy with think:false when the profile reasons and one
  of its think:false calibration probes passed
  (ModelProfile::strategy_thinking). Edit calls keep the saved mode.
- A strategy the model answered but that cannot be used is recorded as
  rejected, and the restart edit runs from the original source without
  it. A strategy call that fails, times out or loses the resident model
  still ends the search.
- A regression test runs both paths against a mock Ollama runtime.
@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-18T09:27:26.889437Z
We couldn’t complete this review. Comment @codex review to try again.
1f80b78 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@mattbaconz
mattbaconz merged commit cc80438 into main Nov 18, 2026
10 checks passed
@mattbaconz
mattbaconz deleted the mbcz/strategy-stop branch November 18, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant