Repository navigation
Keep a local run going when its restart strategy cannot answer - #48
Merged
Merged
Conversation
A reasoning model's first edit can use up its output limit, which leaves nothing to repair, so the search restarts with a strategy call capped at 192 tokens. The model's reasoning used up that cap on all 14 strategy calls in four 20-exercise laptop runs, and each of those runs ended search_stopped with the rest of its calls unused. - Ask for the strategy with think:false when the profile reasons and one of its think:false calibration probes passed (ModelProfile::strategy_thinking). Edit calls keep the saved mode. - A strategy the model answered but that cannot be used is recorded as rejected, and the restart edit runs from the original source without it. A strategy call that fails, times out or loses the resident model still ends the search. - A regression test runs both paths against a mock Ollama runtime.
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This was referenced Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On a model that reasons (calibrated in the runtime-default mode, with thinking
output in its probes), every baseline restart first asks for a source-anchored
strategy under a 192-token limit, and the model's reasoning used up that limit
on every such call: 14 of 14 strategy calls across four 20-exercise runs of
qwen3.5:4bon the 16 GB laptop. Each one ended its runsearch_stopped. In10 of the 14 the restart came right after a first edit that had also reasoned
past its output limit and returned no content, so 6 of the run's 8 calls went
unused. In the 12-of-20 rerun of
mainafter #47 this was 4 of the 8failures, and it cost sum-of-multiples, which every other run passed.
With
think:falsethe same model answered both strategy calls of thethinking-off laptop runs inside the limit (77 and 82 tokens). One was
accepted; the other was rejected as "Strategy phrase appears incomplete",
which also ended its run. In the CI smoke run of
main(qwen2.5-coder:3b,which does not reason) strategies took 70 to 110 tokens, 7 of 10 were
accepted, and the 3 ledger-parens runs ended on an incomplete phrase.
Change
ModelProfile::strategy_thinking(): a profile thatreasons()and has apassing
think:falsecalibration probe asks for strategies withthink:false. Every other profile keeps its saved mode, and edit calls areunchanged.
reviewed spans, malformed, or a repeat of an accepted one) is recorded as
rejected, and the restart edit runs from the original source without it.
The candidate's decision reason says so. A strategy call that fails, times
out or loses the resident model still ends the search.
THINK_OFF_PROBE_PREFIXnames the probe-name prefix calibration alreadyused, so the profile can find those probes.
docs/local-harness.md,docs/local-harness-reference.mdand thechangelog describe the new behaviour.
Tests
restart_after_an_output_limit_keeps_a_reasoning_run_goingruns thewhole loop against a mock Ollama. The first edit runs out of output. With a
passing
think:falseprobe the strategy call sendsthink:falseand itsanswer is accepted; with a failing one the strategy runs out of output too,
is rejected, and the restart edit runs without it. Both cases end
review_readywith candidate 2 after 3 calls, and both endedsearch_stoppedbefore this change.Measured
CI. All 6 checks pass on Linux, macOS and Windows, including the new test.
A
windows-latestrelease build of this commit succeeded and supplied thelaptop binary.
Laptop (
qwen3.5:4b, 16 GB laptop with a 6 GB RTX 4050, the same 20Exercism Python exercises and memory-gated harness as the rerun of
main):mainafter #47its
think:falseprobes passed, so strategy calls sentthink:false.runs (transpose, dominoes) and 2 in attempts the harness stopped for low
memory and ran again (scale-generator, transpose). None reached the
192-token limit. In the rerun of
mainall 4 strategy calls did, and eachended its run. No run on this branch stopped early.
now pass (say, sum-of-multiples, nucleotide-count, killer-sudoku-helper)
passed on their first call or a call-2 repair and never asked for a
strategy. The two counted runs that took the new path (transpose,
dominoes) used all 8 calls and still failed. One run of 20 cannot
separate first-reply variance, so the time and score differences are not
attributed to this change.
CI smoke (
local-smoke.yml,qwen2.5-coder:3bon a CPU runner, 12 tasksby 3 runs, "accepted" means the hidden acceptance tests pass):
review_readybut not acceptedsearch_stoppedmainmain. On this branch all 3 endreview_readyand fail the hidden test(parenthesized negatives). Runs 2 and 3 got there through the new path;
run 1 passed the visible checks on its first reply, without a restart.
main, a first-reply candidate thatfails the hidden test here. Neither run restarted, so this is first-reply
variance; the prompt includes the run folder's name.
mode because those runners generated at 20 tokens per second or faster
(the Make the measured local loop the default for models that reason #47 rule), while
main's run calibrated all three fixtures withthink:false. The model does not reason, so strategy calls keep the savedmode; both fixtures' outcomes match
mainexactly, and shop-top-ties usesthe same output tokens per call. ledger, where every difference is, was
calibrated with
think:falsein both runs.Limits
think:falseand do not take thereasoning path until
phonton models calibrate MODELruns again, as withMake the measured local loop the default for models that reason #47.
run used to stop. In the smoke run that turned 2 stops into candidates that
pass the visible checks and fail the hidden ones; review still has to catch
those.
choices still ends the run ("Strategy source anchor was omitted from the
edit choices"). shop-top-ties stops this way after 3 calls in all 3 runs,
on
mainand on this branch. That is a separate fix.