Skip to content

Make the measured local loop the default for models that reason - #47

Merged
mattbaconz merged 6 commits into
mainfrom
mbcz/local-loop-defaults
Nov 15, 2026
Merged

mattbaconz merged 6 commits into
mainfrom
mbcz/local-loop-defaults

Conversation

@mattbaconz

@mattbaconz mattbaconz commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Makes the local-loop changes from the November Polyglot ablation the default for the kind of model and machine they were measured on: a model that reasons before it answers, on a machine fast enough to afford it.

In that ablation (20 Exercism Python exercises, qwen3.5:4b, 16,384-token context, one laptop with a 6 GB GPU) the changes together took Phonton from 8 to 10 of 20 with thinking off to 14 of 20 with thinking on, with no false claim. Each change alone moved nothing.

What changes

  • Failure text, for every model. A repair prompt carries up to 4,000 characters of the failed checks' output (a quarter of the context in characters, at least 450) instead of 450. When that much does not fit beside the source, the 450-character excerpt is sent as before.
  • Thinking is decided by calibration. Calibration still measures think:false first. When those probes generate at 20 tokens per second or more by the runtime's own timing, or when think:false cannot edit or create a file, it also measures the runtime's own thinking mode, with an output limit of a quarter of the context (512 to 2,048 tokens), and keeps that mode when its new-file probe passes. A slower machine, or a runtime that reports no timing, keeps think:false.
  • Budget, for a model that reasons. A one-path goal with an implicit budget reserves 8 model calls and up to 16,384 output tokens (was 4 calls and 4,096 tokens). A multi-path goal on a unified-diff model keeps four calls and gets one calibrated response of output per call. Other models and explicit budgets are unchanged.
  • Whole-file edits, for a model that reasons. With a SearchReplace model and a scope of one existing file of at most 100 lines, the first edit and every repair ask for the complete corrected file. A restart keeps its span edit. The existing whole-file retry after an unchanged search/replace still applies to every model, up to 300 lines.

"A model that reasons" is a calibrated profile in the runtime-default thinking mode whose probes returned thinking output (ModelProfile::reasons). The reviewed plan records it (ReviewedModelSelection::reasons), so the plan a user approves shows the budget it will run with.

Why the gate

The first version of this branch (9246e29) probed thinking first and applied the budget and the whole-file edits to every model. The CI smoke benchmark (local-smoke.yml: qwen2.5-coder:3b, 4-vCPU runner without a GPU, 32,768-token context, 12 tasks, 3 runs each) showed what that costs a model that does not reason on a slow machine:

Build Accepted Runs at the 600 s wall Thinking mode chosen
main a89864c 14 of 36 0 off
ungated 9246e29 13 of 36 4 runtime default
this branch 412dc35 14 of 36 0 off

The four runs at the wall spent all 600 seconds inside their first model call: the larger budget gave that call an 8,192-token allowance (2,048 on main) and the reply ran to the limit at about 12 tokens per second.

With the gate, the smoke run is the same as main: the same accepted count for each of the 12 tasks, the same end states (21 review-ready, 6 search-stopped, 9 budget-exhausted), the same token count in 33 of 36 runs, and four calibration probes (the runtime-default mode is never requested under 20 tokens per second). Its wall time is not comparable: two of its three jobs landed on slower runner CPUs than on main (the calibration probes took 29.8 s against 19.1 s, and 18.6 s against 13.7 s). The job that ran on the same CPU has the same medians (33, 34, 10 and 3 s).

Verification

  • Unit tests: the one-file scope rule; budget sizing with and without a profile that reasons, with and without an explicit budget; six calibration outcomes (no timing keeps the earlier behaviour in three cases; a fast machine with both modes passing chooses the runtime default; a fast machine whose reasoning uses up the probe limit keeps think:false; a slow machine never requests the runtime default).
  • CI: workspace tests on Linux, macOS and Windows; fmt, clippy and the release build on Linux.
  • CI smoke: the table above (runs 43670372522, 43670375241 and 43676081334).
  • 20-exercise slice on the laptop with this build and a fresh calibration: next section.

Laptop rerun

The 20 exercises of the ablation, on the same laptop (RTX 4050 with 6 GB, Ollama 0.34.2, qwen3.5:4b, 16,384-token context), with this branch's Windows build and a fresh calibration.

Calibration measured think:false first (38 to 50 tokens per second), went on to the runtime default and kept it, with SearchReplace. In that mode the edit probe took 1,175 output tokens and the new-file probe 1,477: the earlier 512-token probe limit would have failed both.

Run Passed Claimed False claims
Ablation, all four changes, thinking on 14 of 20 13 0
Ablation, the same plus one more experimental change 13 of 20 12 0
Ablation, nine other variants (thinking off, or not all four changes) 8 to 10 of 20 7 to 9 0
This branch 12 of 20 11 0
  • The run passed four exercises that the seven thinking-off runs passed once or never: state-of-tic-tac-toe, complex-numbers, satellite and pascals-triangle.
  • Against the 14-of-20 run it lost sum-of-multiples, nucleotide-count and killer-sudoku-helper and gained pascals-triangle. All four diverged on the first model call. That call's prompt is the same in both builds except for the run directory's name inside the failed checks' output: markdown, the one exercise whose checks pass before any edit, got the same 1,158-token prompt and the same 1,662-token reply as in the ablation. So 12, 13 and 14 are three draws of the same loop, not a difference between builds; one run of 20 exercises cannot place it more exactly.
  • 4 of the 8 failures stopped after a single model call: the first reply ran to its 4,096-token limit and the restart strategy call could not answer inside its 192-token cap. That happened once and four times in the two ablation runs above. It is the next thing to fix (see the limits below).
  • The laptop was in use. Phonton stopped 6 calibration attempts and 11 exercise attempts for low host memory; they were run again and are not counted. No retried exercise turned a failure into a pass: say, scale-generator and nucleotide-count failed again, markdown ended as it does in every run (its tests already pass, nothing claimed), and the dropped hexadecimal attempt had made no model call.
  • Times are not comparable with the ablation (median 109 s against 53 s): with about 2 GB of free RAM the weights were read from disk again for every exercise.

Known limits, not in this change

  • Output tokens are charged by reservation, so with 8 calls the last three get 512, 256 and 256 tokens, which a model that reasons first can use up.
  • The restart strategy call keeps its 192-token cap, which a model that reasons first uses up. When the first reply of a run hits its output limit, the run then stops with its other calls unused.
  • A check that times out ends the run as unverified instead of asking for a repair (killer-sudoku-helper in the laptop run).
  • The failed checks' output carries the run directory's absolute path, which costs tokens and makes two runs of one goal differ.
  • At a 32,768-token context a profile that reasons gives its first call up to 8,192 output tokens. That is unmeasured: the laptop runs used 16,384.
  • A profile saved by 0.27.1 or earlier keeps its mode, usually think:false, until phonton models calibrate MODEL runs again.
  • Calibrating a model that reasons takes longer, because its probes reason too.

A SearchReplace answer changes one span, so a stub with two functions took one model call per function and a repair of several failures one call per failure. When the scope is one existing file of at most 300 lines, the first edit and every repair now ask for the complete corrected file. A restart keeps its span edit.
With an implicit budget a one-path goal stopped after 4 calls, well inside its 600 s wall budget, and later calls were cut to 512 output tokens. A multi-path diff plan keeps its calls and gets one calibrated response of output per call. Explicit budgets are unchanged.
Calibration tried think:false first and kept it. It now measures the runtime's own mode first, with probe limits of a quarter of the context (512 to 2,048 tokens) so reasoning does not use up the limit, and falls back to think:false when no new-file format passes.
On a CPU runner with a model that does not reason, the larger budget let a reply that ran to its 8,192-token limit use up the whole 600 s wall budget, and nothing was gained. Calibration now keeps think:false first and measures the runtime's own mode only when those probes generate at 20 tokens per second or more. The larger budget and the whole-file scope edit apply to a profile that reasons, and the whole-file scope edit to files of at most 100 lines, the range the measured slice covered.
@mattbaconz mattbaconz changed the title Make the measured local loop the default Make the measured local loop the default for models that reason Nov 15, 2026
@mattbaconz
mattbaconz marked this pull request as ready for review November 15, 2026 08:05
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T08:08:10.922766Z 412dc35 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@mattbaconz
mattbaconz merged commit 0d5c9ce into main Nov 15, 2026
10 checks passed
@mattbaconz
mattbaconz deleted the mbcz/local-loop-defaults branch November 15, 2026 08:06

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 412dc351c2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1135 to +1137
let assembly = match assemble(&detail) {
Ok(assembly) if assembly.evidence.feedback_omitted => assemble(&diagnostics),
result => result,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retry compact feedback when a creation prompt exceeds context

For a new-file goal with verbose baseline failures and a goal near the context limit, the larger feedback can make a previously fitting creation prompt overflow. In context.rs, assemble_with_feedback returns Creation goal and output reserve exceed calibrated context, which assemble_with_evidence propagates without trimming feedback because its retry condition only handles No relevant source excerpt fits errors. This match then forwards the error instead of trying the compact diagnostics, aborting the run before any model call even when the previous 450-character excerpt would fit. Retry with compact diagnostics for this context-fit error as well.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant