Repository navigation
Make the measured local loop the default for models that reason - #47
Conversation
A SearchReplace answer changes one span, so a stub with two functions took one model call per function and a repair of several failures one call per failure. When the scope is one existing file of at most 300 lines, the first edit and every repair now ask for the complete corrected file. A restart keeps its span edit.
With an implicit budget a one-path goal stopped after 4 calls, well inside its 600 s wall budget, and later calls were cut to 512 output tokens. A multi-path diff plan keeps its calls and gets one calibrated response of output per call. Explicit budgets are unchanged.
Calibration tried think:false first and kept it. It now measures the runtime's own mode first, with probe limits of a quarter of the context (512 to 2,048 tokens) so reasoning does not use up the limit, and falls back to think:false when no new-file format passes.
On a CPU runner with a model that does not reason, the larger budget let a reply that ran to its 8,192-token limit use up the whole 600 s wall budget, and nothing was gained. Calibration now keeps think:false first and measures the runtime's own mode only when those probes generate at 20 tokens per second or more. The larger budget and the whole-file scope edit apply to a profile that reasons, and the whole-file scope edit to files of at most 100 lines, the range the measured slice covered.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 412dc351c2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let assembly = match assemble(&detail) { | ||
| Ok(assembly) if assembly.evidence.feedback_omitted => assemble(&diagnostics), | ||
| result => result, |
There was a problem hiding this comment.
Retry compact feedback when a creation prompt exceeds context
For a new-file goal with verbose baseline failures and a goal near the context limit, the larger feedback can make a previously fitting creation prompt overflow. In context.rs, assemble_with_feedback returns Creation goal and output reserve exceed calibrated context, which assemble_with_evidence propagates without trimming feedback because its retry condition only handles No relevant source excerpt fits errors. This match then forwards the error instead of trying the compact diagnostics, aborting the run before any model call even when the previous 450-character excerpt would fit. Retry with compact diagnostics for this context-fit error as well.
Useful? React with 👍 / 👎.
Makes the local-loop changes from the November Polyglot ablation the default for the kind of model and machine they were measured on: a model that reasons before it answers, on a machine fast enough to afford it.
In that ablation (20 Exercism Python exercises,
qwen3.5:4b, 16,384-token context, one laptop with a 6 GB GPU) the changes together took Phonton from 8 to 10 of 20 with thinking off to 14 of 20 with thinking on, with no false claim. Each change alone moved nothing.What changes
think:falsefirst. When those probes generate at 20 tokens per second or more by the runtime's own timing, or whenthink:falsecannot edit or create a file, it also measures the runtime's own thinking mode, with an output limit of a quarter of the context (512 to 2,048 tokens), and keeps that mode when its new-file probe passes. A slower machine, or a runtime that reports no timing, keepsthink:false."A model that reasons" is a calibrated profile in the runtime-default thinking mode whose probes returned thinking output (
ModelProfile::reasons). The reviewed plan records it (ReviewedModelSelection::reasons), so the plan a user approves shows the budget it will run with.Why the gate
The first version of this branch (9246e29) probed thinking first and applied the budget and the whole-file edits to every model. The CI smoke benchmark (
local-smoke.yml:qwen2.5-coder:3b, 4-vCPU runner without a GPU, 32,768-token context, 12 tasks, 3 runs each) showed what that costs a model that does not reason on a slow machine:The four runs at the wall spent all 600 seconds inside their first model call: the larger budget gave that call an 8,192-token allowance (2,048 on main) and the reply ran to the limit at about 12 tokens per second.
With the gate, the smoke run is the same as main: the same accepted count for each of the 12 tasks, the same end states (21 review-ready, 6 search-stopped, 9 budget-exhausted), the same token count in 33 of 36 runs, and four calibration probes (the runtime-default mode is never requested under 20 tokens per second). Its wall time is not comparable: two of its three jobs landed on slower runner CPUs than on main (the calibration probes took 29.8 s against 19.1 s, and 18.6 s against 13.7 s). The job that ran on the same CPU has the same medians (33, 34, 10 and 3 s).
Verification
think:false; a slow machine never requests the runtime default).Laptop rerun
The 20 exercises of the ablation, on the same laptop (RTX 4050 with 6 GB, Ollama 0.34.2,
qwen3.5:4b, 16,384-token context), with this branch's Windows build and a fresh calibration.Calibration measured
think:falsefirst (38 to 50 tokens per second), went on to the runtime default and kept it, with SearchReplace. In that mode the edit probe took 1,175 output tokens and the new-file probe 1,477: the earlier 512-token probe limit would have failed both.Known limits, not in this change
think:false, untilphonton models calibrate MODELruns again.