Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,12 @@ This project follows pre-1.0 SemVer: minor versions may still include breaking c
- [fixed] The later model calls of a local goal no longer run short of
output after short replies. Each call was charged its whole output limit
even when the model wrote far less, and each call keeps half of what is
left for later ones, so calls 5 to 8 of an 8-call goal got 1,024, 512, 256
and 256 tokens. In a laptop run of 20 exercises with `qwen3.5:4b`, edits
that finished wrote a median of 1,057 output tokens, half of them between
about 880 and 1,310, and three of the four goals that failed ended with
their last calls cut off at 256 to 512 tokens. A call is now charged the
output tokens the runtime reports once it answers, and a reply without a
count stays charged its whole limit. The receipt's
left for later ones, so the four calls of a goal with the default
4,096-token budget got 2,048, 1,024, 512 and 512 tokens however short the
replies were. In a CI run of 36 goals with `qwen2.5-coder:3b`, the calls
were charged 120,160 output tokens and the runtime reported 29,353. A call
is now charged the output tokens the runtime reports once it answers, and
a reply without a count stays charged its whole limit. The receipt's
`generated_tokens_reserved` total follows the same rule.
- [fixed] A restart edit no longer stops when its strategy's one-line source
anchor sits inside a small symbol that the edit prompt offers as one
Expand Down Expand Up @@ -67,13 +66,14 @@ This project follows pre-1.0 SemVer: minor versions may still include breaking c
again. Calibrating a model that reasons takes longer, because its probes
reason too, and its goals take longer per call.
- [changed] On a model that reasons before it answers, a local goal with one
editable path and no explicit budget reserves 8 model calls and up to
16,384 output tokens, instead of 4 calls and 4,096 tokens. The earlier
budget stopped a run after four calls, well inside its 600-second limit,
and cut the later calls to 512 output tokens, which the reasoning used up.
A multi-path goal on a unified-diff model keeps four calls and gets one
calibrated response of output per call. Other models and explicit budgets
are unchanged.
editable path and no explicit budget reserves one calibrated response of
output for each of its 4 model calls, up to 16,384 tokens, instead of
4,096 tokens in all. The smaller reserve cut the later calls to 512 output
tokens, which the reasoning used up. A multi-path goal on a unified-diff
model gets the same reserve. Other models and explicit budgets are
unchanged. More calls did not help: in eight laptop runs of `qwen3.5:4b`
with 8 calls, 1 of the 75 goals that found a passing candidate needed a
fifth call, and a failing goal ran for up to 346 seconds.
- [changed] On a model that reasons before it answers, when a local goal's
scope is one existing file of at most 100 lines and the model uses
search/replace, the first edit and every repair ask for the complete
Expand Down
14 changes: 7 additions & 7 deletions docs/local-harness.md
Original file line number Diff line number Diff line change
Expand Up @@ -412,13 +412,13 @@ and any offline dependency-preparation command. A smaller explicit budget is
shown as a plan warning and refused before host checks run.
On a model that reasons before it answers (a calibrated profile in the
runtime's own thinking mode whose probes returned thinking output), a reviewed
plan with one editable path and an implicit budget reserves 8 model calls and
up to 16,384 output tokens (one calibrated response per call), and raises the
check reserve to cover those calls, up to 32 launches. Reasoning then leaves
room for the edit, and a failed first attempt for repairs and a restart; the
600-second wall budget still bounds the run. Other models keep 4 calls and
4,096 tokens: without reasoning the larger reserve only lengthens a reply
that runs to its output limit.
plan with one editable path and an implicit budget keeps its 4 model calls and
reserves one calibrated response of output for each, up to 16,384 tokens, and
keeps enough check launches for a baseline and 4 candidates, up to 32.
Reasoning then leaves room for the edit in every call, a repair or a restart
included; the 600-second wall budget still bounds the run. Other models keep
4 calls and 4,096 tokens: without reasoning the larger reserve only lengthens
a reply that runs to its output limit.
When a selected model uses SearchReplace, each model response edits one path.
Reviewed plans with an implicit budget reserve at least one call per editable
path and scale check/output reserves up to the worker caps. The displayed plan
Expand Down
36 changes: 24 additions & 12 deletions phonton-cli/src/local_goal_cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -840,13 +840,11 @@ fn adjust_reviewed_plan_budget(
// that ran to an 8,192-token limit used up the whole wall budget.
let reasons = selection.is_some_and(|selected| selected.reasons);
if editable <= 1 {
// A one-path goal stopped after 4 calls, at a median 41 s of its
// 600 s wall budget, with later calls cut to 512 output tokens.
// Give repairs and restarts the worker's caps; the wall clock still
// bounds the run.
// A one-path goal keeps its 4 calls and gets one calibrated response
// of output for each. With 8 calls, 1 of 75 laptop passes came after
// the fourth call, and a failing goal ran up to 346 s of its 600 s.
if !explicit_budget && reasons {
let budget = &mut plan.request.budget;
budget.generations = budget.generations.max(8);
let checks = (budget.generations + 1).saturating_mul(checks_per_candidate);
budget.check_runs = budget.check_runs.max(checks.min(32));
let tokens = u64::from(per_call).saturating_mul(u64::from(budget.generations));
Expand Down Expand Up @@ -1922,7 +1920,10 @@ mod tests {
reasons: false,
profile_sha256: "fixture".into(),
};
let reasoning = ReviewedModelSelection {
// A model that reasons, calibrated as on a 16 GB laptop.
let laptop = ReviewedModelSelection {
context_tokens: 16_384,
output_tokens: 4096,
reasons: true,
..selection.clone()
};
Expand Down Expand Up @@ -1967,22 +1968,33 @@ mod tests {
.iter()
.any(|warning| warning.contains("above the 32-slot cap")));

// A one-path goal gets room to reason only on a model that reasons.
// A one-path goal gets room to reason only on a model that reasons:
// its 4 calls get one calibrated response of output each.
let mut one = make_plan(1, 1);
adjust_reviewed_plan_budget(&mut one, Some(&reasoning), false);
assert_eq!(one.request.budget.generations, 8);
adjust_reviewed_plan_budget(&mut one, Some(&laptop), false);
assert_eq!(one.request.budget.generations, 4);
assert_eq!(one.request.budget.check_runs, 16);
assert_eq!(one.request.budget.generated_tokens, 8192);
assert_eq!(one.request.budget.generated_tokens, 16_384);
assert_eq!(one.request.budget.wall_seconds, 600);
assert!(one.warnings.is_empty());

// The check reserve covers a baseline and 4 candidates.
let mut one_checked = make_plan(1, 4);
adjust_reviewed_plan_budget(&mut one_checked, Some(&laptop), false);
assert_eq!(one_checked.request.budget.generations, 4);
assert_eq!(one_checked.request.budget.check_runs, 20);

let laptop_plain = ReviewedModelSelection {
reasons: false,
..laptop.clone()
};
let mut one_plain = make_plan(1, 1);
adjust_reviewed_plan_budget(&mut one_plain, Some(&selection), false);
adjust_reviewed_plan_budget(&mut one_plain, Some(&laptop_plain), false);
assert_eq!(one_plain.request.budget.generations, 4);
assert_eq!(one_plain.request.budget.generated_tokens, 4096);

let mut one_explicit = make_plan(1, 1);
adjust_reviewed_plan_budget(&mut one_explicit, Some(&reasoning), true);
adjust_reviewed_plan_budget(&mut one_explicit, Some(&laptop), true);
assert_eq!(one_explicit.request.budget.generations, 4);
assert_eq!(one_explicit.request.budget.generated_tokens, 4096);

Expand Down
Loading