diff --git a/CHANGELOG.md b/CHANGELOG.md index 21da80f..9083a8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,13 +11,12 @@ This project follows pre-1.0 SemVer: minor versions may still include breaking c - [fixed] The later model calls of a local goal no longer run short of output after short replies. Each call was charged its whole output limit even when the model wrote far less, and each call keeps half of what is - left for later ones, so calls 5 to 8 of an 8-call goal got 1,024, 512, 256 - and 256 tokens. In a laptop run of 20 exercises with `qwen3.5:4b`, edits - that finished wrote a median of 1,057 output tokens, half of them between - about 880 and 1,310, and three of the four goals that failed ended with - their last calls cut off at 256 to 512 tokens. A call is now charged the - output tokens the runtime reports once it answers, and a reply without a - count stays charged its whole limit. The receipt's + left for later ones, so the four calls of a goal with the default + 4,096-token budget got 2,048, 1,024, 512 and 512 tokens however short the + replies were. In a CI run of 36 goals with `qwen2.5-coder:3b`, the calls + were charged 120,160 output tokens and the runtime reported 29,353. A call + is now charged the output tokens the runtime reports once it answers, and + a reply without a count stays charged its whole limit. The receipt's `generated_tokens_reserved` total follows the same rule. - [fixed] A restart edit no longer stops when its strategy's one-line source anchor sits inside a small symbol that the edit prompt offers as one @@ -67,13 +66,14 @@ This project follows pre-1.0 SemVer: minor versions may still include breaking c again. Calibrating a model that reasons takes longer, because its probes reason too, and its goals take longer per call. - [changed] On a model that reasons before it answers, a local goal with one - editable path and no explicit budget reserves 8 model calls and up to - 16,384 output tokens, instead of 4 calls and 4,096 tokens. The earlier - budget stopped a run after four calls, well inside its 600-second limit, - and cut the later calls to 512 output tokens, which the reasoning used up. - A multi-path goal on a unified-diff model keeps four calls and gets one - calibrated response of output per call. Other models and explicit budgets - are unchanged. + editable path and no explicit budget reserves one calibrated response of + output for each of its 4 model calls, up to 16,384 tokens, instead of + 4,096 tokens in all. The smaller reserve cut the later calls to 512 output + tokens, which the reasoning used up. A multi-path goal on a unified-diff + model gets the same reserve. Other models and explicit budgets are + unchanged. More calls did not help: in eight laptop runs of `qwen3.5:4b` + with 8 calls, 1 of the 75 goals that found a passing candidate needed a + fifth call, and a failing goal ran for up to 346 seconds. - [changed] On a model that reasons before it answers, when a local goal's scope is one existing file of at most 100 lines and the model uses search/replace, the first edit and every repair ask for the complete diff --git a/docs/local-harness.md b/docs/local-harness.md index 323ef11..44fa7f7 100644 --- a/docs/local-harness.md +++ b/docs/local-harness.md @@ -412,13 +412,13 @@ and any offline dependency-preparation command. A smaller explicit budget is shown as a plan warning and refused before host checks run. On a model that reasons before it answers (a calibrated profile in the runtime's own thinking mode whose probes returned thinking output), a reviewed -plan with one editable path and an implicit budget reserves 8 model calls and -up to 16,384 output tokens (one calibrated response per call), and raises the -check reserve to cover those calls, up to 32 launches. Reasoning then leaves -room for the edit, and a failed first attempt for repairs and a restart; the -600-second wall budget still bounds the run. Other models keep 4 calls and -4,096 tokens: without reasoning the larger reserve only lengthens a reply -that runs to its output limit. +plan with one editable path and an implicit budget keeps its 4 model calls and +reserves one calibrated response of output for each, up to 16,384 tokens, and +keeps enough check launches for a baseline and 4 candidates, up to 32. +Reasoning then leaves room for the edit in every call, a repair or a restart +included; the 600-second wall budget still bounds the run. Other models keep +4 calls and 4,096 tokens: without reasoning the larger reserve only lengthens +a reply that runs to its output limit. When a selected model uses SearchReplace, each model response edits one path. Reviewed plans with an implicit budget reserve at least one call per editable path and scale check/output reserves up to the worker caps. The displayed plan diff --git a/phonton-cli/src/local_goal_cli.rs b/phonton-cli/src/local_goal_cli.rs index b4a662a..6fa1425 100644 --- a/phonton-cli/src/local_goal_cli.rs +++ b/phonton-cli/src/local_goal_cli.rs @@ -840,13 +840,11 @@ fn adjust_reviewed_plan_budget( // that ran to an 8,192-token limit used up the whole wall budget. let reasons = selection.is_some_and(|selected| selected.reasons); if editable <= 1 { - // A one-path goal stopped after 4 calls, at a median 41 s of its - // 600 s wall budget, with later calls cut to 512 output tokens. - // Give repairs and restarts the worker's caps; the wall clock still - // bounds the run. + // A one-path goal keeps its 4 calls and gets one calibrated response + // of output for each. With 8 calls, 1 of 75 laptop passes came after + // the fourth call, and a failing goal ran up to 346 s of its 600 s. if !explicit_budget && reasons { let budget = &mut plan.request.budget; - budget.generations = budget.generations.max(8); let checks = (budget.generations + 1).saturating_mul(checks_per_candidate); budget.check_runs = budget.check_runs.max(checks.min(32)); let tokens = u64::from(per_call).saturating_mul(u64::from(budget.generations)); @@ -1922,7 +1920,10 @@ mod tests { reasons: false, profile_sha256: "fixture".into(), }; - let reasoning = ReviewedModelSelection { + // A model that reasons, calibrated as on a 16 GB laptop. + let laptop = ReviewedModelSelection { + context_tokens: 16_384, + output_tokens: 4096, reasons: true, ..selection.clone() }; @@ -1967,22 +1968,33 @@ mod tests { .iter() .any(|warning| warning.contains("above the 32-slot cap"))); - // A one-path goal gets room to reason only on a model that reasons. + // A one-path goal gets room to reason only on a model that reasons: + // its 4 calls get one calibrated response of output each. let mut one = make_plan(1, 1); - adjust_reviewed_plan_budget(&mut one, Some(&reasoning), false); - assert_eq!(one.request.budget.generations, 8); + adjust_reviewed_plan_budget(&mut one, Some(&laptop), false); + assert_eq!(one.request.budget.generations, 4); assert_eq!(one.request.budget.check_runs, 16); - assert_eq!(one.request.budget.generated_tokens, 8192); + assert_eq!(one.request.budget.generated_tokens, 16_384); assert_eq!(one.request.budget.wall_seconds, 600); assert!(one.warnings.is_empty()); + // The check reserve covers a baseline and 4 candidates. + let mut one_checked = make_plan(1, 4); + adjust_reviewed_plan_budget(&mut one_checked, Some(&laptop), false); + assert_eq!(one_checked.request.budget.generations, 4); + assert_eq!(one_checked.request.budget.check_runs, 20); + + let laptop_plain = ReviewedModelSelection { + reasons: false, + ..laptop.clone() + }; let mut one_plain = make_plan(1, 1); - adjust_reviewed_plan_budget(&mut one_plain, Some(&selection), false); + adjust_reviewed_plan_budget(&mut one_plain, Some(&laptop_plain), false); assert_eq!(one_plain.request.budget.generations, 4); assert_eq!(one_plain.request.budget.generated_tokens, 4096); let mut one_explicit = make_plan(1, 1); - adjust_reviewed_plan_budget(&mut one_explicit, Some(&reasoning), true); + adjust_reviewed_plan_budget(&mut one_explicit, Some(&laptop), true); assert_eq!(one_explicit.request.budget.generations, 4); assert_eq!(one_explicit.request.budget.generated_tokens, 4096);