diff --git a/CHANGELOG.md b/CHANGELOG.md index 41de414..24a8153 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,34 @@ This project follows pre-1.0 SemVer: minor versions may still include breaking c ### Changed +- [changed] On a machine that generates 20 tokens per second or more, local + calibration also measures the model's own thinking mode and keeps it when + its edit and new-file probes pass. It used to keep `think:false` whenever + that worked. Probes in the runtime-default mode may generate a quarter of + the context (512 to 2,048 tokens) so reasoning does not use up the limit + before the answer. Slower machines keep `think:false`. A saved profile + keeps its mode: run `phonton models calibrate MODEL` to measure the model + again. Calibrating a model that reasons takes longer, because its probes + reason too, and its goals take longer per call. +- [changed] On a model that reasons before it answers, a local goal with one + editable path and no explicit budget reserves 8 model calls and up to + 16,384 output tokens, instead of 4 calls and 4,096 tokens. The earlier + budget stopped a run after four calls, well inside its 600-second limit, + and cut the later calls to 512 output tokens, which the reasoning used up. + A multi-path goal on a unified-diff model keeps four calls and gets one + calibrated response of output per call. Other models and explicit budgets + are unchanged. +- [changed] On a model that reasons before it answers, when a local goal's + scope is one existing file of at most 100 lines and the model uses + search/replace, the first edit and every repair ask for the complete + corrected file. A search/replace reply changes one span, so a file that + needed several changes took one model call for each. The whole file must + fit the calibrated context with an output reserve that can hold it; + otherwise the span edit runs as before. +- [changed] A repair prompt for a local goal carries up to 4,000 characters + of the failed checks' output (a quarter of the context in characters, at + least 450), instead of 450. A test run with several failures no longer + reaches the model as the first and last few lines of one of them. - [changed] With `PHONTON_LOG=debug`, each worker reply is logged with its prompt size and the token counts the provider reported, so the cost of a goal can be traced request by request. diff --git a/docs/local-harness-reference.md b/docs/local-harness-reference.md index dfa0fbb..ca0cfd0 100644 --- a/docs/local-harness-reference.md +++ b/docs/local-harness-reference.md @@ -160,18 +160,29 @@ profiles. Finished probes are saved after each request, so cancellation, memory pressure or an engine exit does not erase their diagnostic output. A retry runs the probes again; an earlier passing profile remains until recalibration finishes. The attempt's starting digest does not certify each later probe's weights. -Calibration keeps its format probes small. For coding goals, the saved profile +Calibration keeps its format probes small: 512 output tokens for an edit or +creation probe and 256 for the tool-call probe, or a quarter of the context +(at most 2,048) when the model may think first. For coding goals, the saved profile reserves up to one quarter of that context for each complete edit response; the goal's shared token budget can reduce it further. When later attempts are allowed, the runner keeps half the remaining token budget for them. This is a generation allowance, not a measured claim that the model can produce a file of that size. Existing profiles keep their saved ceiling until recalibrated. Calibration first requests `think:false` so a thinking model can spend the -small edit-probe budget on the actual edit. If no edit format or no new-file -format passes, it also probes the runtime's default thinking behavior and -prefers a mode that demonstrated new-file creation. The chosen request mode +small edit-probe budget on the actual edit. It also probes the runtime's +default thinking behavior when no edit format or no new-file format passed, +or when the `think:false` probes generated at 20 tokens per second or more +by the runtime's own timing: a machine that fast can afford a model that +reasons before each edit. It then prefers the runtime default if that mode +demonstrated new-file creation. A slower machine, or a runtime that reports +no generation timing, keeps `think:false`. The chosen request mode is saved in a schema-2 digest-bound profile and reused for tool-call probes, -coding edits, and strategy requests. Older schema-1 profiles retain their +coding edits, and strategy requests. A profile in the runtime-default mode +whose probes returned thinking output is treated as a model that reasons: +its goals get the larger budget and the whole-file edits described in +[local-harness.md](local-harness.md). A profile saved by 0.27.1 or earlier +keeps its mode, usually `think:false`, until `phonton models calibrate MODEL` +runs again. Older schema-1 profiles retain their original runtime-default request behavior until recalibrated. Probe results show format compatibility, not general coding quality or proof that a runtime honored the requested thinking setting. diff --git a/docs/local-harness.md b/docs/local-harness.md index fe70007..a5207d2 100644 --- a/docs/local-harness.md +++ b/docs/local-harness.md @@ -407,12 +407,24 @@ candidate passes. A host-approved request needs enough slots for one complete baseline and one complete candidate pass: twice the total of selected checks and any offline dependency-preparation command. A smaller explicit budget is shown as a plan warning and refused before host checks run. +On a model that reasons before it answers (a calibrated profile in the +runtime's own thinking mode whose probes returned thinking output), a reviewed +plan with one editable path and an implicit budget reserves 8 model calls and +up to 16,384 output tokens (one calibrated response per call), and raises the +check reserve to cover those calls, up to 32 launches. Reasoning then leaves +room for the edit, and a failed first attempt for repairs and a restart; the +600-second wall budget still bounds the run. Other models keep 4 calls and +4,096 tokens: without reasoning the larger reserve only lengthens a reply +that runs to its output limit. When a selected model uses SearchReplace, each model response edits one path. Reviewed plans with an implicit budget reserve at least one call per editable path and scale check/output reserves up to the worker caps. The displayed plan warns about caps and the lack of repair slack; it does not promise that every -path will be edited. Explicit budgets submitted for plan review are preserved -and warned about when they cannot cover even a straight sequence. +path will be edited. A plan with several editable paths on a unified-diff +model that reasons keeps its four calls, with an output reserve of one +calibrated response per call, up to 16,384 tokens. Explicit budgets submitted +for plan review are preserved and warned about when they cannot cover even a +straight sequence. The original Git index is captured separately before checks. Its resolved path and exact byte hash are compared before each generation and at final review. Changes or unavailable observations stop the run and clear candidate selection; @@ -428,7 +440,11 @@ fresh baseline approach unless another selected check newly passes. Losing a pass without either signal also restarts. If a check gains a pass, or a different selected check now fails with the same message, the controller keeps that changed candidate for another repair. Feedback includes the -rejected edit and bounded check output. The most recent failed checks precede +rejected edit and the output of the failed checks: a quarter of the calibrated +context in characters, between 450 and 4,000, shared between the failed checks +and keeping the start and the end of each stream. When that much does not fit +beside the source, the 450-character excerpt is sent instead. The most recent +failed checks precede copied edits so they survive tight context limits; uncheckable proposals retain that evidence with its candidate identity (or baseline evidence if none ran). A reply whose raw output repeats an earlier one in the run byte for byte is @@ -437,11 +453,18 @@ then on every strategy and edit call in that run samples at temperature 0.7 with the candidate number as seed (recorded as the candidate's `sampling`); until then calls stay at the calibrated temperature 0. Two successive repeated outputs or candidate identities stop further inference. -When a search/replace reply changes nothing and its file has at most 300 lines, -the next edit from the same parent shows that file whole and asks for the -complete corrected file in the whole-file JSON shape used for new files. This -happens once per branch, and only when the file and an output reserve that can -hold it fit the calibrated context; otherwise the ordinary edit runs. +A search/replace reply changes one span, so a file that needs several changes +took one model call per change. When the scope is one existing file of at most +100 lines and the model uses SearchReplace and reasons before it answers, the +first edit and every repair show that file whole and ask for the complete +corrected file in the whole-file JSON shape used for new files. A restart +keeps its span edit, anchored to the strategy's source span. For any model, +when a search/replace reply changes nothing and its file has at most 300 +lines, the next edit from the same parent is a whole-file edit, once per +branch. Either way the whole-file edit runs only +when the file and an output reserve that can hold it fit the calibrated +context; otherwise the ordinary edit runs. The candidate is still reviewed as +a diff against the captured source. Different diagnostics are a search signal, not proof that code quality improved. Every branch uses the same run budget. Feedback is shortened when necessary to admit source; the exact transmitted diff --git a/phonton-cli/src/local_goal_cli.rs b/phonton-cli/src/local_goal_cli.rs index 94edc9a..b4a662a 100644 --- a/phonton-cli/src/local_goal_cli.rs +++ b/phonton-cli/src/local_goal_cli.rs @@ -491,6 +491,7 @@ fn reviewed_selection( context_tokens: profile.context_tokens, output_tokens: profile.output_tokens, protocol: profile.protocol, + reasons: profile.reasons(), profile_sha256: crate::models_cli::profile_sha256(profile)?, }) } @@ -824,22 +825,46 @@ fn adjust_reviewed_plan_budget( selection: Option<&ReviewedModelSelection>, explicit_budget: bool, ) { - if selection.and_then(|selected| selected.protocol) != Some(EditProtocol::SearchReplace) { - return; - } let editable = if plan.request.new_file.is_some() { 1 + plan.request.editable_existing.len() } else { plan.request.files.len() } as u32; - if editable <= 1 { - return; - } let checks_per_candidate = plan.request.checks.len() as u32 + u32::from(plan.request.preparation.is_some()); let per_call = selection.map_or(0, |selected| { selected.output_tokens.min(selected.context_tokens / 4) }); + // A model that reasons first spends output tokens before it reaches the + // edit. Other models keep the smaller reserves: on a CPU runner a reply + // that ran to an 8,192-token limit used up the whole wall budget. + let reasons = selection.is_some_and(|selected| selected.reasons); + if editable <= 1 { + // A one-path goal stopped after 4 calls, at a median 41 s of its + // 600 s wall budget, with later calls cut to 512 output tokens. + // Give repairs and restarts the worker's caps; the wall clock still + // bounds the run. + if !explicit_budget && reasons { + let budget = &mut plan.request.budget; + budget.generations = budget.generations.max(8); + let checks = (budget.generations + 1).saturating_mul(checks_per_candidate); + budget.check_runs = budget.check_runs.max(checks.min(32)); + let tokens = u64::from(per_call).saturating_mul(u64::from(budget.generations)); + budget.generated_tokens = budget.generated_tokens.max(tokens.min(16_384)); + } + return; + } + if selection.and_then(|selected| selected.protocol) != Some(EditProtocol::SearchReplace) { + // A multi-path diff plan keeps its calls. The 4,096-token reserve + // gave them 2,048, 1,024, 512 and 512 output tokens, which a model + // that reasons first uses up before it reaches the edit. + if !explicit_budget && reasons { + let budget = &mut plan.request.budget; + let tokens = u64::from(per_call).saturating_mul(u64::from(budget.generations)); + budget.generated_tokens = budget.generated_tokens.max(tokens.min(16_384)); + } + return; + } if explicit_budget { if plan.request.budget.generations < editable { plan.warnings.push(format!( @@ -1894,8 +1919,13 @@ mod tests { context_tokens: 4096, output_tokens: 1024, protocol: Some(EditProtocol::SearchReplace), + reasons: false, profile_sha256: "fixture".into(), }; + let reasoning = ReviewedModelSelection { + reasons: true, + ..selection.clone() + }; let make_plan = |files: usize, checks: usize| { let request: LocalRunRequest = serde_json::from_value(json!({ "goal":"Update the scoped files", @@ -1937,10 +1967,28 @@ mod tests { .iter() .any(|warning| warning.contains("above the 32-slot cap"))); + // A one-path goal gets room to reason only on a model that reasons. let mut one = make_plan(1, 1); - adjust_reviewed_plan_budget(&mut one, Some(&selection), false); - assert_eq!(one.request.budget.generations, 4); - assert_eq!(one.request.budget.generated_tokens, 4096); + adjust_reviewed_plan_budget(&mut one, Some(&reasoning), false); + assert_eq!(one.request.budget.generations, 8); + assert_eq!(one.request.budget.check_runs, 16); + assert_eq!(one.request.budget.generated_tokens, 8192); + assert_eq!(one.request.budget.wall_seconds, 600); + assert!(one.warnings.is_empty()); + + let mut one_plain = make_plan(1, 1); + adjust_reviewed_plan_budget(&mut one_plain, Some(&selection), false); + assert_eq!(one_plain.request.budget.generations, 4); + assert_eq!(one_plain.request.budget.generated_tokens, 4096); + + let mut one_explicit = make_plan(1, 1); + adjust_reviewed_plan_budget(&mut one_explicit, Some(&reasoning), true); + assert_eq!(one_explicit.request.budget.generations, 4); + assert_eq!(one_explicit.request.budget.generated_tokens, 4096); + + let mut one_unselected = make_plan(1, 1); + adjust_reviewed_plan_budget(&mut one_unselected, None, false); + assert_eq!(one_unselected.request.budget.generations, 4); let mut explicit = make_plan(5, 1); adjust_reviewed_plan_budget(&mut explicit, Some(&selection), true); @@ -1974,7 +2022,24 @@ mod tests { diff_selection.protocol = Some(EditProtocol::UnifiedDiff); adjust_reviewed_plan_budget(&mut diff, Some(&diff_selection), false); assert_eq!(diff.request.budget.generations, 4); + assert_eq!(diff.request.budget.generated_tokens, 4096); assert!(diff.warnings.is_empty()); + + // On a model that reasons, a larger calibrated allowance gives each + // of the four calls its share. + diff_selection.context_tokens = 16_384; + diff_selection.output_tokens = 4096; + let mut plain_diff = make_plan(5, 4); + adjust_reviewed_plan_budget(&mut plain_diff, Some(&diff_selection), false); + assert_eq!(plain_diff.request.budget.generated_tokens, 4096); + diff_selection.reasons = true; + let mut roomy = make_plan(5, 4); + adjust_reviewed_plan_budget(&mut roomy, Some(&diff_selection), false); + assert_eq!(roomy.request.budget.generations, 4); + assert_eq!(roomy.request.budget.generated_tokens, 16_384); + let mut roomy_explicit = make_plan(5, 4); + adjust_reviewed_plan_budget(&mut roomy_explicit, Some(&diff_selection), true); + assert_eq!(roomy_explicit.request.budget.generated_tokens, 4096); } #[test] @@ -2261,6 +2326,7 @@ mod tests { context_tokens: 4096, output_tokens: 512, protocol: None, + reasons: false, profile_sha256: "profile-hash".into(), }; let mut reviewed = ReviewedLocalPlan { diff --git a/phonton-local/src/runtime.rs b/phonton-local/src/runtime.rs index 96e4810..50350c5 100644 --- a/phonton-local/src/runtime.rs +++ b/phonton-local/src/runtime.rs @@ -339,10 +339,53 @@ fn with_thinking_mode(mut request: Value, thinking: LocalThinkingMode) -> Value request } +/// Output limit for one calibration probe. A model that reasons first spends +/// part of the limit before it writes the answer, so the runtime-default mode +/// gets room in proportion to the context; `think:false` keeps the small limit. +fn probe_output_tokens(thinking: LocalThinkingMode, context: u32, small: u32) -> u32 { + match thinking { + LocalThinkingMode::Off => small, + LocalThinkingMode::RuntimeDefault => (context / 4).clamp(small, 2048), + } +} + +/// Slowest generation, in tokens per second, at which calibration also +/// measures the runtime's own thinking mode. A goal on a model that reasons +/// first may spend 4,096 output tokens on each of its first three calls; at +/// 20 tokens per second those fit the default 600-second wall budget. A +/// slower machine keeps `think:false`. +const MIN_REASONING_TOKENS_PER_SECOND: f64 = 20.0; + +/// Generation speed summed over probe replies, from the runtime's own +/// `eval_count` and `eval_duration`: model load and prompt time are not in it. +#[derive(Default)] +struct GenerationRate { + tokens: u64, + nanos: u64, +} + +impl GenerationRate { + fn record(&mut self, response: &Value) { + if let (Some(tokens), Some(nanos)) = ( + response["eval_count"].as_u64(), + response["eval_duration"].as_u64(), + ) { + self.tokens += tokens; + self.nanos += nanos; + } + } + + fn tokens_per_second(&self) -> Option { + (self.nanos > 0).then(|| self.tokens as f64 * 1e9 / self.nanos as f64) + } +} + struct ModeCalibration { probes: Vec, edit_protocol: Option, creation_protocol: Option, + /// None when the runtime reported no generation timing. + tokens_per_second: Option, } impl ModeCalibration { @@ -1086,6 +1129,7 @@ impl LocalRuntime { model: &LocalModel, admission: &CalibrationAdmission, thinking: LocalThinkingMode, + rate: &mut GenerationRate, observe_hardware: &F, on_probe: &mut P, ) -> Result<(Vec, Vec)> @@ -1113,7 +1157,7 @@ impl LocalRuntime { system: instruction, user: "File add.py contains exactly:\ndef add(a, b): return a - b\n\nFix add so it adds its two arguments.", context, - output: 512, + output: probe_output_tokens(thinking, context, 512), thinking, paths: &["add.py".into()], searches: &["def add(a, b): return a - b".into()], @@ -1124,6 +1168,7 @@ impl LocalRuntime { .await; let probe = match result { Ok(response) => { + rate.record(&response); let output = response["message"]["content"] .as_str() .unwrap_or("") @@ -1138,7 +1183,7 @@ impl LocalRuntime { .as_str() .is_some_and(|value| !value.is_empty()); let thinking_note = if has_thinking { - " Runtime returned separate thinking output." + THINKING_OUTPUT_NOTE } else { "" }; @@ -1183,8 +1228,16 @@ impl LocalRuntime { { let name = model.name.as_str(); let context = admission.context_tokens; + let mut rate = GenerationRate::default(); let (mut probes, passing_edit_protocols) = self - .calibrate_edit_probes(model, admission, thinking, observe_hardware, on_probe) + .calibrate_edit_probes( + model, + admission, + thinking, + &mut rate, + observe_hardware, + on_probe, + ) .await?; let mut creation_protocol = None; for kind in passing_edit_protocols.iter().copied() { @@ -1210,7 +1263,7 @@ impl LocalRuntime { system: instruction, user: "Create the absent file probe_add.py with exactly this content:\ndef add(a, b):\n return a + b\n", context, - output: 512, + output: probe_output_tokens(thinking, context, 512), thinking, paths: &["probe_add.py".into()], searches: &[], @@ -1220,6 +1273,7 @@ impl LocalRuntime { }).await; let creation_probe = match creation_result { Ok(response) => { + rate.record(&response); let output = response["message"]["content"] .as_str() .unwrap_or("") @@ -1260,6 +1314,7 @@ impl LocalRuntime { probes, edit_protocol: passing_edit_protocols.first().copied(), creation_protocol, + tokens_per_second: rate.tokens_per_second(), }) } @@ -1392,7 +1447,14 @@ impl LocalRuntime { &mut record_probe, ) .await?; - let (thinking, mut selected) = if off.creation_protocol.is_some() { + // A machine that generates fast enough can afford a model that reasons + // before each edit, so the runtime's own mode is measured too and + // preferred when it can also create a file. A slower machine, or a + // runtime that reports no timing, keeps `think:false` as before. + let fast = off + .tokens_per_second + .is_some_and(|rate| rate >= MIN_REASONING_TOKENS_PER_SECOND); + let (thinking, mut selected) = if off.creation_protocol.is_some() && !fast { (LocalThinkingMode::Off, off) } else { let mut default = self @@ -1433,7 +1495,7 @@ impl LocalRuntime { "model": name, "stream": false, "keep_alive": "2m", "messages": [{"role":"user", "content":"Call read_file with path add.py. Do not answer in prose."}], "tools": [{"type":"function", "function": {"name":"read_file", "description":"Read a file", "parameters":{"type":"object", "properties":{"path":{"type":"string", "enum":["add.py"]}}, "required":["path"]}}}], - "options": {"num_ctx": context, "num_predict": 256, "temperature": 0} + "options": {"num_ctx": context, "num_predict": probe_output_tokens(thinking, context, 256), "temperature": 0} }), thinking)).await; let tool_probe = match tool_result { Ok(response) => { @@ -2116,6 +2178,8 @@ mod tests { let server_seen = Arc::clone(&seen); let thinking_requests = Arc::new(Mutex::new(Vec::::new())); let server_thinking_requests = Arc::clone(&thinking_requests); + let output_limits = Arc::new(Mutex::new(Vec::::new())); + let server_output_limits = Arc::clone(&output_limits); let thinking_case = Arc::new(AtomicU8::new(0)); let server_thinking_case = Arc::clone(&thinking_case); let server = tokio::spawn(async move { @@ -2169,6 +2233,10 @@ mod tests { .lock() .unwrap() .push(thinking_disabled); + server_output_limits + .lock() + .unwrap() + .push(input["options"]["num_predict"].as_u64().unwrap_or(0)); let instruction = input["messages"][0]["content"].as_str().unwrap_or(""); let (kind, output) = if instruction.contains("exactly path and text") { ( @@ -2189,16 +2257,27 @@ mod tests { }; server_seen.lock().unwrap().push(kind.into()); let case = server_thinking_case.load(Ordering::Relaxed); + // Cases 3 and 4 report 50 tokens per second, case 5 + // reports 5; the earlier cases report no timing. if thinking_disabled && case == 1 { json!({"error":"think=false unsupported"}) - } else if !thinking_disabled && case == 0 { + } else if !thinking_disabled && matches!(case, 0 | 4) { json!({"message":{"thinking":"reasoning used the output limit","content":""},"done":true,"done_reason":"length","eval_count":512}) } else if thinking_disabled && case == 2 && kind.contains("creation") { json!({"message":{"content":"creation did not match"},"done":true}) } else if kind == "tool call" { json!({"message":{"content":"","tool_calls":[{"function":{"name":"read_file","arguments":{"path":"add.py"}}}]},"done":true}) } else { - json!({"message":{"content":output},"done":true,"prompt_eval_count":12,"eval_count":20}) + let mut reply = json!({"message":{"content":output},"done":true,"prompt_eval_count":12,"eval_count":20}); + match case { + 3 | 4 => reply["eval_duration"] = json!(400_000_000_u64), + 5 => reply["eval_duration"] = json!(4_000_000_000_u64), + _ => {} + } + if !thinking_disabled && case == 3 { + reply["message"]["thinking"] = json!("the operator is wrong"); + } + reply } } _ => json!({"error":"Unexpected fixture endpoint"}), @@ -2340,7 +2419,6 @@ mod tests { .await .unwrap() .unwrap(); - server.abort(); assert_eq!( new_file_profile.thinking, Some(LocalThinkingMode::RuntimeDefault) @@ -2356,6 +2434,79 @@ mod tests { &thinking_requests.lock().unwrap()[13..], [true, true, true, true, false, false, false, false, false], ); + // None of these runs reported generation timing, and none reasons. + assert!(!profile.reasons() && !default_profile.reasons() && !new_file_profile.reasons()); + + let calibrate = |case: u8| { + thinking_case.store(case, Ordering::Relaxed); + let endpoint = endpoint.clone(); + let observe_hardware = &observe_hardware; + async move { + let runtime = LocalRuntime::new(&endpoint).unwrap(); + let hardware = HardwareSnapshot { + ram_total_bytes: Some(16u64 << 30), + ram_available_bytes: Some(8u64 << 30), + ..Default::default() + }; + let calibration = runtime.calibrate_with_observer( + "fixture:small", + Some(4096), + hardware, + observe_hardware, + ); + let result = tokio::time::timeout(Duration::from_secs(10), calibration).await; + result.unwrap().unwrap() + } + }; + + // A machine measured at 50 tokens per second can afford reasoning: the + // runtime's own mode is probed too, with room to reason, and kept. + let fast_profile = calibrate(3).await; + assert_eq!( + fast_profile.thinking, + Some(LocalThinkingMode::RuntimeDefault) + ); + assert_eq!(fast_profile.protocol, Some(EditProtocol::UnifiedDiff)); + assert_eq!(fast_profile.creation_status(), CheckStatus::Passed); + assert_eq!(fast_profile.probes[0].name, "Think off: SearchReplace edit"); + assert_eq!(fast_profile.probes[0].status, CheckStatus::Passed); + assert!(fast_profile.reasons()); + assert_eq!( + &thinking_requests.lock().unwrap()[22..], + [true, true, true, true, false, false, false, false, false], + ); + assert_eq!( + &output_limits.lock().unwrap()[22..], + [512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024], + ); + + // As fast, but reasoning uses up the probe limit: `think:false` stays. + let cut_profile = calibrate(4).await; + assert_eq!(cut_profile.thinking, Some(LocalThinkingMode::Off)); + assert_eq!(cut_profile.creation_status(), CheckStatus::Passed); + assert!(cut_profile + .probes + .iter() + .any(|probe| probe.name == "Runtime default: SearchReplace edit" + && probe.status == CheckStatus::Failed)); + assert!(!cut_profile.reasons()); + assert_eq!( + &thinking_requests.lock().unwrap()[31..], + [true, true, true, true, false, false, true], + ); + + // At 5 tokens per second the runtime's own mode is never requested. + let slow_profile = calibrate(5).await; + server.abort(); + assert_eq!(slow_profile.thinking, Some(LocalThinkingMode::Off)); + assert!(slow_profile + .probes + .iter() + .all(|probe| !probe.name.starts_with("Runtime default"))); + assert_eq!( + &thinking_requests.lock().unwrap()[38..], + [true, true, true, true, true], + ); } #[tokio::test] diff --git a/phonton-types/src/local.rs b/phonton-types/src/local.rs index ca78727..2c597f5 100644 --- a/phonton-types/src/local.rs +++ b/phonton-types/src/local.rs @@ -147,6 +147,10 @@ pub const CREATION_PROBE_NAME: &str = "Structured create"; /// Name of the unified-diff new-file transport probe in a model profile. pub const DIFF_CREATION_PROBE_NAME: &str = "UnifiedDiff create"; +/// Appended to an edit probe's detail when the runtime returned thinking +/// output beside the answer. [`ModelProfile::reasons`] reads it back. +pub const THINKING_OUTPUT_NOTE: &str = " Runtime returned separate thinking output."; + /// Editing wire protocol demonstrated by a compatibility probe. #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] @@ -228,6 +232,17 @@ pub struct ModelProfile { } impl ModelProfile { + /// Whether coding requests with this profile reason before they answer: + /// the saved mode leaves thinking to the runtime, and a probe returned + /// thinking output. Goals budget output for that reasoning. + pub fn reasons(&self) -> bool { + self.thinking == Some(LocalThinkingMode::RuntimeDefault) + && self + .probes + .iter() + .any(|probe| probe.detail.contains(THINKING_OUTPUT_NOTE)) + } + /// Keep absent probes on older profiles distinct from measured success. pub fn creation_status(&self) -> CheckStatus { let name = match self.protocol { diff --git a/phonton-types/src/local_run.rs b/phonton-types/src/local_run.rs index ca91a97..b61e71b 100644 --- a/phonton-types/src/local_run.rs +++ b/phonton-types/src/local_run.rs @@ -117,6 +117,10 @@ pub struct ReviewedModelSelection { pub context_tokens: u32, pub output_tokens: u32, pub protocol: Option, + /// Whether the profile reasons before it answers + /// ([`crate::local::ModelProfile::reasons`]); the plan budgets for it. + #[serde(default)] + pub reasons: bool, /// Hash of the complete saved profile, including probes and calibration. pub profile_sha256: String, } diff --git a/phonton-worker/src/local_run.rs b/phonton-worker/src/local_run.rs index 20ee49c..0d55407 100644 --- a/phonton-worker/src/local_run.rs +++ b/phonton-worker/src/local_run.rs @@ -134,9 +134,15 @@ fn creation_system_instruction(path: &Path, protocol: EditProtocol) -> String { } /// Whole-file rewrite of one existing file, using the creation transport. -fn whole_file_system_instruction(path: &Path) -> String { +/// `after_noop` tells the model that its search and replacement changed nothing. +fn whole_file_system_instruction(path: &Path, after_noop: bool) -> String { let path = model_relative_path(path); - format!("Return only JSON with keys path and text. Set path to exactly {path} and text to the complete corrected file, newline-terminated, with every line you do not change kept exactly. Your previous search and replacement changed nothing. No prose, commands, test changes, or other paths.") + let noop = if after_noop { + " Your previous search and replacement changed nothing." + } else { + "" + }; + format!("Return only JSON with keys path and text. Set path to exactly {path} and text to the complete corrected file, newline-terminated, with every line you do not change kept exactly.{noop} No prose, commands, test changes, or other paths.") } /// Whether `resident` may serve the next chat. @@ -586,6 +592,8 @@ async fn run_inner( seen.insert(receipt.baseline_sha256.clone()); let mut unproductive_searches = BTreeSet::new(); let baseline_diagnostics = search::failed_checks_feedback(&receipt.baseline_checks); + let check_chars = search::check_feedback_chars(receipt.profile.context_tokens); + let baseline_detail = search::failed_checks_excerpt(&receipt.baseline_checks, check_chars); for number in 1..=receipt.request.budget.generations { if let Err(error) = observe_original_source( &repository, @@ -625,6 +633,7 @@ async fn run_inner( receipt.request.new_file.is_some() && !receipt.request.editable_existing.is_empty(), ); let mut whole_file = None; + let mut after_noop = false; if decision.action != SearchAction::Stop && receipt.profile.protocol == Some(EditProtocol::SearchReplace) && receipt.request.new_file.is_none() @@ -634,6 +643,14 @@ async fn run_inner( { decision = retry; whole_file = Some(path); + after_noop = true; + } else if receipt.profile.reasons() { + whole_file = search::whole_file_scope( + &receipt.candidates, + &decision, + &receipt.request.files, + &baseline, + ); } } if decision.action == SearchAction::Stop { @@ -768,6 +785,9 @@ async fn run_inner( .and_then(|candidate| candidate.content_sha256.clone()) .unwrap_or_else(|| receipt.baseline_sha256.clone()); let diagnostics = search::feedback(&receipt.candidates, &baseline_diagnostics); + // Edit prompts try this larger excerpt of the failed checks first and + // keep `diagnostics` for a source that does not fit beside it. + let detail = search::feedback_with(&receipt.candidates, &baseline_detail, check_chars); let verifier_diagnostics = search::verifier_feedback(&receipt.candidates, &baseline_diagnostics); let mut strategy_note = None; @@ -1048,18 +1068,23 @@ async fn run_inner( }.to_owned() }; let strategy_reserve = strategy_note.as_ref().map_or(0, String::len); - // A small file the last SearchReplace answer could not change is shown - // whole and rewritten whole; if it cannot fit, the ordinary edit runs. + // A small one-file scope on a model that reasons, or a small file the + // last SearchReplace answer could not change, is shown whole and + // rewritten whole; if it cannot fit, the ordinary edit runs. let whole_file_assembly = whole_file.as_ref().and_then(|path| { - let system = whole_file_system_instruction(path); + let system = whole_file_system_instruction(path, after_noop); let model = context::ModelContext { protocol: EditProtocol::SearchReplace, capacity: receipt.profile.context_tokens, output, }; - context::whole_file(&edit_request, &parent, &system, model, &diagnostics, path) - .ok() - .map(|assembly| (system, assembly)) + let fit = |feedback: &str| { + context::whole_file(&edit_request, &parent, &system, model, feedback, path).ok() + }; + let assembly = fit(&detail) + .filter(|assembly| !assembly.evidence.feedback_omitted) + .or_else(|| fit(&diagnostics)); + assembly.map(|assembly| (system, assembly)) }); if whole_file_assembly.is_none() { // ponytail: the retry's parent stays; restoring the controller's @@ -1069,39 +1094,48 @@ async fn run_inner( "; the whole file did not fit the calibrated context, so an ordinary edit ran", ); } - } else { + } else if after_noop { approach = "whole-file rewrite after an unchanged search/replace".into(); + } else { + approach.push_str(", file rewritten whole"); } let (system, assembly_result) = if let Some((system, assembly)) = whole_file_assembly { (system, Ok(assembly)) } else { - let assembly = context::assemble_with_evidence( - &edit_request, - &parent, - &system, - context::ModelContext { - protocol: receipt - .profile - .protocol - .ok_or_else(|| invalid("Missing measured protocol"))?, - capacity: receipt - .profile - .context_tokens - .saturating_sub(strategy_reserve as u32), - output, - }, - context::EvidenceInput { - feedback: &diagnostics, - verifier: &verifier_diagnostics, - focus_path: if repairing_created_file { - receipt.request.new_file.as_deref() - } else { - None + let model = context::ModelContext { + protocol: receipt + .profile + .protocol + .ok_or_else(|| invalid("Missing measured protocol"))?, + capacity: receipt + .profile + .context_tokens + .saturating_sub(strategy_reserve as u32), + output, + }; + let assemble = |feedback: &str| { + context::assemble_with_evidence( + &edit_request, + &parent, + &system, + model, + context::EvidenceInput { + feedback, + verifier: &verifier_diagnostics, + focus_path: if repairing_created_file { + receipt.request.new_file.as_deref() + } else { + None + }, }, - }, - false, - excluded_searches, - ); + false, + excluded_searches, + ) + }; + let assembly = match assemble(&detail) { + Ok(assembly) if assembly.evidence.feedback_omitted => assemble(&diagnostics), + result => result, + }; (system, assembly) }; let mut assembly = match assembly_result { diff --git a/phonton-worker/src/local_run/search.rs b/phonton-worker/src/local_run/search.rs index 768bf5c..bf02d1f 100644 --- a/phonton-worker/src/local_run/search.rs +++ b/phonton-worker/src/local_run/search.rs @@ -375,6 +375,44 @@ pub(super) fn whole_file_retry( }) } +/// Largest file a one-file scope rewrites whole from its first edit. The +/// slice that measured the rewrite (Polyglot, 2026-11-11) had no file over +/// 96 lines, and a longer file costs a rewrite far more output than a span. +pub(super) const WHOLE_FILE_SCOPE_MAX_LINES: usize = 100; + +/// The file of a one-file scope when `decision`'s edit should rewrite it +/// whole. A SearchReplace answer changes one span, so a stub with two +/// functions took one model call per function, and a repair of several +/// failures one call per failure. A restart keeps its source anchor, and a +/// file over [`WHOLE_FILE_SCOPE_MAX_LINES`] keeps span edits. The caller +/// asks only for a model that reasons: the rewrite was measured with one. +/// `baseline` is the parent directory of a first edit. +pub(super) fn whole_file_scope( + candidates: &[CandidateEvidence], + decision: &SearchDecision, + scope: &[PathBuf], + baseline: &Path, +) -> Option { + let [path] = scope else { + return None; + }; + if !matches!( + decision.action, + SearchAction::Initial | SearchAction::Repair + ) { + return None; + } + let directory = match decision.parent_candidate { + Some(number) => &candidates.iter().find(|c| c.number == number)?.directory, + None => baseline, + }; + let lines = std::fs::read_to_string(directory.join(path)) + .ok()? + .lines() + .count(); + (lines <= WHOLE_FILE_SCOPE_MAX_LINES).then(|| path.clone()) +} + /// Rejection for a generation whose raw output equals an earlier one in the /// same run. Its verdict is already known, so it never reaches the checks. pub(super) const REPEATED_OUTPUT: &str = @@ -522,8 +560,20 @@ pub(super) fn next(candidates: &[CandidateEvidence]) -> SearchDecision { "Rejected edit did not produce a checkable change; revise it without altering the source scope") } -fn failures(candidate: &CandidateEvidence) -> String { - failed_checks_feedback(&candidate.checks) +fn failures(candidate: &CandidateEvidence, chars: usize) -> String { + failed_checks_excerpt(&candidate.checks, chars) +} + +/// Characters of failed check output kept for retrieval anchors, and for a +/// prompt whose source does not fit beside the detailed excerpt. +const COMPACT_CHECK_CHARS: usize = 450; + +/// Characters of failed check output an edit prompt tries first: one per four +/// tokens of calibrated context, at most 4,000. At 16,384 tokens the compact +/// excerpt held one traceback of a ten-test failure, and repairs by a 4B model +/// passed 1 time in 25 (Polyglot slice, 2026-11-11). +pub(super) fn check_feedback_chars(context_tokens: u32) -> usize { + (context_tokens as usize / 4).clamp(COMPACT_CHECK_CHARS, 4000) } fn feedback_output_excerpt(text: &str, max: usize) -> String { @@ -549,6 +599,11 @@ fn feedback_output_excerpt(text: &str, max: usize) -> String { } pub(super) fn failed_checks_feedback(checks: &[CheckEvidence]) -> String { + failed_checks_excerpt(checks, COMPACT_CHECK_CHARS) +} + +/// The failed checks' output in `chars` characters, shared between them. +pub(super) fn failed_checks_excerpt(checks: &[CheckEvidence], chars: usize) -> String { let failed: Vec<_> = checks .iter() .filter(|c| c.status == CheckStatus::Failed) @@ -558,7 +613,7 @@ pub(super) fn failed_checks_feedback(checks: &[CheckEvidence]) -> String { } // Retain both output streams independently. Concatenating first can hide // a useful stdout tail inside the omitted middle of verbose stderr. - let available = 450usize.saturating_sub(failed.len().saturating_sub(1)); + let available = chars.saturating_sub(failed.len().saturating_sub(1)); let per_check = available / failed.len(); let remainder = available % failed.len(); failed @@ -753,6 +808,15 @@ fn comparison_output(output: &str, directory: &std::path::Path) -> String { /// Retain the rejected edit as well as its outcome; a failure label alone cannot /// explain what the next generation must improve. All content remains untrusted. pub(super) fn feedback(candidates: &[CandidateEvidence], baseline: &str) -> String { + feedback_with(candidates, baseline, COMPACT_CHECK_CHARS) +} + +/// `feedback` with `check_chars` characters of failed check output. +pub(super) fn feedback_with( + candidates: &[CandidateEvidence], + baseline: &str, + check_chars: usize, +) -> String { let Some(last) = candidates.last() else { return baseline.into(); }; @@ -781,7 +845,7 @@ pub(super) fn feedback(candidates: &[CandidateEvidence], baseline: &str) -> Stri sections.push(format!( "Check evidence from candidate {} (data):\n{}", checked.number, - failures(checked) + failures(checked, check_chars) )); } else if !baseline.is_empty() { sections.push(format!("Baseline check evidence (data):\n{}", baseline)); @@ -798,7 +862,7 @@ pub(super) fn feedback(candidates: &[CandidateEvidence], baseline: &str) -> Stri super::truncate(&c.raw_output, 350) )); } - super::truncate(§ions.join("\n"), 1400) + super::truncate(§ions.join("\n"), 950 + check_chars) } /// Only executed check output may provide verifier context anchors. @@ -812,8 +876,8 @@ pub(super) fn verifier_feedback(candidates: &[CandidateEvidence], baseline: &str .iter() .any(|check| check.status == CheckStatus::Failed) }) - .map(failures) - .unwrap_or_else(|| super::truncate(baseline, 450)) + .map(|candidate| failures(candidate, COMPACT_CHECK_CHARS)) + .unwrap_or_else(|| super::truncate(baseline, COMPACT_CHECK_CHARS)) } #[cfg(test)] @@ -1435,6 +1499,42 @@ mod tests { assert!(verifier.len() <= 450); } #[test] + fn detailed_feedback_shows_failures_the_compact_excerpt_drops() { + assert_eq!(check_feedback_chars(1024), 450); + assert_eq!(check_feedback_chars(4096), 1024); + assert_eq!(check_feedback_chars(16_384), 4000); + // Ten failing tests of about 300 characters each, as a unittest run prints them. + let output: String = (1..=10) + .map(|n| { + format!( + "FAIL: test_case_{n}\n{}\nAssertionError: case {n}\n", + "-".repeat(260) + ) + }) + .collect(); + let candidate = failed(1, None, &output); + let compact = feedback(std::slice::from_ref(&candidate), ""); + assert!(compact.contains("AssertionError: case 10")); + assert!(!compact.contains("AssertionError: case 5")); + let detailed = feedback_with( + std::slice::from_ref(&candidate), + "", + check_feedback_chars(16_384), + ); + for n in 1..=10 { + assert!( + detailed.contains(&format!("AssertionError: case {n}\n")), + "{n}" + ); + } + assert!(detailed.len() <= 4950); + // No candidate yet: the baseline excerpt is the feedback, at either size. + let baseline = failed_checks_excerpt(&candidate.checks, 4000); + assert!(baseline.contains("AssertionError: case 1\n")); + assert_eq!(feedback_with(&[], &baseline, 4000), baseline); + assert!(failed_checks_feedback(&candidate.checks).len() <= 450); + } + #[test] fn four_verbose_failures_keep_each_stream_tail() { let mut candidate = failed(1, None, ""); candidate.checks.clear(); @@ -1554,6 +1654,69 @@ mod tests { assert!(whole_file_retry(&[failed_checks], &scope, &baseline).is_none()); } + #[test] + fn a_small_one_file_scope_is_rewritten_whole_in_direct_edits_and_repairs() { + let root = tempfile::tempdir().unwrap(); + let baseline = root.path().join("baseline"); + let repaired = root.path().join("candidate-1"); + for directory in [&baseline, &repaired] { + std::fs::create_dir_all(directory).unwrap(); + } + std::fs::write( + baseline.join("stub.py"), + "def count():\n pass\n\ndef total():\n pass\n", + ) + .unwrap(); + std::fs::write( + baseline.join("big.py"), + "x = 1\n".repeat(WHOLE_FILE_SCOPE_MAX_LINES + 1), + ) + .unwrap(); + // The failed candidate outgrew the limit; the baseline copy did not. + std::fs::write( + repaired.join("stub.py"), + "x = 1\n".repeat(WHOLE_FILE_SCOPE_MAX_LINES + 1), + ) + .unwrap(); + let decide = |action, parent_candidate| SearchDecision { + action, + parent_candidate, + reason: String::new(), + }; + let one = [PathBuf::from("stub.py")]; + let initial = decide(SearchAction::Initial, None); + assert_eq!( + whole_file_scope(&[], &initial, &one, &baseline), + Some(one[0].clone()) + ); + // Two files, a file over the line limit, a restart or a stop keep span edits. + let two = [PathBuf::from("stub.py"), PathBuf::from("big.py")]; + assert_eq!(whole_file_scope(&[], &initial, &two, &baseline), None); + let big = [PathBuf::from("big.py")]; + assert_eq!(whole_file_scope(&[], &initial, &big, &baseline), None); + for action in [SearchAction::Restart, SearchAction::Stop] { + assert_eq!( + whole_file_scope(&[], &decide(action, None), &one, &baseline), + None + ); + } + // A repair is sized by the candidate it repairs, not by the baseline. + let mut parent = failed(1, None, "FAIL"); + parent.directory = repaired.clone(); + let repair = decide(SearchAction::Repair, Some(1)); + assert_eq!( + whole_file_scope(std::slice::from_ref(&parent), &repair, &one, &baseline), + None + ); + std::fs::write(repaired.join("stub.py"), "def count():\n return 0\n").unwrap(); + assert_eq!( + whole_file_scope(std::slice::from_ref(&parent), &repair, &one, &baseline), + Some(one[0].clone()) + ); + // A parent that is not in the run has no file to size. + assert_eq!(whole_file_scope(&[], &repair, &one, &baseline), None); + } + #[test] fn repeated_output_resends_the_same_request_with_seeded_sampling() { let parent = failed(1, None, "failure");