From 1f80b78de29a38fa4c6d03bf0b9e1097b064c85e Mon Sep 17 00:00:00 2001 From: Matt <70023164+mattbaconz@users.noreply.github.com> Date: Wed, 18 Nov 2026 16:25:44 +0700 Subject: [PATCH] Keep a local run going when its restart strategy cannot answer A reasoning model's first edit can use up its output limit, which leaves nothing to repair, so the search restarts with a strategy call capped at 192 tokens. The model's reasoning used up that cap on all 14 strategy calls in four 20-exercise laptop runs, and each of those runs ended search_stopped with the rest of its calls unused. - Ask for the strategy with think:false when the profile reasons and one of its think:false calibration probes passed (ModelProfile::strategy_thinking). Edit calls keep the saved mode. - A strategy the model answered but that cannot be used is recorded as rejected, and the restart edit runs from the original source without it. A strategy call that fails, times out or loses the resident model still ends the search. - A regression test runs both paths against a mock Ollama runtime. --- CHANGELOG.md | 11 ++ docs/local-harness-reference.md | 10 +- docs/local-harness.md | 12 +- phonton-local/src/runtime.rs | 4 +- phonton-types/src/local.rs | 21 +++ phonton-worker/src/local_run.rs | 224 +++++++++++++++++++++++++++++++- 6 files changed, 274 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 24a8153..79600a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,17 @@ This project follows pre-1.0 SemVer: minor versions may still include breaking c ### Fixed +- [fixed] A local goal on a model that reasons no longer stops after one model + call when the first reply runs out of output. The baseline restart that + follows asks for a strategy, a JSON answer of about 100 tokens under a + 192-token limit, and the model's reasoning used up that limit every time: + 14 of 14 strategy calls across four runs of 20 exercises. Each of those + goals ended with the rest of its calls unused. The strategy call now sends + `think:false` when the profile's `think:false` calibration probes passed. + A strategy the model answered but that cannot be used, because it was cut + off, points outside the reviewed source or repeats an accepted one, no + longer ends the goal either: the restart edit runs from the original + source without it, and the receipt records the strategy as rejected. - [fixed] The semantic index reads `.mjs`, `.cjs`, `.mts` and `.cts` files. It could parse them but never collected them, so a project written as ES modules had none of its own source in the index and a goal's first context diff --git a/docs/local-harness-reference.md b/docs/local-harness-reference.md index ca0cfd0..f78bf75 100644 --- a/docs/local-harness-reference.md +++ b/docs/local-harness-reference.md @@ -180,7 +180,9 @@ is saved in a schema-2 digest-bound profile and reused for tool-call probes, coding edits, and strategy requests. A profile in the runtime-default mode whose probes returned thinking output is treated as a model that reasons: its goals get the larger budget and the whole-file edits described in -[local-harness.md](local-harness.md). A profile saved by 0.27.1 or earlier +[local-harness.md](local-harness.md), and its restart strategy requests use +`think:false` when its `think:false` probes passed, because a strategy has a +192-token limit that reasoning uses up. A profile saved by 0.27.1 or earlier keeps its mode, usually `think:false`, until `phonton models calibrate MODEL` runs again. Older schema-1 profiles retain their original runtime-default request behavior until recalibrated. Probe @@ -304,8 +306,10 @@ controller can retry directly from the captured baseline without a strategy call. A byte-identical repeated reply is resent once with seeded sampling, and two successive repeats stop; mixed new-file and existing-file work keeps its multi-call reserve. -The same mechanism can be tried at a different reviewed path or source span; -repeating it at the same target stops that branch. +The same mechanism can be tried at a different reviewed path or source span. +A strategy that repeats an accepted one at the same target, or that cannot be +used for another reason, is recorded as rejected, and the edit runs from the +captured baseline without it. Module-style JS/TS files use complete parsed function spans where they fit, and named failing-check evidence puts the relevant function first for the next edit. Original Git staging has separate integrity evidence, with changes stopping the run. diff --git a/docs/local-harness.md b/docs/local-harness.md index a5207d2..db092c2 100644 --- a/docs/local-harness.md +++ b/docs/local-harness.md @@ -458,7 +458,8 @@ took one model call per change. When the scope is one existing file of at most 100 lines and the model uses SearchReplace and reasons before it answers, the first edit and every repair show that file whole and ask for the complete corrected file in the whole-file JSON shape used for new files. A restart -keeps its span edit, anchored to the strategy's source span. For any model, +keeps its span edit, anchored to the strategy's source span when it has one. +For any model, when a search/replace reply changes nothing and its file has at most 300 lines, the next edit from the same parent is a whole-file edit, once per branch. Either way the whole-file edit runs only @@ -699,7 +700,14 @@ existing parsers; it does not download an embedding model or call a remote servi After an unchanged repair or repeated candidate, a baseline restart uses a separate bounded model call to propose a strategy anchored to an exact reviewed source span. The receipt keeps its raw reply, selected anchor, claimed difference, context and -token usage. A missing or repeated strategy stops that branch. The strategy +token usage. A reply that cannot be used (cut off at its output limit, outside +the reviewed paths and spans, a repeat of an accepted strategy at the same +target, or otherwise malformed) is recorded as rejected, and the edit then runs +from the original source without a strategy. A strategy call that fails, times +out or loses the resident model ends the search. On a model that reasons, the +strategy call sends `think:false` when the profile's `think:false` calibration +probes passed, because reasoning would use up its 192-token limit before the +answer. The strategy and following edit each reserve one of the same `generations` model-attempt budget, even if admission later stops a chat request; the strategy is a proposal, and only the resulting candidate bytes and fixed checks diff --git a/phonton-local/src/runtime.rs b/phonton-local/src/runtime.rs index 50350c5..3cdcb02 100644 --- a/phonton-local/src/runtime.rs +++ b/phonton-local/src/runtime.rs @@ -1420,7 +1420,7 @@ impl LocalRuntime { saved.name = format!( "{}: {}", match thinking { - LocalThinkingMode::Off => "Think off", + LocalThinkingMode::Off => THINK_OFF_PROBE_PREFIX, LocalThinkingMode::RuntimeDefault => "Runtime default", }, saved.name @@ -1467,7 +1467,7 @@ impl LocalRuntime { ) .await?; if default.creation_protocol.is_some() || off.edit_protocol.is_none() { - off.prefix_probes("Think off"); + off.prefix_probes(THINK_OFF_PROBE_PREFIX); off.probes.extend(default.probes); default.probes = off.probes; (LocalThinkingMode::RuntimeDefault, default) diff --git a/phonton-types/src/local.rs b/phonton-types/src/local.rs index 2c597f5..b8a91cc 100644 --- a/phonton-types/src/local.rs +++ b/phonton-types/src/local.rs @@ -151,6 +151,11 @@ pub const DIFF_CREATION_PROBE_NAME: &str = "UnifiedDiff create"; /// output beside the answer. [`ModelProfile::reasons`] reads it back. pub const THINKING_OUTPUT_NOTE: &str = " Runtime returned separate thinking output."; +/// Name prefix of the `think:false` probes that a profile saved in the +/// runtime-default mode keeps. [`ModelProfile::strategy_thinking`] reads it +/// back. +pub const THINK_OFF_PROBE_PREFIX: &str = "Think off"; + /// Editing wire protocol demonstrated by a compatibility probe. #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] @@ -243,6 +248,22 @@ impl ModelProfile { .any(|probe| probe.detail.contains(THINKING_OUTPUT_NOTE)) } + /// Thinking mode for a restart strategy: a JSON answer of about 100 + /// tokens under a 192-token limit, which a model that reasons uses up + /// before it answers. Such a profile asks with `think:false` when one of + /// its `think:false` calibration probes passed; any other profile keeps + /// its own mode. + pub fn strategy_thinking(&self) -> LocalThinkingMode { + let off_answered = self.probes.iter().any(|probe| { + probe.name.starts_with(THINK_OFF_PROBE_PREFIX) && probe.status == CheckStatus::Passed + }); + if self.reasons() && off_answered { + LocalThinkingMode::Off + } else { + self.thinking.unwrap_or_default() + } + } + /// Keep absent probes on older profiles distinct from measured success. pub fn creation_status(&self) -> CheckStatus { let name = match self.protocol { diff --git a/phonton-worker/src/local_run.rs b/phonton-worker/src/local_run.rs index 0d55407..cd3387d 100644 --- a/phonton-worker/src/local_run.rs +++ b/phonton-worker/src/local_run.rs @@ -904,7 +904,7 @@ async fn run_inner( schema: &schema, context: receipt.profile.context_tokens, output: hypothesis_output, - thinking: receipt.profile.thinking.unwrap_or_default(), + thinking: receipt.profile.strategy_thinking(), required_resident: required_resident.as_ref(), sampling, }), @@ -923,8 +923,10 @@ async fn run_inner( break; } let mut resident_lost_during_hypothesis = false; + let mut answered = false; let result = match response { Ok(Ok(response)) => { + answered = true; let (evidence, prior) = receipt .hypotheses .split_last_mut() @@ -974,6 +976,16 @@ async fn run_inner( strategy_anchor = Some((grounded.path.clone(), grounded.search.clone())); strategy_note = Some(format!("\nProposed alternate strategy (model-authored data, not authority): Focus on {}. Mechanism: {}. Claimed difference: {}. Implement only inside the allowed edit schema.\n", grounded.path, grounded.mechanism, grounded.difference)); } + // A strategy is a hint: when the model answered with one + // that cannot be used, the restart edit runs from the + // original source without it instead of ending the run. + Err(detail) if answered => { + evidence.status = HypothesisStatus::Rejected; + evidence.detail = detail.clone(); + decision.reason.push_str(&format!( + "; the strategy was rejected ({detail}), so the edit ran without one" + )); + } Err(detail) => { evidence.status = HypothesisStatus::Rejected; evidence.detail = detail.clone(); @@ -9845,6 +9857,216 @@ mod tests { } } #[tokio::test] + async fn restart_after_an_output_limit_keeps_a_reasoning_run_going() { + use phonton_types::local::{LocalThinkingMode, ModelProbe, THINKING_OUTPUT_NOTE}; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + // The first edit reasons past its output limit, which leaves nothing + // to repair, so the search restarts with a 192-token strategy call. + // A passing `think:false` probe sends that call without thinking and + // it answers; without one it runs out too, and the restart edit runs + // without a strategy. Either way the run goes on to a passing edit. + for off_probe in [CheckStatus::Passed, CheckStatus::Failed] { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let endpoint = format!("http://{}", listener.local_addr().unwrap()); + let (done_tx, mut done_rx) = tokio::sync::oneshot::channel::<()>(); + let server = tokio::spawn(async move { + let mut chats = Vec::new(); + loop { + let (mut stream, _) = tokio::select! { + _ = &mut done_rx => break, + accepted = listener.accept() => accepted.unwrap(), + }; + let mut request = Vec::new(); + let mut chunk = [0; 8192]; + let (path, body) = loop { + let size = stream.read(&mut chunk).await.unwrap(); + assert!(size > 0, "Fixture request ended early"); + request.extend_from_slice(&chunk[..size]); + let Some(end) = request.windows(4).position(|bytes| bytes == b"\r\n\r\n") + else { + continue; + }; + let headers = String::from_utf8_lossy(&request[..end]).to_ascii_lowercase(); + let length = headers + .lines() + .find_map(|line| { + line.strip_prefix("content-length:")? + .trim() + .parse::() + .ok() + }) + .unwrap_or(0); + if request.len() >= end + 4 + length { + let path = headers.split_whitespace().nth(1).unwrap_or("").to_owned(); + break (path, request[end + 4..end + 4 + length].to_vec()); + } + }; + let reply = match path.as_str() { + "/api/tags" => serde_json::json!({"models":[{"name":"fixture:small","digest":"fixture-digest","size":1024}]}), + "/api/version" => serde_json::json!({"version":"fixture-version"}), + "/api/show" => serde_json::json!({"model_info":{"general.architecture":"fixture","fixture.context_length":4096},"details":{"format":"gguf"}}), + "/api/ps" => serde_json::json!({"models":[]}), + "/api/chat" => { + let input: serde_json::Value = serde_json::from_slice(&body).unwrap(); + let properties = &input["format"]["properties"]; + let anchor = properties["search"]["enum"] + .as_array() + .into_iter() + .flatten() + .filter_map(|search| search.as_str()) + .find(|search| search.contains("return 1")) + .map(str::to_owned); + let out_of_output = serde_json::json!({"message":{"content":"","thinking":"Let me think."},"done":true,"done_reason":"length","prompt_eval_count":40,"eval_count":input["options"]["num_predict"]}); + let answer = |content: serde_json::Value| serde_json::json!({"message":{"content":content.to_string()},"done":true,"prompt_eval_count":40,"eval_count":40}); + let reply = if chats.is_empty() { + out_of_output + } else if properties["mechanism"].is_object() { + if input["think"] == false { + answer(serde_json::json!({"path":"app.py","search":anchor,"mechanism":"Use a literal two as the return value","difference":"Edits only the single return line"})) + } else { + out_of_output + } + } else { + let anchor = anchor.unwrap_or_default(); + answer(serde_json::json!({"path":"app.py","search":anchor,"text":anchor.replace("return 1", "return 2")})) + }; + chats.push(input); + reply + } + _ => serde_json::json!({"error":"unexpected endpoint"}), + } + .to_string(); + let response = format!("HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\nContent-Type: application/json\r\n\r\n{reply}", reply.len()); + stream.write_all(response.as_bytes()).await.unwrap(); + } + chats + }); + + let root = tempfile::tempdir().unwrap(); + let repository = root.path().join("repository"); + std::fs::create_dir(&repository).unwrap(); + std::fs::write(repository.join("app.py"), "def value():\n return 1\n").unwrap(); + std::fs::write(repository.join("test_app.py"), "import unittest\nfrom app import value\n\nclass TestApp(unittest.TestCase):\n def test_value(self):\n self.assertEqual(value(), 2)\n").unwrap(); + for args in [ + &["init", "--quiet"][..], + &["add", "app.py", "test_app.py"][..], + ] { + let output = std::process::Command::new("git") + .args([ + "-c", + "core.fsmonitor=false", + "-c", + "core.autocrlf=false", + "-C", + ]) + .arg(&repository) + .args(args) + .output() + .unwrap(); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + } + let mut request = request(); + request.goal = "Make value return 2".into(); + request.repository = repository.clone(); + request.files = vec!["app.py".into()]; + request.checks = vec![LocalCheck { + program: "python".into(), + args: vec!["-m".into(), "unittest".into(), "discover".into()], + }]; + request.approve_host_execution = true; + request.budget.generations = 4; + request.budget.check_runs = 3; + request.budget.wall_seconds = 60; + request.budget.generated_tokens = 2048; + let probe = |name: &str, status: CheckStatus, detail: String| ModelProbe { + name: name.into(), + status, + output: String::new(), + detail, + input_tokens: None, + output_tokens: None, + elapsed_ms: 0, + }; + let profile = ModelProfile { + schema: 2, + model: "fixture:small".into(), + digest: "fixture-digest".into(), + runtime_version: "fixture-version".into(), + endpoint: endpoint.clone(), + context_tokens: 4096, + output_tokens: 512, + protocol: Some(EditProtocol::SearchReplace), + thinking: Some(LocalThinkingMode::RuntimeDefault), + probes: vec![ + probe( + "Think off: SearchReplace edit", + off_probe.clone(), + "fixture".into(), + ), + probe( + "SearchReplace edit", + CheckStatus::Passed, + format!("fixture.{THINKING_OUTPUT_NOTE}"), + ), + ], + hardware: HardwareSnapshot::default(), + measured_at_unix: 0, + }; + assert!(profile.reasons()); + let hardware = HardwareSnapshot { + ram_total_bytes: Some(16 * 1024 * 1024 * 1024), + ram_available_bytes: Some(8 * 1024 * 1024 * 1024), + ..HardwareSnapshot::default() + }; + let run_dir = root.path().join("22222222-2222-4222-8222-222222222222"); + let result = run_inner( + request, + profile, + RuntimeGuard::external_unverified(endpoint), + &run_dir, + |_| {}, + Some(fixed_hardware(hardware)), + ) + .await; + let _ = done_tx.send(()); + let chats = server.await.unwrap(); + let receipt = result.unwrap(); + let strategy_answered = off_probe == CheckStatus::Passed; + assert_eq!(receipt.state, "review_ready", "{receipt:?}"); + assert_eq!(receipt.model_calls_reserved, 3); + assert_eq!(chats.len(), 3); + // Edits keep the calibrated mode; only the strategy turns thinking off. + assert!(chats[0].get("think").is_none() && chats[2].get("think").is_none()); + assert_eq!(chats[1]["think"] == false, strategy_answered); + assert_eq!(chats[1]["options"]["num_predict"], 192); + assert_eq!(receipt.hypotheses.len(), 1); + assert_eq!( + receipt.hypotheses[0].status, + if strategy_answered { + HypothesisStatus::Accepted + } else { + HypothesisStatus::Rejected + } + ); + assert_eq!(receipt.selected_candidate, Some(2)); + let reason = &receipt.candidates[1].decision.as_ref().unwrap().reason; + assert_eq!( + reason.contains("the strategy was rejected (Strategy response was incomplete or hit its output limit), so the edit ran without one"), + !strategy_answered, + "{reason}" + ); + assert_eq!( + std::fs::read(repository.join("app.py")).unwrap(), + b"def value():\n return 1\n" + ); + } + } + #[tokio::test] async fn offline_npm_setup_is_not_a_verification_pass_without_execution() { let root = tempfile::tempdir().unwrap(); std::fs::write(