diff --git a/CHANGELOG.md b/CHANGELOG.md index 79600a6..21da80f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,25 @@ This project follows pre-1.0 SemVer: minor versions may still include breaking c ### Fixed +- [fixed] The later model calls of a local goal no longer run short of + output after short replies. Each call was charged its whole output limit + even when the model wrote far less, and each call keeps half of what is + left for later ones, so calls 5 to 8 of an 8-call goal got 1,024, 512, 256 + and 256 tokens. In a laptop run of 20 exercises with `qwen3.5:4b`, edits + that finished wrote a median of 1,057 output tokens, half of them between + about 880 and 1,310, and three of the four goals that failed ended with + their last calls cut off at 256 to 512 tokens. A call is now charged the + output tokens the runtime reports once it answers, and a reply without a + count stays charged its whole limit. The receipt's + `generated_tokens_reserved` total follows the same rule. +- [fixed] A restart edit no longer stops when its strategy's one-line source + anchor sits inside a small symbol that the edit prompt offers as one + whole-symbol choice. The edit call usually has a larger output limit than + the strategy call, so the symbol the strategy saw as separate lines could + become one choice, the anchor line was no longer a choice of its own, and + the goal ended with "Strategy source anchor was omitted from the edit + choices". The edit is now bound to the smallest choice that contains the + anchor, and the candidate's decision reason says so. - [fixed] A local goal on a model that reasons no longer stops after one model call when the first reply runs out of output. The baseline restart that follows asks for a strategy, a JSON answer of about 100 tokens under a diff --git a/docs/local-harness-reference.md b/docs/local-harness-reference.md index f78bf75..fabecac 100644 --- a/docs/local-harness-reference.md +++ b/docs/local-harness-reference.md @@ -165,7 +165,9 @@ creation probe and 256 for the tool-call probe, or a quarter of the context (at most 2,048) when the model may think first. For coding goals, the saved profile reserves up to one quarter of that context for each complete edit response; the goal's shared token budget can reduce it further. When later attempts are -allowed, the runner keeps half the remaining token budget for them. This is a +allowed, the runner keeps half the remaining token budget for them. Each call +is charged the output tokens the runtime reports once it answers, not its +whole limit; a reply without a count stays charged its limit. This is a generation allowance, not a measured claim that the model can produce a file of that size. Existing profiles keep their saved ceiling until recalibrated. Calibration first requests `think:false` so a thinking model can spend the diff --git a/docs/local-harness.md b/docs/local-harness.md index db092c2..323ef11 100644 --- a/docs/local-harness.md +++ b/docs/local-harness.md @@ -200,7 +200,10 @@ The small probe output caps do not set the ceiling for a coding goal. The saved profile allows up to one quarter of its calibrated context for one complete edit or new-file response, then the shared goal budget can reduce that reservation. The worker keeps half the remaining token budget for a -later permitted model call. This is a context-based allowance, not measured +later permitted model call. A call is charged its whole limit while it runs, +then the output tokens the runtime reports once it answers, so a short reply +leaves the rest for later calls; a reply without a count stays charged its +limit. This is a context-based allowance, not measured model generation capacity. A truncated runtime response is still rejected. Previously saved profiles retain their smaller ceiling until recalibrated. If neither edit format passes, calibration still saves and prints the raw probe @@ -712,8 +715,13 @@ and following edit each reserve one of the same `generations` model-attempt budget, even if admission later stops a chat request; the strategy is a proposal, and only the resulting candidate bytes and fixed checks can justify review. The edit's actual returned path and source anchor are -checked before materialization. An interrupted strategy remains pending evidence, not a -candidate. +checked before materialization. The edit call usually has a larger output +limit than the strategy call, so a small symbol the strategy saw as separate +lines can be one whole-symbol choice in the edit prompt. A one-line anchor +inside such a symbol binds the edit to the smallest choice that contains it, +and the candidate's decision reason says so. An anchor that no edit choice +contains still ends the search. An interrupted strategy remains pending +evidence, not a candidate. Cancellation retains evidence. Reopening an unfinished receipt labels it interrupted; it does not silently resume or mark it complete. Inspect the selected candidate and its diff before applying it. diff --git a/phonton-types/src/local_run.rs b/phonton-types/src/local_run.rs index b61e71b..2c62e02 100644 --- a/phonton-types/src/local_run.rs +++ b/phonton-types/src/local_run.rs @@ -323,7 +323,9 @@ pub struct LocalRunReceipt { pub hypotheses: Vec, pub selected_candidate: Option, pub checks_used: u32, - /// Missing runtime counters are unknown; admission reserves max output. + /// Output tokens charged to the shared budget: each call's whole output + /// limit while it runs, then the runtime's reported count once it + /// answers. A call without a reported count stays charged its limit. pub generated_tokens_reserved: u64, /// All reserved model attempts, including those stopped before chat transport. #[serde(default, alias = "inference_calls_used")] diff --git a/phonton-worker/src/local_run.rs b/phonton-worker/src/local_run.rs index cd3387d..6041e44 100644 --- a/phonton-worker/src/local_run.rs +++ b/phonton-worker/src/local_run.rs @@ -959,6 +959,8 @@ async fn run_inner( Ok(Err(error)) => Err(format!("Strategy inference failed: {error}")), Err(_) => Err("Run wall-time budget exhausted during strategy inference".into()), }; + let reported = receipt.hypotheses.last().and_then(|h| h.output_tokens); + settle_output(&mut receipt, hypothesis_output, reported); let evidence = receipt .hypotheses .last_mut() @@ -1043,8 +1045,9 @@ async fn run_inner( receipt.state = "budget_exhausted".into(); break; } - // Reservations are charged before inference and never refunded. Leave - // half the remaining budget for a later call when one is still allowed. + // A call's whole output limit is charged before inference and settled + // to the runtime's reported count once it answers. Leave half the + // remaining budget for a later call when one is still allowed. let calls_left = receipt .request .budget @@ -1224,6 +1227,15 @@ async fn run_inner( phonton_local::runtime::edit_schema(&assembly.paths, &assembly.searches) }; assembly.evidence.constraint_bytes = serde_json::to_vec(&schema)?.len(); + // The edit may use only its bound choice, which can + // be the whole symbol around a one-line anchor, so + // the returned edit is checked against that choice. + if let Some(bound) = assembly.searches.first() { + if search.as_ref() != Some(bound) { + decision.reason.push_str("; the strategy's source anchor is inside a larger edit choice, so the edit was bound to that choice"); + strategy_anchor = Some((path.clone(), Some(bound.clone()))); + } + } } } Err(detail) => { @@ -1378,6 +1390,7 @@ async fn run_inner( Some("Run wall-time budget exhausted during generation".into()) } } + settle_output(&mut receipt, output, candidate.output_tokens); if search::repeats_output(&receipt.candidates, &candidate.raw_output) { // Same bytes, same verdict: never spend checks on it again. candidate.rejection = Some(search::REPEATED_OUTPUT.into()); @@ -6135,6 +6148,16 @@ fn go_true_flag(arg: &str, flag: &str) -> bool { fn remaining_time(request: &LocalRunRequest, started: Instant) -> Duration { Duration::from_secs(request.budget.wall_seconds).saturating_sub(started.elapsed()) } +/// Once a model call answers, charge the output tokens the runtime reports +/// instead of the call's whole limit. Without a count the limit stays charged. +fn settle_output(receipt: &mut LocalRunReceipt, reserved: u32, reported: Option) { + if let Some(reported) = reported { + receipt.generated_tokens_reserved = receipt + .generated_tokens_reserved + .saturating_sub(u64::from(reserved)) + .saturating_add(reported); + } +} fn truncate(text: &str, max: usize) -> String { let mut end = text.len().min(max); while !text.is_char_boundary(end) { @@ -9811,7 +9834,9 @@ mod tests { assert_eq!(chats, selected_number, "{paths:?}"); assert_eq!(receipt.model_calls_reserved, selected_number); if large_context_budget { - assert_eq!(receipt.generated_tokens_reserved, 3072); + // Both replies reported 50 output tokens, and each call is + // charged that instead of its 2,048- or 2,023-token limit. + assert_eq!(receipt.generated_tokens_reserved, 100); } assert_eq!(receipt.baseline_checks[0].status, CheckStatus::Failed); assert_eq!( @@ -9856,10 +9881,109 @@ mod tests { assert_eq!(std::fs::read(&index_path).unwrap(), index_before); } } + /// A fake Ollama serving one 4,096-token fixture model. `chat` answers + /// each `/api/chat` body, given how many chats came before it; the + /// server returns every chat body once `done` is sent. + async fn fixture_runtime( + mut chat: impl FnMut(&serde_json::Value, usize) -> serde_json::Value + Send + 'static, + ) -> ( + String, + tokio::sync::oneshot::Sender<()>, + tokio::task::JoinHandle>, + ) { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let endpoint = format!("http://{}", listener.local_addr().unwrap()); + let (done_tx, mut done_rx) = tokio::sync::oneshot::channel::<()>(); + let server = tokio::spawn(async move { + let mut chats = Vec::new(); + loop { + let (mut stream, _) = tokio::select! { + _ = &mut done_rx => break, + accepted = listener.accept() => accepted.unwrap(), + }; + let mut request = Vec::new(); + let mut chunk = [0; 8192]; + let (path, body) = loop { + let size = stream.read(&mut chunk).await.unwrap(); + assert!(size > 0, "Fixture request ended early"); + request.extend_from_slice(&chunk[..size]); + let Some(end) = request.windows(4).position(|bytes| bytes == b"\r\n\r\n") + else { + continue; + }; + let headers = String::from_utf8_lossy(&request[..end]).to_ascii_lowercase(); + let length = headers + .lines() + .find_map(|line| { + line.strip_prefix("content-length:")? + .trim() + .parse::() + .ok() + }) + .unwrap_or(0); + if request.len() >= end + 4 + length { + let path = headers.split_whitespace().nth(1).unwrap_or("").to_owned(); + break (path, request[end + 4..end + 4 + length].to_vec()); + } + }; + let reply = match path.as_str() { + "/api/tags" => serde_json::json!({"models":[{"name":"fixture:small","digest":"fixture-digest","size":1024}]}), + "/api/version" => serde_json::json!({"version":"fixture-version"}), + "/api/show" => serde_json::json!({"model_info":{"general.architecture":"fixture","fixture.context_length":4096},"details":{"format":"gguf"}}), + "/api/ps" => serde_json::json!({"models":[]}), + "/api/chat" => { + let input: serde_json::Value = serde_json::from_slice(&body).unwrap(); + let reply = chat(&input, chats.len()); + chats.push(input); + reply + } + _ => serde_json::json!({"error":"unexpected endpoint"}), + } + .to_string(); + let response = format!("HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\nContent-Type: application/json\r\n\r\n{reply}", reply.len()); + stream.write_all(response.as_bytes()).await.unwrap(); + } + chats + }); + (endpoint, done_tx, server) + } + + /// A Git repository whose `app.py` is `app`, a `value()` that returns 1, + /// while its test expects 2. + fn value_fixture(root: &Path, app: &str) -> PathBuf { + let repository = root.join("repository"); + std::fs::create_dir(&repository).unwrap(); + std::fs::write(repository.join("app.py"), app).unwrap(); + std::fs::write(repository.join("test_app.py"), "import unittest\nfrom app import value\n\nclass TestApp(unittest.TestCase):\n def test_value(self):\n self.assertEqual(value(), 2)\n").unwrap(); + for args in [ + &["init", "--quiet"][..], + &["add", "app.py", "test_app.py"][..], + ] { + let output = std::process::Command::new("git") + .args([ + "-c", + "core.fsmonitor=false", + "-c", + "core.autocrlf=false", + "-C", + ]) + .arg(&repository) + .args(args) + .output() + .unwrap(); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + } + repository + } + #[tokio::test] async fn restart_after_an_output_limit_keeps_a_reasoning_run_going() { use phonton_types::local::{LocalThinkingMode, ModelProbe, THINKING_OUTPUT_NOTE}; - use tokio::io::{AsyncReadExt, AsyncWriteExt}; // The first edit reasons past its output limit, which leaves nothing // to repair, so the search restarts with a 192-token strategy call. @@ -9867,109 +9991,33 @@ mod tests { // it answers; without one it runs out too, and the restart edit runs // without a strategy. Either way the run goes on to a passing edit. for off_probe in [CheckStatus::Passed, CheckStatus::Failed] { - let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); - let endpoint = format!("http://{}", listener.local_addr().unwrap()); - let (done_tx, mut done_rx) = tokio::sync::oneshot::channel::<()>(); - let server = tokio::spawn(async move { - let mut chats = Vec::new(); - loop { - let (mut stream, _) = tokio::select! { - _ = &mut done_rx => break, - accepted = listener.accept() => accepted.unwrap(), - }; - let mut request = Vec::new(); - let mut chunk = [0; 8192]; - let (path, body) = loop { - let size = stream.read(&mut chunk).await.unwrap(); - assert!(size > 0, "Fixture request ended early"); - request.extend_from_slice(&chunk[..size]); - let Some(end) = request.windows(4).position(|bytes| bytes == b"\r\n\r\n") - else { - continue; - }; - let headers = String::from_utf8_lossy(&request[..end]).to_ascii_lowercase(); - let length = headers - .lines() - .find_map(|line| { - line.strip_prefix("content-length:")? - .trim() - .parse::() - .ok() - }) - .unwrap_or(0); - if request.len() >= end + 4 + length { - let path = headers.split_whitespace().nth(1).unwrap_or("").to_owned(); - break (path, request[end + 4..end + 4 + length].to_vec()); - } - }; - let reply = match path.as_str() { - "/api/tags" => serde_json::json!({"models":[{"name":"fixture:small","digest":"fixture-digest","size":1024}]}), - "/api/version" => serde_json::json!({"version":"fixture-version"}), - "/api/show" => serde_json::json!({"model_info":{"general.architecture":"fixture","fixture.context_length":4096},"details":{"format":"gguf"}}), - "/api/ps" => serde_json::json!({"models":[]}), - "/api/chat" => { - let input: serde_json::Value = serde_json::from_slice(&body).unwrap(); - let properties = &input["format"]["properties"]; - let anchor = properties["search"]["enum"] - .as_array() - .into_iter() - .flatten() - .filter_map(|search| search.as_str()) - .find(|search| search.contains("return 1")) - .map(str::to_owned); - let out_of_output = serde_json::json!({"message":{"content":"","thinking":"Let me think."},"done":true,"done_reason":"length","prompt_eval_count":40,"eval_count":input["options"]["num_predict"]}); - let answer = |content: serde_json::Value| serde_json::json!({"message":{"content":content.to_string()},"done":true,"prompt_eval_count":40,"eval_count":40}); - let reply = if chats.is_empty() { - out_of_output - } else if properties["mechanism"].is_object() { - if input["think"] == false { - answer(serde_json::json!({"path":"app.py","search":anchor,"mechanism":"Use a literal two as the return value","difference":"Edits only the single return line"})) - } else { - out_of_output - } - } else { - let anchor = anchor.unwrap_or_default(); - answer(serde_json::json!({"path":"app.py","search":anchor,"text":anchor.replace("return 1", "return 2")})) - }; - chats.push(input); - reply - } - _ => serde_json::json!({"error":"unexpected endpoint"}), + let (endpoint, done_tx, server) = fixture_runtime(|input, earlier| { + let properties = &input["format"]["properties"]; + let anchor = properties["search"]["enum"] + .as_array() + .into_iter() + .flatten() + .filter_map(|search| search.as_str()) + .find(|search| search.contains("return 1")) + .map(str::to_owned); + let out_of_output = serde_json::json!({"message":{"content":"","thinking":"Let me think."},"done":true,"done_reason":"length","prompt_eval_count":40,"eval_count":input["options"]["num_predict"]}); + let answer = |content: serde_json::Value| serde_json::json!({"message":{"content":content.to_string()},"done":true,"prompt_eval_count":40,"eval_count":40}); + if earlier == 0 { + out_of_output + } else if properties["mechanism"].is_object() { + if input["think"] == false { + answer(serde_json::json!({"path":"app.py","search":anchor,"mechanism":"Use a literal two as the return value","difference":"Edits only the single return line"})) + } else { + out_of_output } - .to_string(); - let response = format!("HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\nContent-Type: application/json\r\n\r\n{reply}", reply.len()); - stream.write_all(response.as_bytes()).await.unwrap(); + } else { + let anchor = anchor.unwrap_or_default(); + answer(serde_json::json!({"path":"app.py","search":anchor,"text":anchor.replace("return 1", "return 2")})) } - chats - }); - + }) + .await; let root = tempfile::tempdir().unwrap(); - let repository = root.path().join("repository"); - std::fs::create_dir(&repository).unwrap(); - std::fs::write(repository.join("app.py"), "def value():\n return 1\n").unwrap(); - std::fs::write(repository.join("test_app.py"), "import unittest\nfrom app import value\n\nclass TestApp(unittest.TestCase):\n def test_value(self):\n self.assertEqual(value(), 2)\n").unwrap(); - for args in [ - &["init", "--quiet"][..], - &["add", "app.py", "test_app.py"][..], - ] { - let output = std::process::Command::new("git") - .args([ - "-c", - "core.fsmonitor=false", - "-c", - "core.autocrlf=false", - "-C", - ]) - .arg(&repository) - .args(args) - .output() - .unwrap(); - assert!( - output.status.success(), - "{}", - String::from_utf8_lossy(&output.stderr) - ); - } + let repository = value_fixture(root.path(), "def value():\n return 1\n"); let mut request = request(); request.goal = "Make value return 2".into(); request.repository = repository.clone(); @@ -10067,6 +10115,202 @@ mod tests { } } #[tokio::test] + async fn short_replies_leave_later_calls_their_output_limit() { + use phonton_types::local::ModelProbe; + + // Three edits finish in 100 tokens and fail the test; the fourth is + // right. A call given under 600 tokens is cut off. When each call + // was charged its whole limit, calls 2 to 4 got 512, 256 and 256 + // tokens and the run ended `budget_exhausted` without a candidate. + let (endpoint, done_tx, server) = fixture_runtime(|input, earlier| { + if input["options"]["num_predict"].as_u64().unwrap_or(0) < 600 { + return serde_json::json!({"message":{"content":"{\"path\":\"app.py\""},"done":true,"done_reason":"length","prompt_eval_count":40,"eval_count":input["options"]["num_predict"]}); + } + let anchor = input["format"]["properties"]["search"]["enum"] + .as_array() + .into_iter() + .flatten() + .filter_map(|search| search.as_str()) + .find(|search| search.contains("return ")) + .unwrap_or("return 0") + .to_owned(); + let digit = anchor.find("return ").unwrap_or(0) + "return ".len(); + let value = [3, 4, 5, 2][earlier.min(3)]; + let text = format!("{}{value}{}", &anchor[..digit], &anchor[digit + 1..]); + serde_json::json!({"message":{"content":serde_json::json!({"path":"app.py","search":anchor,"text":text}).to_string()},"done":true,"prompt_eval_count":40,"eval_count":100}) + }) + .await; + let root = tempfile::tempdir().unwrap(); + let repository = value_fixture(root.path(), "def value():\n return 1\n"); + let mut request = request(); + request.goal = "Make value return 2".into(); + request.repository = repository; + request.files = vec!["app.py".into()]; + request.checks = vec![LocalCheck { + program: "python".into(), + args: vec!["-m".into(), "unittest".into(), "discover".into()], + }]; + request.approve_host_execution = true; + request.budget.generations = 4; + request.budget.check_runs = 5; + request.budget.wall_seconds = 60; + request.budget.generated_tokens = 2048; + let profile = ModelProfile { + schema: 2, + model: "fixture:small".into(), + digest: "fixture-digest".into(), + runtime_version: "fixture-version".into(), + endpoint: endpoint.clone(), + context_tokens: 4096, + output_tokens: 1024, + protocol: Some(EditProtocol::SearchReplace), + thinking: Some(phonton_types::local::LocalThinkingMode::Off), + probes: vec![ModelProbe { + name: "SearchReplace edit".into(), + status: CheckStatus::Passed, + output: String::new(), + detail: "fixture transport".into(), + input_tokens: None, + output_tokens: None, + elapsed_ms: 0, + }], + hardware: HardwareSnapshot::default(), + measured_at_unix: 0, + }; + let hardware = HardwareSnapshot { + ram_total_bytes: Some(16 * 1024 * 1024 * 1024), + ram_available_bytes: Some(8 * 1024 * 1024 * 1024), + ..HardwareSnapshot::default() + }; + let run_dir = root.path().join("33333333-3333-4333-8333-333333333333"); + let result = run_inner( + request, + profile, + RuntimeGuard::external_unverified(endpoint), + &run_dir, + |_| {}, + Some(fixed_hardware(hardware)), + ) + .await; + let _ = done_tx.send(()); + let chats = server.await.unwrap(); + let receipt = result.unwrap(); + let limits: Vec<_> = chats + .iter() + .map(|chat| chat["options"]["num_predict"].as_u64().unwrap_or(0)) + .collect(); + // Each call keeps half of what is left for later ones; the last + // gets the profile's whole limit. + assert_eq!(limits, [1024, 974, 924, 1024], "{receipt:?}"); + assert_eq!(receipt.state, "review_ready", "{receipt:?}"); + assert_eq!(receipt.selected_candidate, Some(4)); + assert_eq!(receipt.generated_tokens_reserved, 400); + } + #[tokio::test] + async fn restart_line_anchor_inside_a_symbol_choice_binds_to_that_symbol() { + use phonton_types::local::ModelProbe; + + // The 230-byte function is one choice at the edit's 512-token limit + // but separate lines at the strategy's 192. The strategy picks the + // return line, which the restart edit no longer offers on its own. + const APP: &str = "def value():\n # Callers compare against this number, so it lives in one place.\n # Keep it a plain literal: no lookups, no side effects, no I/O.\n # The unit test in test_app.py pins the number callers expect.\n return 1\n"; + let (endpoint, done_tx, server) = fixture_runtime(|input, earlier| { + let answer = |content: serde_json::Value| serde_json::json!({"message":{"content":content.to_string()},"done":true,"prompt_eval_count":40,"eval_count":40}); + if earlier == 0 { + return serde_json::json!({"message":{"content":"{\"path\":\"app.py\""},"done":true,"done_reason":"length","prompt_eval_count":40,"eval_count":input["options"]["num_predict"]}); + } + let properties = &input["format"]["properties"]; + if properties["mechanism"].is_object() { + return answer(serde_json::json!({"path":"app.py","search":" return 1","mechanism":"Use a literal two as the return value","difference":"Edits only the single return line"})); + } + let choice = properties["search"]["enum"][0].as_str().unwrap_or_default(); + answer(serde_json::json!({"path":"app.py","search":choice,"text":choice.replace("return 1", "return 2")})) + }) + .await; + let root = tempfile::tempdir().unwrap(); + let repository = value_fixture(root.path(), APP); + let mut request = request(); + request.goal = "Make value return 2".into(); + request.repository = repository.clone(); + request.files = vec!["app.py".into()]; + request.checks = vec![LocalCheck { + program: "python".into(), + args: vec!["-m".into(), "unittest".into(), "discover".into()], + }]; + request.approve_host_execution = true; + request.budget.generations = 4; + request.budget.check_runs = 3; + request.budget.wall_seconds = 60; + request.budget.generated_tokens = 2048; + let profile = ModelProfile { + schema: 2, + model: "fixture:small".into(), + digest: "fixture-digest".into(), + runtime_version: "fixture-version".into(), + endpoint: endpoint.clone(), + context_tokens: 4096, + output_tokens: 512, + protocol: Some(EditProtocol::SearchReplace), + thinking: Some(phonton_types::local::LocalThinkingMode::Off), + probes: vec![ModelProbe { + name: "SearchReplace edit".into(), + status: CheckStatus::Passed, + output: String::new(), + detail: "fixture transport".into(), + input_tokens: None, + output_tokens: None, + elapsed_ms: 0, + }], + hardware: HardwareSnapshot::default(), + measured_at_unix: 0, + }; + let hardware = HardwareSnapshot { + ram_total_bytes: Some(16 * 1024 * 1024 * 1024), + ram_available_bytes: Some(8 * 1024 * 1024 * 1024), + ..HardwareSnapshot::default() + }; + let run_dir = root.path().join("44444444-4444-4444-8444-444444444444"); + let result = run_inner( + request, + profile, + RuntimeGuard::external_unverified(endpoint), + &run_dir, + |_| {}, + Some(fixed_hardware(hardware)), + ) + .await; + let _ = done_tx.send(()); + let chats = server.await.unwrap(); + let receipt = result.unwrap(); + let choices = |chat: &serde_json::Value| -> Vec { + chat["format"]["properties"]["search"]["enum"] + .as_array() + .into_iter() + .flatten() + .filter_map(|choice| choice.as_str().map(str::to_owned)) + .collect() + }; + let function = APP.trim_end(); + assert_eq!(chats.len(), 3, "{receipt:?}"); + assert_eq!(chats[1]["options"]["num_predict"], 192); + assert!(choices(&chats[1]).contains(&" return 1".to_owned())); + assert!(!choices(&chats[1]).contains(&function.to_owned())); + assert_eq!(chats[2]["options"]["num_predict"], 512); + assert_eq!(choices(&chats[2]), [function]); + assert_eq!(receipt.state, "review_ready", "{receipt:?}"); + assert_eq!(receipt.hypotheses[0].status, HypothesisStatus::Accepted); + assert_eq!(receipt.selected_candidate, Some(2)); + let reason = &receipt.candidates[1].decision.as_ref().unwrap().reason; + assert!( + reason.contains("the edit was bound to that choice"), + "{reason}" + ); + assert_eq!( + std::fs::read_to_string(repository.join("app.py")).unwrap(), + APP + ); + } + #[tokio::test] async fn offline_npm_setup_is_not_a_verification_pass_without_execution() { let root = tempfile::tempdir().unwrap(); std::fs::write( diff --git a/phonton-worker/src/local_run/search.rs b/phonton-worker/src/local_run/search.rs index bf02d1f..c1f9732 100644 --- a/phonton-worker/src/local_run/search.rs +++ b/phonton-worker/src/local_run/search.rs @@ -87,11 +87,17 @@ pub(super) fn bound_edit_choices( return Ok((vec![path.into()], Vec::new())); } let search = search.ok_or("Strategy has no source anchor")?; - if !searches.iter().any(|allowed| allowed == search) { - return Err("Strategy source anchor was omitted from the edit choices".into()); - } + // The edit call's larger output limit can turn the symbol around a + // one-line strategy anchor into one whole-block choice. Both are unique + // in the source, so the smallest choice containing the anchor is the + // anchor itself or the tightest block around it. + let bound = searches + .iter() + .filter(|allowed| allowed.contains(search)) + .min_by_key(|allowed| allowed.len()) + .ok_or("Strategy source anchor was omitted from the edit choices")?; let constrained = if protocol == EditProtocol::SearchReplace { - vec![search.into()] + vec![bound.clone()] } else { Vec::new() }; @@ -1130,6 +1136,36 @@ mod tests { .is_err()); } #[test] + fn restart_line_anchor_binds_to_the_smallest_block_choice_around_it() { + let paths = vec!["src/stock.ts".into()]; + let searches = vec![ + "class Stock {\n restock(n) {\n this.n += n;\n }\n}".into(), + " restock(n) {\n this.n += n;\n }".into(), + "export const unit = 1;".into(), + ]; + let bind = |search, protocol| { + bound_edit_choices( + "src/stock.ts", + Some(search), + false, + protocol, + &paths, + &searches, + ) + }; + let (_, bound) = bind("class Stock {", EditProtocol::SearchReplace).unwrap(); + assert_eq!(bound, [searches[0].clone()]); + let (_, bound) = bind(" this.n += n;", EditProtocol::SearchReplace).unwrap(); + assert_eq!(bound, [searches[1].clone()]); + let (_, bound) = bind("export const unit = 1;", EditProtocol::SearchReplace).unwrap(); + assert_eq!(bound, ["export const unit = 1;"]); + let (_, bound) = bind("class Stock {", EditProtocol::UnifiedDiff).unwrap(); + assert!(bound.is_empty()); + assert!(bind("class Order {", EditProtocol::SearchReplace) + .unwrap_err() + .contains("omitted from the edit choices")); + } + #[test] fn restart_checks_actual_edit_even_if_runtime_ignores_schema() { let root = tempfile::tempdir().unwrap(); std::fs::write(root.path().join("add.rs"), "first\nsecond\nthird\n").unwrap();