Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,34 @@ This project follows pre-1.0 SemVer: minor versions may still include breaking c

### Changed

- [changed] On a machine that generates 20 tokens per second or more, local
calibration also measures the model's own thinking mode and keeps it when
its edit and new-file probes pass. It used to keep `think:false` whenever
that worked. Probes in the runtime-default mode may generate a quarter of
the context (512 to 2,048 tokens) so reasoning does not use up the limit
before the answer. Slower machines keep `think:false`. A saved profile
keeps its mode: run `phonton models calibrate MODEL` to measure the model
again. Calibrating a model that reasons takes longer, because its probes
reason too, and its goals take longer per call.
- [changed] On a model that reasons before it answers, a local goal with one
editable path and no explicit budget reserves 8 model calls and up to
16,384 output tokens, instead of 4 calls and 4,096 tokens. The earlier
budget stopped a run after four calls, well inside its 600-second limit,
and cut the later calls to 512 output tokens, which the reasoning used up.
A multi-path goal on a unified-diff model keeps four calls and gets one
calibrated response of output per call. Other models and explicit budgets
are unchanged.
- [changed] On a model that reasons before it answers, when a local goal's
scope is one existing file of at most 100 lines and the model uses
search/replace, the first edit and every repair ask for the complete
corrected file. A search/replace reply changes one span, so a file that
needed several changes took one model call for each. The whole file must
fit the calibrated context with an output reserve that can hold it;
otherwise the span edit runs as before.
- [changed] A repair prompt for a local goal carries up to 4,000 characters
of the failed checks' output (a quarter of the context in characters, at
least 450), instead of 450. A test run with several failures no longer
reaches the model as the first and last few lines of one of them.
- [changed] With `PHONTON_LOG=debug`, each worker reply is logged with its
prompt size and the token counts the provider reported, so the cost of a
goal can be traced request by request.
Expand Down
21 changes: 16 additions & 5 deletions docs/local-harness-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,18 +160,29 @@ profiles. Finished probes are saved after each request, so cancellation, memory
pressure or an engine exit does not erase their diagnostic output. A retry runs
the probes again; an earlier passing profile remains until recalibration finishes.
The attempt's starting digest does not certify each later probe's weights.
Calibration keeps its format probes small. For coding goals, the saved profile
Calibration keeps its format probes small: 512 output tokens for an edit or
creation probe and 256 for the tool-call probe, or a quarter of the context
(at most 2,048) when the model may think first. For coding goals, the saved profile
reserves up to one quarter of that context for each complete edit response;
the goal's shared token budget can reduce it further. When later attempts are
allowed, the runner keeps half the remaining token budget for them. This is a
generation allowance, not a measured claim that the model can produce a file
of that size. Existing profiles keep their saved ceiling until recalibrated.
Calibration first requests `think:false` so a thinking model can spend the
small edit-probe budget on the actual edit. If no edit format or no new-file
format passes, it also probes the runtime's default thinking behavior and
prefers a mode that demonstrated new-file creation. The chosen request mode
small edit-probe budget on the actual edit. It also probes the runtime's
default thinking behavior when no edit format or no new-file format passed,
or when the `think:false` probes generated at 20 tokens per second or more
by the runtime's own timing: a machine that fast can afford a model that
reasons before each edit. It then prefers the runtime default if that mode
demonstrated new-file creation. A slower machine, or a runtime that reports
no generation timing, keeps `think:false`. The chosen request mode
is saved in a schema-2 digest-bound profile and reused for tool-call probes,
coding edits, and strategy requests. Older schema-1 profiles retain their
coding edits, and strategy requests. A profile in the runtime-default mode
whose probes returned thinking output is treated as a model that reasons:
its goals get the larger budget and the whole-file edits described in
[local-harness.md](local-harness.md). A profile saved by 0.27.1 or earlier
keeps its mode, usually `think:false`, until `phonton models calibrate MODEL`
runs again. Older schema-1 profiles retain their
original runtime-default request behavior until recalibrated. Probe
results show format compatibility, not general coding quality or proof that a
runtime honored the requested thinking setting.
Expand Down
39 changes: 31 additions & 8 deletions docs/local-harness.md
Original file line number Diff line number Diff line change
Expand Up @@ -407,12 +407,24 @@ candidate passes. A host-approved request needs enough slots for one complete
baseline and one complete candidate pass: twice the total of selected checks
and any offline dependency-preparation command. A smaller explicit budget is
shown as a plan warning and refused before host checks run.
On a model that reasons before it answers (a calibrated profile in the
runtime's own thinking mode whose probes returned thinking output), a reviewed
plan with one editable path and an implicit budget reserves 8 model calls and
up to 16,384 output tokens (one calibrated response per call), and raises the
check reserve to cover those calls, up to 32 launches. Reasoning then leaves
room for the edit, and a failed first attempt for repairs and a restart; the
600-second wall budget still bounds the run. Other models keep 4 calls and
4,096 tokens: without reasoning the larger reserve only lengthens a reply
that runs to its output limit.
When a selected model uses SearchReplace, each model response edits one path.
Reviewed plans with an implicit budget reserve at least one call per editable
path and scale check/output reserves up to the worker caps. The displayed plan
warns about caps and the lack of repair slack; it does not promise that every
path will be edited. Explicit budgets submitted for plan review are preserved
and warned about when they cannot cover even a straight sequence.
path will be edited. A plan with several editable paths on a unified-diff
model that reasons keeps its four calls, with an output reserve of one
calibrated response per call, up to 16,384 tokens. Explicit budgets submitted
for plan review are preserved and warned about when they cannot cover even a
straight sequence.
The original Git index is captured separately before checks. Its resolved path
and exact byte hash are compared before each generation and at final review.
Changes or unavailable observations stop the run and clear candidate selection;
Expand All @@ -428,7 +440,11 @@ fresh baseline approach unless another selected check newly passes. Losing a
pass without either signal also restarts. If a check gains a pass, or a
different selected check now fails with the same message, the controller keeps
that changed candidate for another repair. Feedback includes the
rejected edit and bounded check output. The most recent failed checks precede
rejected edit and the output of the failed checks: a quarter of the calibrated
context in characters, between 450 and 4,000, shared between the failed checks
and keeping the start and the end of each stream. When that much does not fit
beside the source, the 450-character excerpt is sent instead. The most recent
failed checks precede
copied edits so they survive tight context limits; uncheckable proposals retain
that evidence with its candidate identity (or baseline evidence if none ran).
A reply whose raw output repeats an earlier one in the run byte for byte is
Expand All @@ -437,11 +453,18 @@ then on every strategy and edit call in that run samples at temperature 0.7
with the candidate number as seed (recorded as the candidate's `sampling`);
until then calls stay at the calibrated temperature 0. Two successive repeated
outputs or candidate identities stop further inference.
When a search/replace reply changes nothing and its file has at most 300 lines,
the next edit from the same parent shows that file whole and asks for the
complete corrected file in the whole-file JSON shape used for new files. This
happens once per branch, and only when the file and an output reserve that can
hold it fit the calibrated context; otherwise the ordinary edit runs.
A search/replace reply changes one span, so a file that needs several changes
took one model call per change. When the scope is one existing file of at most
100 lines and the model uses SearchReplace and reasons before it answers, the
first edit and every repair show that file whole and ask for the complete
corrected file in the whole-file JSON shape used for new files. A restart
keeps its span edit, anchored to the strategy's source span. For any model,
when a search/replace reply changes nothing and its file has at most 300
lines, the next edit from the same parent is a whole-file edit, once per
branch. Either way the whole-file edit runs only
when the file and an output reserve that can hold it fit the calibrated
context; otherwise the ordinary edit runs. The candidate is still reviewed as
a diff against the captured source.
Different diagnostics are a search signal,
not proof that code quality improved. Every branch uses the same run budget.
Feedback is shortened when necessary to admit source; the exact transmitted
Expand Down
84 changes: 75 additions & 9 deletions phonton-cli/src/local_goal_cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -491,6 +491,7 @@ fn reviewed_selection(
context_tokens: profile.context_tokens,
output_tokens: profile.output_tokens,
protocol: profile.protocol,
reasons: profile.reasons(),
profile_sha256: crate::models_cli::profile_sha256(profile)?,
})
}
Expand Down Expand Up @@ -824,22 +825,46 @@ fn adjust_reviewed_plan_budget(
selection: Option<&ReviewedModelSelection>,
explicit_budget: bool,
) {
if selection.and_then(|selected| selected.protocol) != Some(EditProtocol::SearchReplace) {
return;
}
let editable = if plan.request.new_file.is_some() {
1 + plan.request.editable_existing.len()
} else {
plan.request.files.len()
} as u32;
if editable <= 1 {
return;
}
let checks_per_candidate =
plan.request.checks.len() as u32 + u32::from(plan.request.preparation.is_some());
let per_call = selection.map_or(0, |selected| {
selected.output_tokens.min(selected.context_tokens / 4)
});
// A model that reasons first spends output tokens before it reaches the
// edit. Other models keep the smaller reserves: on a CPU runner a reply
// that ran to an 8,192-token limit used up the whole wall budget.
let reasons = selection.is_some_and(|selected| selected.reasons);
if editable <= 1 {
// A one-path goal stopped after 4 calls, at a median 41 s of its
// 600 s wall budget, with later calls cut to 512 output tokens.
// Give repairs and restarts the worker's caps; the wall clock still
// bounds the run.
if !explicit_budget && reasons {
let budget = &mut plan.request.budget;
budget.generations = budget.generations.max(8);
let checks = (budget.generations + 1).saturating_mul(checks_per_candidate);
budget.check_runs = budget.check_runs.max(checks.min(32));
let tokens = u64::from(per_call).saturating_mul(u64::from(budget.generations));
budget.generated_tokens = budget.generated_tokens.max(tokens.min(16_384));
}
return;
}
if selection.and_then(|selected| selected.protocol) != Some(EditProtocol::SearchReplace) {
// A multi-path diff plan keeps its calls. The 4,096-token reserve
// gave them 2,048, 1,024, 512 and 512 output tokens, which a model
// that reasons first uses up before it reaches the edit.
if !explicit_budget && reasons {
let budget = &mut plan.request.budget;
let tokens = u64::from(per_call).saturating_mul(u64::from(budget.generations));
budget.generated_tokens = budget.generated_tokens.max(tokens.min(16_384));
}
return;
}
if explicit_budget {
if plan.request.budget.generations < editable {
plan.warnings.push(format!(
Expand Down Expand Up @@ -1894,8 +1919,13 @@ mod tests {
context_tokens: 4096,
output_tokens: 1024,
protocol: Some(EditProtocol::SearchReplace),
reasons: false,
profile_sha256: "fixture".into(),
};
let reasoning = ReviewedModelSelection {
reasons: true,
..selection.clone()
};
let make_plan = |files: usize, checks: usize| {
let request: LocalRunRequest = serde_json::from_value(json!({
"goal":"Update the scoped files",
Expand Down Expand Up @@ -1937,10 +1967,28 @@ mod tests {
.iter()
.any(|warning| warning.contains("above the 32-slot cap")));

// A one-path goal gets room to reason only on a model that reasons.
let mut one = make_plan(1, 1);
adjust_reviewed_plan_budget(&mut one, Some(&selection), false);
assert_eq!(one.request.budget.generations, 4);
assert_eq!(one.request.budget.generated_tokens, 4096);
adjust_reviewed_plan_budget(&mut one, Some(&reasoning), false);
assert_eq!(one.request.budget.generations, 8);
assert_eq!(one.request.budget.check_runs, 16);
assert_eq!(one.request.budget.generated_tokens, 8192);
assert_eq!(one.request.budget.wall_seconds, 600);
assert!(one.warnings.is_empty());

let mut one_plain = make_plan(1, 1);
adjust_reviewed_plan_budget(&mut one_plain, Some(&selection), false);
assert_eq!(one_plain.request.budget.generations, 4);
assert_eq!(one_plain.request.budget.generated_tokens, 4096);

let mut one_explicit = make_plan(1, 1);
adjust_reviewed_plan_budget(&mut one_explicit, Some(&reasoning), true);
assert_eq!(one_explicit.request.budget.generations, 4);
assert_eq!(one_explicit.request.budget.generated_tokens, 4096);

let mut one_unselected = make_plan(1, 1);
adjust_reviewed_plan_budget(&mut one_unselected, None, false);
assert_eq!(one_unselected.request.budget.generations, 4);

let mut explicit = make_plan(5, 1);
adjust_reviewed_plan_budget(&mut explicit, Some(&selection), true);
Expand Down Expand Up @@ -1974,7 +2022,24 @@ mod tests {
diff_selection.protocol = Some(EditProtocol::UnifiedDiff);
adjust_reviewed_plan_budget(&mut diff, Some(&diff_selection), false);
assert_eq!(diff.request.budget.generations, 4);
assert_eq!(diff.request.budget.generated_tokens, 4096);
assert!(diff.warnings.is_empty());

// On a model that reasons, a larger calibrated allowance gives each
// of the four calls its share.
diff_selection.context_tokens = 16_384;
diff_selection.output_tokens = 4096;
let mut plain_diff = make_plan(5, 4);
adjust_reviewed_plan_budget(&mut plain_diff, Some(&diff_selection), false);
assert_eq!(plain_diff.request.budget.generated_tokens, 4096);
diff_selection.reasons = true;
let mut roomy = make_plan(5, 4);
adjust_reviewed_plan_budget(&mut roomy, Some(&diff_selection), false);
assert_eq!(roomy.request.budget.generations, 4);
assert_eq!(roomy.request.budget.generated_tokens, 16_384);
let mut roomy_explicit = make_plan(5, 4);
adjust_reviewed_plan_budget(&mut roomy_explicit, Some(&diff_selection), true);
assert_eq!(roomy_explicit.request.budget.generated_tokens, 4096);
}

#[test]
Expand Down Expand Up @@ -2261,6 +2326,7 @@ mod tests {
context_tokens: 4096,
output_tokens: 512,
protocol: None,
reasons: false,
profile_sha256: "profile-hash".into(),
};
let mut reviewed = ReviewedLocalPlan {
Expand Down
Loading
Loading