Skip to content

fix(orchestrator): refuse a contradicted skip_to_close and show the initial-roofline hold - #2175

Merged
chaojhou merged 3 commits into
mainfrom
fix/refuse-contradicted-skip-to-close
Oct 26, 2026
Merged

chaojhou merged 3 commits into
mainfrom
fix/refuse-contradicted-skip-to-close

Conversation

@fengshaoyi-amd

@fengshaoyi-amd fengshaoyi-amd commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor
  • Description: what and why

    skip_to_close ends the run on the agent's word. Until now the Coordinator honoured it unconditionally and booked it as global_converged whenever budget was left. In the release/v1.1.5 release qualification (session d11ccead), the orchestrator emitted it in PRELUDE at cycle 0 with baseline_tput=1078.29 measured and 9.3 h of budget left; the session closed as global_converged and the qualification row read PASSED.

    The intent router now refuses the hint when the session's own facts contradict it: a measured baseline, and a remaining budget that still funds another cycle (the cycle_reloop_min_remaining_sec floor the phase machine already uses). The refusal goes through record_policy_denied with rule skip_to_close_contradicted, so the agent gets a policy_denied observation with a hint, and the denial lands in policy_denial_history and the breakdown phase events. A skip_to_close that reaches the phase machine can therefore only be booked as baseline_failed or time_exhausted. GPU work that keeps failing after the baseline still ends the run, through the FRAMEWORK plateau or skip_to_sweep and then SWEEP's sweep_failed. The PHASE CONTRACT block, orchestration.md and SKILL.md now state the rule.

    The agent made that close from inside a wait it could not see. After the baseline landed, PRELUDE stayed open for 55 minutes while the initial roofline it had enqueued ran. The === Phase === block showed the phase's 0.03 share as remaining_sec=0, the PHASE CONTRACT said baseline_tput > 0 exits PRELUDE, and orchestration.md told the agent to read that remaining_sec as its urgency signal. While PRELUDE or ENABLEMENT holds for the initial roofline, the block now carries a hold line. It names the roofline and how long the hold can still last (it ends at the optimization reserve), says that no hint advances the phase and that remaining_sec is the phase's share rather than the session's, and names a send_message observation as the expected answer each tick. The PHASE CONTRACT and orchestration.md now say the same. One helper reads the hold's two inputs for the phase predicates and for the line, so the line and the exit decision cannot disagree.

    • compute_next_phase is unchanged: recorded decisions replay as they were made, and a hint an older build left pending still closes the run. The hold's predicate inputs keep their keys and values.
    • The hold itself is unchanged: when it starts, how long it can last, and what ends it.
    • Deliberately not added: an allow condition for "GPU work keeps failing". It is not a fact the session can freeze without a new predicate, and the sweep already owns that close.
  • Linked issue(s): Part of release/v1.1.5: session closed at cycle 0 on a skip_to_close escalation that contradicts session state; PRELUDE stalled 55 min after its exit gate #2167. This PR now covers both defects the issue reports: the contradicted close, and the PRELUDE hold the agent could not see. It stays "Part of" so that the issue remains open until the v1.1.5 qualification row has been rerun on a build that carries it. Not covered: the router still drops a skip_to_kernel emitted outside FRAMEWORK_AGENT without telling the agent; fix(agentx): serve the replay with its prefix cache, and stop reading an unmeasured round as a result #1867 replaces that drop with a denial.

  • Tests: replaced test_escalate_skip_to_close_sets_pending_hint, which pinned the unconditional behaviour, with test_escalate_skip_to_close_with_baseline_and_budget_is_denied and test_escalate_skip_to_close_the_facts_admit_is_deferred[no_baseline, budget_below_cycle_floor]. Added test_the_phase_block_shows_the_initial_roofline_hold_exactly_while_it_holds: the line appears while the hold is on, naming the roofline and its bound, and disappears both at the optimization reserve and once the roofline books. Run locally on Windows, with the POSIX-only stdlib modules the package imports shimmed outside the repo, so CI is the authoritative run:

    • test_coordinator_async_methods_coverage_unit.py -k escalate: 10 passed
    • test_longrun_phase1.py, test_orchestration_prune_branch_permission.py, test_phase_state_plateau.py, test_close_phase_sequencer.py, test_workflow_evaluation_contract.py: 197 passed
    • orchestration prompt and doc tests (test_prompt_phase_scoping.py, test_role_realignment.py, test_prompt_kernel_lane_ownership.py, test_per_domain_prompts.py, test_prompt_reference_index.py, test_agentx_launch_ownership_prompt.py, test_drop_scoreboard.py, test_grid_size_prompt_unit.py, test_codex_prompt_transport.py, scripts/tests/test_check_cli_references.py): 240 passed; 3 specialist-runner cases in test_per_domain_prompts.py fail only under the Windows shim (os.open on a directory)
    • on the hold-line commit: test_phase_state_machine.py, test_prompt_phase_scoping.py, test_role_realignment.py, test_longrun_phase1.py, test_prompt_kernel_lane_ownership.py, test_codex_prompt_transport.py, test_prompt_reference_index.py, scripts/tests/test_check_cli_references.py: 244 passed
    • ruff check and ruff format --check (0.16.2) on the changed files: clean
    • scripts/code_metrics.py cannot run on Windows (it hands about 1,000 paths to one ruff command line, past the 32K limit), so it is left to the code-metrics job
  • Size/complexity triggers crossed: module length only, in three modules that were over the 1,200-line limit before this change. machine_state.py grows by 40 lines (1975 to 2015) and intent_router.py by 10 (1380 to 1390). prompt_builder.py gains nothing from this diff: main already carries it at 1212 lines against a recorded 1205, and both PHASE CONTRACT rewordings keep _section_phase_semantics at its 52 lines. The refusal's query sits beside _skip_to_close_reason, whose labelling it reuses; the hold line sits beside _holds_for_initial_analysis, whose rule it reuses; the enforcement sits in the handler that already routes escalate hints. Splitting either module is a refactor of its own, not part of this fix, so baseline-raise is applied for these three module-length entries only. Every changed or added function stays at cyclomatic complexity 5 or below, and phase_status_summary stays under 50 lines (48).

  • If this simplifies or refactors: not a refactor. The hold's two inputs, previously built in two places, now come from one helper.

  • Observable effect:

    • An orchestrator skip_to_close emitted while a baseline is measured and the budget still funds another cycle no longer closes the session. The agent receives a policy_denied observation (rule=skip_to_close_contradicted, with a reason naming the measured baseline and the minutes left against the minutes a cycle needs), the denial is recorded in state.json under policy_denial_history, and the run continues in its current phase. Before, such a session closed on the spot and was booked as global_converged: d11ccead stopped at cycle 0 with 9.3 h unused, and release qualification reported it PASSED. A skip_to_close with no measured baseline (baseline_failed) or below the cycle floor (time_exhausted) behaves as before.
    • While PRELUDE or ENABLEMENT waits for its initial roofline, the orchestration prompt's === Phase === block carries one more line, for example hold : waiting for initial roofline 59d9e666, for at most 12180s more. No hint advances PRELUDE, and remaining_sec above is this phase's share, not the session's; answer each tick with a send_message observation until it moves on. Before, the block showed only the phase's spent share (remaining_sec=0), which the agent read as a spent session and tried to escalate out of.
  • Breaking changes: no. No schema, CLI or replay change; agents see one new denial rule and one new Phase block line.

  • PR addresses single concern: yes, the false close of release/v1.1.5: session closed at cycle 0 on a skip_to_close escalation that contradicts session state; PRELUDE stalled 55 min after its exit gate #2167. Seven files carry the refusal: where it is decided (machine_state.py), where it is enforced (intent_router.py), the three prompt contracts that describe it, and two test files. The hold line is the same incident's first link and was raised in this PR's review, so it lands here rather than in a follow-up: the v1.1.5 cherry-pick then carries both.

  • Root cause is upstream (Magpie/TraceLens/GEAK/IntelliKit/AgentKernelArena), ticket filed: no, the cause is in the Coordinator.

…tradict

A skip_to_close ends the run on the agent's word and is booked as
global_converged whenever budget is left. Session d11ccead closed at
cycle 0 with a measured baseline and 9.3 h of budget left, and release
qualification read that close as PASSED.

The intent router now denies the hint (rule skip_to_close_contradicted)
when a baseline is measured and the remaining budget still funds another
cycle, so a live skip_to_close can only earn baseline_failed or
time_exhausted. GPU work that keeps failing after the baseline still ends
the run, through the sweep. compute_next_phase is unchanged: recorded
decisions replay as they were made, and a hint already pending still
closes the run.

Part of #2167.

Co-authored-by: Cursor <cursoragent@cursor.com>
@fengshaoyi-amd
fengshaoyi-amd requested a review from a team as a code owner October 25, 2026 16:56
@fengshaoyi-amd fengshaoyi-amd added domain:inference Related to inference optimization type:bug Something is broken or not working as expected labels Oct 25, 2026
@fengshaoyi-amd fengshaoyi-amd self-assigned this Oct 25, 2026
@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Code metrics gate: PASSED with waived findings (baseline-raise)

Judged: the 8 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension Limit New Worse Baseline out of date Baseline entries vs base
Cyclomatic complexity fails > 20 0 0 0 6 +0
Cognitive complexity fails > 30 0 0 0 53 +0
Function length (lines) fails > 50 0 0 0 545 +0
Nested block depth fails > 4 0 0 0 44 +0
Parameter count fails > 5 0 0 0 3 +0
Module length (lines) warns > 800, fails > 1200 0 2 0 43 +0
Duplicated lines per file fails > 0 0 0 0 95 +0
Dead code (vulture) fails > 0 0 0 0 2 +0
Directory size (entries) fails > 50 0 0 0 2 +0
Check Findings Effect
Comments 0
Broad catch without a reason 0
English only 0
Production code imports test code 0
Repeated literals 0
A1 Private name of another package 0
A8 Flag argument 0
A9 Tuple return 0
A10 Raw vocabulary comparison 0
A11 Hidden global write 0
A12 Pure forwarder 0
A12 Single-implementation abstraction 0

Worse than the baseline (waived by the baseline-raise label): 2

Unit Dimension Baseline Now
src/hyperloom/orchestrator/loop/intent_router.py:1 Module length (lines) 1380 1390
src/hyperloom/orchestrator/phases/machine_state.py:1 Module length (lines) 1975 2015

How to fix

  • New or worse: split the unit (extract a function, a class, a module) until it is back under the limit, or no worse than its recorded value. The baseline cannot absorb a new violation: CI refuses any baseline entry added or raised relative to the base branch.
  • Baseline out of date: something improved. Run python scripts/code_metrics.py --update-baseline and commit scripts/code_metrics_baseline.txt; the command only lowers or removes entries.
  • Override: the baseline-raise PR label waives new, worse and baseline-growth findings and the failing design checks (they stay listed here). Use it only with the reason in the PR description; it waives nothing else.
  • Tools for a local run: pip install ruff==0.16.2 complexipy==8.0.1 vulture==2.16 and npm install -g jscpd@5.4.1 (or point CODE_METRICS_JSCPD at another jscpd command, such as npx --yes jscpd@<version>).

Already over at the base (backlog, not this change's): 1

Unit Dimension Base Now
src/hyperloom/orchestrator/prompts/prompt_builder.py:1 Module length (lines) 1212 1212
18 findings in files this change does not touch

Outside this change (informational): 18

Unit Dimension Baseline Now
src/hyperloom/orchestrator/actions/executors/_integrate_apply.py ApplyStageMixin._stage_apply Cognitive complexity absent 32
src/hyperloom/orchestrator/kernel/serving_log_evidence.py resolve_forge_server_log Cognitive complexity absent 49
src/hyperloom_kb/collect/mapping.py _experience Cognitive complexity absent 31
src/kernelforge/cli/forge_loop_command.py forge_loop Cognitive complexity absent 31
src/kernelforge/loop/runner.py IterationLoop._collect_session_candidate Cognitive complexity absent 44
src/kernelforge/loop/runner.py IterationLoop._rule_on_attempt Cognitive complexity absent 38
src/hyperloom/orchestrator/kernel/integrate_accuracy.py grade_integrate_accuracy Function length (lines) absent 56
src/hyperloom/orchestrator/loop/_resume_recovery.py ResumeRecoveryMixin._resume_recover_pending_warm_replay Function length (lines) absent 75
src/hyperloom_kb/http_service.py Module length (lines) absent 1525
src/kernelforge/cli/forge_loop_command.py Module length (lines) absent 2003
src/hyperloom/inference_optimizer/cli/preflight.py _probe_rocm_build Function length (lines) 54 57
src/kernelforge/gemm_tune/tuners/_aiter_dense_common.py run_aiter_dense_tuner Function length (lines) 118 119
src/hyperloom/inference_optimizer/breakdown/schema.py Module length (lines) 2681 2686
src/hyperloom/orchestrator/prompts/specialist_prompt_builder.py Module length (lines) 2556 2567
src/kernelforge/fusion/command.py Module length (lines) 1584 1708
src/kernelforge/loop/runner.py Module length (lines) 6051 6510
src/kernelforge/loop/task_preparer.py Module length (lines) 1299 1557
src/hyperloom/inference_optimizer/cli/__init__.py Module length (lines) 2727 2723

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, ruff 0.16.2, vulture 2.16.

Drop the unread now_unix parameter, build the denial before recording it, and keep the PHASE CONTRACT rewording to the lines it replaces.

Co-authored-by: Cursor <cursoragent@cursor.com>
@fengshaoyi-amd fengshaoyi-amd added the baseline-raise Waives new, worse and baseline-growth code-metrics findings; reason required in the PR description label Oct 25, 2026
@fengshaoyi-amd

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown

Automated review started for head e5297e6. Progress: https://github.com/AMD-AGI/Hyperloom/actions/runs/40215439467

@github-actions

Copy link
Copy Markdown

PR #2175 -- fix(orchestrator): refuse a skip_to_close the session's own facts contradict

What it does: An orchestrator agent's skip_to_close escalate hint closed the session unconditionally and was booked global_converged even when the session's own state contradicted abandonment — release-qualification session d11ccead closed at cycle 0 with a measured baseline, 9.3 h of budget unused and zero optimization cycles, and the qualification row read PASSED. The intent router (_handle_escalate_strategy_change, the single dispatch site of escalate_strategy_change) now asks skip_to_close_contradiction — a new query in the phase machine's own module, beside the _skip_to_close_reason labelling the booking path already uses — and refuses the hint through record_policy_denied (rule skip_to_close_contradicted) exactly when the booking would mislabel a never-converged run: a measured baseline and a remaining budget that still funds another cycle. A hint the facts admit (no baseline, booked baseline_failed; below the cycle_reloop_min_remaining_sec floor, booked time_exhausted) is deferred as before, compute_next_phase replays recorded decisions unchanged, and the agent-facing contracts (SKILL.md, orchestration.md, the PHASE CONTRACT block) state the new rule.

Blocking issues: none

Checked: intent_router.py:474/1271-1305/1321-1391 (handler, single dispatch entry, record_policy_denied), machine_state.py:104-183/446-486/901-910/954-960/1436-1481/1496-1525/1691-1720 (_cycle_reloop_min_remaining_sec, session_remaining_seconds, _number, the frozen booking inputs, _skip_to_close_reason, skip_to_close_contradiction, _terminal_close_transition, replay_next_phase), machine.py:203-246, close.py:875-878, shared_state.py:65-82/656/1360-1374/1434-1455 (hint vocab, policy_denial_history, pending_escalate_hint writers), policy/denial.py (free-form rule identifier), prompt_builder.py:234-308/695-730, orchestration.md:140-180, SKILL.md:415-435, references/internals.md:45-80, _baseline_promotion.py:270-330, proposal_bench.py:99, message_bus.py:20-58, _subprocess_kill.py:276-304 (resolve_benchmark_timeouts); tree-wide greps over src/ and scripts/ for skip_to_close_contradicted, skip_to_close_contradiction, skip_to_close, _handle_escalate_strategy_change, set_pending_escalate_hint, pending_escalate_hint, strategy_change; base-tree vs pr-head diffs of intent_router.py and machine_state.py; PR #2033 body and issue #2167 via gh | Ran: pytest src/hyperloom/inference_optimizer/tests/test_coordinator_async_methods_coverage_unit.py -k escalate (10 passed) and pytest src/hyperloom/inference_optimizer/tests/test_longrun_phase1.py -k "skip_to_close or cycle_reloop_floor" (16 passed) at head e5297e6 from pr-head; ruff 0.16.2 --select C901,PLR0913 on the three changed runtime files at head and on base copies (skip_to_close_contradiction complexity 2, _handle_escalate_strategy_change 4 at head vs 3 at base, ceiling 20, max 5 params) | Base: 69546c4 | Head: e5297e6

LGTM

@xiaofei-zheng

Copy link
Copy Markdown
Collaborator

PR #2175 -- fix(orchestrator): refuse a skip_to_close the session's own facts contradict

What it does. An orchestration escalate_strategy_change{next_action_hint='skip_to_close'} was stored as a pending hint without ever being checked against the session state the Coordinator already held, so a run with a measured baseline and most of its budget left closed on the agent's word and was booked as global_converged — session d11ccead stopped at cycle 0 with 9.3 h unused and release qualification read PASSED. The intent router now asks the phase machine what that close would be booked as before accepting the hint: when the answer is global_converged (baseline measured, remaining budget still above the cycle_reloop_min_remaining_sec floor the reloop decision already uses) the hint is refused through record_policy_denied with rule skip_to_close_contradicted instead of being stored, so a skip_to_close that reaches the phase machine can now only close as baseline_failed or time_exhausted. compute_next_phase is untouched, so recorded decisions replay unchanged.

Blocking issues: none.

Checked: intent_router.py _handle_escalate_strategy_change / record_policy_denied; machine_state.py skip_to_close_contradiction, _skip_to_close_reason, _base_workflow_predicate_inputs, session_remaining_seconds, _cycle_reloop_min_remaining_sec, compute_next_phase; shared_state.py ESCALATE_HINT_VOCAB, set_pending_escalate_hint, record_policy_denial; policy/gate.py PolicyGate; phases/proposal_bench.py:99, the only other writer of a pending escalate hint (it writes skip_to_sweep); denial.rule in session_breakdown.v6.workflow-evaluation.v3.schema.json:988, which is a free string, so no accept-set had to widen for the new rule id; the prompt copies at SKILL.md:425, orchestration.md:159-173 and :424, prompt_builder.py:296-306 and :708; the --max-hours backfill at cli/__init__.py:1977 and :2468; state.json of session 20261024T085938Z-d11ccead; and the full changed-file lists of the overlapping open PRs #1867 and #1869.

Ran: the escalate tests at head (10 passed) and again against the merge-base copies of intent_router.py and machine_state.py, where test_escalate_skip_to_close_with_baseline_and_budget_is_denied fails with assert 'skip_to_close' == '' — the regression test falsifies rather than restating the implementation; test_longrun_phase1.py, test_close_phase_sequencer.py, test_phase_state_plateau.py, test_workflow_evaluation_contract.py, test_orchestration_prune_branch_permission.py (197 passed); the prompt-contract tests (101 passed); and skip_to_close_contradiction directly against session d11ccead's own facts (max_minutes=720, baseline_tput=1078.28978, 33665 s remaining), which returns baseline_tput=1078.29 is measured and 561 min remain; a new cycle needs 130 min with a floor of 7800 s — the fix catches the exact case the issue reports. The router and compute_next_phase build the same three predicate inputs from the same two helpers, so they cannot disagree about the floor or the remaining seconds.

Base: 69546c4a66869f610b8c8da0d539240a0099e4e3 | Head: e5297e6c4b41962a3ba8bde28e3b61dd341f885c

LGTM

@xiaofei-zheng

Copy link
Copy Markdown
Collaborator

Follow-up to my review above. The diff itself still looks right to me; this is about scope against #2167.

I walked session d11ccead end to end. The close at 11:38 is the last of four links, and this PR covers only the last one:

  1. The baseline landed at 10:43:36 with benchmark_valid=True, but PRELUDE stayed held for 55 minutes while roofline 59d9e666 ran — last_roofline.error_excerpt is "the orchestrator cancelled this action while this round was running", i.e. it was still running when CLOSE killed it. Throughout the hold the === Phase === block showed remaining_sec=0 against a 0.03 share, so a spent phase share read as a spent session.
  2. The agent's only idea for getting out, escalate_strategy_change{skip_to_kernel}, is dropped in PRELUDE by a bare log.info with no observation back. It emitted it twice (seq 429/430) and recorded at tick 127 that both went "without a route".
  3. From tick 111 to tick 138 it re-derived at eleven consecutive ticks that no hint can advance PRELUDE, while the prompt requires it to emit something every tick. At tick 140 it produced premises its own prompt contradicted (7/7 api_error 404s, "no baseline on this stack", 610 min elapsed of a 360-min cap, against elapsed=157.0min remaining=563.0min baseline_tput=1078.28978).
  4. The Coordinator accepted that word without checking it. This is the link the PR fixes.

With this PR in, the tick-140 hint is denied — I ran skip_to_close_contradiction against the session's own facts and it refuses. So no false global_converged, no PASSED row, and the roofline is not cancelled. But links 1-3 are untouched: PRELUDE is still held, there is still no legal way to ask for an advance, and the agent is still required to emit each tick, so it keeps re-sending skip_to_close and collecting denials (the streak is a fact for self-correction only, nothing acts on it). Worst case the run now reaches time_exhausted instead of a false global_converged — an honest terminal, which is the improvement — but the 9.3 h still goes unused. #2167 should not be closed on this PR.

Two things worth settling before this lands. #2033 already blocks a skip_to_close in PRELUDE once a baseline has landed, and additionally adds the hold line to the phase block, which is the part that addresses link 1; its description reports the same shape (10 h session, --phase-budget-prelude-pct 0.03, an 89-minute hold read as a spent session). #1867 replaces the silent skip_to_kernel drop with an escalate_hint_phase denial, which is link 2, and its comment names this exact failure. All three PRs edit the same hunk of _handle_escalate_strategy_change and the same paragraph of orchestration.md, so a landing order needs agreeing rather than being settled by whoever rebases last.

@fengshaoyi-amd

Copy link
Copy Markdown
Contributor Author

@xiaofei-zheng Agreed that #2167 stays open; this PR is linked as "Part of", so merging it does not close the issue. Links 1 to 3 stay tracked there.

One correction on the effect of this PR alone: the worst case is not time_exhausted. The hold is bounded by analysis_affordable_sec, the session's usable time minus the optimization reserve (OPTIMIZATION_RESERVE_PCT = 0.50). Once only the reserve is left, the hold stops and PRELUDE leaves through prelude_done. So d11ccead with this PR would have denied tick 140, kept the roofline, and entered the optimization phases when the roofline booked, or at the latest with about 360 of its 720 minutes left. Until links 1 to 3 are fixed, the hold still costs the agent's turns and the hold's share of the budget, but it no longer costs the run.

On landing order, this PR does not need to wait for #2033 or #1867. Neither replaces it: it refuses the hint in every phase and answers the agent with a denial. It also merges cleanly with #1867's escalate_hint_phase change in the router (git merge-tree). The textual overlaps that remain are one orchestration.md paragraph with #1867 and adjacent lines in machine_state.py with #2033. Both PRs already conflict with main and need a rebase regardless, so whichever lands later can take these lines with it.

…ase block

While PRELUDE or ENABLEMENT waits for the initial roofline it enqueued, the
`=== Phase ===` block now carries a `hold` line: the roofline it waits for,
how long the wait can still last, that no hint advances the phase, that
`remaining_sec` is the phase's share rather than the session's, and that a
`send_message` observation is the expected answer each tick. The PHASE
CONTRACT no longer says `baseline_tput > 0` alone exits PRELUDE, and
orchestration.md stops reading that share as urgency under a hold.

One helper reads the hold's two inputs for both phase predicates and for the
block, so the line the agent is shown and the exit decision cannot disagree.

Part of #2167.

Co-authored-by: Cursor <cursoragent@cursor.com>
@fengshaoyi-amd fengshaoyi-amd changed the title fix(orchestrator): refuse a skip_to_close the session's own facts contradict fix(orchestrator): refuse a contradicted skip_to_close and show the initial-roofline hold Oct 26, 2026
@fengshaoyi-amd

Copy link
Copy Markdown
Contributor Author

@xiaofei-zheng Pushed 0ea9dc6 for link 1, and for the part of link 3 the prompt owns. While PRELUDE or ENABLEMENT waits for its initial roofline, the === Phase === block now carries a hold line; for a d11ccead-like state it reads:

hold      : waiting for initial roofline 59d9e666, for at most 12180s more. No hint advances PRELUDE, and remaining_sec above is this phase's share, not the session's; answer each tick with a send_message observation until it moves on.

The PHASE CONTRACT no longer says baseline_tput > 0 alone exits PRELUDE, and orchestration.md no longer treats remaining_sec as the urgency signal under a hold. The line and the exit decision read the hold's two inputs through one helper, so they cannot disagree. Link 2 stays with #1867. Could you take another look?

@chaojhou chaojhou left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #2175 -- fix(orchestrator): refuse a skip_to_close the session's own facts contradict

What it does: Before this PR, an orchestrator skip_to_close hint was always stored as pending. It was booked as global_converged even when a baseline had been measured and hours of budget were left (session d11ccead). The intent router now calls a new phase-machine query, skip_to_close_contradiction (machine_state.py:1465-1481). The query reuses _skip_to_close_reason and the cycle_reloop_min_remaining_sec floor, the same inputs the booking path builds at machine_state.py:954-960. When the close would be booked global_converged, the router refuses the hint through record_policy_denied with rule skip_to_close_contradicted (intent_router.py:1293-1301). Hints the facts admit (baseline_failed, time_exhausted) are still deferred as before. compute_next_phase and replay are unchanged. SKILL.md, orchestration.md and the PHASE CONTRACT block state the rule.

Blocking issues: none

SKIPPED: overlap with #2033 / #1867. They share the intent (refuse a post-baseline skip_to_close) and the overlap is textual only. The landing order is already discussed in the PR comments.

Checked: intent_router.py:1271-1305, 1321-1391 (handler, record_policy_denied) and every other record_policy_denied call site; machine_state.py:156-183, 446-486, 901-905, 940-965, 1436-1525 (floor, session_remaining_seconds, _number, frozen skip_to_close inputs, _skip_to_close_reason, skip_to_close_contradiction, _terminal_close_transition); the writers of set_pending_escalate_hint (intent_router.py:1303, proposal_bench.py:99); consumers of the strategy_change topic (message_bus.py); skip_to_close prose across src/ and docs/ (internals.md:59, orchestration.md:161-166 and 425, prompt_builder.py:299-305 and 709, optimization-loop.md:99); open PRs #2033 and #1867 for overlap | Ran: skip_to_close_contradiction called directly on the head tree for six fact sets (180 min with 1800 s spent and baseline 1078.29 -> denied; 10200 s spent -> admitted; baseline 0.0 and None -> admitted; max_minutes=0 -> denied; 720 min with 9420 s spent, the d11ccead shape -> denied, 563 min left against a 130 min floor); git merge-tree of the head onto main 32e80a9 (includes #2159): clean | Base: 69546c4 | Head: 0ea9dc6 | Re-checked at 0ea9dc6, which adds the PRELUDE/ENABLEMENT initial-roofline hold line to the Phase block (machine_state._initial_analysis_hold_line, shown only when _holds_for_initial_analysis holds, with its two inputs now read by one helper _initial_analysis_inputs and the predicate inputs unchanged in keys and values), the matching orchestration.md wording and test_the_phase_block_shows_the_initial_roofline_hold_exactly_while_it_holds; the description states it and baseline-raise is applied for the three module-length entries; CI green

LGTM

LGTM

@chaojhou
chaojhou merged commit e7b8bc7 into main Oct 26, 2026
44 checks passed
@chaojhou
chaojhou deleted the fix/refuse-contradicted-skip-to-close branch October 26, 2026 08:06
@xiaofei-zheng

Copy link
Copy Markdown
Collaborator

Post-merge review of 0ea9dc624, which landed after my review of e5297e6c and was merged at the new head with no review against it.

What it does. While PRELUDE or ENABLEMENT is held for the initial roofline it enqueued, the === Phase === block now carries a hold line naming the roofline, how long the hold can still last, that no hint advances the phase, that remaining_sec is the phase's share rather than the session's, and that a send_message observation is the expected answer each tick. The PHASE CONTRACT drops the claim that baseline_tput > 0 alone exits PRELUDE, and orchestration.md stops reading that share as urgency under a hold. initial_analysis_pending and analysis_affordable_sec move into one helper read by both phase predicates and the line.

Blocking issues: none.

One correction to my earlier comment: I wrote that with the refusal alone the worst case is time_exhausted. That was wrong, and @fengshaoyi-amd's reply has it right — prelude_affordable_seconds bounds the hold at OPTIMIZATION_RESERVE_PCT = 0.50, so the hold ends and PRELUDE leaves through prelude_done with roughly half the budget still unspent.

Checked: _initial_analysis_hold_line and _initial_analysis_inputs against _holds_for_initial_analysis, _baseline_is_anchored and the two hold sites in replay_next_phase and _enablement_transition; the helper's keys and values against the two inline dicts it replaces in _prelude_predicate_inputs and workflow_predicate_inputs, which are identical, so recorded inputs replay unchanged; prelude.py:2346, which skips enqueueing the roofline while the warm replay is in flight; the ≤7 lines budget in the phase_status_summary docstring, which still holds — PRELUDE renders 6, ENABLEMENT 6 base plus reloop plus hold for 7, every other phase 6; and state.auto_roofline_pending_task_id.strip() in the f-string, which initial_analysis_pending guarantees is a non-empty string before it is reached.

Ran: test_phase_state_machine.py at head (59 passed) and test_the_phase_block_shows_the_initial_roofline_hold_exactly_while_it_holds against 0ea9dc624~1, where it fails on the missing line, so it falsifies rather than restating the implementation; and phase_status_summary next to compute_next_phase over the hold with the warm replay in flight and with the measure round dropped — compute_next_phase returns None in each, so the phase is held and the line's instruction holds in all of them. CI is green at 0ea9dc624 across all twelve shards, coverage, code-metrics and the static gates.

ZhengGong-amd added a commit that referenced this pull request Oct 26, 2026
An `escalate_strategy_change` carrying `skip_to_close` ended the run on the
orchestration model's word. A real MuseSpark session was closed this way after
the model retracted its own hint; the run reported global_converged with 0%
validated gain and more than 13 hours of budget left.

The hint is removed from the closed vocabulary together with its terminal
branch in the phase machine, the #2175 contradiction rule, and the four
workflow-evaluation `global` inputs that only existed to price it. A run now
closes only on a recorded stop reason, a target reached, convergence of the
macro-cycles, an exhausted budget or the closing deadline.

Observable effect: a hint outside the vocabulary, `skip_to_close` included,
is answered with a `policy_denied` event (rule `escalate_hint_not_in_vocab`)
instead of being dropped silently, so the model sees why nothing happened. A
`pending_escalate_hint` of `skip_to_close` persisted by an earlier version is
ignored on resume. The workflow-evaluation v3 schema loses the `skip_to_close`
hint value and the `sweep_closeout_status`, `session_remaining_sec`,
`cycle_reloop_min_remaining_sec` and `baseline_tput` global inputs.

Co-authored-by: Cursor <cursoragent@cursor.com>
chaojhou pushed a commit that referenced this pull request Oct 26, 2026
…ved framework tree (#2191)

* fix(orchestrator): stop the orchestration LLM from closing the run

An `escalate_strategy_change` carrying `skip_to_close` ended the run on the
orchestration model's word. A real MuseSpark session was closed this way after
the model retracted its own hint; the run reported global_converged with 0%
validated gain and more than 13 hours of budget left.

The hint is removed from the closed vocabulary together with its terminal
branch in the phase machine, the #2175 contradiction rule, and the four
workflow-evaluation `global` inputs that only existed to price it. A run now
closes only on a recorded stop reason, a target reached, convergence of the
macro-cycles, an exhausted budget or the closing deadline.

Observable effect: a hint outside the vocabulary, `skip_to_close` included,
is answered with a `policy_denied` event (rule `escalate_hint_not_in_vocab`)
instead of being dropped silently, so the model sees why nothing happened. A
`pending_escalate_hint` of `skip_to_close` persisted by an earlier version is
ignored on resume. The workflow-evaluation v3 schema loses the `skip_to_close`
hint value and the `sweep_closeout_status`, `session_remaining_sec`,
`cycle_reloop_min_remaining_sec` and `baseline_tput` global inputs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(prompts): drop skip_to_close from the orchestration contract

The orchestration prompt, the PHASE CONTRACT block, SKILL.md, the internals
reference and the optimization-loop concept page still described a hint that
no longer exists. They now say that no hint ends the run, that the Coordinator
closes it with a terminal stop reason, and that a hint outside the closed
vocabulary is denied with `escalate_hint_not_in_vocab`.

Prompt and documentation text only.

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(install): share the deterministic image-baseline commit with the vLLM source route

prepare_vllm_image_git_tree both staged the image's tracked deltas and
committed them as a synthetic baseline. The commit half (fixed identity, the
parent's date, refs/hyperloom/{upstream,image-baseline}, HEAD pinned and
verified) is now commit_image_baseline, so another framework tree can be
pinned the same way.

The post-pin verification ignores uncommitted content inside submodules,
which a parent commit cannot record. No other observable change: the vLLM
baseline commit has the same tree, parent, date, identity and subject as
before.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(install): pin each framework source tree's served state as its image baseline

A prebuilt image's SGLang and aiter checkouts carry hundreds of uncommitted
changes and untracked source files, and that working tree is what the server
runs. Nothing committed it, so integrate_patch stashed it away before every
candidate and graded the candidate on a different SGLang than the baseline
measured: a MuseSpark session reverted a candidate on a spurious -18% because
the 1105 modified files of the image were not in the tree it ran on.
Specialist worktrees, cut from HEAD, were also authored against that other
tree.

install.sh now pins every discovered framework checkout the way it already
pins the vLLM image source. Tracked changes and untracked, non-ignored files
become one `Hyperloom image baseline` commit on top of HEAD, recorded as
refs/hyperloom/image-baseline, with a fixed identity and the parent's date so
two containers of one image pin the same hash. HEAD is therefore the served
code before the baseline is measured, before any specialist is cut, and before
a warm replay or a candidate touches the tree.

Observable effect: after install, `git status` in /sgl-workspace/sglang is
clean and HEAD is the baseline commit; a checkout that is already pinned or
already clean is left alone, so a second install changes nothing. aiter's
jit/flydsl_cache runtime cache is added to the checkout's info/exclude so it is
neither committed nor stashed. Install stops, with the tree untouched, when a
checkout has unmerged paths or more than 256 MiB of untracked files.
--check-only and --dry-run change nothing.
Checkouts outside install.sh's roots and multi-node pod trees are not pinned.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test: pin that candidates and specialists run on the pinned image tree

On a checkout pinned by install.sh, an integrate candidate finds nothing to
stash and a revert returns to the served files, and a specialist worktree is
cut from the served tree rather than from the stock commit beneath it.

Test-only change.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(ci): lower the code-metrics baseline and give the unmerged-path test a git identity

Removing skip_to_close shrank _section_phase_semantics under the function-length
limit, _section_decision_framework to 84 lines, machine_state.py to 1928 lines
and shared_state.py to 2453 lines; the gate requires the baseline to follow.

The unmerged-path test ran `git merge` without the committer identity its other
git calls carry. On a runner with no global identity the merge refused to start,
no conflict was created and the pin succeeded. The merge now uses the same
identity, and the test asserts the conflict exists before pinning.

Test and CI baseline only.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
xiaofei-zheng pushed a commit that referenced this pull request Oct 26, 2026
…nitial-roofline hold (#2175) (#2185)

* fix(orchestrator): refuse a skip_to_close the session's own facts contradict

A skip_to_close ends the run on the agent's word and is booked as
global_converged whenever budget is left. Session d11ccead closed at
cycle 0 with a measured baseline and 9.3 h of budget left, and release
qualification read that close as PASSED.

The intent router now denies the hint (rule skip_to_close_contradicted)
when a baseline is measured and the remaining budget still funds another
cycle, so a live skip_to_close can only earn baseline_failed or
time_exhausted. GPU work that keeps failing after the baseline still ends
the run, through the sweep. compute_next_phase is unchanged: recorded
decisions replay as they were made, and a hint already pending still
closes the run.

Part of #2167.



* fix(orchestrator): tighten the skip_to_close refusal

Drop the unread now_unix parameter, build the denial before recording it, and keep the PHASE CONTRACT rewording to the lines it replaces.



* fix(orchestrator): show the agent the initial-roofline hold in the Phase block

While PRELUDE or ENABLEMENT waits for the initial roofline it enqueued, the
`=== Phase ===` block now carries a `hold` line: the roofline it waits for,
how long the wait can still last, that no hint advances the phase, that
`remaining_sec` is the phase's share rather than the session's, and that a
`send_message` observation is the expected answer each tick. The PHASE
CONTRACT no longer says `baseline_tput > 0` alone exits PRELUDE, and
orchestration.md stops reading that share as urgency under a hold.

One helper reads the hold's two inputs for both phase predicates and for the
block, so the line the agent is shown and the exit decision cannot disagree.

Part of #2167.



---------


(cherry picked from commit e7b8bc7)

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

baseline-raise Waives new, worse and baseline-growth code-metrics findings; reason required in the PR description domain:inference Related to inference optimization type:bug Something is broken or not working as expected

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants