Skip to content

feat: add cost-aware executor tier routing - #53

Open
aaaazqh wants to merge 1 commit into
AMAP-ML:mainfrom
aaaazqh:codex/executor-tier-routing-13
Open

aaaazqh wants to merge 1 commit into
AMAP-ML:mainfrom
aaaazqh:codex/executor-tier-routing-13

Conversation

@aaaazqh

@aaaazqh aaaazqh commented Aug 19, 2026

Copy link
Copy Markdown

Summary

Adds cost-aware executor tier routing to LongHorizon Harness.

A Manager can choose the initial executor tier, while the harness tracks valid Auditor failures for the same subtask and automatically escalates the next execution after the configured threshold. The implementation remains backend-agnostic and preserves the existing single-executor configuration.

Closes #13.

What changed

  • Added named executor tiers under [run.roles.executor.<tier>] and routing policy under [run.executor_routing].
  • Added strict configuration and programmatic validation for tier names, defaults, escalation targets, and thresholds.
  • Extended the English and Chinese Manager protocol with an optional Executor tier: field.
  • Added a shared structural parser authority for route, tier, task identity, ASK fields, related references, state, and task contract, including adversarial nested-protocol protection.
  • Added backend-agnostic GUI/CLI tier-to-adapter binding with permission isolation and field-level backend/model fallback.
  • Added durable routing evidence to managed rounds, events, JSONL, and reports.
  • Counted only accepted Auditor incomplete/blocked outcomes; provider failures, timeouts, cancellation, and rejected format repair are recorded as not_counted.
  • Added sticky escalation plus reset behavior for completion, route changes, and new task identities.
  • Updated English and Chinese documentation and initialization examples, including the current Resume limitation: a retry run is created and routing counters reset; this is not checkpoint continuation.

Compatibility

  • Existing [run.roles.executor] and single-executor runs continue to work unchanged.
  • Routing is config-only; no new public CLI flags were added.
  • GUI and CLI tiers build separate permission-scoped adapters.
  • Unknown Manager tiers are recoverable invalid plans and never silently fall back to another agent.

Validation

  • Executor routing focused suite: 132 passed.
  • Executor routing + role prompt suite: 145 passed.
  • Legacy focused comparison: 14 passed, 3 unchanged Windows secure-filesystem failures, 6 skipped because pytest-asyncio is unavailable.
  • Python compile check, CLI help, generated init template, diff, status, and process checks passed.
  • Three-agent planning, implementation, and independent final review completed with no P0-P3 findings.

Residual risk

  • No paid-provider end-to-end run was performed.
  • This Windows environment cannot exercise POSIX secure-filesystem behavior.
  • The full Windows suite is not green because of the repository's existing POSIX/secure-filesystem, symlink-privilege, and missing async-plugin baseline failures; no new failure class was introduced.
  • Before merge, Linux CI with full extras and a smoke run against two real backends is recommended.

AI assistance

The implementation was generated with AI assistance under separate planner, executor, and reviewer roles. The repository owner authorized the scope and contribution; upstream maintainer review is requested.

@Upper9527

Copy link
Copy Markdown
Collaborator

Thanks for putting together the focused executor-routing implementation. I ran the exact current head (c2605bd) against its base (be2e7b4) with the repository's full test environment. The result was 332 passed, 3 failed, 1 skipped.

The failing existing regression tests are:

tests/test_manager_hardening.py::test_executor_timeout_preserves_partial_output_for_recovery
tests/test_manager_hardening.py::test_auditor_timeout_keeps_executor_result_and_recovers
tests/test_manager_hardening.py::test_late_crash_report_preserves_completed_rounds

The first two lose the expected status field because the executor/auditor round is never reached. The third records 4 rounds instead of the expected 2 and exhausts the scripted episode sequence. The common cause is the new structural Manager parser rejecting previously accepted short plans such as Next: cli followed by Current Task State: when no Task: header is present. Those plans become invalid rounds, shift the episode sequence, and break the PR's backward-compatibility claim.

Since this test run, #52, #33, and #28 have been merged. GitHub now reports this PR as conflicting with the latest main.

Please:

  1. Rebase onto the current main.
  2. Preserve the existing Manager-plan compatibility used by the current suite, or introduce the stricter protocol without losing the timeout/recovery behavior.
  3. Restore the timeout status records and correct round accounting.
  4. Run the full suite, not only the routing-focused tests.

Once the full existing suite and routing tests are green on current main, the feature can be reviewed again.

@aaaazqh
aaaazqh force-pushed the codex/executor-tier-routing-13 branch from c2605bd to 8135632 Compare August 21, 2026 15:42

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Dynamic Executor Model Routing with Cost-Aware Escalation

2 participants