Skip to content

feat(scheduler): add maxConcurrentWorkers global dispatch cap (refs #97) - #144

Open
spacexun2 wants to merge 2 commits into
NanmiCoder:mainfrom
spacexun2:fix/max-concurrent-workers
Open

spacexun2 wants to merge 2 commits into
NanmiCoder:mainfrom
spacexun2:fix/max-concurrent-workers

Conversation

@spacexun2

@spacexun2 spacexun2 commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Background: #97 — when members are all ready the whole team starts working in parallel, hits 429/quota, and then stalls until a human intervenes. In the #97 design comment the maintainer drew the boundary as three things (concurrency permits, queue fairness, failure release) and separated retryable 429s from insufficient_quota that needs a route change (the latter belongs to host fallback; FALLBACK_FAILURE_CODES in members.ts already covers it). This PR implements the first slice: an opt-in per-team dispatch concurrency limit (one global config value). 429 backoff is deliberately left out of the plugin to avoid overlapping host llm-retry.

Changes:

  • Config/ToolsConfig gains maxConcurrentWorkers (z.natural().default(0), 0 = unlimited). At the default the behavior is path-for-path identical to today (the whole guard is gated on cap > 0).
  • The count and gate live inside the ticket transaction: after a task is selected and before beginTaskAttempt, count claimed/in_progress tasks held by non-removed members; at the cap, a new dispatch returns undefined — the task stays pending, the member stays idle, nothing is written. Dispatch is already event-driven, so graph changes and member idle edges re-kick; queueing and backfill need no new polling.
  • The recoverOwned recovery path is exempt from the cap so the parked recovery from fix: recover parked tasks after member loss #86 does not get blocked.
  • Explicit decision (documented in code comments): only member-held tasks are counted; captain takeover is not — the captain's take-over and cleanup actions should not be blocked by a member concurrency limit.
  • Docs: one row each in README.md / README_ZH.md / docs/usage.md.

Two semantic boundaries, stated explicitly:

  1. On a cap hit the gate writes nothing; member state is not rewritten through this path. If a member has a stale working state, the existing status sync converges it — the gate does not correct it.
  2. recoverOwned recovery ignores the cap, and a recovered task counts toward the in-flight number like any member task — a long-parked task therefore keeps holding a permit slot. That is the current, deliberate semantics; happy to adjust in discussion if it does not match expectations.

Verification:

  • lifecycle-verify gains cap scenarios (a second plugin instance on the shared fake harness, separate stateDir): 7 members with 7 ready tasks and cap=2 → exactly 2 dispatched, 5 pending; complete one → exactly the third gets dispatched; a member failure → permit released, queue not stuck; cold restore still rotates attempts while the cap is full.
  • Two red-test mutations against a build without the feature: deleting the guard → exactly the 4 cap checks FAIL; deleting only the recoverOwned exemption → exactly the fourth FAILs, the rest PASS. The assertions target the feature and its exemption specifically.
  • tsc --noEmit on both tsconfigs, the full build, the verify chain item by item (verify main gate, fallback-tdd, member-failure-tdd, quality-gates-tdd, stress-verify, web-routes, harness-compat-tdd, etc.) and lifecycle-verify in both classic and --modern-harness variants, all green.

Known scope note: queue fairness (ordering when several tasks contend for one permit) is not changed by this PR — it keeps nextReadyTask's assigned-first ordering. If #97 wants explicit fairness semantics later, that deserves its own discussion.

@spacexun2
spacexun2 force-pushed the fix/max-concurrent-workers branch from 54a5444 to 2f5573a Compare September 10, 2026 03:14
kickTeam dispatched every ready task to every idle member with no
concurrency budget, so a 7-member team immediately ran 7 model turns.
Config now accepts maxConcurrentWorkers (z.natural().default(0), 0 =
unlimited, matching the previous behavior exactly).

Inside the ticket transaction, after a task is selected and before
beginTaskAttempt, new dispatches are blocked while the count of
claimed/in_progress tasks held by current members has reached the cap:
the task stays pending, the member stays idle, and nothing is written,
so later idle/graph kicks re-run the decision for free. Two explicit
decisions: captain takeovers do not count toward the budget (the
captain is one extra lane outside the member worker pool), and cold
recovery of an unobserved durable attempt bypasses the cap so a
saturated budget can never wedge restart recovery (NanmiCoder#86).

The lifecycle verification gains a scenario that registers a second
plugin instance with maxConcurrentWorkers=2: 7 members and 7 ready
tasks admit exactly 2 concurrent dispatches, completion and failure
each release exactly one license to the queued tasks, and a saturated
budget never blocks recovery of a disposed owner's open attempt.
@spacexun2
spacexun2 force-pushed the fix/max-concurrent-workers branch from 2f5573a to c2d8d80 Compare September 11, 2026 02:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant