diff --git a/.agents/skills/afk/SKILL.md b/.agents/skills/afk/SKILL.md index b411c0f2d83..c9c125a2604 100644 --- a/.agents/skills/afk/SKILL.md +++ b/.agents/skills/afk/SKILL.md @@ -235,8 +235,9 @@ The single-line format makes submission unambiguous across harnesses; the carrie ### Stale-artifact lifecycle Treat `state/.subsuper-escalations`, its `.since` sidecar, `state/.subsuper-inject-wedged`, and `state/.subsuper-unknown-acked` as session-scoped delivery artifacts, not as the durable work record. -Always enter through `bin/fm-afk-launch.sh`, which clears prior-session artifacts only for a fresh entry and preserves the current session's buffer on refresh. -Always exit through `bin/fm-afk-launch.sh stop`, which keeps `state/.afk` present through the daemon's shutdown flush, clears it, and archives the posture record last. +The per-task markers `state/.subsuper-stale-`, `.subsuper-reported-stale-`, `.subsuper-dialog-`, `.subsuper-paused-`, and `.subsuper-pause-until-due-` are the same kind of session-scoped episode state: a still-true condition is re-derived fresh by the next session, and a declared wait spanning two away sessions restarts its recheck window at re-entry. +Always enter through `bin/fm-afk-launch.sh`, which clears prior-session artifacts and markers only for a fresh entry and preserves the current session's buffer on refresh. +Always exit through `bin/fm-afk-launch.sh stop`, which keeps `state/.afk` present through the daemon's shutdown flush, clears it with the session's per-task markers, and archives the posture record last. `docs/herdr-backend.md` "Away-mode supervisor support" and `docs/t3code-backend.md` "Away mode" own the backend mechanisms, and `docs/verification/runtime-backends.md` "Away-mode transport" owns active evidence. ### Reliability properties diff --git a/.agents/skills/bootstrap-diagnostics/SKILL.md b/.agents/skills/bootstrap-diagnostics/SKILL.md index 1bb3669e57a..57b5e822187 100644 --- a/.agents/skills/bootstrap-diagnostics/SKILL.md +++ b/.agents/skills/bootstrap-diagnostics/SKILL.md @@ -1,7 +1,7 @@ --- name: bootstrap-diagnostics description: >- - Load when the session-start digest's bootstrap or network-checks section, or a standalone bin/fm-bootstrap.sh or bin/fm-startup-network.sh run, prints an actionable diagnostic line - MISSING, MISSING_MANUAL, OUTDATED, PRESENTATION_UNAVAILABLE, BACKEND_INVALID, NEEDS_GH_AUTH, TANGLE, STARTUP_MEMORY_BUDGET, CREW_DISPATCH invalid, FLEET_SYNC, NETWORK_CHECKS, HOME_SUMMARY, BACKLOG_RECONCILE, SECONDMATE_SYNC, SECONDMATE_LIVENESS, SECONDMATE_HANDOFF, NUDGE_SECONDMATES, or FMX - or reports that an interrupted backlog cleanup may have left an endpoint or local copy. + Load when the session-start digest's bootstrap or network-checks section, or a standalone bin/fm-bootstrap.sh or bin/fm-startup-network.sh run, prints an actionable diagnostic line - MISSING, MISSING_MANUAL, OUTDATED, PRESENTATION_UNAVAILABLE, BACKEND_INVALID, NEEDS_GH_AUTH, TANGLE, STARTUP_MEMORY_BUDGET, CREW_DISPATCH invalid, FLEET_SYNC, NETWORK_CHECKS, HOME_SUMMARY, BACKLOG_RECONCILE, SECONDMATE_SYNC, SECONDMATE_LIVENESS, SECONDMATE_HANDOFF, NUDGE_SECONDMATES, RECORD_HYGIENE, or FMX - or reports that an interrupted backlog cleanup may have left an endpoint or local copy. A silent bootstrap section, or any other BOOTSTRAP_INFO fact, means no skill load. user-invocable: false metadata: @@ -67,6 +67,9 @@ When any diagnostic needs captain attention, report the plain consequence and re - `BACKLOG_RECONCILE: code-root is not this home's ; ...` - a tasks-axi write addressed the code root instead of this home, so the queue has already forked and either copy may hold rows the other lacks; [`docs/configuration.md`](../../../docs/configuration.md) ("Backlog backend") owns why. Neither copy is a safe winner: union-merge them into this home's file by task id, resolve each conflicting id to its most recent real transition, check this home's archive before treating a missing Done row as lost, and verify the merged id set equals the union of both inputs before installing it. Then move the code-root file aside rather than deleting it, tell the captain which rows were recovered, and run every later backlog command through `bin/fm-tasks-axi.sh`; re-linking the code-root copy is never the fix, because the next cwd-relative tasks-axi write replaces the link again. +- `RECORD_HYGIENE: ` - report the finding to the captain. + For an orphan status log, say whether it was archived to `data//status.log` by a mutating session start or only reported by a detect-only session start or heartbeat scan; `state/.status` still existing means it was not archived. + Leave any further cleanup for an explicit instruction. - `SECONDMATE_SYNC: secondmate : skipped: ` - secondmate convergence left a live home on its existing checkout because the home was dirty, diverged, unsafe, on the wrong branch, missing its placement-specific target commit, unreachable, or otherwise not fast-forwardable, or because inherited local-material propagation failed; bootstrap continued, but inspect the reason because the secondmate's tracked instructions, inherited settings, or shared captain preferences may be stale after a primary update. - `SECONDMATE_LIVENESS: secondmate : skipped: |respawn failed after : ` - the session-start liveness sweep could not guarantee that the registered secondmate is running a real agent process. Investigate the reason because that secondmate is not guaranteed live. diff --git a/.agents/skills/captain-hold-lifecycle/SKILL.md b/.agents/skills/captain-hold-lifecycle/SKILL.md index 4aec330a976..af669d3ac15 100644 --- a/.agents/skills/captain-hold-lifecycle/SKILL.md +++ b/.agents/skills/captain-hold-lifecycle/SKILL.md @@ -28,9 +28,17 @@ A completed investigation and an ended visual review use this same owner and com Run the command in the originating work's authoritative `FM_HOME`; secondmate-owned work registers in that secondmate home's backlog, and a question already held anywhere is never re-registered as a second row. Do not close a captain-held task merely because the originating investigation completed, its report was archived, its visual review ended, or its task was torn down. Holding the work item the question gates is safe for exactly that reason: cleanup keeps such a row open with the finished work's deliverable recorded and returns it to the queue, so it still reads as the captain's own call. -Only `answer` with the captain's words or an evidence-backed `reconcile close` may resolve it. +Only `answer` with the captain's words, an evidence-backed `reconcile close`, or the declared PR settlement policy below may resolve it. -Never close anything the captain owns without recording what he actually said: `bin/fm-captain-hold.sh answer` writes his exact words into the task and closes a question-shaped call, while `--release` frees a captain-gated work item to proceed. +When [PR hold settlement](../../../docs/configuration.md#pr-hold-settlement-confighold-pr-settle) is enabled, declare a `pr-policy` only when its specified forge event answers the whole held question. +Use `close` only for a decision-only task, and `release` when the task still contains unfinished work. +A structured PR link alone never authorizes settlement. +The contribution observer records the event as evidence, not as a captain answer, and settles matching holds without a model turn. +An unmatched unmerged closure remains an open question and arrives as a contribution notification. +Read that fact before presenting the remaining choice, rather than asking whether to merge the closed PR. +The script's help owns policy registration, supported events, and retry mechanics. + +When recording a captain answer, preserve what he actually said: `bin/fm-captain-hold.sh answer` writes his exact words into the task and closes a question-shaped call, while `--release` frees a captain-gated work item to proceed. A merge approval uses that existing release path because approval permits the merge to proceed; cleanup closes the work only after it lands and records what shipped. Closing a held row at merge approval instead records completion before landing, so the backlog claims completion before the work actually ships. When the answer changes what a task must build, follow `AGENTS.md` section 7's mid-task ask rule to preserve the captain's words in the brief and steer the worker. @@ -64,7 +72,7 @@ The absence of a routed work item is not a divergence and the guard never requir 3. Firstmate holds that task - or creates one captain-held task for the review's open questions - with a concise reason carrying the question and options. 4. Attest the full inventory for that review pass under the caller authority in Policy above. 5. Relay the choices to the captain as decisions from Bearings' Captain's Call section under `AGENTS.md` section 9; do not use the word hold in captain chat. -6. Close each call only through `answer` (or a channel that feeds `answers`), close a board-requested moot call through evidence-backed `reconcile close`, record a still-active reconciliation through `reconcile note`, use `--until` when the captain defers it, or confirm a channel already closed it. +6. Resolve each call through `answer`, board-requested reconciliation, or the declared PR settlement policy above, and use `--until` when the captain defers it. 7. Confirm Bearings reflects the outcome: answered or reconciled-moot calls leave Captain's Call, released work resumes, active reconciliations remain held, and deferred calls sit in Charted Next with their date. `bin/fm-captain-hold.sh --help` owns command syntax, close modes, legacy-identity compatibility, completion attestation, retry behavior, and close ordering. diff --git a/.agents/skills/harness-adapters/SKILL.md b/.agents/skills/harness-adapters/SKILL.md index 886bc49664a..633b6fd5e68 100644 --- a/.agents/skills/harness-adapters/SKILL.md +++ b/.agents/skills/harness-adapters/SKILL.md @@ -40,7 +40,7 @@ Muse, Gemini, Devin, Command Code, Polytoken, Cortex, Kiro, and Hermes are verif `../../../bin/fm-harness.sh` prints firstmate's own harness from verified environment markers and process ancestry, and owns how they combine. A marker names its harness, but a structural ancestor of a different harness outranks it, because a marker is ordinary environment state a child or a multiplexer can retain while ancestry is what proves who owns the process tree. Only `FM_PI_HARNESS=pi-signed` at the launch boundary together with `PI_CODING_AGENT=true` selects Pi-signed; shared unmarked launcher ancestry remains Pi. -omp publishes no marker of its own; `FM_OMP_HARNESS=omp` is Firstmate's launch marker and the anchored process name `omp` is its ancestry evidence, as `references/harness/omp.md` records. +omp publishes no marker of its own; `FM_OMP_HARNESS=omp` is Firstmate's launch marker and the anchored identity in `bin/fm-omp-process-lib.sh` (process name `omp`, or `bun` running the omp launcher) is its ancestry evidence, as `references/harness/omp.md` records. `../../../bin/fm-spawn.sh` owns worker marker establishment, while the README launch command owns the signed-primary boundary. `../../../bin/fm-harness.sh crew` resolves `config/crew-harness`, where absent or `default` means firstmate's own harness. [`docs/configuration.md`](../../../docs/configuration.md#choose-the-secondmate-harness) owns secondmate pin resolution, including per-mate overrides; [`bin/fm-harness.sh`](../../../bin/fm-harness.sh) owns the resolver commands. diff --git a/.agents/skills/harness-adapters/references/harness/omp.md b/.agents/skills/harness-adapters/references/harness/omp.md index 297f50bfd35..aee7237a88a 100644 --- a/.agents/skills/harness-adapters/references/harness/omp.md +++ b/.agents/skills/harness-adapters/references/harness/omp.md @@ -17,7 +17,7 @@ Cross-harness provider and credential identity is owned by `references/common/mo | Model flag | `--model /` (fuzzy patterns are accepted by omp but bypass Firstmate's pre-launch check). | | Effort flag | `--thinking `, a superset of the shared vocabulary, so every level including `max` maps straight across. | | Model discovery | `omp models [--json]` lists built-in and auto-discovered providers only; extension-registered providers such as `claude-bridge` never appear, so those models pass through the spawn unvalidated with a stderr notice. `omp usage` shows provider windows; `quota-axi` covers the `claude` provider when the bridge is in use. | -| Marker | None of omp's own (verified: `PI_CODING_AGENT` absent from the binary, no `PI_CODING_AGENT_DIR` or `OMP_PROFILE` in the default profile). `FM_OMP_HARNESS=omp` is Firstmate's launch marker; ancestry matches the exact process name `omp`. | +| Marker | None of omp's own (verified: `PI_CODING_AGENT` absent from the binary, no `PI_CODING_AGENT_DIR` or `OMP_PROFILE` in the default profile). `FM_OMP_HARNESS=omp` is Firstmate's launch marker; ancestry matches the exact process name `omp` or bun running the package's `omp` launcher. | | Composer | Pinned to `composer.shape: borderless` by the overlay, a bare `❯` (U+276F) row the shared classifier already reads; busy text is `Working…` (U+2026), the only spelling the omp busy regex accepts (the three-dot form its headless `-p` mode writes never reaches a supervised pane), with the status row's braille spinner plus elapsed cell as the second signal. | | Autonomy | `--auto-approve` owns approval (omp forces `tools.approvalMode: yolo` for the session under it); the overlay pins `plan.defaultOnStartup: false`, `prewalk.enabled: false`, `retry.usageReservePolicy: auto`. | | Trust | No project-trust gate at all; a fresh profile shows a provider-login wizard instead, suppressed by `OMP_SKIP_SETUP=1`. | @@ -29,9 +29,9 @@ omp cold start is roughly twenty seconds to the first agent turn, paid once per ## Detection -`../../../bin/fm-harness.sh` tests `FM_OMP_HARNESS=omp` before `CLAUDECODE`, like Cursor's markers, and its ancestry walk matches the anchored process name `omp` above the interpreter fallback. +`../../../bin/fm-harness.sh` tests `FM_OMP_HARNESS=omp` before `CLAUDECODE`, like Cursor's markers, and its ancestry walk matches the anchored identity from `../../../bin/fm-omp-process-lib.sh` (the exact process name `omp`, or `bun` whose next argument is a script named exactly `omp`) above the interpreter fallback. The omp template in `../../../bin/fm-spawn.sh` clears every foreign marker at its own launch boundary, and `FM_OMP_HARNESS=omp` counts only under a real `omp` ancestor, so the marker inherited by any other launch is inert: an omp secondmate's workers keep their own identity and an inherited `CLAUDECODE` cannot outrank a worker that omp launched. -`../../../bin/fm-session-lock-lib.sh` matches the same anchored name for session-lock ownership, and `../../../bin/backends/tmux.sh` classifies it `agent` for liveness. +`../../../bin/fm-session-lock-lib.sh` uses the same `fm_omp_process_matches` identity for session-lock ownership, while `../../../bin/backends/tmux.sh` classifies only the exact name `omp` as `agent` for liveness. The optional claude-bridge extension runs a nested executable literally named `claude` as a sibling of tool execution, never an ancestor of it, so omp's own tool calls detect as omp; that subtree is never walked by a Firstmate script. ## Worker posture overlay diff --git a/.agents/skills/operational-home-layout/SKILL.md b/.agents/skills/operational-home-layout/SKILL.md index f005ecf2a3e..20916f1e972 100644 --- a/.agents/skills/operational-home-layout/SKILL.md +++ b/.agents/skills/operational-home-layout/SKILL.md @@ -56,6 +56,8 @@ config/fleet-ledger optional presence flag opting this home in to the default-o config/firstmate-role optional LOCAL, gitignored firstmate-repository publication role; see docs/configuration.md "Firstmate role" for operator opt-in and secondmate inheritance config/wait-no-turns optional presence flag opting this home into default-off waiting-worker behavior (brief waiting section, foreground pipeline drive, pending-reply hold, one fire-and-forget retry ring); LOCAL, gitignored, and not inherited; see docs/configuration.md "Waiting worker spends no turns" config/pr-health-monitor optional presence flag opting this home into the default-off monitor of its owned PRs; LOCAL, gitignored, and not inherited; see docs/configuration.md "PR health monitor" +config/park-on-review config/auto-land optional presence flags: the first opts this home into default-off parking of idle ship lanes whose PR only awaits human review, the second opts this home into merge-time cleanup of unparked ship lanes; LOCAL, gitignored, and not inherited; see docs/configuration.md "Park on review" +config/hold-pr-settle optional presence flag opting this home into default-off settlement of captain holds that declare a PR policy; LOCAL, gitignored, and not inherited; see docs/configuration.md "PR hold settlement" config/contribution-review-bots optional exact GitHub-login allowlist for review-bot signals on owned pull requests; LOCAL, gitignored, and not inherited; see docs/configuration.md "Contribution review bots" config/pipeline-spend optional presence flag opting this home in to default-off per-task no-mistakes spend recording in data/pipeline-spend.jsonl; LOCAL, gitignored, and not inherited; see docs/configuration.md "No-mistakes pipeline spend" config/nm-home-resolver optional one-line absolute path of a site-provided executable that maps a task copy to its no-mistakes home for host-side reads when NM_HOME is unset; LOCAL, gitignored, and not inherited; see docs/configuration.md "No-mistakes home resolver" @@ -67,6 +69,7 @@ config/wedge-defer-parked-gate optional presence flag opting this home into the config/wedge-evidence-guards optional list of absolute operator guard executable paths, one per line, consulted by the watcher before a wedge escalation where a `CRITICAL` answer defers a bounded run of windows; absent means off; LOCAL, gitignored, and not inherited; see docs/configuration.md "Wedge evidence guards" config/wedge-defer-worker-wait optional presence flag opting this home into the default-off deferral of a wedge escalation for a lane whose newest status line is its own still-open blocked: or needs-decision: line; LOCAL, gitignored, and not inherited; see docs/configuration.md "Worker-declared wait deferral" config/wedge-defer-validating optional presence flag opting this home into the default-off deferral of a wedge escalation for a lane whose attributed no-mistakes run is validating (running) or validating (fixing); LOCAL, gitignored, and not inherited; see docs/configuration.md "Validating-run deferral" +config/paused-wait-supersession optional presence flag opting this home into the default-off wake that surfaces a declared `paused:` wait ahead of its recheck cadence once its run is parked at an unreported human-owed gate or terminal; LOCAL, gitignored, and not inherited; see docs/configuration.md "Superseded declared-wait wake" config/spawn-base-primary-checkout optional presence flag opting this home into the default-off spawn base that starts a fresh local-only ship or scout from the primary checkout's default branch when it strictly contains origin's tip; LOCAL, gitignored, and not inherited; see docs/configuration.md "Spawn base from the primary checkout" config/cmux-socket-password optional cmux control-socket password; LOCAL, gitignored; read fresh on every cmux CLI call and passed through without ever overriding an operator's own ambient CMUX_SOCKET_PASSWORD when absent (docs/cmux-backend.md "Setup") config/wedge-alarm optional away-mode wedge-alarm active-alert directives; LOCAL, gitignored; absent means auto (macOS Notification Center when available); see docs/wedge-alarm.md @@ -112,6 +115,7 @@ state/ runtime records and signals; gitignored .herdr-presentation quarantinable attempt and restart-binding journal for Herdr's optional visual projection; never task or endpoint authority; see docs/herdr-backend.md "Presentation spaces" .check.sh authenticated slow poll; the watcher dispatches validated PR data and the byte-identified Relay shim through trusted repository scripts, runs registered custom checks from hash-validated private snapshots, and rejects every other state check without execution .check-trust private content binding created by fm-check-register.sh for an intentional custom check + Review-park records (`.review-park` and `.review-park-land`), locks, reports, and tick handoff files are owned by `bin/fm-park.sh`; see [Park on review](../../../docs/configuration.md#park-on-review) for the behavior contract. .pr-poll private validated data sidecar for the byte-static PR merge poll .pr-poll-registration private transactional provenance record binding the task, canonical metadata identity, sidecar, and static poll publication .pr-poll-retirement private identity-bound crash-recovery receipt for one exact validated merged result; removed after its poll artifacts retire diff --git a/.agents/skills/ship-landing/SKILL.md b/.agents/skills/ship-landing/SKILL.md index b9ebee177d8..00212b2c68d 100644 --- a/.agents/skills/ship-landing/SKILL.md +++ b/.agents/skills/ship-landing/SKILL.md @@ -10,9 +10,11 @@ metadata: For PR-based ship tasks, the ready signal depends on mode: `no-mistakes` reports `done [at=]: PR checks green` after CI is green, while `direct-PR` reports `done [at=]: PR ` after opening the PR, each only for a non-draft PR; a lane that deliberately holds a draft declares a wait instead, and `bin/fm-pr-check.sh` refuses to arm merge monitoring on a draft. Run `bin/fm-pr-check.sh ` with the URL copied from that ready signal or the resolved checks-green `fm-crew-state.sh` line - it records `pr=` and the forge's `pr_head=` when available in the task's meta and arms the watcher's merge poll, which also wakes the supervisor if the ready PR/MR later degrades (failing check, failed head pipeline, merge-train drop) so the worker can be steered to fix it. -With `config/pr-register-scan` present, the watcher's check sweep also records a PR that a `done:` line names when no `pr=` exists, and reports a refused registration as a `check: pr-register:` notification; handle that refusal as you would a refusal from a manual run. +With `config/pr-register-scan` present, the watcher can record ready task PRs during its check sweep, including replacing a named PR; see [`docs/configuration.md`](../../../docs/configuration.md#pr-registration-scan-configpr-register-scan) for the full rule. Handle a `check: pr-register:` notification as you would the same result from a manual run. `bin/fm-dod-lib.sh` owns the named-head gate on that ready signal: a ship `done:` whose named head exists only in the worker's disposable copy is not ready (`bin/fm-crew-state.sh` reports blocked, `bin/fm-pr-check.sh` refuses to register, and a secondmate does not publish that done upstream). -That blocked reading is the gate working, not a stuck worker, so steer the worker on the commit the refusal names rather than waiting. +Of those readers only `bin/fm-pr-check.sh` fetches the copy's remotes before it decides, and a no-mistakes pipeline pushes from its own clone, so a no-mistakes ready signal can read blocked until that registration has run: register it before steering. +A refusal that says it could not refresh a remote decided on the last-known refs, so retry it once that remote is reachable. +Otherwise that blocked reading is the gate working, not a stuck worker, so steer the worker on the commit the refusal names rather than waiting. A `forge=external` task reports `done [at=]: review uploaded for review` instead: run `bin/fm-review-check.sh ` rather than `bin/fm-pr-check.sh`, which arms no merge watch, then always escalate the review URL to the captain whatever the yolo setting, because firstmate can neither observe nor perform its approval or merge. A direct-PR worker pushes that commit to its PR branch, and a local-only worker commits it on its ship branch. A no-mistakes worker re-validates it with /no-mistakes so the pipeline stays the one publisher; it never pushes from its copy. @@ -22,7 +24,10 @@ A captain instruction to merge is explicit authority; `yolo` is the only standin For any custom `state/.check.sh` you write yourself, keep it an ordinary single-link mode-`0700` file, print one line only when firstmate should wake, print nothing otherwise, finish before `FM_CHECK_TIMEOUT`, then bind its current bytes with `bin/fm-check-register.sh ` before the watcher may execute it. Retire a custom check only through `bin/fm-check-unregister.sh ` (or `bin/fm-teardown.sh` for a spawned task); never hand-compose an `rm` with `$STATE`/`$ID`. -Tear down a ship task only after landing is confirmed. +The watcher owns teardown of a merged lane whose merge poll is armed, including a PR merged through `bin/fm-pr-merge.sh`, when the lane is parked or its home has `config/auto-land` ([`docs/configuration.md`](../../../docs/configuration.md#park-on-review)). Otherwise, firstmate tears it down after landing is confirmed. +A lane parked on review (`bin/fm-park.sh`) has no agent and relaunches itself on review activity; `bin/fm-park.sh park ` parks one clean lane on request and `bin/fm-park.sh resume --trigger ` relaunches one by hand. + +Tear down any other ship task only after landing is confirmed. A teardown refusal for uncommitted or unlanded work is a stop-and-investigate result, never an obstacle to bypass. Never force teardown without explicit discard authority. After successful teardown, record completion, retain only the configured recent Done history, and re-evaluate queued work whose blockers and time gates have cleared. diff --git a/.agents/skills/stuck-crewmate-recovery/SKILL.md b/.agents/skills/stuck-crewmate-recovery/SKILL.md index 5d3c0172d82..b366ab8cbff 100644 --- a/.agents/skills/stuck-crewmate-recovery/SKILL.md +++ b/.agents/skills/stuck-crewmate-recovery/SKILL.md @@ -12,6 +12,8 @@ metadata: Use this playbook when the session-start digest reports an ordinary direct report's endpoint dead or its metadata has no window, or when a direct report is stale, looping, repeatedly confused, asking a question its brief already answers, unresponsive, or when a steer failed to land. A stale or dead-endpoint report for a worker whose pull request has already landed is not a recovery case: the work is finished, so close the task through ordinary teardown (`AGENTS.md` section 7 for firstmate, the landed-work rule in `bin/fm-branch-prompt.sh` for the supervision branch) instead of this playbook, never with `--force`. +A lane parked on review (`bin/fm-crew-state.sh` reads `source: review-park`) stopped its agent on purpose and is not a recovery case; relaunch it only through `bin/fm-park.sh resume`. + Follow the crew-hosted Lavish board contract in [`docs/configuration.md`](../../../docs/configuration.md#crew-hosted-lavish-review-boards) when recovering a worker that hosts a board. Interrupt, stop, and relaunch a worker through `bin/fm-control.sh interrupt|exit|relaunch`, which resolves the recorded runtime itself, verifies each action, and never tears down or discards anything ([`docs/agent-control.md`](../../../docs/agent-control.md)). diff --git a/FORK.md b/FORK.md index 629ba28a7c7..f6bd536b94a 100644 --- a/FORK.md +++ b/FORK.md @@ -2,7 +2,7 @@ This is a fork of [kunchenguid/firstmate](https://github.com/kunchenguid/firstmate), maintained by Andrew Smith. It does everything upstream does and adds the capabilities below. -It is current with upstream through commit `913bb7a7` (2026-11-09). +It is current with upstream through commit `427056c4` (2026-11-12). ## What you gain diff --git a/bin/fm-afk-contract.sh b/bin/fm-afk-contract.sh index b4a103017a3..d367bec00cf 100755 --- a/bin/fm-afk-contract.sh +++ b/bin/fm-afk-contract.sh @@ -247,12 +247,13 @@ fm_afk_contract_lock_helpers() { # fm_afk_contract_lock_hold [state-dir]: take the cross-subsystem lock described # in the header. The acquire is bounded so a wedged holder is refused instead of # blocking a merge or a captain return forever, and returns 1 WITHOUT the lock so -# every caller refuses rather than proceeding unlocked. -fm_afk_contract_lock_hold() { # [state-dir] +# every caller refuses rather than proceeding unlocked. A caller that defers on +# contention instead of waiting for a merge passes its own shorter bound. +fm_afk_contract_lock_hold() { # [state-dir] [timeout-seconds] local lock rc=0 STATE timeout STATE=${1:-$FM_AFK_CONTRACT_STATE} lock=$(fm_afk_contract_lock_path "$STATE") - timeout=${FM_TEST_AFK_CONTRACT_LOCK_TIMEOUT:-$_FM_AFK_CONTRACT_LOCK_TIMEOUT} + timeout=${2:-${FM_TEST_AFK_CONTRACT_LOCK_TIMEOUT:-$_FM_AFK_CONTRACT_LOCK_TIMEOUT}} fm_afk_contract_lock_helpers || { fm_afk_contract_log "could not load the lock primitive for $lock" return 1 diff --git a/bin/fm-afk-launch.sh b/bin/fm-afk-launch.sh index 9b4f831b994..4299e125552 100755 --- a/bin/fm-afk-launch.sh +++ b/bin/fm-afk-launch.sh @@ -633,11 +633,15 @@ fm_afk_launch_restore_backup() { # "$FM_AFK_LAUNCH_STATE/.subsuper-escalations.since" \ "$FM_AFK_LAUNCH_STATE/.subsuper-inject-wedged" \ "$FM_AFK_LAUNCH_STATE/.subsuper-unknown-acked" \ - "$FM_AFK_LAUNCH_STATE"/.subsuper-reported-stale-* || result=1 + "$FM_AFK_LAUNCH_STATE"/.subsuper-stale-* \ + "$FM_AFK_LAUNCH_STATE"/.subsuper-reported-stale-* \ + "$FM_AFK_LAUNCH_STATE"/.subsuper-dialog-* \ + "$FM_AFK_LAUNCH_STATE"/.subsuper-paused-* \ + "$FM_AFK_LAUNCH_STATE"/.subsuper-pause-until-due-* || result=1 if [ "$had_afk" -eq 1 ]; then cp "$backup/.afk" "$FM_AFK_LAUNCH_STATE/.afk" || result=1 fi - for artifact in .subsuper-escalations .subsuper-escalations.since .subsuper-inject-wedged .subsuper-unknown-acked "$backup"/.subsuper-reported-stale-*; do + for artifact in .subsuper-escalations .subsuper-escalations.since .subsuper-inject-wedged .subsuper-unknown-acked "$backup"/.subsuper-stale-* "$backup"/.subsuper-reported-stale-* "$backup"/.subsuper-dialog-* "$backup"/.subsuper-paused-* "$backup"/.subsuper-pause-until-due-*; do artifact=${artifact##*/} if [ -e "$backup/$artifact" ]; then cp -p "$backup/$artifact" "$FM_AFK_LAUNCH_STATE/$artifact" || result=1 @@ -799,7 +803,7 @@ fm_afk_launch_start() { had_afk=1 cp "$FM_AFK_LAUNCH_STATE/.afk" "$backup/.afk" || { rm -rf "$backup"; return 1; } fi - for artifact in .subsuper-escalations .subsuper-escalations.since .subsuper-inject-wedged .subsuper-unknown-acked "$FM_AFK_LAUNCH_STATE"/.subsuper-reported-stale-*; do + for artifact in .subsuper-escalations .subsuper-escalations.since .subsuper-inject-wedged .subsuper-unknown-acked "$FM_AFK_LAUNCH_STATE"/.subsuper-stale-* "$FM_AFK_LAUNCH_STATE"/.subsuper-reported-stale-* "$FM_AFK_LAUNCH_STATE"/.subsuper-dialog-* "$FM_AFK_LAUNCH_STATE"/.subsuper-paused-* "$FM_AFK_LAUNCH_STATE"/.subsuper-pause-until-due-*; do artifact=${artifact##*/} if [ -e "$FM_AFK_LAUNCH_STATE/$artifact" ]; then cp -p "$FM_AFK_LAUNCH_STATE/$artifact" "$backup/$artifact" || { rm -rf "$backup"; return 1; } @@ -858,7 +862,7 @@ fm_afk_launch_start_native() { had_afk=1 cp "$FM_AFK_LAUNCH_STATE/.afk" "$backup/.afk" || { rm -rf "$backup"; return 1; } fi - for artifact in .subsuper-escalations .subsuper-escalations.since .subsuper-inject-wedged .subsuper-unknown-acked "$FM_AFK_LAUNCH_STATE"/.subsuper-reported-stale-*; do + for artifact in .subsuper-escalations .subsuper-escalations.since .subsuper-inject-wedged .subsuper-unknown-acked "$FM_AFK_LAUNCH_STATE"/.subsuper-stale-* "$FM_AFK_LAUNCH_STATE"/.subsuper-reported-stale-* "$FM_AFK_LAUNCH_STATE"/.subsuper-dialog-* "$FM_AFK_LAUNCH_STATE"/.subsuper-paused-* "$FM_AFK_LAUNCH_STATE"/.subsuper-pause-until-due-*; do artifact=${artifact##*/} if [ -e "$FM_AFK_LAUNCH_STATE/$artifact" ]; then cp -p "$FM_AFK_LAUNCH_STATE/$artifact" "$backup/$artifact" || { rm -rf "$backup"; return 1; } @@ -937,6 +941,18 @@ fm_afk_launch_stop() { fm_afk_launch_log "failed to clear away-mode flag" result=1 fi + # The session's per-task supervision markers are episode state that ends with + # it: left behind, the NEXT session's first housekeeping reads the stale/pause + # clocks as its own and reports a possible wedge whose duration never + # happened. A wait that still stands is re-declared and re-windowed fresh. + if ! rm -f "$FM_AFK_LAUNCH_STATE"/.subsuper-stale-* \ + "$FM_AFK_LAUNCH_STATE"/.subsuper-reported-stale-* \ + "$FM_AFK_LAUNCH_STATE"/.subsuper-dialog-* \ + "$FM_AFK_LAUNCH_STATE"/.subsuper-paused-* \ + "$FM_AFK_LAUNCH_STATE"/.subsuper-pause-until-due-*; then + fm_afk_launch_log "failed to clear the session's per-task supervision markers" + result=1 + fi if [ "$result" -eq 0 ] && fm_afk_contract_present "$FM_AFK_LAUNCH_STATE"; then if archived=$("$FM_AFK_CONTRACT_CMD" archive); then fm_afk_launch_log "away-posture record archived at $archived" diff --git a/bin/fm-afk-start.sh b/bin/fm-afk-start.sh index 6b9d76c4ffc..f8a7a911d17 100755 --- a/bin/fm-afk-start.sh +++ b/bin/fm-afk-start.sh @@ -7,7 +7,7 @@ # FM_AFK_STATE_PREPARED=1, checks state/.supervise-daemon.lock, and: # - prints "afk: daemon already running pid=" then exits 0 when that # lock is held by a live daemon (a REFRESH: no stale-artifact clear); -# - otherwise clears any prior away session's stale escalation artifacts +# - otherwise clears prior-session delivery caches and per-task markers # (fm_afk_clear_stale_artifacts) for a direct, non-prepared start, then # execs bin/fm-supervise-daemon.sh in the foreground. A prepared start was # already cleared transactionally by bin/fm-afk-launch.sh. @@ -46,39 +46,22 @@ fm_afk_start_usage() { sed -n '2,14p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' } -# fm_afk_clear_stale_artifacts: on a FRESH away-session entry (the daemon is not -# already running), drop the previous away session's leftover escalation-delivery -# artifacts so they cannot surface as stale escalations under the new session. -# These are session-scoped by timing: a fresh entry owns a new supervision -# session and the new daemon has not produced anything yet, so anything present -# here belongs to a PRIOR session. This never drops a genuinely-pending -# escalation - the delivery buffer is a transient cache, and any condition still -# true (a crew still blocked, a check still firing) is re-derived and re-escalated -# fresh by the daemon's heartbeat catch-all scan and the durable -# state/.wake-queue replay (see docs/herdr-backend.md "Away-mode stale-artifact -# lifecycle" and bin/fm-supervise-daemon.sh's escalate_add/inject_wedge_alarm). -# NOT called on a refresh (daemon already alive), so the current session's own -# buffered escalations are preserved. +# Call only before a fresh daemon can produce state, never on a live refresh: +# timing is what proves these caches and episode clocks belong to a prior session. +# Durable wake-queue rows and classification positions must remain intact for +# replay and catch-all reconciliation by bin/fm-supervise-daemon.sh. +# .agents/skills/afk/SKILL.md "Stale-artifact lifecycle" owns the session contract. fm_afk_clear_stale_artifacts() { # - local state=$1 line win key - # Unknown-state warnings suppress repeats even after delivery. Re-arm only - # warnings still in the buffer being discarded, preserving delivered ones. - if [ -s "$state/.subsuper-escalations" ] && compgen -G "$state/.subsuper-reported-stale-*" >/dev/null; then - _fm_wake_require_classify || return 1 - while IFS= read -r line || [ -n "$line" ]; do - case "$line" in - stale\ persisted\ *s\ \(possible\ wedge\):\ *) - win=${line#* (possible wedge): } - key=$(window_to_task "$win" "$state" | tr ':/.' '___') - rm -f "$state/.subsuper-reported-stale-$key" || return 1 - ;; - esac - done < "$state/.subsuper-escalations" - fi + local state=$1 rm -f "$state/.subsuper-escalations" \ "$state/.subsuper-escalations.since" \ "$state/.subsuper-inject-wedged" \ - "$state/.subsuper-unknown-acked" 2>/dev/null + "$state/.subsuper-unknown-acked" \ + "$state"/.subsuper-stale-* \ + "$state"/.subsuper-reported-stale-* \ + "$state"/.subsuper-dialog-* \ + "$state"/.subsuper-paused-* \ + "$state"/.subsuper-pause-until-due-* 2>/dev/null } daemon_lock_owner() { @@ -182,8 +165,7 @@ fm_afk_start_main() { fm_lock_remove_path "$FM_AFK_LOCK" 2>/dev/null || true fi - # Fresh start: clear the previous away session's stale delivery artifacts - # before the new daemon can surface them (fix for the leaked-artifact defect). + # Clear prior-session state before the new daemon can observe or produce it. if [ "${FM_AFK_STATE_PREPARED:-0}" != 1 ]; then fm_afk_clear_stale_artifacts "$FM_AFK_STATE" fi diff --git a/bin/fm-bootstrap.sh b/bin/fm-bootstrap.sh index 26c383bfb8f..e1cb54fcac5 100755 --- a/bin/fm-bootstrap.sh +++ b/bin/fm-bootstrap.sh @@ -16,6 +16,8 @@ # "HOME_SUMMARY: >; failed attempt(s) ... last: ", # "BACKLOG_RECONCILE: : ", +# "RECORD_HYGIENE: " +# (only with config/record-hygiene present), # "BACKLOG_RECONCILE: code-root is not this home's ; ...", # "TANGLE: ", # "SECONDMATE_SYNC: secondmate : skipped: ", @@ -23,7 +25,9 @@ # "BOOTSTRAP_INFO: nudged fm- with ''", # "SECONDMATE_LIVENESS: secondmate : skipped: |respawn failed after : ", # "SECONDMATE_HANDOFF: secondmate : pending delivery: item(s)", -# "FMX: X mode on ..." or "FMX: X mode off ...". +# "FMX: X mode on ..." or "FMX: X mode off ...", +# "TREEHOUSE_POOL_REPORT: " (only when +# the config/treehouse-pool-report presence flag exists). # When a RUNNING secondmate home is fast-forwarded, its target is # firstmate's own current default-branch commit. A local worktree uses # a purely local fast-forward with no origin fetch; a remote route hands @@ -181,6 +185,8 @@ PROJECTS="${FM_PROJECTS_OVERRIDE:-$FM_HOME/projects}" CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" +# shellcheck source=bin/fm-classify-lib.sh disable=SC1091 +. "$SCRIPT_DIR/fm-classify-lib.sh" # shellcheck source=bin/fm-tasks-axi-lib.sh disable=SC1091 . "$SCRIPT_DIR/fm-tasks-axi-lib.sh" # shellcheck source=bin/fm-backlog-transition-lib.sh disable=SC1091 @@ -1557,7 +1563,96 @@ detect_local_tools() { fi } +treehouse_pool_report() { + local root=${TREEHOUSE_ROOT:-$HOME/.treehouse} state pool records slot repo slot_root identity created_at slot_leased lease_epoch age_days owner_live pool_has_owner pool_count=0 threshold=${FM_POOL_ORPHAN_DAYS:-30} duplicate duplicate_reported seen_pair i now + local -a seen_repos=() seen_pools=() + [ -d "$root" ] && command -v jq >/dev/null 2>&1 || return 0 + case "$threshold" in ''|*[!0-9]*) threshold=30 ;; esac + now=$(date +%s) + records=$(mktemp "${TMPDIR:-/tmp}/fm-pool-records.XXXXXX") || return 0 + for state in "$root"/*/treehouse-state.json; do + [ -f "$state" ] || continue + pool_count=$((pool_count + 1)) + pool=${state%/treehouse-state.json} + pool_has_owner=0 + duplicate_reported=0 + if ! jq -j '.worktrees[]? | (.path // ""), "\u0000", (.created_at // ""), "\u0000", (((.leased == true) or ((.lease_id // "") != "") or ((.lease_holder // "") != "")) | tostring), "\u0000", (.owner_pid // ""), "\u0000"' "$state" > "$records" 2>/dev/null; then + continue + fi + while IFS= read -r -d '' slot && IFS= read -r -d '' created_at \ + && IFS= read -r -d '' slot_leased && IFS= read -r -d '' slot_owner; do + owner_live=0 + case "$slot_owner" in ''|*[!0-9]*) ;; *) kill -0 "$slot_owner" 2>/dev/null && owner_live=1 ;; esac + if [ -z "$slot" ]; then + if [ "$owner_live" -eq 1 ] || [ "$slot_leased" = true ]; then pool_has_owner=1; fi + continue + fi + repo=$(git -C "$slot" rev-parse --show-toplevel 2>/dev/null) || repo= + slot_root=$(cd -P -- "$slot" 2>/dev/null && pwd -P) || slot_root= + repo=$(cd -P -- "$repo" 2>/dev/null && pwd -P) || repo= + [ "$repo" = "$slot_root" ] || repo= + if [ -e "${slot%/*}/.fm-slot-owner" ] || [ -e "$slot/.fm-slot-owner" ] || [ "$owner_live" -eq 1 ] || [ "$slot_leased" = true ]; then + pool_has_owner=1 + fi + if [ -z "$repo" ]; then + if [ ! -e "${slot%/*}/.fm-slot-owner" ] && [ ! -e "$slot/.fm-slot-owner" ] && [ "$owner_live" -eq 0 ] && [ "$slot_leased" != true ]; then + echo "TREEHOUSE_POOL_REPORT: unregistered pool $pool (project path unavailable: $slot)" + fi + continue + fi + if [ -n "$repo" ]; then + identity=$(git -C "$slot" remote get-url origin 2>/dev/null) || identity=$repo + [ -n "$identity" ] || identity=$repo + duplicate=0 + seen_pair=0 + for ((i = 0; i < ${#seen_repos[@]}; i++)); do + if [ "${seen_repos[$i]}" = "$identity" ]; then + if [ "${seen_pools[$i]}" = "$pool" ]; then + seen_pair=1 + else + duplicate=1 + fi + fi + done + if [ "$duplicate" -eq 1 ]; then + if [ "$duplicate_reported" -eq 0 ]; then + echo "TREEHOUSE_POOL_REPORT: duplicate-clone pool $pool (project $repo)" + duplicate_reported=1 + fi + fi + if [ "$seen_pair" -eq 0 ]; then + seen_repos+=("$identity") + seen_pools+=("$pool") + fi + fi + if [ ! -e "${slot%/*}/.fm-slot-owner" ] && [ ! -e "$slot/.fm-slot-owner" ] \ + && [ "$owner_live" -eq 0 ] && [ "$slot_leased" != true ]; then + lease_epoch= + if [ -n "$created_at" ] && [ "$created_at" != null ]; then + lease_date=$(printf '%s\n' "$created_at" | sed -E 's/\.[0-9]+([+-][0-9]{2}:[0-9]{2}|Z)$/\1/; s/Z$/+0000/; s/([+-][0-9]{2}):([0-9]{2})$/\1\2/') + lease_epoch=$(date -j -u -f '%Y-%m-%dT%H:%M:%S%z' "$lease_date" '+%s' 2>/dev/null || date -u -d "$lease_date" '+%s' 2>/dev/null || true) + fi + case "$lease_epoch" in ''|*[!0-9]*) continue ;; esac + age_days=$(( (now - lease_epoch) / 86400 )) + [ "$age_days" -lt "$threshold" ] || echo "TREEHOUSE_POOL_REPORT: unowned slot older than ${threshold}d: $slot (age ${age_days}d)" + fi + done < "$records" + case "${pool##*/}" in + project-*|scratch-project-*) + [ "$pool_has_owner" -eq 1 ] || echo "TREEHOUSE_POOL_REPORT: test-leftover pool $pool" + ;; + esac + done + [ "$pool_count" -eq 0 ] || echo "TREEHOUSE_POOL_REPORT: scanned $pool_count pools under $root" + rm -f "$records" +} + +# shellcheck source=bin/fm-record-hygiene-lib.sh +. "$SCRIPT_DIR/fm-record-hygiene-lib.sh" + detect_local_config() { + # Opt-in only: without the flag bootstrap reads no Treehouse state. + [ ! -e "$CONFIG/treehouse-pool-report" ] || treehouse_pool_report # Worktree-tangle check: the firstmate primary checkout (FM_ROOT) must sit on its # default branch, not a feature branch (see fm-tangle-lib.sh). Scoped to the # primary only; detached-HEAD worktrees and secondmate homes never trip it. @@ -1679,6 +1774,7 @@ if network_phase; then fm_timing_record phase gh-auth "$__fm_timing_stamp" fi local_phase && detect_local_config +local_phase && [ -e "$CONFIG/record-hygiene" ] && record_hygiene_report if [ "${FM_BOOTSTRAP_DETECT_ONLY:-0}" != 1 ]; then # secondmate_sync consumes SECONDMATE_RESPAWNED_IDS from the liveness sweep, so diff --git a/bin/fm-captain-hold.sh b/bin/fm-captain-hold.sh index ce58834f703..1c1caadab10 100755 --- a/bin/fm-captain-hold.sh +++ b/bin/fm-captain-hold.sh @@ -23,6 +23,8 @@ # fm-captain-hold.sh hold --reason \ # [--title ] [--repo <repo>] [--origin <origin-id>] [--until YYYY-MM-DD] # fm-captain-hold.sh answer <task-id> --decision-file <path> [--release] +# fm-captain-hold.sh pr-policy <task-id> <PR-url> <merged|closed|terminal> <close|release> +# fm-captain-hold.sh settle-pr <task-id> <PR-url> # fm-captain-hold.sh answers [<legacy-origin> | --any-origin] --source <provenance> (keyed answers on stdin) # fm-captain-hold.sh reconcile-requests --source-id <source-id> --source <provenance> (task ids on stdin) # fm-captain-hold.sh bind <source-id> [<legacy-origin> | --any-origin] @@ -195,7 +197,8 @@ # backlog close and, on 0, returns the row to Queued with its deliverable # recorded instead (bin/fm-backlog-transition-lib.sh owns that transition), so # holding the very work item a question gates is safe; only `answer` with the -# captain's words or evidence-backed `reconcile close` closes the call. +# captain's words, evidence-backed `reconcile close`, or the opt-in PR policy +# below settles the call. # bin/fm-watch.sh asks it when an ordinary # crew task reaches a due stale alarm - its open backlog hold need not appear in # the task's last status line - and on a 0 bounds repeated alarms from new pane @@ -204,9 +207,31 @@ # `diverged` is the read-only guard over the seam between the two records of # one captain call. See "record divergence" beside command_diverged below. # +# With config/hold-pr-settle present, pr-policy declares which forge events +# answer this particular hold. It adds the structured PR link and atomically +# records data/<task>/hold-pr-policy.json: schema fm-hold-pr.v1, task, url, on, +# action, fingerprint (SHA-256 of the shown reason, title without its structured +# links, hold-set timestamp and resolution count), occurrence. +# `close` asserts the task contains only the decision; `release` preserves +# unfinished work. `terminal` accepts either merge or unmerged closure. +# No policy is inferred from links or prose. Re-holding invalidates the policy, +# and editing the reason or title makes it stale; linking another PR does not. +# A policy that can no longer settle is removed: silently once the task is done +# or no longer a captain hold, and with a non-zero exit naming the cause when +# the question changed or the PR ended in the other terminal state. +# A `release` policy waits while the home is away, like `answer --release`, and +# a later settle-pr applies it. Only github.com PRs supported +# by the contribution observer qualify. settle-pr reads that observer's durable, +# validated terminal evidence, never caller-supplied state or a captain answer. +# It records forge-closed/forge-released reconciliation evidence before changing +# the task, and retries interrupted settlement without duplicating the record. +# Successful settlement removes the policy after parent-channel publication. +# Removing the presence flag disables policy registration and settlement. +# # Resolution records: the block written into the body names this script, the -# decision digest, and a `Resolution mode:` of answered, released, repaired, or -# reconciled. Records written by the retired fm-decision-hold.sh (routed, +# decision digest, and a `Resolution mode:` of answered, released, repaired, +# reconciled, forge-closed, or forge-released. +# Records written by the retired fm-decision-hold.sh (routed, # declined, answered, repaired) are recognized everywhere a record is read, so # nothing already closed needs rewriting. # @@ -234,6 +259,7 @@ FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" +CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" # shellcheck source=bin/fm-classify-lib.sh # shellcheck disable=SC1091 @@ -535,11 +561,11 @@ closed_answer_replay_mode_compatible() { # <mode> <task-body> } # The record's label is what keeps an evidence-backed reconciliation from -# reading as the captain's own words. `reconciled` closes a call that went moot -# and carries verified evidence; every other mode carries what the captain said. +# reading as the captain's own words. Reconciliation modes carry evidence; +# answer modes carry what the captain said. resolution_block() { # <mode> local label='Captain decision:' - [ "$1" != reconciled ] || label='Reconciliation evidence:' + case "$1" in reconciled|forge-closed|forge-released) label='Reconciliation evidence:' ;; esac printf 'Resolution recorded by fm-captain-hold.\nDecision digest: %s\nResolution mode: %s\n\n%s\n%s\n' \ "$DECISION_DIGEST" "$1" "$label" "$DECISION_TEXT" } @@ -1162,6 +1188,8 @@ command_hold() { || fail "could not create task $id" fi fi + # A repeated hold may ask a different question even if it keeps its timestamp. + rm -f -- "$DATA/$id/hold-pr-policy.json" # Publish the timestamp before the captain-hold annotation. A concurrent # snapshot may see the harmless stamp by itself, but can never see a newly # held task without the timestamp that defines this hold lifecycle's age. @@ -1795,6 +1823,127 @@ command_reconcile_requests() { [ "$skipped" -eq 0 ] } +pr_hold_fingerprint() { # <show-output> + local links=, word title='' + local -a words=() + # The markdown backend shows structured links inside the title, and linking a + # follow-up PR or a retained report does not change the question. + IFS=, read -r -a words <<< "$(show_field_value "$1" links)" + for word in ${words[@]+"${words[@]}"}; do links="$links${word#*:},"; done + read -r -a words <<< "$(show_field_value "$1" title)" + for word in ${words[@]+"${words[@]}"}; do + case "$links" in *",$word,"*) ;; *) title="$title $word" ;; esac + done + sha256_text "$(show_field "$1" hold_reason) +$title +$(body_hold_set_timestamp "$(show_field_value "$1" body)") +$(resolution_record_count "$(show_field "$1" body)")" +} + +command_pr_policy() { + local id=${1:-} url=${2:-} event=${3:-} action=${4:-} show dir tmp occurrence fingerprint + [ "$#" -eq 4 ] || { usage >&2; exit 2; } + [ -f "$CONFIG/hold-pr-settle" ] || fail 'config/hold-pr-settle is required' + validate_slug task-id "$id" + jq -ne --arg url "$url" '$url | test("^https://github[.]com/[A-Za-z0-9-]+/[A-Za-z0-9._-]+/pull/[1-9][0-9]*$")' >/dev/null \ + || fail 'pr-policy requires a canonical github.com PR URL' + case "$event" in merged|closed|terminal) ;; *) fail 'event must be merged, closed, or terminal' ;; esac + case "$action" in close|release) ;; *) fail 'action must be close or release' ;; esac + acquire_task_control_lock "$id" + require_tasks_axi + command_open "$id" || fail "task $id is not an open captain hold" + tasks_axi update "$id" --pr "$url" >/dev/null || fail "could not link the PR on $id" + task_show_or_fail "$id" "task $id disappeared while declaring its PR policy" + fingerprint=$(pr_hold_fingerprint "$show") + occurrence=$(( $(resolution_record_count "$(show_field "$show" body)") + 1 )) + dir="$DATA/$id" + [ ! -L "$dir" ] || fail 'PR policy directory is a symlink' + mkdir -p "$dir" + [ ! -L "$dir/hold-pr-policy.json" ] || fail 'PR policy is a symlink' + [ ! -e "$dir/hold-pr-policy.json" ] || [ -f "$dir/hold-pr-policy.json" ] || fail 'PR policy is not a regular file' + tmp=$(umask 077; mktemp "$dir/.hold-pr-policy.XXXXXX") + jq -n --arg task "$id" --arg url "$url" --arg on "$event" --arg action "$action" \ + --arg fingerprint "$fingerprint" --argjson occurrence "$occurrence" \ + '{schema:"fm-hold-pr.v1",task:$task,url:$url,on:$on,action:$action,fingerprint:$fingerprint,occurrence:$occurrence}' > "$tmp" + mv -f -- "$tmp" "$dir/hold-pr-policy.json" + printf 'pr-policy: %s %s %s %s\n' "$id" "$url" "$event" "$action" +} + +command_settle_pr() { + local id=${1:-} url=${2:-} policy record show body event action mode occurrence observed fingerprint evidence + [ "$#" -eq 2 ] || { usage >&2; exit 2; } + [ -f "$CONFIG/hold-pr-settle" ] || return 0 + validate_slug task-id "$id" + policy="$DATA/$id/hold-pr-policy.json" + [ -f "$policy" ] || return 0 + # An answer can hold this lock for minutes while it waits on the away record; + # the poll that calls this is cut after 30 seconds, so a busy task defers. + CAPTAIN_CONTROL_LOCK="$STATE/.control-$id.lock" + fm_lock_acquire_wait_bounded "$CAPTAIN_CONTROL_LOCK" 1 2>/dev/null || return 0 + CAPTAIN_CONTROL_LOCK_HELD=1 + [ -f "$policy" ] || return 0 + [ ! -L "$DATA/$id" ] && [ ! -L "$policy" ] || fail 'unsafe PR policy path' + jq -e --arg task "$id" '.schema == "fm-hold-pr.v1" and .task == $task + and (.on | IN("merged","closed","terminal")) and (.action | IN("close","release")) + and (.fingerprint | test("^[a-f0-9]{64}$")) + and (.occurrence | type == "number" and . > 0 and floor == .)' "$policy" >/dev/null \ + || fail "invalid PR settlement policy for $id" + [ "$(jq -r .url "$policy")" = "$url" ] || return 0 + record="$DATA/$id/contributions.json" + [ -f "$record" ] && [ ! -L "$record" ] || return 0 + jq -L "$SCRIPT_DIR" -e --arg task "$id" 'include "fm-contributions"; valid_record and .task == $task' "$record" >/dev/null \ + || fail "invalid forge evidence for $id" + evidence=$(jq -c --arg url "$url" '[.records[] | select(.url == $url and .kind == "pr" and .error == null + and .retired == null and .checked_at != null and (.observation.state | IN("merged","closed")))] | first // empty' "$record") + [ -n "$evidence" ] || return 0 + event=$(printf '%s' "$evidence" | jq -r .observation.state) + case "$(jq -r .on "$policy")" in + "$event"|terminal) ;; + *) + rm -f -- "$policy" + fail "dropped the PR policy for $id: the PR is $event and can no longer reach the declared event; the hold stays open" + ;; + esac + observed=$(printf '%s' "$evidence" | jq -r .checked_at) + action=$(jq -r .action "$policy") + if [ "$action" = release ]; then + release_attended && fm_afk_contract_lock_hold "$STATE" 1 2>/dev/null && release_attended || return 0 + fi + occurrence=$(jq -r .occurrence "$policy") + fingerprint=$(jq -r .fingerprint "$policy") + if [ "$action" = close ]; then mode=forge-closed; else mode=forge-released; fi + DECISION_TEXT="Forge PR $event: $url +Observed at: $observed +Settlement policy: $(jq -r .on "$policy") -> $action" + DECISION_DIGEST=$(sha256_text "$DECISION_TEXT") + require_tasks_axi + task_show_or_fail "$id" "task $id is absent while settling its PR hold" + body=$(show_field "$show" body) + if [ "$(recorded_decision_digest "$body" || true)" = "$DECISION_DIGEST" ] \ + && [ "$(recorded_resolution_mode "$body" || true)" = "$mode" ] \ + && [ "$(resolution_record_count "$body")" = "$occurrence" ]; then + : # Resume an interrupted settlement after its evidence was recorded. + else + if [ "$(show_field "$show" state)" = "done" ] || [ "$(show_field_value "$show" hold_kind)" != captain ]; then + rm -f -- "$policy" + return 0 + fi + if [ "$(pr_hold_fingerprint "$show")" != "$fingerprint" ]; then + rm -f -- "$policy" + fail "dropped the PR policy for $id: its question changed after the policy was declared; the hold stays open" + fi + write_resolution_record "$id" "$mode" "$body" + fi + if [ "$action" = close ]; then close_answered "$id" 0 + else tasks_axi unhold "$id" >/dev/null; fi + remove_interrupted_answer_stamp "$id" + publish_parent_hold "$id" "$occurrence" resolved "PR $event: $url ($action)" + [ "$PARENT_HOLD_PUBLISHED" = 1 ] || fail "could not publish PR settlement for $id" + reconcile_request_retire "$id" + rm -f -- "$policy" + printf 'pr-settled: %s %s %s %s\n' "$id" "$url" "$event" "$action" +} + command_reconcile() { local action=${1:-} [ "$#" -ge 1 ] || { usage >&2; exit 2; } @@ -2279,6 +2428,8 @@ case "${1:-}" in open) shift; command_open "$@" ;; diverged) shift; command_diverged "$@" ;; reconcile) shift; command_reconcile "$@" ;; + pr-policy) shift; command_pr_policy "$@" ;; + settle-pr) shift; command_settle_pr "$@" ;; -h|--help) usage ;; *) usage >&2; exit 2 ;; esac diff --git a/bin/fm-classify-lib.sh b/bin/fm-classify-lib.sh index a02a12c73ed..f234e110625 100755 --- a/bin/fm-classify-lib.sh +++ b/bin/fm-classify-lib.sh @@ -172,6 +172,19 @@ last_status_line() { # <status-file> [<previous-event-var>] printf '%s\n' "${scan##*$'\n'}" } +# The newest event the crew itself declared: last_status_line past the resolved +# and captain-held lines a supervisor writes about it. Bounded the same way as +# last_status_line. +last_crew_status_line() { # <status-file> + local f=$1 scan='' annotations + [ -f "$f" ] && [ -r "$f" ] || return 0 + annotations="^[[:space:]]*(${FM_CLASSIFY_RESOLVE_VERB:-$FM_CLASSIFY_RESOLVE_VERB_DEFAULT}|${FM_CLASSIFY_CAPTAIN_HELD_VERB:-$FM_CLASSIFY_CAPTAIN_HELD_VERB_DEFAULT})[[:space:]:[]" + if ! scan=$(tail -n "$FM_CLASSIFY_EVENT_WINDOW_LINES" "$f" 2>/dev/null | grep -Ev "$annotations" | _fm_status_event_scan); then + scan=$(grep -Ev "$annotations" "$f" | _fm_status_event_scan) || : + fi + printf '%s\n' "${scan##*$'\n'}" +} + # 0 when <verb> is exactly one recognized status verb, with no leftover token. _fm_status_verb_recognized() { # <verb> case "$1" in @@ -757,17 +770,18 @@ _fm_key_before_colon() { # <status-line> # the line has no colon or no complete token there; slug charset validity is # the caller's check via _fm_decision_slug_ok, exactly as for the before-colon # position. -_fm_key_at_note_head() { # <status-line> -> raw slug - local rest +_fm_key_at_note_head() { # <status-line> [<out-var>] -> raw slug + local __fm_head_rest case "$1" in - *:*) rest=${1#*:} ;; + *:*) __fm_head_rest=${1#*:} ;; *) return 1 ;; esac - rest=${rest#"${rest%%[![:space:]]*}"} - case "$rest" in - \[key=*\]*) rest=${rest#\[key=}; printf '%s' "${rest%%\]*}" ;; + __fm_head_rest=${__fm_head_rest#"${__fm_head_rest%%[![:space:]]*}"} + case "$__fm_head_rest" in + \[key=*\]*) __fm_head_rest=${__fm_head_rest#\[key=}; __fm_head_rest=${__fm_head_rest%%\]*} ;; *) return 1 ;; esac + if [ "$#" -gt 1 ]; then printf -v "$2" '%s' "$__fm_head_rest"; else printf '%s' "$__fm_head_rest"; fi } # 0 when a stated key slug is well-formed: nonempty, A-Za-z0-9._- only. _fm_decision_slug_ok() { # <slug> @@ -780,50 +794,60 @@ _fm_decision_slug_ok() { # <slug> # worker-written stamp cannot move it: a readable time like [at=10:30] carries # colons that would otherwise end the head mid-tag and hand the caller a note # and a key sliced out of the timestamp. The line's own bytes are never altered. -status_line_note() { # <status-line> -> text after the first colon, trimmed - local n k unstamped - _fm_status_unstamped "$1" unstamped - case "$unstamped" in - *:*) n=${unstamped#*:}; n=${n#"${n%%[![:space:]]*}"} ;; - *) printf '%s' "$unstamped"; return 0 ;; +# Like status_line_verb, an <out-var> receives the note without forking a +# command substitution, and must not be named like one of these locals. +status_line_note() { # <status-line> [<out-var>] -> text after the first colon, trimmed + local __fm_note_text __fm_note_key __fm_note_unstamped + _fm_status_unstamped "$1" __fm_note_unstamped + case "$__fm_note_unstamped" in + *:*) + __fm_note_text=${__fm_note_unstamped#*:} + __fm_note_text=${__fm_note_text#"${__fm_note_text%%[![:space:]]*}"} + # A note-head token that states this line's key (no before-colon token, + # valid slug) is key metadata, not note text: strip it so both stated-key + # positions yield the same note. + if ! _fm_key_before_colon "$__fm_note_unstamped" \ + && _fm_key_at_note_head "$__fm_note_unstamped" __fm_note_key \ + && _fm_decision_slug_ok "$__fm_note_key"; then + __fm_note_text=${__fm_note_text#"[key=$__fm_note_key]"} + __fm_note_text=${__fm_note_text#"${__fm_note_text%%[![:space:]]*}"} + fi + ;; + *) __fm_note_text=$__fm_note_unstamped ;; esac - # A note-head token that states this line's key (no before-colon token, valid - # slug) is key metadata, not note text: strip it so both stated-key positions - # yield the same note. - if ! _fm_key_before_colon "$unstamped" && k=$(_fm_key_at_note_head "$unstamped") \ - && _fm_decision_slug_ok "$k"; then - n=${n#"[key=$k]"} - n=${n#"${n%%[![:space:]]*}"} - fi - printf '%s' "$n" -} -_fm_decision_key() { # <status-line> [<keyless>] -> key slug, or <keyless> (default "default") when no token - local k unstamped - _fm_status_unstamped "$1" unstamped - if _fm_key_before_colon "$unstamped"; then - k=${unstamped%%:*} - k=${k#*\[key=} - k=${k%%\]*} + if [ "$#" -gt 1 ]; then printf -v "$2" '%s' "$__fm_note_text"; else printf '%s' "$__fm_note_text"; fi +} +_fm_decision_key() { # <status-line> [<keyless>] [<out-var>] -> key slug, or <keyless> (default "default") when no token + local __fm_key_slug __fm_key_unstamped + _fm_status_unstamped "$1" __fm_key_unstamped + if _fm_key_before_colon "$__fm_key_unstamped"; then + __fm_key_slug=${__fm_key_unstamped%%:*} + __fm_key_slug=${__fm_key_slug#*\[key=} + __fm_key_slug=${__fm_key_slug%%\]*} + _fm_decision_slug_ok "$__fm_key_slug" || return 1 + elif _fm_key_at_note_head "$__fm_key_unstamped" __fm_key_slug; then + _fm_decision_slug_ok "$__fm_key_slug" || return 1 else - k=$(_fm_key_at_note_head "$unstamped") || { printf '%s' "${2-default}"; return 0; } + __fm_key_slug=${2-default} fi - _fm_decision_slug_ok "$k" || return 1 - printf '%s' "$k" + if [ "$#" -gt 2 ]; then printf -v "$3" '%s' "$__fm_key_slug"; else printf '%s' "$__fm_key_slug"; fi } # Drop the record for <key> from a newline-terminated "<key>\t<verb>\t<note>" set. # Portable (no associative arrays) so the fold runs on bash 3.2 as well as 4+. -_fm_decision_drop() { # <open-set> <key> - local set=$1 key=$2 line out='' - while IFS= read -r line; do - [ -n "$line" ] || continue - case "$line" in - "$key"$'\t'*) : ;; - *) out="${out}${line}"$'\n' ;; +# An <out-var> receives the set without its trailing newline, exactly as a +# command substitution of the printed form would. +_fm_decision_drop() { # <open-set> <key> [<out-var>] + local __fm_drop_line __fm_drop_out='' + while IFS= read -r __fm_drop_line; do + [ -n "$__fm_drop_line" ] || continue + case "$__fm_drop_line" in + "$2"$'\t'*) : ;; + *) __fm_drop_out="${__fm_drop_out}${__fm_drop_line}"$'\n' ;; esac done <<EOF -$set +$1 EOF - printf '%s' "$out" + if [ "$#" -gt 2 ]; then printf -v "$3" '%s' "${__fm_drop_out%$'\n'}"; else printf '%s' "$__fm_drop_out"; fi } # Fold ONE status line into an existing "<key>\t<verb>\t<note>\n"-per-line open # set, applying the same needs-decision/blocked-opens, resolved/captain-held-closes @@ -888,9 +912,26 @@ _fm_status_kind() { case "$kind" in ship|scout|secondmate) printf '%s' "$kind" ;; *) printf unknown ;; esac } -_fm_decision_fold_line() { # <open-set> <status-line> <resolve-verb> <held-verb> <kind> [timed] - local open=$1 line=$2 resolve=$3 held=$4 kind=$5 verb key note unstamped - local timed=${6:-} epoch until old_key old_verb old_epoch old_note +# Print <value>, or assign it to <out-var> without its trailing newlines, which +# is exactly what a command substitution of the printed form would capture. +_fm_decision_fold_emit() { # <value> [<out-var>] + local __fm_emit_value=$1 + [ -n "${2-}" ] || { printf '%s' "$__fm_emit_value"; return 0; } + while :; do + case "$__fm_emit_value" in + *$'\n') __fm_emit_value=${__fm_emit_value%$'\n'} ;; + *) break ;; + esac + done + printf -v "$2" '%s' "$__fm_emit_value" +} + +# An optional <out-var> receives the folded set without forking, so a whole-file +# fold costs no process per status line; it must not be named like a local here. +# A caller that wants the out-var without the timed view passes '' as [timed]. +_fm_decision_fold_line() { # <open-set> <status-line> <resolve-verb> <held-verb> <kind> [timed] [<out-var>] + local open=$1 line=$2 resolve=$3 held=$4 kind=$5 timed=${6:-} out=${7-} + local verb key note unstamped epoch stated until old_key old_verb old_epoch old_note # Both colon tests below ask where the head ends, the same question the note # and key readers ask, so they read the same unstamped copy those readers do. # A worker-written time tag must never decide whether a decision opens or @@ -908,22 +949,22 @@ _fm_decision_fold_line() { # <open-set> <status-line> <resolve-verb> <held-verb # both folds on a status log of ordinary width. Same verdict, bounded cost. case "$unstamped" in *:*|*\[key=*\]*) ;; - *) printf '%s' "$open"; return 0 ;; + *) _fm_decision_fold_emit "$open" "$out"; return 0 ;; esac status_line_verb "$line" verb case "$unstamped" in - *:*) case "$verb:$kind" in done:ship|done:scout|failed:ship|failed:scout) return 0 ;; esac ;; + *:*) case "$verb:$kind" in done:ship|done:scout|failed:ship|failed:scout) _fm_decision_fold_emit '' "$out"; return 0 ;; esac ;; esac case "$verb" in needs-decision|blocked|"$resolve"|"$held") ;; - *) printf '%s' "$open"; return 0 ;; + *) _fm_decision_fold_emit "$open" "$out"; return 0 ;; esac - key=$(_fm_decision_key "$line") || { printf '%s' "$open"; return 0; } - _fm_decision_key_transition_allowed "$key" "$(status_line_note "$line")" \ - || { printf '%s' "$open"; return 0; } + _fm_decision_key "$line" default key || { _fm_decision_fold_emit "$open" "$out"; return 0; } + status_line_note "$line" note + _fm_decision_key_transition_allowed "$key" "$note" \ + || { _fm_decision_fold_emit "$open" "$out"; return 0; } case "$verb" in needs-decision|blocked) - note=$(status_line_note "$line") if [ "$timed" = timed ]; then # The timed view adds emission-time and deferral-date columns, without # changing any open/closed rule. Keyless or unstamped events have @@ -934,7 +975,8 @@ _fm_decision_fold_line() { # <open-set> <status-line> <resolve-verb> <held-verb # newest line; it is `-` when absent and `?` when malformed, never # empty, so a TAB-split read cannot collapse it. epoch=unknown - if [ -n "$(_fm_decision_key "$line" '')" ]; then + _fm_decision_key "$line" '' stated + if [ -n "$stated" ]; then _fm_status_at_epoch "$line" epoch || epoch=unknown fi _fm_status_until_date "$line" until || case $? in 1) until=- ;; *) until='?' ;; esac @@ -950,16 +992,16 @@ $open EOF note="$epoch"$'\t'"$until"$'\t'"$note" fi - open=$(_fm_decision_drop "$open" "$key") + _fm_decision_drop "$open" "$key" open [ -n "$open" ] && open="${open}"$'\n' open="${open}${key}"$'\t'"${verb}"$'\t'"${note}"$'\n' ;; "$resolve"|"$held") - open=$(_fm_decision_drop "$open" "$key") + _fm_decision_drop "$open" "$key" open [ -n "$open" ] && open="${open}"$'\n' ;; esac - printf '%s' "$open" + _fm_decision_fold_emit "$open" "$out" } # Fold the WHOLE status stream into the set of decisions still open. Prints one @@ -977,7 +1019,7 @@ EOF # subprocess read, which exists for that function's much narrower payload-driven # path resolution rather than this directory-local glob. status_open_decisions() { # <status-file> [<kind>] - local f=$1 kind=${2:-} line resolve held open='' verb + local f=$1 kind=${2:-} line resolve held open='' folded verb [ -f "$f" ] && [ -r "$f" ] && [ ! -L "$f" ] || return 0 kind=$(_fm_status_kind "$f" "$kind") resolve=${FM_CLASSIFY_RESOLVE_VERB:-$FM_CLASSIFY_RESOLVE_VERB_DEFAULT} @@ -986,7 +1028,8 @@ status_open_decisions() { # <status-file> [<kind>] status_line_verb "$line" verb case "$verb" in needs-decision|blocked|done|failed|"$resolve"|"$held") - open=$(_fm_decision_fold_line "$open" "$line" "$resolve" "$held" "$kind") + _fm_decision_fold_line "$open" "$line" "$resolve" "$held" "$kind" '' folded + open=$folded ;; esac done < "$f" @@ -2726,10 +2769,10 @@ FM_NO_LIVE_MONITOR='open PR, no live monitor' # Same cost and the same caveat as crew_absorb_class: one fm-crew-state.sh read, # which may make a bounded no-mistakes call, so callers take it only where they # already accept that cost. -crew_gate_awaits_human_decision() { # <id> -> <run-id> on stdout - local id=$1 line state src rest part human='' run='' +crew_gate_awaits_human_decision() { # <id> [state-line] -> <run-id> on stdout + local id=$1 line=${2-} state src rest part human='' run='' [ -n "$id" ] || return 1 - line=$("$FM_CREW_STATE_BIN" "$id" 2>/dev/null) || true + [ -n "$line" ] || line=$("$FM_CREW_STATE_BIN" "$id" 2>/dev/null) || true case "$line" in state:*) ;; *) return 1 ;; esac state=${line#state: }; state=${state%% *} [ "$state" = parked ] || return 1 diff --git a/bin/fm-claude-stop-autoarm.sh b/bin/fm-claude-stop-autoarm.sh index 6ea78fac919..64c33245f3e 100755 --- a/bin/fm-claude-stop-autoarm.sh +++ b/bin/fm-claude-stop-autoarm.sh @@ -33,9 +33,10 @@ # - Single-flight: Claude does not dedupe async hooks, so exactly one # GENERATION owner arms per event epoch: the epoch ledger's monotonic # sequence is the claim generation, every firing defers (exit 0) to a live -# open claim, and a stuck, dead, identity-mismatched, or finished claim is -# superseded by taking the next generation instead of being unlocked or -# revoked. No mutex is ever held across arming or output - the owner lock +# open claim, and a stuck, dead, identity-mismatched, or finished claim - +# or one an earlier Claude session took, as the Stop payload's session_id +# tells - is superseded by taking the next generation instead of being +# unlocked or revoked. No mutex is ever held across arming or output - the owner lock # survives only as the micro-mutex serializing individual ledger writes - # and a superseded owner goes completely silent: ownership is re-verified # before every arm invocation, episode-state mutation, ledger write, and @@ -296,15 +297,27 @@ SESSION_ID=$(fm_session_lock_recorded_session_id "$STATE" 2>/dev/null || true) # pre-generation build (or the guard's own terminal-check), which the legacy # shim defers to while genuinely deciding and reclaims once when proven # abandoned. -fm_autoarm_claim_open "$STATE" "$GRACE" && exit 0 -fm_autoarm_claim_next "$STATE" "$GRACE" +# The payload's session_id names the Claude session this firing belongs to, +# which is the session Claude delivers its exit-2 rewake to, so a claim an +# earlier session took stops deferring this one (fm_autoarm_claim_open). No +# payload, no jq, or an id the ledger cannot carry means no session, which +# keeps the claim's previous behavior. +HOOK_SESSION= +if [ -n "$PAYLOAD" ] && command -v jq >/dev/null 2>&1; then + HOOK_SESSION=$(printf '%s' "$PAYLOAD" | jq -r '.session_id // empty | strings' 2>/dev/null || true) +fi +case "$HOOK_SESSION" in + *[!A-Za-z0-9._-]*) HOOK_SESSION= ;; +esac +fm_autoarm_claim_open "$STATE" "$GRACE" "$HOOK_SESSION" && exit 0 +fm_autoarm_claim_next "$STATE" "$GRACE" "$HOOK_SESSION" CLAIM_RC=$? if [ "$CLAIM_RC" -ne 0 ]; then [ "$CLAIM_RC" -eq 2 ] && exit 0 ROLE=$(fm_lock_role "$OWNER_LOCK" 2>/dev/null || true) [ -n "$ROLE" ] || exit 0 fm_autoarm_release_abandoned "$STATE" "$GRACE" || exit 0 - fm_autoarm_claim_next "$STATE" "$GRACE" || exit 0 + fm_autoarm_claim_next "$STATE" "$GRACE" "$HOOK_SESSION" || exit 0 fi MY_GEN=$FM_AUTOARM_MY_GEN [ -n "$MY_GEN" ] || exit 0 diff --git a/bin/fm-composer-lib.sh b/bin/fm-composer-lib.sh index f52eea08d40..33e2967219d 100644 --- a/bin/fm-composer-lib.sh +++ b/bin/fm-composer-lib.sh @@ -65,7 +65,8 @@ # older claude). The bottom border may carry a TITLE (grok # writes its model name there). A title embedded in the family's # rule is tolerated at the aligned width, including Grok 1.0.44's -# rounded typed model/effort title with ` · always-approve`; Grok 1.0.5's +# rounded typed model/effort title with ` · always-approve` +# (Grok 1.0.77's `<version> Fast` model name too); Grok 1.0.5's # typed title also permits its proven three-column overhang. # bare - an agent prompt glyph row with no border at all (claude `❯`, # codex `›`, muse `⟩`, cursor `→`). The agent glyph is itself the container @@ -1298,9 +1299,11 @@ EOF fi } -# 0 when <title> is exactly the typed Grok model/effort title shape. -_fm_composer_grok_typed_title_ok() { # <title> - local title=$1 effort model +# 0 when <title> is exactly the typed Grok model/effort title shape. With +# <fast>=1 the model may also carry Grok 1.0.77's single ` Fast` word, the +# title it draws for grok-4.7-build-fast (`Grok 4.7 Fast (medium)`). +_fm_composer_grok_typed_title_ok() { # <title> [fast] + local title=$1 fast=${2:-0} effort model case "$title" in 'Grok '*\ \(low\)) effort=low ;; 'Grok '*\ \(medium\)) effort=medium ;; @@ -1310,6 +1313,7 @@ _fm_composer_grok_typed_title_ok() { # <title> esac model=${title#Grok } model=${model%" ($effort)"} + [ "$fast" != 1 ] || model=${model%' Fast'} [ -n "$model" ] || return 1 case "$model" in *[!A-Za-z0-9._-]*) return 1 ;; esac return 0 @@ -1358,11 +1362,12 @@ _fm_composer_titled_bottom_ok() { # <family> <bottom-inner> <top-spaces> # mid-dot. Only that exact suffix on a typed Grok title is known; the single # mid-dot column is swapped for an ASCII one so it cannot survive as residue # in the geometry comparison, while any other mid-dot stays non-space. + # Grok 1.0.77's ` Fast` model word was observed only with this suffix. case "$title" in *' · always-approve') # Only the rounded family was observed for this approval-title shape. [ "$family" = rounded ] || return 1 - _fm_composer_grok_typed_title_ok "${title%' · always-approve'}" || return 1 + _fm_composer_grok_typed_title_ok "${title%' · always-approve'}" 1 || return 1 inner=${inner/' · always-approve'/' . always-approve'} ;; esac diff --git a/bin/fm-contributions.sh b/bin/fm-contributions.sh index 1390befc946..ab907642710 100755 --- a/bin/fm-contributions.sh +++ b/bin/fm-contributions.sh @@ -106,6 +106,14 @@ # structural check that refused the observation. A later good read clears the # record's error, so the unavailable line is the copy that outlives it. # FM_CONTRIBUTIONS_MAX_AGE (default 900 seconds) bounds freshness. +# config/hold-pr-settle opts into applying declared captain-hold PR policies +# after terminal observation, including retained final observations. The policy +# and evidence transition are owned by fm-captain-hold.sh; a settlement prints +# nothing, and a policy that could not be applied prints one diagnostic line. +# Unmerged closure of a PR owned by a task that is still an open captain hold +# becomes one durable `closed` pending event per owner, acknowledged like a review. +# With the flag absent, neither settlement nor this extra notification runs. +# # A supported URL whose last good observation is merged or closed is final: it is # never re-read, stays fresh, and every owner's saved row converges on that # observation, with a stale error beside it cleared. @@ -677,6 +685,36 @@ publish_pending() { # task canonical-url record-file | [.token, .type, (.source // "-"), (.age // "-")] | @tsv' "$record") } +terminal_hold_event() { # task canonical-url + local task=$1 url=$2 policy="$DATA/$1/hold-pr-policy.json" err + [ -f "$CONFIG/hold-pr-settle" ] || return 0 + jq -e --arg url "$url" '.records[] | select(.url == $url and .kind == "pr" + and .error == null and .retired == null and (.observation.state | IN("merged","closed")))' \ + "$DATA/$task/contributions.json" > "$TMP/terminal.json" || return 0 + if [ -f "$policy" ]; then + # The watcher discards a check's stderr, so the cause rides the poll line. + if err=$("$SCRIPT_DIR/fm-captain-hold.sh" settle-pr "$task" "$url" 2>&1 >/dev/null); then + # A removed policy means no open hold remains to report the closure to. + [ -f "$policy" ] || return 0 + else + printf 'contributions: PR hold settlement failed for %s %s: %s\n' "$task" "$url" "${err##*$'\n'}" + fi + fi + # Closure is actionable even when no policy says it answers the question. + jq --arg task "$task" --slurpfile input "$TMP/input.json" ' + select(.observation.state == "closed" and (.seen | index("closed")) == null + and any($input[0].backlog.records[]?; .structured == true and .id == $task and .hold_bucket != null)) + | .seen += ["closed"] + | .pending += [{token:"closed",type:"closed",source:.url,head:.observation.head,body:"PR closed without merging"}]' \ + "$TMP/terminal.json" > "$TMP/terminal-event.json" + [ -s "$TMP/terminal-event.json" ] || return 0 + # The held list is the poll-start snapshot; a settlement earlier in this poll + # may have closed or released this task since. + "$SCRIPT_DIR/fm-captain-hold.sh" open "$task" 2>/dev/null || return 0 + write_record "$task" "$TMP/terminal-event.json" + publish_pending "$task" "$url" "$TMP/terminal-event.json" +} + settle_final() { # canonical-url task... : copy the URL's final observation to every owner local url=$1 task shift @@ -690,7 +728,7 @@ settle_final() { # canonical-url task... : copy the URL's final observation to e [$saved[0][] | select(.task == $task) | .records[] | select(.url == $url)] | first' > "$TMP/old.json" if jq -e '. == null' "$TMP/old.json" >/dev/null; then jq -n --slurpfile final "$TMP/final.json" ' - $final[0] + {error:null,pending:[],notified:[]} | del(.retired)' > "$TMP/row.json" + $final[0] + {error:null,pending:[],notified:[]} | del(.retired) | .seen -= ["closed"]' > "$TMP/row.json" write_record "$task" "$TMP/row.json" elif jq -e '(.observation.state | IN("merged","closed") | not) or .error != null' "$TMP/old.json" >/dev/null; then jq -n --slurpfile final "$TMP/final.json" --slurpfile old "$TMP/old.json" ' @@ -717,7 +755,7 @@ clear_unsupported_errors() { } poll() { - local task url old kind error observed why + local task url old kind error observed why unreported=$'\n' local -a row acquire get_input @@ -747,11 +785,30 @@ poll() { + $known.tasks | @tsv' >> "$TMP/known.tsv" fi : > "$TMP/live.tsv" + # One pass picks the retained closures still owed to an open captain hold, so + # a home's settled history costs no process per row. + if [ -f "$CONFIG/hold-pr-settle" ]; then + unreported+=$(jq -nr --slurpfile input "$TMP/input.json" --slurpfile saved "$TMP/saved.json" ' + [$input[0].backlog.records[]? | select(.structured == true and .hold_bucket != null) | .id] as $held + | $saved[0][] | .task as $task | select($held | index($task) != null) | .records[] + | select(.kind == "pr" and .error == null and .retired == null + and .observation.state == "closed" and (.seen | index("closed")) == null) + | "\($task)\t\(.url)"')$'\n' + fi while IFS=$'\t' read -r -a row; do [ "${#row[@]}" -ge 3 ] || continue case "${row[0]}" in - settled) ;; - final) settle_final "${row[@]:1}" ;; + settled|final) + [ "${row[0]}" != final ] || settle_final "${row[@]:1}" + [ -f "$CONFIG/hold-pr-settle" ] || continue + for task in "${row[@]:2}"; do + case "${row[0]}:$unreported" in + final:*|*$'\n'"$task"$'\t'"${row[1]}"$'\n'*) ;; + *) [ -f "$DATA/$task/hold-pr-policy.json" ] || continue ;; + esac + terminal_hold_event "$task" "${row[1]}" + done + ;; *) (IFS=$'\t'; printf '%s\n' "${row[*]:1}") >> "$TMP/live.tsv" ;; esac done < "$TMP/known.tsv" @@ -811,6 +868,7 @@ poll() { fi write_record "$task" "$TMP/row.json" publish_pending "$task" "$url" "$TMP/row.json" + terminal_hold_event "$task" "$url" done done < "$TMP/known.tsv" } diff --git a/bin/fm-crew-state.sh b/bin/fm-crew-state.sh index be9d972559c..0269891945e 100755 --- a/bin/fm-crew-state.sh +++ b/bin/fm-crew-state.sh @@ -24,7 +24,7 @@ # Output is one stable, parseable, token-tight line firstmate can read every # heartbeat: # -# state: <working|parked|done|blocked|paused|failed|unknown> · source: <run-step|pane|status-log|remote-endpoint|none> · <detail> +# state: <working|parked|done|blocked|paused|failed|unknown> · source: <run-step|pane|status-log|remote-endpoint|review-park|none> · <detail> # # A run-step reading whose run came from inventory selection ends its detail # with a `run: <id>` field naming that run, and a terminal one appends @@ -35,7 +35,9 @@ # from (rule 2b). # # Logic, numbered for reference; rules 2b and 3 each record that rule 3's -# status-log reconciliation is applied before rule 2b: +# status-log reconciliation is applied before rule 2b. A lane parked on review +# by bin/fm-park.sh short-circuits after the worktree check: paused, or blocked +# when its relaunch failed. # 1. Resolve worktree + backend target + kind from state/<id>.meta. A meta # recording remote_host= is a remote secondmate: its worktree and endpoint # live on that host, so the local worktree and pane reads are skipped and @@ -323,6 +325,23 @@ if [ -z "$REMOTE_HOST" ] && { [ -z "$WT" ] || [ ! -d "$WT" ]; }; then emit unknown none "worktree gone (torn down?)" fi +# A lane parked on review (bin/fm-park.sh) stopped its agent on purpose; its +# record, not the stopped pane or an old run, is the current state. +PARK_RECORD=$STATE/$ID.review-park +if [ -f "$PARK_RECORD" ]; then + PARK_PR=$(fm_meta_get "$PARK_RECORD" pr) + PARK_HEAD=$(fm_meta_get "$PARK_RECORD" head) + PARK_AT=$(fm_meta_get "$PARK_RECORD" parked_at) + PARK_WHEN=$(date -u -r "$PARK_AT" +%Y-%m-%dT%H:%MZ 2>/dev/null || date -u -d "@$PARK_AT" +%Y-%m-%dT%H:%MZ 2>/dev/null || printf '%s' "$PARK_AT") + if [ -n "$(fm_meta_get "$PARK_RECORD" resume_error)" ]; then + emit blocked review-park "parked on review: PR $PARK_PR at ${PARK_HEAD:0:12}; the relaunch on review activity failed - run bin/fm-park.sh resume" + fi + if [ -n "$(fm_meta_get "$PARK_RECORD" pending)" ]; then + emit paused review-park "parking on review: PR $PARK_PR at ${PARK_HEAD:0:12} since $PARK_WHEN; the agent's stop is not yet confirmed" + fi + emit paused review-park "parked on review: PR $PARK_PR at ${PARK_HEAD:0:12} since $PARK_WHEN; agent stopped, relaunches on review activity" +fi + # --- status log ------------------------------------------------------------ # Map a status-log verb onto a canonical state for the fallback path. `paused` is @@ -333,6 +352,12 @@ fi # A ship `done:` is not current-state done while bin/fm-dod-lib.sh refuses the # named-head reachability gate: that claim is blocked so a disposable copy is # not treated as finished-and-safe. +# This read never refreshes remote-tracking refs: a fetch bounded at 30 seconds +# per remote does not fit a caller's per-task bound (bin/fm-fleet-snapshot.sh +# defaults to 10 seconds) and would repeat on every read. bin/fm-pr-check.sh, +# which runs when a ready report is registered, and on each watcher sweep when +# config/pr-register-scan is on, refreshes the same copy before it refuses, so +# a read after it sees the fetched head. emit_ship_status_done() { # [extra-detail] local extra=${1:-} reason if reason=$(fm_dod_accept_ship_done "$KIND" "$(meta_value mode)" "$WT" "$(meta_value project)" "$LOG_LINE" "$STATE" "$ID" "$META"); then diff --git a/bin/fm-dod-lib.sh b/bin/fm-dod-lib.sh index dd93d353087..294e6a4509b 100755 --- a/bin/fm-dod-lib.sh +++ b/bin/fm-dod-lib.sh @@ -36,6 +36,10 @@ # bin/fm-pr-check.sh (PR registration), and bin/fm-inactive-reconcile.sh # (secondmate ledger-first publish of a child done). A ship `done:` is not # accepted while the named head exists only in the worker's disposable copy. +# Only PR registration, including the merge-time re-record bin/fm-pr-merge.sh +# makes through it, refreshes remote-tracking refs before it refuses +# (fm_dod_named_head_reachable_outside_worktree); the other callers read the +# refs as that refresh left them. # The check tests that head, not whether some branch moved. In no-mistakes # mode the pre-validation `done: {summary}` is the pipeline handoff and is # not gated; only the later CI-ready `done: PR <url> checks green` is, or on a @@ -700,15 +704,24 @@ EOF esac } -# 0 when <sha> is contained in a ref under <namespace> in <repo>. +# 0 when <sha> is contained in a ref under <namespace> in <repo>, ignoring +# refs that match the optional <excluded-ref-glob>. # --contains tests that exact commit, so a branch that moved to a different # tip does not count. -fm_dod_ref_contains() { # <repo> <ref-namespace> <sha> - local repo=$1 ns=$2 sha=$3 hit +fm_dod_ref_contains() { # <repo> <ref-namespace> <sha> [<excluded-ref-glob>] + local repo=$1 ns=$2 sha=$3 exclude=${4:-} refs ref [ -n "$repo" ] && [ -d "$repo" ] || return 1 [ -n "$sha" ] || return 1 - hit=$(git -C "$repo" for-each-ref --format='%(refname)' --contains="$sha" --count=1 "$ns" 2>/dev/null) || return 1 - [ -n "$hit" ] + refs=$(git -C "$repo" for-each-ref --format='%(refname)' --contains="$sha" "$ns" 2>/dev/null) || return 1 + while IFS= read -r ref; do + [ -n "$ref" ] || continue + # shellcheck disable=SC2254 # the exclusion is a glob by contract + [ -n "$exclude" ] && case "$ref" in $exclude) continue ;; esac + return 0 + done <<EOF +$refs +EOF + return 1 } # 0 when a done: note reports the no-mistakes CI-ready PR (`PR <url> checks @@ -818,6 +831,28 @@ fm_dod_pr_url_from_done_note() { # <note> printf '%s\n' "$url" } +# The URL of a note that is exactly a ready report (`PR <url>`, `PR <url> checks +# green`, or Gerrit's `PR <url> published for review`), or empty. Anything else +# around the URL is prose, so the caller claims no delivered PR from it. +fm_dod_ready_note_url() { # <note> + local url=$1 + case "$url" in 'PR '*) url=${url#PR } ;; *) return 1 ;; esac + case "$url" in + *' checks green') url=${url% checks green} ;; + *' published for review') url=${url% published for review} ;; + esac + case "$url" in ''|*[[:space:]]*) return 1 ;; esac + printf '%s\n' "$url" +} + +# 0 when the status log names <url>. A task's newest ready line replaces its +# recorded PR only on this proof that the line is newer than that PR; a recorded +# PR the log never named is the supervisor's own choice. +fm_dod_status_names_pr() { # <status-file> <url> + [ -f "$1" ] && [ -r "$1" ] || return 1 + tr -s '[:space:]' '\n' < "$1" | grep -Fx -- "$2" > /dev/null +} + # The last recorded <key>= value in <meta>, or empty. fm_dod_meta_value() { # <meta> <key> grep "^$2=" "$1" 2>/dev/null | tail -1 | cut -d= -f2- @@ -966,13 +1001,108 @@ fm_dod_nm_custody_returned() { # <worktree> return 0 } +# Refreshes <repo>'s remote-tracking refs from every remote except the local +# no-mistakes gate, which is a validation copy on this machine: fetching it +# would count work that no real remote holds. When <project> is given, its +# remotes are refreshed the same way after <repo>'s. In each repository origin, +# when there is one, is fetched first, then the other remotes. Each fetch is +# bounded to <seconds> (default 30) and is non-interactive. When <deadline> +# (epoch seconds) is given, each fetch is also bounded to an even share of the +# time left before it among the remotes still to fetch in both repositories, +# so one remote that hangs cannot take the time of the ones after it, <project>'s +# included; a remote whose turn comes after <deadline> is not fetched. +# --no-prune keeps a branch a remote has dropped visible whatever fetch.prune +# says, and the explicit refspec covers every branch even when a clone tracks +# only some of them. +# A fetch that fails, times out, or is not made only leaves the last-known refs +# in place; its remote is appended to FM_DOD_REFRESH_FAILED (", "-separated), +# with " (project clone)" after a <project> remote, so a refusal can say its +# verdict used those refs. Always returns 0. +# bin/fm-teardown.sh and the named-head gate below share this one refresh. +FM_DOD_REFRESH_FAILED= +fm_dod_refresh_remote_refs() { # <repo> [<seconds>] [<deadline>] [<project>] + local repo=$1 seconds=${2:-30} deadline=${3:-} project=${4:-} where dir label remote first rest remotes='' pending=0 bound left + [ -n "$repo" ] && [ -d "$repo" ] || return 0 + for where in repo project; do + dir=$repo + [ "$where" = repo ] || dir=$project + [ -n "$dir" ] && [ -d "$dir" ] || continue + first='' rest='' + while IFS= read -r remote; do + case "$remote" in + ''|no-mistakes) continue ;; + origin) first="$where origin"$'\n' ;; + *) rest="$rest$where $remote"$'\n' ;; + esac + pending=$((pending + 1)) + done <<EOF +$(git -C "$dir" remote 2>/dev/null) +EOF + remotes="$remotes$first$rest" + done + while read -r where remote; do + [ -n "$remote" ] || continue + dir=$repo label='' + [ "$where" = repo ] || { dir=$project; label=' (project clone)'; } + bound=$seconds + if [ -n "$deadline" ]; then + left=$((deadline - $(date +%s))) + [ "$left" -le 0 ] || left=$(((left + pending - 1) / pending)) + [ "$left" -ge "$bound" ] || bound=$left + fi + pending=$((pending - 1)) + [ "$bound" -gt 0 ] \ + && GIT_TERMINAL_PROMPT=0 fm_run_timed "$bound" git -C "$dir" fetch --quiet --no-prune --no-tags --no-recurse-submodules \ + "$remote" "+refs/heads/*:refs/remotes/$remote/*" </dev/null >/dev/null 2>&1 \ + || FM_DOD_REFRESH_FAILED="$FM_DOD_REFRESH_FAILED${FM_DOD_REFRESH_FAILED:+, }$remote$label" + done <<EOF +$remotes +EOF + return 0 +} + +# 0 when <sha> is on a ref that survives the disposable worktree, read from the +# refs as they are now. +fm_dod_named_head_on_surviving_ref() { # <worktree> <project> <mode> <sha> + local wt=$1 project=$2 mode=$3 sha=$4 gate_mirror='refs/remotes/no-mistakes/*' + fm_dod_ref_contains "$wt" refs/remotes "$sha" "$gate_mirror" && return 0 + fm_dod_ref_contains "$project" refs/remotes "$sha" "$gate_mirror" && return 0 + [ "$mode" = local-only ] && fm_dod_ref_contains "$project" refs/heads "$sha" +} + # 0 when <sha> is reachable from a ref that survives the disposable worktree: -# any remote-tracking ref, or - for local-only - heads in the project clone. +# any remote-tracking ref except the local no-mistakes gate mirror +# (refs/remotes/no-mistakes/*, which never leaves this machine), or - for +# local-only - heads in the project clone. The gate mirror does not count even +# after a refresh. +# In no-mistakes mode the pipeline pushes from its own gate clone, so origin +# can hold a head that neither this copy nor the project clone has fetched. +# When the caller sets FM_DOD_REFRESH_REMOTES=1, a head those refs do not show +# triggers one fm_dod_refresh_remote_refs of the copy, and of the project clone +# when its git directory differs, bounded at 30 seconds per fetch, and is then +# checked again. When FM_DOD_REFRESH_DEADLINE (epoch seconds) is also set, the +# whole refresh, every fetch in both repositories together, ends by then, and +# none is made once it has passed. Without FM_DOD_REFRESH_REMOTES the check +# reads the refs as they are and makes no network call. FM_DOD_REFRESH_FAILED +# names any remote that could not be refreshed, including one left unfetched +# because the deadline had passed. fm_dod_named_head_reachable_outside_worktree() { # <worktree> <project> <mode> <sha> - local wt=$1 project=$2 mode=$3 sha=$4 - fm_dod_ref_contains "$wt" refs/remotes "$sha" && return 0 - fm_dod_ref_contains "$project" refs/remotes "$sha" && return 0 - [ "$mode" = local-only ] && fm_dod_ref_contains "$project" refs/heads "$sha" + local wt=$1 project=$2 deadline=${FM_DOD_REFRESH_DEADLINE:-} wt_common project_common separate='' + FM_DOD_REFRESH_FAILED= + fm_dod_named_head_on_surviving_ref "$@" && return 0 + [ "${FM_DOD_REFRESH_REMOTES:-0}" = 1 ] || return 1 + case "$deadline" in + 0*|*[!0-9]*) return 1 ;; + esac + if [ -n "$project" ] && [ -d "$project" ]; then + wt_common=$(git -C "$wt" rev-parse --path-format=absolute --git-common-dir 2>/dev/null) || wt_common= + project_common=$(git -C "$project" rev-parse --path-format=absolute --git-common-dir 2>/dev/null) || project_common= + if [ -n "$project_common" ] && [ "$project_common" != "$wt_common" ]; then + separate=$project + fi + fi + fm_dod_refresh_remote_refs "$wt" 30 "$deadline" "$separate" + fm_dod_named_head_on_surviving_ref "$@" } # 0 when <line> is not a ship done: to gate, when the task's meta records @@ -1045,6 +1175,10 @@ fm_dod_accept_ship_done() { # <kind> <mode> <worktree> <project> <line> [<state fi return 0 fi + if [ -n "$FM_DOD_REFRESH_FAILED" ]; then + printf '%s\n' "named head $sha is unreachable outside the worker copy; could not refresh remote: $FM_DOD_REFRESH_FAILED, so this verdict used the last-known refs and the work may be there; retry once the remote is reachable" + return 1 + fi printf '%s\n' "named head $sha is unreachable outside the worker copy" return 1 } diff --git a/bin/fm-exclude-tools-lib.sh b/bin/fm-exclude-tools-lib.sh index 0306a7e8026..d99dcc76a0e 100644 --- a/bin/fm-exclude-tools-lib.sh +++ b/bin/fm-exclude-tools-lib.sh @@ -49,13 +49,13 @@ fm_exclude_tools_check() { names=$(fm_exclude_tools_names "$config") || return 1 if [ -n "$names" ]; then if [ "$raw" = 1 ]; then - echo "error: config/crew-exclude-tools lists tools to hide, but a raw launch command cannot hide tools; remove the entries or launch a runtime that supports them (pi, pi-signed)" >&2 + echo "error: config/crew-exclude-tools lists tools to hide, but a raw launch command cannot hide tools; remove the entries or launch a runtime that supports them (claude, pi, pi-signed)" >&2 return 1 fi case "$harness" in - pi | pi-signed) ;; + claude | pi | pi-signed) ;; *) - echo "error: config/crew-exclude-tools lists tools to hide, but the $harness runtime cannot hide tools, so this launch is refused rather than running with them available; empty the file or launch a runtime that supports exclusion (pi, pi-signed)" >&2 + echo "error: config/crew-exclude-tools lists tools to hide, but the $harness runtime cannot hide tools, so this launch is refused rather than running with them available; empty the file or launch a runtime that supports exclusion (claude, pi, pi-signed)" >&2 return 1 ;; esac diff --git a/bin/fm-extension-deadline.mjs b/bin/fm-extension-deadline.mjs new file mode 100644 index 00000000000..a23a9b44598 --- /dev/null +++ b/bin/fm-extension-deadline.mjs @@ -0,0 +1,55 @@ +// Host-owned extension deadlines. A callback over two seconds late gets one +// second to drain work frozen with the host. Ordinary stalls keep their bound, +// and a second suspension cannot renew the grace. +import { performance } from "node:perf_hooks"; + +const LATE_SLACK_MS = 2000; +const RESUME_GRACE_MS = 1000; + +export class ExtensionDeadline { + constructor(bound, timeoutMs, operation) { + this.bound = bound; + this.timeoutMs = timeoutMs; + this.operation = operation; + this.started = performance.now(); + this.due = this.started + timeoutMs; + this.graceMs = 0; + this.timer = null; + } + + expired() { + const now = performance.now(); + if (now < this.due) return false; + if (!this.graceMs && now - this.due > LATE_SLACK_MS) { + this.graceMs = RESUME_GRACE_MS; + this.due = now + RESUME_GRACE_MS; + return false; + } + return true; + } + + arm(onTimeout) { + const check = () => { + if (this.expired()) onTimeout(this.evidence()); + else this.timer = setTimeout(check, Math.max(1, this.due - performance.now())); + }; + this.timer = setTimeout(check, this.timeoutMs); + return this; + } + + cancel() { + clearTimeout(this.timer); + } + + evidence() { + const elapsedMs = Math.max(0, Math.floor(performance.now() - this.started)); + return { + bound: this.bound, + operation: this.operation, + timeout_ms: this.timeoutMs, + elapsed_ms: elapsedMs, + overrun_ms: Math.max(0, elapsedMs - this.timeoutMs), + grace_ms: this.graceMs, + }; + } +} diff --git a/bin/fm-extension-launch-barrier.mjs b/bin/fm-extension-launch-barrier.mjs index ce3e7799cc2..a7df258ed1f 100755 --- a/bin/fm-extension-launch-barrier.mjs +++ b/bin/fm-extension-launch-barrier.mjs @@ -10,6 +10,7 @@ import { spawn } from "node:child_process"; import { open, readFile, rename } from "node:fs/promises"; import path from "node:path"; +import { ExtensionDeadline } from "./fm-extension-deadline.mjs"; const READY_SCHEMA = "firstmate.extension-invocation-ready.v1"; const OWNER_SCHEMA = "firstmate.extension-invocation-owner.v1"; @@ -86,9 +87,9 @@ async function main() { group_identity: identity, }); - const deadline = Date.now() + STARTUP_WAIT_MS; + const deadline = new ExtensionDeadline("startup-release", STARTUP_WAIT_MS, verb); let release; - while (Date.now() < deadline) { + while (!deadline.expired()) { if (!pidAlive(hostPid)) process.exit(125); try { release = await readControl(releaseFile); @@ -98,7 +99,7 @@ async function main() { } await sleep(POLL_MS); } - if (!release) die("host did not release the launch barrier"); + if (!release) die(`host did not release the launch barrier ${JSON.stringify(deadline.evidence())}`); if (!exactKeys(release, ["schema", "token"]) || release.schema !== RELEASE_SCHEMA || release.token !== token) { die("launch release identity is invalid"); } diff --git a/bin/fm-extension.mjs b/bin/fm-extension.mjs index 9f63fbf4e80..a1082e03ca4 100755 --- a/bin/fm-extension.mjs +++ b/bin/fm-extension.mjs @@ -78,6 +78,7 @@ import { createHash, randomBytes } from "node:crypto"; import path from "node:path"; import { fileURLToPath } from "node:url"; import { TextDecoder, promisify } from "node:util"; +import { ExtensionDeadline } from "./fm-extension-deadline.mjs"; const SELF = fileURLToPath(import.meta.url); const CODE_ROOT = path.dirname(path.dirname(SELF)); @@ -130,10 +131,11 @@ const fstatAsync = promisify(fstat); const readAsync = promisify(read); class HostError extends Error { - constructor(code, message) { + constructor(code, message, deadline = null) { super(message); this.name = "HostError"; this.code = code; + this.deadline = deadline; } } @@ -1301,9 +1303,9 @@ async function reserveInvocation(home, record, verb, request, statePath) { } async function publishInvocationGroup(invocation, owner) { - const deadline = Date.now() + LAUNCH_READY_WAIT_MS; + const deadline = new ExtensionDeadline("launch-ready", LAUNCH_READY_WAIT_MS, owner.operation); let ready = null; - while (Date.now() < deadline) { + while (!deadline.expired()) { const value = await readPrivateJson(invocation.readyFile, "extension invocation readiness"); if (value) { ready = validateInvocationReady(value, invocation.token); @@ -1312,7 +1314,7 @@ async function publishInvocationGroup(invocation, owner) { if (!pidAlive(invocation.pid)) fail("entrypoint-missing", "extension launch barrier exited before publishing ownership"); await sleep(INVOCATION_POLL_MS); } - if (!ready) fail("timeout", "extension launch barrier did not publish ownership in time"); + if (!ready) throw new HostError("timeout", "extension launch barrier did not publish ownership in time", deadline.evidence()); if (ready.group_pid !== invocation.pid) fail("process-cleanup-failed", "extension launch barrier published a different process group"); // The tracked barrier is the exact detached child this host just created. // Package code cannot run until after this ready record is accepted and the @@ -1409,12 +1411,14 @@ async function runExtensionProcess(home, record, verb, request, timeoutMs, state const stdout = []; let forcedCode = ""; let forcedMessage = ""; + let forcedDeadline = null; let killTimer = null; - const forceStop = (code, message) => { + const forceStop = (code, message, deadline = null) => { if (forcedCode) return; forcedCode = code; forcedMessage = message; + forcedDeadline = deadline; signalProcessGroup(invocation, "SIGTERM"); killTimer = setTimeout(() => signalProcessGroup(invocation, "SIGKILL"), TERMINATE_GRACE_MS); }; @@ -1442,7 +1446,8 @@ async function runExtensionProcess(home, record, verb, request, timeoutMs, state await finalizeInvocation(invocation); throw error; } - const timeout = setTimeout(() => forceStop("timeout", `extension ${verb} exceeded ${timeoutMs} ms`), timeoutMs); + const timeout = new ExtensionDeadline(verb, timeoutMs, verb === "handshake" ? verb : request.operation) + .arm((deadline) => forceStop("timeout", `extension ${verb} exceeded ${timeoutMs} ms`, deadline)); child.stdin.on("error", () => {}); child.stdin.end(requestBytes); @@ -1450,16 +1455,16 @@ async function runExtensionProcess(home, record, verb, request, timeoutMs, state try { outcome = await completion; } catch (error) { - clearTimeout(timeout); + timeout.cancel(); if (killTimer) clearTimeout(killTimer); await finalizeInvocation(invocation); throw error; } - clearTimeout(timeout); + timeout.cancel(); if (killTimer) clearTimeout(killTimer); const leakedProcessGroup = !forcedCode && groupAlive(invocation.pid); await finalizeInvocation(invocation); - if (forcedCode) fail(forcedCode, forcedMessage); + if (forcedCode) throw new HostError(forcedCode, forcedMessage, forcedDeadline); if (leakedProcessGroup) fail("process-leak", `extension ${verb} left a background process in its invocation group`); if (outcome.signal || outcome.code !== 0) fail("process-failed", `extension ${verb} exited nonzero`); return parseStrictJson(Buffer.concat(stdout), `extension ${verb} response`); @@ -1799,6 +1804,7 @@ function errorEvidence(error, extensionId, operation) { extension_id: safeExtensionId, operation, code: allowedCode, + ...(error instanceof HostError && error.deadline ? { deadline: error.deadline } : {}), })}\n`; } diff --git a/bin/fm-fleet-snapshot.sh b/bin/fm-fleet-snapshot.sh index 3349f464272..a4d018846c9 100755 --- a/bin/fm-fleet-snapshot.sh +++ b/bin/fm-fleet-snapshot.sh @@ -269,7 +269,11 @@ reported unreadable with the reason; collection never computes a summary in that home. Each local per-task current-state read is bounded by FM_SNAPSHOT_CREW_STATE_TIMEOUT (default 10 seconds); a read that hits the bound reports state unknown. Local task -observations run concurrently, up to FM_SNAPSHOT_LOCAL_READ_CONCURRENCY (default 8). +observations run in batches of up to FM_SNAPSHOT_LOCAL_READ_CONCURRENCY tasks +(default 8), each task's status-history fold and row composition running in its +own observation worker, and each batch waits for its slowest task before the next +batch starts. That fold and the bounded current-state read both scan the task's +whole status history, so collection time still grows with accumulated history. Remote secondmate endpoint liveness is not probed by this command. Terminal contradiction evidence uses FM_SNAPSHOT_TERMINAL_LINES, FM_SNAPSHOT_TERMINAL_BYTES, and @@ -634,15 +638,18 @@ snapshot_task_generation_is_current() { # <captured-meta> <id> prefetch_task_observations() { # <meta> <id> local meta=$1 id=$2 remote_host current_file endpoint_file current_pid='' current_rc=0 - local status_log status_capture report_path report_capture + local status_log status_capture report_path report_capture open_file local kind backend target endpoint_exists=null agent_alive=not_checked generation_current=1 remote_host=$(meta_value "$meta" remote_host) current_file="$SNAPSHOT_TASK_DIR/$id.json" endpoint_file="$SNAPSHOT_TASK_DIR/$id.endpoint" + open_file="$SNAPSHOT_TASK_DIR/$id.open-decisions" status_log="$STATE/$id.status" status_capture="$SNAPSHOT_TASK_DIR/$id.status" report_path="$DATA/$id/report.md" report_capture="$SNAPSHOT_TASK_DIR/$id.report" + kind=$(meta_value "$meta" kind) + [ -n "$kind" ] || kind=ship snapshot_task_generation_is_current "$meta" "$id" || generation_current=0 if [ "$generation_current" = 1 ]; then @@ -657,7 +664,6 @@ prefetch_task_observations() { # <meta> <id> elif [ "$generation_current" = 1 ]; then crew_state_json "$id" "$meta" "$status_capture" > "$current_file" & current_pid=$! - kind=$(meta_value "$meta" kind) backend=$(fm_backend_of_meta "$meta") target=$(fm_backend_target_of_meta "$meta") if [ -n "$target" ]; then @@ -676,24 +682,35 @@ prefetch_task_observations() { # <meta> <id> agent_alive=unknown fi + # The whole-stream open-decision fold costs time proportional to the task's + # status history, so it runs here while the bounded current-state read is + # still outstanding rather than after every read has finished. + status_open_decisions "$status_capture" "$kind" > "$open_file" || current_rc=1 [ -z "$current_pid" ] || wait "$current_pid" || current_rc=1 # All mutable observations must belong to the metadata generation captured in # the manifest. If teardown/relaunch raced any read, discard the whole sample. if ! snapshot_task_generation_is_current "$meta" "$id"; then rm -f -- "$status_capture" "$report_capture" + : > "$open_file" || current_rc=1 jq -n '{state:"unknown",source:"none",detail:"task generation changed during snapshot",raw:""}' \ > "$current_file" || current_rc=1 endpoint_exists=null agent_alive=unknown fi printf 'endpoint_exists=%s\nagent_alive=%s\n' "$endpoint_exists" "$agent_alive" > "$endpoint_file" || current_rc=1 - return "$current_rc" + [ "$current_rc" -eq 0 ] || return "$current_rc" + task_json_row "$meta" "$id" > "$SNAPSHOT_TASK_DIR/$id.row" } -# Current-state and endpoint reads are independent observations. Start each -# task's pair together so five local workers pay one slow no-mistakes response -# window rather than five in series, while every command bound remains owned by -# fm-timeout-lib.sh. +# Each task's observations and row composition are independent of every other +# task's. Start each task's current-state and endpoint reads together, fold its +# status history while they run, and compose its row in the same worker. Workers +# run in batches of up to FM_SNAPSHOT_LOCAL_READ_CONCURRENCY tasks and each batch +# waits for its slowest task, so a fleet pays its slowest task once per batch +# rather than every task's no-mistakes response window and status history in +# series, while every command bound remains owned by fm-timeout-lib.sh. The fold +# still scans each task's whole history, so a batch's cost grows with its longest +# accumulated history. prefetch_task_current_states() { local meta captured_meta id active=0 index=0 rc=0 local -a pids=() @@ -744,197 +761,198 @@ prefetch_task_current_states() { fi } -task_json_lines() { - local meta original_meta id kind harness mode yolo project worktree home projects spawn_gen backend target status_log report_path - local remote_host remote_root current_file endpoint_file observation_line index=0 +# Compose one task row from the observations its own worker captured. +task_json_row() { # <captured-meta> <id> + local meta=$1 id=$2 original_meta kind harness mode yolo project worktree home projects spawn_gen branch backend target status_log report_path + local remote_host remote_root current_file endpoint_file observation_line local pr pr_source event_json current_json endpoint_exists agent_alive meta_json status_json report_json worktree_json home_json local last_event_raw current_state current_source pending_decision blocked_event report_present=0 pr_from_status local open_decisions_tsv open_decisions_json - while [ "$index" -lt "$SNAPSHOT_TASK_META_COUNT" ]; do - meta=${SNAPSHOT_TASK_METAS[index]} - index=$((index + 1)) - id=$(basename "$meta" .meta) - original_meta="$STATE/$id.meta" - kind=$(meta_value "$meta" kind) - [ -n "$kind" ] || kind=ship - harness=$(meta_value "$meta" harness) - mode=$(meta_value "$meta" mode) - yolo=$(meta_value "$meta" yolo) - project=$(meta_value "$meta" project) - worktree=$(meta_value "$meta" worktree) - home=$(meta_value "$meta" home) - projects=$(meta_value "$meta" projects) - spawn_gen=$(meta_value "$meta" spawn_gen) - branch=$(meta_value "$meta" branch) - remote_host=$(meta_value "$meta" remote_host) - remote_root=$(meta_value "$meta" remote_root) - if [ -n "$remote_host" ]; then - backend=$(meta_value "$meta" remote_backend) - [ -n "$backend" ] || backend=unknown - target=$(meta_value "$meta" remote_target) - else - backend=$(fm_backend_of_meta "$meta") - target=$(fm_backend_target_of_meta "$meta") - fi - status_log="$SNAPSHOT_TASK_DIR/$id.status" - report_path="$SNAPSHOT_TASK_DIR/$id.report" - pr=$(meta_value "$meta" pr) - pr_source=meta - if [ -z "$pr" ]; then - pr_from_status=$(first_pr_url_in_file "$status_log" || true) - pr=$pr_from_status - pr_source=status_event - fi - if [ -z "$pr" ]; then - pr_source=absent - fi + original_meta="$STATE/$id.meta" + kind=$(meta_value "$meta" kind) + [ -n "$kind" ] || kind=ship + harness=$(meta_value "$meta" harness) + mode=$(meta_value "$meta" mode) + yolo=$(meta_value "$meta" yolo) + project=$(meta_value "$meta" project) + worktree=$(meta_value "$meta" worktree) + home=$(meta_value "$meta" home) + projects=$(meta_value "$meta" projects) + spawn_gen=$(meta_value "$meta" spawn_gen) + branch=$(meta_value "$meta" branch) + remote_host=$(meta_value "$meta" remote_host) + remote_root=$(meta_value "$meta" remote_root) + if [ -n "$remote_host" ]; then + backend=$(meta_value "$meta" remote_backend) + [ -n "$backend" ] || backend=unknown + target=$(meta_value "$meta" remote_target) + else + backend=$(fm_backend_of_meta "$meta") + target=$(fm_backend_target_of_meta "$meta") + fi + status_log="$SNAPSHOT_TASK_DIR/$id.status" + report_path="$SNAPSHOT_TASK_DIR/$id.report" + pr=$(meta_value "$meta" pr) + pr_source=meta + if [ -z "$pr" ]; then + pr_from_status=$(first_pr_url_in_file "$status_log" || true) + pr=$pr_from_status + pr_source=status_event + fi + if [ -z "$pr" ]; then + pr_source=absent + fi - current_file="$SNAPSHOT_TASK_DIR/$id.json" - current_json=$(<"$current_file") || { - snapshot_task_cleanup - return 1 - } - event_json=$(status_event_json "$status_log" "$STATE/$id.status") - last_event_raw=$(printf '%s' "$event_json" | jq -r '.last_event.raw // ""') - read -r current_state current_source < <( - printf '%s' "$current_json" | jq -r '[.state // "", .source // ""] | @tsv' - ) - - # Durable keyed open-decision set: fold the WHOLE status stream - # (fm-classify-lib.sh's status_open_decisions) so a later unrelated event can - # never mask a still-open captain decision. The set is derived purely from the - # keyed fold - never from report bodies or decision-like prose - and then - # reconciled against the crew LIFECYCLE, which only clears a stale decision the - # crew has provably moved past. Two lifecycle signals clear it, neither of which - # reads any report content: - # - a live activity read (run-step or busy pane) that is working/done, so a - # crew that resumed past a gate is not still reported as parked; and - # - a TERMINAL done/failed state on a single-owner task (scout or ship), whose - # deliverable is its report or PR, so a COMPLETED scout surfaces only as a - # report POINTER, never as a reopened pending decision. - # Secondmates are excluded from lifecycle clearing: they are persistent and - # multiplex many concerns onto one stream, so activity on one concern must - # never clear another concern's keyed decision. A parked/blocked state, or a - # non-authoritative status-log/none read on a still-live task, keeps the fold's - # open decision surfacing. - open_decisions_tsv=$(status_open_decisions "$status_log" "$kind") - if [ "$kind" != secondmate ] && \ - { { { [ "$current_source" = run-step ] || [ "$current_source" = pane ]; } \ - && [ "$current_state" != parked ] && [ "$current_state" != blocked ]; } \ - || { [ "$current_state" = "done" ] || [ "$current_state" = "failed" ]; }; }; then - open_decisions_tsv="" - fi - open_decisions_json=$(printf '%s' "$open_decisions_tsv" | jq -R -s ' - [ splits("\n") | select(length > 0) - | (capture("^(?<key>[^\t]*)\t(?<verb>[^\t]*)\t(?<summary>.*)$")?) - | select(. != null) ]') - pending_decision=$(printf '%s' "$open_decisions_json" | jq 'if any(.[]; .verb == "needs-decision") then 1 else 0 end') - blocked_event=$(printf '%s' "$open_decisions_json" | jq 'if any(.[]; .verb == "blocked") then 1 else 0 end') + current_file="$SNAPSHOT_TASK_DIR/$id.json" + current_json=$(<"$current_file") || return 1 + event_json=$(status_event_json "$status_log" "$STATE/$id.status") + last_event_raw=$(printf '%s' "$event_json" | jq -r '.last_event.raw // ""') + read -r current_state current_source < <( + printf '%s' "$current_json" | jq -r '[.state // "", .source // ""] | @tsv' + ) - endpoint_exists=null - agent_alive=not_checked - endpoint_file="$SNAPSHOT_TASK_DIR/$id.endpoint" - while IFS= read -r observation_line || [ -n "$observation_line" ]; do - case "$observation_line" in - endpoint_exists=*) endpoint_exists=${observation_line#*=} ;; - agent_alive=*) agent_alive=${observation_line#*=} ;; - esac - done < "$endpoint_file" || { - snapshot_task_cleanup - return 1 - } - [ -f "$report_path" ] && report_present=1 || report_present=0 - meta_json=$(path_present_json "$original_meta" "$meta") - status_json=$event_json - report_json=$(path_present_json "$DATA/$id/report.md" "$report_path") - if [ -n "$worktree" ]; then worktree_json=$(path_present_json "$worktree"); else worktree_json=$(jq -n '{path:null,present:false}'); fi - if [ -n "$home" ] && [ -n "$remote_host" ]; then - home_json=$(jq -n --arg path "$home" '{path:$path,present:null}') - elif [ -n "$home" ]; then - home_json=$(path_present_json "$home") - else - home_json=$(jq -n '{path:null,present:false}') - fi + # Durable keyed open-decision set: the worker folded the WHOLE status stream + # (fm-classify-lib.sh's status_open_decisions) so a later unrelated event can + # never mask a still-open captain decision. The set is derived purely from the + # keyed fold - never from report bodies or decision-like prose - and then + # reconciled against the crew LIFECYCLE, which only clears a stale decision the + # crew has provably moved past. Two lifecycle signals clear it, neither of which + # reads any report content: + # - a live activity read (run-step or busy pane) that is working/done, so a + # crew that resumed past a gate is not still reported as parked; and + # - a TERMINAL done/failed state on a single-owner task (scout or ship), whose + # deliverable is its report or PR, so a COMPLETED scout surfaces only as a + # report POINTER, never as a reopened pending decision. + # Secondmates are excluded from lifecycle clearing: they are persistent and + # multiplex many concerns onto one stream, so activity on one concern must + # never clear another concern's keyed decision. A parked/blocked state, or a + # non-authoritative status-log/none read on a still-live task, keeps the fold's + # open decision surfacing. + open_decisions_tsv=$(<"$SNAPSHOT_TASK_DIR/$id.open-decisions") || return 1 + if [ "$kind" != secondmate ] && \ + { { { [ "$current_source" = run-step ] || [ "$current_source" = pane ]; } \ + && [ "$current_state" != parked ] && [ "$current_state" != blocked ]; } \ + || { [ "$current_state" = "done" ] || [ "$current_state" = "failed" ]; }; }; then + open_decisions_tsv="" + fi + open_decisions_json=$(printf '%s' "$open_decisions_tsv" | jq -R -s ' + [ splits("\n") | select(length > 0) + | (capture("^(?<key>[^\t]*)\t(?<verb>[^\t]*)\t(?<summary>.*)$")?) + | select(. != null) ]') + pending_decision=$(printf '%s' "$open_decisions_json" | jq 'if any(.[]; .verb == "needs-decision") then 1 else 0 end') + blocked_event=$(printf '%s' "$open_decisions_json" | jq 'if any(.[]; .verb == "blocked") then 1 else 0 end') + + endpoint_exists=null + agent_alive=not_checked + endpoint_file="$SNAPSHOT_TASK_DIR/$id.endpoint" + while IFS= read -r observation_line || [ -n "$observation_line" ]; do + case "$observation_line" in + endpoint_exists=*) endpoint_exists=${observation_line#*=} ;; + agent_alive=*) agent_alive=${observation_line#*=} ;; + esac + done < "$endpoint_file" || return 1 + [ -f "$report_path" ] && report_present=1 || report_present=0 + meta_json=$(path_present_json "$original_meta" "$meta") + status_json=$event_json + report_json=$(path_present_json "$DATA/$id/report.md" "$report_path") + if [ -n "$worktree" ]; then worktree_json=$(path_present_json "$worktree"); else worktree_json=$(jq -n '{path:null,present:false}'); fi + if [ -n "$home" ] && [ -n "$remote_host" ]; then + home_json=$(jq -n --arg path "$home" '{path:$path,present:null}') + elif [ -n "$home" ]; then + home_json=$(path_present_json "$home") + else + home_json=$(jq -n '{path:null,present:false}') + fi - jq -n \ - --arg id "$id" \ - --arg kind "$kind" \ - --arg harness "$harness" \ - --arg mode "$mode" \ - --arg yolo "$yolo" \ - --arg branch "$branch" \ - --arg project "$project" \ - --arg worktree "$worktree" \ - --arg home "$home" \ - --arg projects "$projects" \ - --arg spawn_gen "$spawn_gen" \ - --arg backend "$backend" \ - --arg target "$target" \ - --arg remote_host "$remote_host" \ - --arg remote_root "$remote_root" \ - --arg pr "$pr" \ - --arg pr_source "$pr_source" \ - --arg pr_head "$(meta_value "$meta" pr_head)" \ - --arg agent_alive "$agent_alive" \ - --arg observed_at "$SNAPSHOT_NOW" \ - --arg last_event_raw "$last_event_raw" \ - --argjson current_state "$current_json" \ - --argjson meta_path "$meta_json" \ - --argjson status_log "$status_json" \ - --argjson report "$report_json" \ - --argjson worktree_path "$worktree_json" \ - --argjson home_path "$home_json" \ - --argjson endpoint_exists "$endpoint_exists" \ - --argjson open_decisions "$open_decisions_json" \ - --argjson pending_decision "$(bool_json "$pending_decision")" \ - --argjson blocked_event "$(bool_json "$blocked_event")" \ - --argjson report_present "$(bool_json "$report_present")" \ - '{ - id:$id, - kind:$kind, - harness:($harness // ""), - mode:($mode // ""), - yolo:($yolo // ""), - branch:($branch | if . == "" then null else . end), - project:($project // ""), - spawn_gen:($spawn_gen | if . == "" then null else . end), - backend:$backend, - remote:(if $remote_host == "" then null else {host:$remote_host,root:$remote_root} end), - paths:{ - meta:$meta_path, - status_log:$status_log, - worktree:$worktree_path, - home:$home_path, - report:$report - }, - secondmate_projects:($projects | if . == "" then [] else split(",") | map(gsub("^[[:space:]]+|[[:space:]]+$"; "")) | map(select(. != "")) end), - current_state:($current_state + {observed_at:$observed_at,freshness:"fresh"}), - endpoint:{target:($target | if . == "" then null else . end),exists:$endpoint_exists,agent_alive:$agent_alive, - status:(if $endpoint_exists == false then "absent" - elif $agent_alive == "alive" or $agent_alive == "dead" then $agent_alive - else "unknown" end), - observed_at:$observed_at,freshness:"fresh"}, - pr:{url:($pr | if . == "" then null else . end),source:$pr_source,head:($pr_head | if . == "" then null else . end)}, - hints:{ - pending_decision:$pending_decision, - blocked_event:$blocked_event, - open_decisions:$open_decisions, - scout_report_present:$report_present, - last_event_text:$last_event_raw - }, - actions:( - if $kind == "secondmate" then - {send:"bin/fm-send.sh fm-\($id) \u0027<request>\u0027", - watch:"read status/doc return channel; do not routinely fm-peek a secondmate for answers", - return_channel_note:"Secondmate answers come back through status/doc paths after a marked fm-send request."} - else - {watch:"bin/fm-peek.sh fm-\($id)", - steer:"bin/fm-send.sh fm-\($id) \u0027<instruction>\u0027", - return_channel_note:null} - end) - }' - done | jq -s 'sort_by(.id)' + jq -n \ + --arg id "$id" \ + --arg kind "$kind" \ + --arg harness "$harness" \ + --arg mode "$mode" \ + --arg yolo "$yolo" \ + --arg branch "$branch" \ + --arg project "$project" \ + --arg worktree "$worktree" \ + --arg home "$home" \ + --arg projects "$projects" \ + --arg spawn_gen "$spawn_gen" \ + --arg backend "$backend" \ + --arg target "$target" \ + --arg remote_host "$remote_host" \ + --arg remote_root "$remote_root" \ + --arg pr "$pr" \ + --arg pr_source "$pr_source" \ + --arg pr_head "$(meta_value "$meta" pr_head)" \ + --arg agent_alive "$agent_alive" \ + --arg observed_at "$SNAPSHOT_NOW" \ + --arg last_event_raw "$last_event_raw" \ + --argjson current_state "$current_json" \ + --argjson meta_path "$meta_json" \ + --argjson status_log "$status_json" \ + --argjson report "$report_json" \ + --argjson worktree_path "$worktree_json" \ + --argjson home_path "$home_json" \ + --argjson endpoint_exists "$endpoint_exists" \ + --argjson open_decisions "$open_decisions_json" \ + --argjson pending_decision "$(bool_json "$pending_decision")" \ + --argjson blocked_event "$(bool_json "$blocked_event")" \ + --argjson report_present "$(bool_json "$report_present")" \ + '{ + id:$id, + kind:$kind, + harness:($harness // ""), + mode:($mode // ""), + yolo:($yolo // ""), + branch:($branch | if . == "" then null else . end), + project:($project // ""), + spawn_gen:($spawn_gen | if . == "" then null else . end), + backend:$backend, + remote:(if $remote_host == "" then null else {host:$remote_host,root:$remote_root} end), + paths:{ + meta:$meta_path, + status_log:$status_log, + worktree:$worktree_path, + home:$home_path, + report:$report + }, + secondmate_projects:($projects | if . == "" then [] else split(",") | map(gsub("^[[:space:]]+|[[:space:]]+$"; "")) | map(select(. != "")) end), + current_state:($current_state + {observed_at:$observed_at,freshness:"fresh"}), + endpoint:{target:($target | if . == "" then null else . end),exists:$endpoint_exists,agent_alive:$agent_alive, + status:(if $endpoint_exists == false then "absent" + elif $agent_alive == "alive" or $agent_alive == "dead" then $agent_alive + else "unknown" end), + observed_at:$observed_at,freshness:"fresh"}, + pr:{url:($pr | if . == "" then null else . end),source:$pr_source,head:($pr_head | if . == "" then null else . end)}, + hints:{ + pending_decision:$pending_decision, + blocked_event:$blocked_event, + open_decisions:$open_decisions, + scout_report_present:$report_present, + last_event_text:$last_event_raw + }, + actions:( + if $kind == "secondmate" then + {send:"bin/fm-send.sh fm-\($id) \u0027<request>\u0027", + watch:"read status/doc return channel; do not routinely fm-peek a secondmate for answers", + return_channel_note:"Secondmate answers come back through status/doc paths after a marked fm-send request."} + else + {watch:"bin/fm-peek.sh fm-\($id)", + steer:"bin/fm-send.sh fm-\($id) \u0027<instruction>\u0027", + return_channel_note:null} + end) + }' +} + +task_json_lines() { + local index=0 + local -a row_files=() + [ "$SNAPSHOT_TASK_META_COUNT" -gt 0 ] || { jq -n '[]'; return; } + while [ "$index" -lt "$SNAPSHOT_TASK_META_COUNT" ]; do + row_files[index]="$SNAPSHOT_TASK_DIR/$(basename "${SNAPSHOT_TASK_METAS[index]}" .meta).row" + index=$((index + 1)) + done + jq -s 'sort_by(.id)' "${row_files[@]}" } # Main-home current-inventory validity: same orphan / unstructured-current checks diff --git a/bin/fm-harness.sh b/bin/fm-harness.sh index 8c8e7a5ade9..57c30b88034 100755 --- a/bin/fm-harness.sh +++ b/bin/fm-harness.sh @@ -81,6 +81,8 @@ CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" . "$SCRIPT_DIR/fm-gemini-lib.sh" # shellcheck source=bin/fm-hermes-process-lib.sh . "${SCRIPT_DIR:-/}/fm-hermes-process-lib.sh" +# shellcheck source=bin/fm-omp-process-lib.sh +. "${SCRIPT_DIR:-/}/fm-omp-process-lib.sh" # Print the harness named by a verified environment marker, or nothing when no # marker is present. Markers only report what the environment CLAIMS; detect_own @@ -163,14 +165,16 @@ harness_marker() { return 0 } -# True when an exact `omp` process sits within eight parents of this one. The -# same anchored match as the ancestry walk below, kept separate so the marker -# precedence above can demand real process evidence before trusting FM_OMP_HARNESS. +# True when an omp process sits within eight parents of this one. The same +# identity as the ancestry walk below (bin/fm-omp-process-lib.sh), kept separate +# so the marker precedence above can demand real process evidence before +# trusting FM_OMP_HARNESS. ancestry_names_omp() { - local pid=$$ comm + local pid=$$ comm args for _ in 1 2 3 4 5 6 7 8; do comm=$(ps -o comm= -p "$pid" 2>/dev/null) || return 1 - [ "$(basename -- "$comm")" = omp ] && return 0 + args=$(ps -o args= -p "$pid" 2>/dev/null) || args= + fm_omp_process_matches "$comm" "$args" && return 0 pid=$(ps -o ppid= -p "$pid" 2>/dev/null | tr -d ' ') [ -n "$pid" ] && [ "$pid" -gt 1 ] || return 1 done @@ -202,6 +206,15 @@ harness_process_verdict() { # <pid> echo "comm hermes" return fi + # omp, compiled binary or bun package (bin/fm-omp-process-lib.sh owns both + # shapes). It sits above the node*|python* interpreter fallback deliberately: + # the optional claude-bridge extension runs a nested executable literally + # named `claude` with its own node child, and that fallback's *claude* args + # glob would otherwise claim it if that subtree were ever walked. + if fm_omp_process_matches "$comm" "$args"; then + echo "comm omp" + return + fi case "$(basename -- "$comm")" in # gemini precedes claude here for the same precedence reason as the # marker layer above, so a gemini worker under a claude primary is never @@ -233,15 +246,6 @@ harness_process_verdict() { # <pid> # is why detect_own keeps a marker that agrees on the family. pi-signed) echo "comm pi"; return ;; pi) echo "comm pi"; return ;; - # omp is a Bun-compiled single binary whose process name is exactly `omp` - # (verified, omp 18.1.11: `ps -o comm=` reports omp from both its `!` - # bash path and the model's bash tool). Anchored, never *omp*, so ompd, - # comp, and similar unrelated commands are not misread as this harness. - # It sits above the node*|python* interpreter fallback deliberately: the - # optional claude-bridge extension runs a nested executable literally - # named `claude` with its own node child, and that fallback's *claude* - # args glob would otherwise claim it if that subtree were ever walked. - omp) echo "comm omp"; return ;; # agy (Antigravity CLI) is a Go-compiled single binary whose process name # is exactly `agy` (verified, agy 1.2.0: `ps -o comm=` reports agy and # Herdr's process-info reports name agy with argv[0] agy). Anchored, never diff --git a/bin/fm-herdr-lab.sh b/bin/fm-herdr-lab.sh index d4ce86b30e4..23321d59e07 100755 --- a/bin/fm-herdr-lab.sh +++ b/bin/fm-herdr-lab.sh @@ -15,6 +15,17 @@ # Session names must begin with "fm-lab-" and can never be "default". # The name command sanitizes the label, caps it at 16 characters, and appends # process/random suffixes to keep generated socket paths short. +# A session whose <herdr config>/sessions/<session>/herdr-client.sock path +# would exceed the 103-byte sun_path limit is refused by name, prepare and +# provision. +# The herdr config root is $XDG_CONFIG_HOME/herdr when XDG_CONFIG_HOME is set, +# otherwise $HOME/.config/herdr, the same root herdr resolves. +# The length is counted in bytes, not characters. +# Provision keeps the lab server output in <session>.server.log in the state +# directory, truncated on every provision and capped at 4096 bytes (later output +# is discarded), and shown only as a bounded tail. +# Provision also stops waiting as soon as the server process exits, and prints +# the log tail when the session never reports running. # Every Herdr call made here carries --session <session>: trailing, or # immediately before the first -- delimiter so it stays a Herdr option instead # of becoming a passthrough argument such as an agent start argument. @@ -61,6 +72,22 @@ fm_herdr_lab_validate_name() { # <session> return 1 } +fm_herdr_lab_sun_path_limit=103 + +fm_herdr_lab_config_dir() { + printf '%s/herdr' "${XDG_CONFIG_HOME:-${HOME:-}/.config}" +} + +fm_herdr_lab_check_socket_path() { # <session> + local name=$1 path bytes + path="$(fm_herdr_lab_config_dir)/sessions/$name/herdr-client.sock" + bytes=$(printf '%s' "$path" | LC_ALL=C wc -c | tr -d ' ') + if [ "$bytes" -gt "$fm_herdr_lab_sun_path_limit" ]; then + fm_herdr_lab_error "session '$name' would bind a ${bytes}-byte socket path, over the $fm_herdr_lab_sun_path_limit-byte sun_path limit: $path" + return 1 + fi +} + fm_herdr_lab_state_dir() { printf '%s' "${FM_HERDR_LAB_STATE_DIR:-${TMPDIR:-/tmp}/fm-herdr-lab-${UID}}" } @@ -128,6 +155,7 @@ fm_herdr_lab_fleet_state() { # <session> fm_herdr_lab_prepare() { # <session> local name=$1 sessions state_dir tripwire fm_herdr_lab_validate_name "$name" || return 1 + fm_herdr_lab_check_socket_path "$name" || return 1 command -v herdr >/dev/null 2>&1 || { fm_herdr_lab_error "herdr is required"; return 1; } command -v jq >/dev/null 2>&1 || { fm_herdr_lab_error "jq is required"; return 1; } @@ -436,9 +464,23 @@ fm_herdr_lab_cancel_provision() { # <pid> wait "$pid" 2>/dev/null || true } +fm_herdr_lab_server_log_cap=4096 + +fm_herdr_lab_server_log_path() { # <session> + printf '%s/%s.server.log' "$(fm_herdr_lab_state_dir)" "$1" +} + +fm_herdr_lab_server_log_tail() { # <log> + local log=$1 + [ -s "$log" ] || return 0 + fm_herdr_lab_error "server output (last 20 lines of $log):" + tail -n 20 "$log" | tail -c "$fm_herdr_lab_server_log_cap" | sed 's/^/fm-herdr-lab: /' >&2 +} + fm_herdr_lab_provision() { # <session> - local name=$1 sessions tripwire running attempt server_pid max_attempts timeout_seconds + local name=$1 sessions tripwire running attempt server_pid max_attempts timeout_seconds server_log fm_herdr_lab_validate_name "$name" || return 1 + fm_herdr_lab_check_socket_path "$name" || return 1 command -v herdr >/dev/null 2>&1 || { fm_herdr_lab_error "herdr is required"; return 1; } command -v jq >/dev/null 2>&1 || { fm_herdr_lab_error "jq is required"; return 1; } @@ -463,7 +505,10 @@ fm_herdr_lab_provision() { # <session> else fm_herdr_lab_prepare "$name" || return 1 fi - fm_herdr_lab_raw "$name" server >/dev/null 2>&1 & + server_log=$(fm_herdr_lab_server_log_path "$name") + mkdir -p "$(dirname "$server_log")" + : > "$server_log" + fm_herdr_lab_raw "$name" server > >(exec >/dev/null 2>&1; head -c "$fm_herdr_lab_server_log_cap" > "$server_log"; cat) 2>&1 & server_pid=$! attempt=0 max_attempts=300 @@ -477,11 +522,21 @@ fm_herdr_lab_provision() { # <session> } return 0 fi + if ! kill -0 "$server_pid" 2>/dev/null; then + wait "$server_pid" 2>/dev/null || true + sleep 0.2 + running=$(fm_herdr_lab_cli "$name" status --json 2>/dev/null | jq -r '.server.running // false' 2>/dev/null) || running=false + [ "$running" = true ] && continue + fm_herdr_lab_error "lab server for '$name' exited before reporting running" + fm_herdr_lab_server_log_tail "$server_log" + return 1 + fi sleep 0.2 attempt=$((attempt + 1)) done fm_herdr_lab_cancel_provision "$server_pid" fm_herdr_lab_error "lab session '$name' did not report running within $timeout_seconds seconds" + fm_herdr_lab_server_log_tail "$server_log" return 1 } @@ -568,7 +623,10 @@ fm_herdr_lab_name() { # <label> label=${label:0:16} label=${label%-} [ -n "$label" ] || label=lab - printf 'fm-lab-%s-%s-%s\n' "$label" "$$" "$RANDOM" + local name + name=$(printf 'fm-lab-%s-%s-%s' "$label" "$$" "$RANDOM") + fm_herdr_lab_check_socket_path "$name" || return 1 + printf '%s\n' "$name" } fm_herdr_lab_usage() { diff --git a/bin/fm-hermes-process-lib.sh b/bin/fm-hermes-process-lib.sh index e29aacbf66b..c5752d31d88 100644 --- a/bin/fm-hermes-process-lib.sh +++ b/bin/fm-hermes-process-lib.sh @@ -11,7 +11,7 @@ fm_hermes_process_matches() { # <args> [pid] else read -r -a words <<< "$args" fi - for word in "${words[@]}"; do + for word in ${words[@]+"${words[@]}"}; do if [ "$first" = 1 ]; then first=0 case "${word##*/}" in diff --git a/bin/fm-inactive-reconcile.sh b/bin/fm-inactive-reconcile.sh index 1d1c14681ab..6b9ce030f96 100755 --- a/bin/fm-inactive-reconcile.sh +++ b/bin/fm-inactive-reconcile.sh @@ -22,6 +22,13 @@ # pre-validation `done: {summary}` still reads done (the pipeline handoff), # while a CI-ready or direct-PR/local-only done whose head lives only in the # disposable copy reads blocked and is not a terminal inactive outcome. +# Neither path refreshes remote-tracking refs before that gate: it runs under +# the child's meta lock inside a scan budget of at most 30 seconds, which a +# fetch bounded at 30 seconds per remote does not fit. The home's own +# bin/fm-pr-check.sh, which runs when the child's ready report is registered, +# and on each watcher sweep when config/pr-register-scan is on, refreshes the +# child's copy before it refuses, and a later scan publishes the done once that +# fetch shows its head. # A line still being appended (no trailing newline yet) # is left for the next poll. This is what keeps a mate's PR-ready, finding, # and failure outcomes from depending on the mate model appending them @@ -397,12 +404,7 @@ terminal_outcome_pr() { # <state-line> <kind> clean_field "$(state_line_pr "$line")" return 0 fi - value=$(state_line_note "$line") - case "$value" in 'PR '*) value=${value#PR } ;; *) return 0 ;; esac - case "$value" in - *' checks green') value=${value% checks green} ;; - *' published for review') value=${value% published for review} ;; - esac + value=$(fm_dod_ready_note_url "$(state_line_note "$line")") || return 0 fm_pr_url_parse "$value" || return 0 clean_field "$FM_PR_URL" } @@ -431,21 +433,40 @@ terminal_outcome_corroborated() { # <state> <last-status-line> <state-line> [ "$declared" = "$state" ] } -# The task's delivered PR. Recorded meta pr= is the only authoritative source; -# the fallback scrape accepts only a preferred terminal line in a mode's -# ready-signal shape (`done: PR <url>` or `done: PR <url> checks green`, -# optionally carrying an emission-time tag this scrape steps over without -# reading), so a PR a worker merely mentioned in prose is never claimed as the -# delivery. +# The task's delivered PR. Recorded meta pr= is authoritative; without one, +# the fallback scrape accepts only a terminal line in a mode's ready-signal +# shape (`done: PR <url>` or `done: PR <url> checks green`, optionally carrying +# an emission-time tag this scrape steps over without reading), so a PR a +# worker merely mentioned in prose is never claimed as the delivery. +# With the home's config/pr-register-scan opt-in present, the terminal line +# instead counts when its note is a ready report accepted by +# fm_dod_ready_note_url, naming a GitHub-shaped or fm_pr_url_parse forge URL, +# and it also replaces a recorded PR the status log named +# (fm_dod_status_names_pr); a recorded PR the log never named stays. # A scout never delivers a PR, so it never carries one. -pr_for_task() { # <meta> [preferred-line] - local meta=$1 preferred=${2:-} value +pr_for_task() { # <meta> <terminal-line> <status-file> + local meta=$1 terminal=$2 status=$3 value ready= + local github_shape='^https?://[^[:space:])"]+/pull/[0-9]+$' [ "$(meta_field "$meta" kind)" != scout ] || return 0 value=$(meta_field "$meta" pr) - if [ -z "$value" ] && [ -n "$preferred" ]; then - value=$(printf '%s\n' "$preferred" \ - | sed -nE 's|^done( \[at=[^]]*\])?: PR (https?://[^[:space:])"]+/pull/[0-9]+)( checks green)?$|\2|p' \ - | head -1 || true) + if [ ! -e "${FM_CONFIG_OVERRIDE:-$FM_HOME/config}/pr-register-scan" ]; then + if [ -z "$value" ] && [ -n "$terminal" ]; then + value=$(printf '%s\n' "$terminal" \ + | sed -nE 's|^done( \[at=[^]]*\])?: PR (https?://[^[:space:])"]+/pull/[0-9]+)( checks green)?$|\2|p' \ + | head -1 || true) + fi + clean_field "$value" + return 0 + fi + if [ "$(status_line_verb "$terminal")" = "done" ]; then + ready=$(fm_dod_ready_note_url "$(status_line_note "$terminal")") || ready= + if [ -n "$ready" ] && ! fm_pr_url_parse "$ready" \ + && ! [[ "$ready" =~ $github_shape ]]; then + ready= + fi + fi + if [ -n "$ready" ] && { [ -z "$value" ] || fm_dod_status_names_pr "$status" "$value"; }; then + value=$ready fi clean_field "$value" } @@ -527,7 +548,7 @@ report_child_ledger_locked() { # <id> <meta> status="$STATE/$id.status" last=$(child_terminal_ledger_line "$status") || return 0 state=$(status_line_verb "$last") - pr=$(pr_for_task "$meta" "$last") + pr=$(pr_for_task "$meta" "$last" "$status") incarnation=$(meta_incarnation "$meta") fingerprint=$(sha256_text "$incarnation|$id|$state|ledger|$last") if [ "$state" = "done" ] && [ ! -f "$(record_path "$fingerprint" reported)" ] \ diff --git a/bin/fm-lock.sh b/bin/fm-lock.sh index f00754f8d72..22d1c4c72a5 100755 --- a/bin/fm-lock.sh +++ b/bin/fm-lock.sh @@ -13,6 +13,13 @@ # POSIX. Line 1 keeps its whole-line pid format because every other reader # takes the first line as the pid. # +# Acquisition serializes through state/.lock.acquire. Both claim paths use +# fm-wake-lib.sh's stale real-directory recovery for that lock and its .steal +# mutex, and opt into fm_lock_acquire_wait's --refuse-unreapable mode. When +# recovery reports an unremovable stale directory, acquisition exits 1 naming +# the path to inspect for contents or permission problems. Live contention has no new +# deadline. tests/fm-watcher-lock.test.sh covers recovery and refusal. +# # The trusted id itself is recorded beside the lock in state/.lock-session, a # sidecar written only here and only under the claim lock: refreshed on every # confirmed-own acquisition, including the early already-mine exit that waits @@ -213,7 +220,7 @@ publish_lock_session_or_die() { confirm_own_lock() { # <recorded-pid> local recorded pids waited=0 if [ "$CLAIM_LOCK_HELD" -ne 1 ]; then - fm_lock_acquire_wait "$CLAIM_LOCK" + fm_lock_acquire_wait "$CLAIM_LOCK" --refuse-unreapable || exit 1 CLAIM_LOCK_HELD=1 waited=1 fi @@ -266,7 +273,7 @@ if [ "$CLAIM_LOCK_HELD" -ne 1 ]; then echo "error: the prior session's bounded startup sweep is finishing; operate read-only until it releases the fleet lock" >&2 exit 1 fi - fm_lock_acquire_wait "$CLAIM_LOCK" + fm_lock_acquire_wait "$CLAIM_LOCK" --refuse-unreapable || exit 1 fi CLAIM_LOCK_HELD=1 fi diff --git a/bin/fm-omp-process-lib.sh b/bin/fm-omp-process-lib.sh new file mode 100755 index 00000000000..96319e40d6e --- /dev/null +++ b/bin/fm-omp-process-lib.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# omp (Oh My Pi) process identity, shared by harness detection and the +# session-lock ancestry walk. This file has no side effects on source. +# +# omp ships two ways, and the live process looks different for each: +# - the Bun-compiled single binary, whose process name is exactly `omp` +# (verified, omp 18.1.11); +# - the bun global package (`bun install -g @oh-my-pi/pi-coding-agent`), whose +# ~/.bun/bin/omp launcher is a symlink to dist/cli.js with a +# `#!/usr/bin/env bun` shebang, so `ps -o comm=` reports `bun` and +# `ps -o args=` reports `bun <path>/omp` (verified on macOS). +# Match the exact compiled name, or bun whose argv0 is bun and whose very next +# argument is a script named exactly `omp`. +# ps prints argv space-joined, so a script path containing spaces is rebuilt by +# extending that word until it names an existing file; with no such file the +# word alone is judged. +# Anchored throughout, so ompd, comp, and similar names are never this harness. +fm_omp_process_matches() { # <comm> <args> + local comm=${1:-} args=${2:-} i n script + local words=() + case "${comm##*/}" in + omp) return 0 ;; + bun) ;; + *) return 1 ;; + esac + read -r -a words <<< "$args" + n=${#words[@]} + [ "$n" -ge 2 ] && [ "${words[0]##*/}" = bun ] || return 1 + script=${words[1]} + for ((i = 2; i < n; i++)); do + [ -f "$script" ] && break + script+=" ${words[i]}" + done + [ -f "$script" ] || script=${words[1]} + [ "${script##*/}" = omp ] +} diff --git a/bin/fm-on.sh b/bin/fm-on.sh index eff02f7c350..81bc85510c1 100755 --- a/bin/fm-on.sh +++ b/bin/fm-on.sh @@ -22,8 +22,8 @@ # # The SSH alias keeps normal public-key and strict host-key policy in ~/.ssh. # This command explicitly disables agent forwarding, forwarding setup, and -# configured SendEnv patterns. The remote entrypoint executes the selected -# command under an empty environment with only its fixed runtime values. +# configured SendEnv patterns. The shared fm-remote-job-lib.sh header owns +# the remote command's child environment contract. # # ServerAliveInterval/ServerAliveCountMax arm dead-peer detection so a vanished # peer (a reboot, a dropped link) becomes a bounded ssh failure (exit 255) diff --git a/bin/fm-park.sh b/bin/fm-park.sh new file mode 100755 index 00000000000..8441fd8e6d8 --- /dev/null +++ b/bin/fm-park.sh @@ -0,0 +1,938 @@ +#!/usr/bin/env bash +# fm-park.sh - park a ship lane whose PR or MR only awaits human review, watch +# it while its agent is stopped, relaunch a fresh worker on review activity, +# and clean the lane up once the merge poll sees it merged. +# +# Usage: +# fm-park.sh gate <task-id> [--expect-head <sha>] +# fm-park.sh park <task-id> +# fm-park.sh resume <task-id> --trigger <text> +# fm-park.sh land <task-id> [--if-eligible] [--detach] +# fm-park.sh tick +# fm-park.sh list +# fm-park.sh run <task-id> park|resume|land [<args>...] (internal) +# +# gate is read-only: one fresh forge read of the task's recorded pr= (GitHub +# through one gh GraphQL read, GitLab through glab api for the merge request and +# all pages of its discussions), never a cached verdict. It prints "clean head=<sha>" and +# exits 0, or prints one "unclean: <reason>" line per failing condition and +# exits 3; an unreadable forge, an unsupported provider, or a missing pr= exits 1 +# and is never clean. Clean means ready to merge except for a human approval: +# - open and not a draft; +# - no conflict, not behind its base, mergeability computed (GitHub mergeable +# MERGEABLE with mergeStateStatus CLEAN, BLOCKED or HAS_HOOKS; GitLab +# has_conflicts false with detailed_merge_status mergeable or not_approved); +# - CI green on the current head (GitHub statusCheckRollup SUCCESS on the +# head commit; GitLab head_pipeline success with its sha the MR's sha); no +# CI reported on the current head is not green; on GitHub every check the +# base branch requires has also reported on the head, read with +# bin/fm-pr-merge.sh's required-check reading (bin/fm-pr-lib.sh), because +# the rollup holds only checks that reported; an unreadable requirement +# is unclean; +# - no changes requested (GitHub reviewDecision; GitLab requested_changes); +# - zero unresolved review threads, all of them read (a failed GitLab +# discussions read counts as not all read); +# - automated reviewers settled: every reviewer bot that has ever posted on +# the PR has submitted a review bound to the current head (a comment, or a +# comment edited later, does not prove its review finished), or the quiet +# window has elapsed since the anchor. The push time is the +# earliest check suite created on the head commit on GitHub (its committer +# time only when no check suite exists) and the head pipeline's creation on +# GitLab; the anchor is the later of that and the newest CI completion on +# the head, so the window never starts before CI finished. A reviewer bot is a GitHub Bot account or +# [bot] login, a GitLab group or project access-token bot +# (group_<n>_bot_..., how CodeRabbit appears on gitlab.com), or a login +# listed in config/contribution-review-bots. The window is +# FM_PARK_QUIET_SECS seconds (default 900), so a PR no bot touched parks +# only after it, never the moment CI turns green. +# With --expect-head, a head other than <sha> is unclean too. +# +# park re-establishes everything itself and accepts no verdict as input. It +# refuses unless: the task is a ship with pr= and an armed merge poll +# (state/<id>.check.sh); its backend can prove a stop and launch a replacement; +# its local copy is clean with HEAD equal to the forge +# head; bin/fm-crew-state.sh reads done or paused; the agent is idle; the +# steering inbox holds no unhandled message; and the gate is clean. It then +# durably leases a Treehouse slot to the task (treehouse lease, state-only, so a +# later get never hands the copy out and prune never removes it even if the +# endpoint shell dies; bin/fm-teardown.sh's treehouse return releases it), +# evaluates the gate again with --expect-head immediately before the stop, +# acquires the same inbox-delivery lock as fm-send, checks once more that the +# agent is idle and the inbox empty, writes +# state/<id>.review-park marked pending=1, stops the agent with +# bin/fm-control.sh exit (endpoint, branch, PR, task record, brief, and copy all +# stay), rewrites the record without pending once the stop is confirmed, and +# appends a declared paused: line to the task's status log (reported upward in a +# secondmate home). A refusal before the pending record leaves no record and the +# agent running; a lease already taken stays until teardown, which is harmless. +# A pending record is watched, relaunches on feedback only once its stop is +# confirmed, because a relaunch replaces the agent, and crew state reads it as +# parking. Each tick promotes a pending record to parked once its agent is +# confirmed dead and drops it only if still alive after the stop timeout. +# +# resume writes a note naming the PR, the parked head and the trigger, then runs +# bin/fm-control.sh relaunch --note-file, which starts a fresh agent of the +# recorded adapter in the same copy on the same branch with its original brief +# plus that note. A working: line superseding the declared wait is appended just +# before the launch, so the fresh worker's own reports always follow it. Success +# removes the park record; failure keeps it with resume_error=, which crew state +# reports as blocked and which stops tick retrying until resume runs by hand. +# +# land waits for any other park action on the lane to finish, runs +# bin/fm-teardown.sh <task-id> (never --force), so every teardown gate still +# applies, and prints one outcome line naming the lane and its PR. With +# --if-eligible it does nothing unless the lane is eligible for merge-time +# cleanup (land_eligible below: parked, or config/auto-land present). With +# --detach it starts the land as a detached action (below), prints "automatic +# cleanup started", and returns at once; with --queue it only saves the request. +# The request (state/<id>.review-park-land) records the merged PR as pr=, and +# a land holding the lane lock drops a request whose pr= is not the lane's +# current pr= without acting, so a stale request never tears down a lane that +# has moved on to another PR. Only the detached cleanup removes its request, +# after its outcome report is saved (teardown leaves it), so a cleanup killed +# mid-teardown is retried; a request whose task record is gone is dropped by +# the tick. bin/fm-watch.sh runs +# land --if-eligible --detach for every merged lane from its merged-poll path +# after the merge outcome is recorded, also when bin/fm-pr-merge.sh recorded it +# first, which makes the watcher the one owner of a merged lane's teardown. +# +# tick is what bin/fm-watch.sh runs every check interval. With +# neither config/auto-land nor config/park-on-review, and no park record, it exits +# after those file tests. Otherwise, holding state/.review-park-tick.lock (a +# held lock exits silently): +# - pending merged-lane cleanup requests (state/<id>.review-park-land) are +# retried when the lane lock is free; outcome reports survive task teardown; +# - the outcome lines detached actions published since the last tick are +# collected with this tick's close lines; +# - each parked lane gets one forge read; a trigger relaunches it, a merge is +# left to the merge poll, and a close without merge is reported once; +# - with config/auto-land or config/park-on-review, each other idle ship lane +# with pr= and an armed merge poll gets one forge read: +# a close without merge is reported once (marker +# state/.review-park-closed-<id>), and with config/park-on-review a clean +# gate parks it. +# Relaunch triggers on a parked lane: the head moved, checks failing, changes +# requested, a merge conflict, any unresolved review thread, or any new +# comment (inline review-thread replies included), review or note created or +# edited after the park (an approval with +# no body is not a comment). A branch merely behind its base is not a trigger. It +# records its outcome and close lines as one check park-on-review wake before +# it removes the reports they came from or marks a close as reported, then +# prints them for the watcher to ring; when the wake cannot be recorded it +# removes and prints nothing, so the next tick retries. A quiet tick prints +# nothing. +# +# A lane action started by tick (park, resume) or by the merge poll (land) runs +# detached, as fm-park.sh run, in its own process group with stdio closed, so it +# outlives the watcher iteration that started it and no tick or watcher budget +# can cut it short. It takes the lane's lock itself (state/.review-park-<id>.lock, +# held by every park, resume and land), so a lane runs one action at a time and +# tick skips a lane whose lock is held. Its stdout outcome line, if any, is +# published as state/.review-park-report-<id>-<epoch>-<pid> (a park refusal +# prints none), and the next tick reports it. +# Forge reads are bounded by FM_PARK_READ_SECS (default 15). Each lifecycle +# call an action makes (bin/fm-control.sh exit or relaunch, bin/fm-teardown.sh) +# is bounded by FM_PARK_ACTION_SECS (default 900), a hang backstop well above +# the bounded waits those owners keep themselves. A whole tick holds only reads +# and action starts and ends within FM_PARK_TICK_SECS (default 180), well under +# the watcher's stale grace: every read is cut to what the tick has left. A tick +# that runs out of time saves the lane it did not reach +# (state/.review-park-tick-next) and the next tick starts there. +# Parked lanes use only the first half of the tick, reserving the rest for +# unparked lanes. +# +# list prints each parked lane: id, PR, parked head, and when it parked. +# +# The record is key=value lines (pr, head, parked_at, slot_leased, pending +# until the stop is confirmed, and resume_error after a failed relaunch), mode +# 0600, published by rename. +# GraphQL and jq programs below are literal, not shell expressions. +# shellcheck disable=SC2016 +set -eu + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-$FM_ROOT}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" +export FM_HOME FM_STATE_OVERRIDE="$STATE" FM_CONFIG_OVERRIDE="$CONFIG" + +# shellcheck source=bin/fm-pr-lib.sh +. "$SCRIPT_DIR/fm-pr-lib.sh" +# shellcheck source=bin/fm-wake-lib.sh +. "$SCRIPT_DIR/fm-wake-lib.sh" +# shellcheck source=bin/fm-task-inbox-lib.sh +. "$SCRIPT_DIR/fm-task-inbox-lib.sh" +# shellcheck source=bin/fm-backend.sh +. "$SCRIPT_DIR/fm-backend.sh" +# shellcheck source=bin/fm-control-lib.sh +. "$SCRIPT_DIR/fm-control-lib.sh" +# shellcheck source=bin/fm-busy-lib.sh +. "$SCRIPT_DIR/fm-busy-lib.sh" +# shellcheck source=bin/fm-timeout-lib.sh +. "$SCRIPT_DIR/fm-timeout-lib.sh" +# shellcheck source=bin/fm-parent-channel-lib.sh +. "$SCRIPT_DIR/fm-parent-channel-lib.sh" + +# The lifecycle owners this script drives; the overrides exist for tests. +CONTROL_BIN="${FM_PARK_CONTROL_BIN:-$SCRIPT_DIR/fm-control.sh}" +TEARDOWN_BIN="${FM_PARK_TEARDOWN_BIN:-$SCRIPT_DIR/fm-teardown.sh}" +CREW_STATE_BIN="${FM_PARK_CREW_STATE_BIN:-$SCRIPT_DIR/fm-crew-state.sh}" +QUIET_SECS=${FM_PARK_QUIET_SECS:-900} +READ_SECS=${FM_PARK_READ_SECS:-15} +ACTION_SECS=${FM_PARK_ACTION_SECS:-900} +TICK_SECS=${FM_PARK_TICK_SECS:-180} +for v in "$QUIET_SECS" "$READ_SECS" "$ACTION_SECS" "$TICK_SECS"; do + case "$v" in ''|*[!0-9]*) printf 'fm-park: timing overrides must be whole seconds\n' >&2; exit 2 ;; esac +done + +usage() { sed -n '2,/^set -eu$/s/^# \{0,1\}//p' "$0"; } +die() { printf 'fm-park: %s\n' "$*" >&2; exit "${2:-1}"; } + +LOCKS_HELD= +release_locks() { + local lock + for lock in $LOCKS_HELD; do fm_lock_release "$lock" || true; done + LOCKS_HELD= +} +cleanup() { + release_locks + [ -z "${PARK_TMP:-}" ] || rm -rf -- "$PARK_TMP" +} +trap cleanup EXIT +trap 'exit 1' HUP INT TERM +PARK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/fm-park.XXXXXX") || die "cannot create a temporary directory" + +lock_try() { # <lock-path> + fm_lock_try_acquire "$1" || return 1 + LOCKS_HELD="$LOCKS_HELD $1" +} + +# The epoch second a tick must end by; empty outside tick. +DEADLINE= +# step_secs <secs> prints that bound cut to what the tick has left, and fails +# once nothing is left. +step_secs() { # <secs> + local left + [ -n "$DEADLINE" ] || { printf '%s\n' "$1"; return 0; } + left=$(( DEADLINE - $(date +%s) )) + [ "$left" -gt 0 ] || return 1 + if [ "$left" -lt "$1" ]; then printf '%s\n' "$left"; else printf '%s\n' "$1"; fi +} +lane_lock() { printf '%s/.review-park-%s.lock' "$STATE" "$1"; } + +# Succeeds while another park action holds the lane. +lane_busy() { # <id> + local lock + lock=$(lane_lock "$1") + fm_lock_try_acquire "$lock" || return 0 + fm_lock_release "$lock" || true + return 1 +} + +# Starts fm-park.sh run <id> <verb> <args...> detached: its own process group +# (so a bound on the caller's group cannot reach it), nohup, and stdio closed +# (so it holds no pipe of the watcher open), in a subshell that returns at once. +action_start() { # <id> <verb> <args...> + ( set -m 2>/dev/null || true + nohup "$SCRIPT_DIR/fm-park.sh" run "$@" </dev/null >/dev/null 2>&1 & ) +} + +# Publishes one outcome line for the next tick to report. +report_write() { # <id> <line> + local tmp + tmp=$(mktemp "$STATE/.review-park.XXXXXX") || return 1 + if ! { printf '%s\n' "$2" > "$tmp" && mv -f -- "$tmp" "$STATE/.review-park-report-$1-$(date +%s)-$$"; }; then + rm -f -- "$tmp" + return 1 + fi +} + +agent_idle() { # <id> + local verdict + verdict=$(fm_busy_classify_meta "$STATE/$1.meta" "$1" "$STATE" 2>/dev/null || true) + case "$verdict" in idle\ *) return 0 ;; *) return 1 ;; esac +} + +# An idle agent with no instruction waiting for it; prints the refusal and +# returns 1 otherwise. +agent_quiet() { # <id> + if find "$STATE/$1.inbox" -maxdepth 1 -name '*.msg' 2>/dev/null | grep -q .; then + echo "an instruction is waiting in the steering inbox"; return 1 + fi + agent_idle "$1" || { echo "the agent is not idle ($(fm_busy_classify_meta "$STATE/$1.meta" "$1" "$STATE" 2>/dev/null || echo unknown))"; return 1; } +} + +task_id_ok() { [ -n "$1" ] && fm_pr_task_id_valid "$1"; } + +record_path() { printf '%s/%s.review-park' "$STATE" "$1"; } +record_get() { fm_meta_get "$(record_path "$1")" "$2"; } + +record_write() { # <id> <key=value>... + local id=$1 tmp dest + shift + dest=$(record_path "$id") + tmp=$(mktemp "$STATE/.review-park.XXXXXX") || return 1 + if ! { printf '%s\n' "$@" > "$tmp" && chmod 0600 "$tmp" && mv -f -- "$tmp" "$dest"; }; then + rm -f -- "$tmp" + return 1 + fi +} + +status_append() { # <id> <line> + printf '%s\n' "$2" >> "$STATE/$1.status" +} + +review_bots_json() { + [ -f "$CONFIG/contribution-review-bots" ] || { printf '[]\n'; return 0; } + jq -Rn '[inputs | gsub("^\\s+|\\s+$"; "") | select(length > 0 and (startswith("#") | not))]' \ + "$CONFIG/contribution-review-bots" +} + +# --- forge snapshot --------------------------------------------------------- +# snapshot <pr-url> prints one normalized JSON object, the only input the gate +# and trigger programs read, so both providers share one decision procedure. + +GITHUB_QUERY='query($owner:String!,$repo:String!,$number:Int!,$endCursor:String){repository(owner:$owner,name:$repo){pullRequest(number:$number){ + state isDraft headRefOid baseRefName mergeable mergeStateStatus reviewDecision + commits(last:1){nodes{commit{oid committedDate checkSuites(first:100){nodes{createdAt}} statusCheckRollup{state contexts(first:100){nodes{__typename ... on CheckRun{name completedAt} ... on StatusContext{context createdAt}}}}}}} + reviewThreads(first:100){totalCount nodes{isResolved comments(last:100){totalCount nodes{author{login __typename} createdAt updatedAt}}}} + reviews(last:100){totalCount nodes{author{login __typename} state body submittedAt updatedAt commit{oid}}} + comments(first:100,after:$endCursor){totalCount nodes{author{login __typename} createdAt updatedAt} pageInfo{hasNextPage endCursor}} +}}}' + +GITHUB_NORMALIZE=' + .data.repository.pullRequest as $p + | if ($p | type) != "object" then error("no pull request") else . end + | ($p.commits.nodes[0].commit) as $c + | if ($c.oid // "") != $p.headRefOid then error("head commit read is incoherent") else . end + | ($c.statusCheckRollup) as $r + | { + provider: "github", + state: ($p.state | ascii_downcase), + draft: ($p.isDraft == true), + head: $p.headRefOid, + merge_status: ($p.mergeStateStatus // "UNKNOWN"), + conflict: (if $p.mergeable == "CONFLICTING" or $p.mergeStateStatus == "DIRTY" then "conflict" + elif $p.mergeStateStatus == "BEHIND" then "behind" + elif $p.mergeable == "UNKNOWN" or $p.mergeStateStatus == "UNKNOWN" then "unknown" + else "none" end), + blocked_other: (if ($p.mergeStateStatus // "UNKNOWN") | IN("CLEAN","BLOCKED","HAS_HOOKS","DIRTY","BEHIND","UNKNOWN","DRAFT","UNSTABLE") + then null else $p.mergeStateStatus end), + ci: (if $r == null then "none" elif $r.state == "SUCCESS" then "success" + elif ($r.state | IN("FAILURE","ERROR")) then "failure" else "pending" end), + ci_done_at: ([$r.contexts.nodes[]? | (.completedAt // .createdAt)] | map(select(. != null)) | max), + pushed_at: ([$c.checkSuites.nodes[]?.createdAt | select(. != null)] | min // $c.committedDate), + changes_requested: ($p.reviewDecision == "CHANGES_REQUESTED"), + threads_unresolved: ([$p.reviewThreads.nodes[] | select(.isResolved | not)] | length), + threads_complete: ($p.reviewThreads.totalCount <= ($p.reviewThreads.nodes | length)), + comments_complete: (($p.comments.totalCount // -1) <= ($p.comments.nodes | length) + and $p.comments.pageInfo.hasNextPage == false + and ($p.reviews.totalCount // -1) <= ($p.reviews.nodes | length) + and ([$p.reviewThreads.nodes[]? | (.comments.totalCount // -1) <= (.comments.nodes | length)] | all)), + activity: ( + [$p.comments.nodes[] | {author: (.author.login // "ghost"), bot: (.author.__typename == "Bot"), + at: .createdAt, touched: (.updatedAt // .createdAt), on_head: false, approval_only: false}] + + [$p.reviews.nodes[] | {author: (.author.login // "ghost"), bot: (.author.__typename == "Bot"), + at: .submittedAt, touched: (.updatedAt // .submittedAt), on_head: (.commit.oid == $p.headRefOid), + approval_only: (.state == "APPROVED" and ((.body // "") | length) == 0)}] + + [$p.reviewThreads.nodes[].comments.nodes[]? | {author: (.author.login // "ghost"), bot: (.author.__typename == "Bot"), + at: .createdAt, touched: (.updatedAt // .createdAt), on_head: false, approval_only: false}] + | map(select(.at != null))) + }' + +# The third input is false when the discussions read failed, so threads_complete +# reports that the same way a truncated GitHub read does. +GITLAB_NORMALIZE=' + .[0] as $mr | .[1] as $d + | if ($mr | type) != "object" or ($d | type) != "array" then error("invalid merge request read") else . end + | ($mr.detailed_merge_status // "unchecked") as $s + | ($mr.head_pipeline | if . != null and .sha == $mr.sha then . else null end) as $hp + | ($hp.status // null) as $ps + | { + provider: "gitlab", + state: (if $mr.state == "opened" then "open" else $mr.state end), + draft: (($mr.draft // $mr.work_in_progress // false) == true), + head: $mr.sha, + merge_status: $s, + conflict: (if $mr.has_conflicts == true or ($s | IN("conflict","broken_status")) then "conflict" + elif $s == "need_rebase" then "behind" + elif ($s | IN("checking","unchecked","preparing","approvals_syncing")) then "unknown" + else "none" end), + blocked_other: (if $s | IN("mergeable","not_approved","conflict","broken_status","need_rebase","checking", + "unchecked","preparing","approvals_syncing","requested_changes","draft_status", + "ci_must_pass","ci_still_running","discussions_not_resolved","not_open") + then null else $s end), + ci: (if $ps == null then "none" elif $ps == "success" then "success" + elif ($ps | IN("failed","canceled")) then "failure" else "pending" end), + ci_done_at: ($hp.finished_at // null), + pushed_at: ($hp.created_at // null), + changes_requested: ($s == "requested_changes"), + threads_unresolved: ([$d[].notes[]? | select(.resolvable == true and .resolved != true)] | length), + threads_complete: (.[2] == true), + comments_complete: true, + activity: [$d[] | .notes[]? | select(.system != true) + | {author: (.author.username // "ghost"), bot: false, at: .created_at, + touched: (.updated_at // .created_at), on_head: false, approval_only: false}] + }' + +# The required checks of the PR's base branch that have not reported on its +# head, as a JSON array, read with the same required-check reading +# bin/fm-pr-merge.sh does; fails when any requirement cannot be read. +github_required_unreported() { # <graphql-json> + local raw=$1 base head rollup secs missing + base=$(printf '%s' "$raw" | jq -er '.data.repository.pullRequest.baseRefName | select(type == "string" and length > 0)') || return 1 + head=$(printf '%s' "$raw" | jq -r '.data.repository.pullRequest.headRefOid') + rollup=$(printf '%s' "$raw" | jq -c '{statusCheckRollup: (.data.repository.pullRequest.commits.nodes[0].commit.statusCheckRollup.contexts.nodes // [])}') || return 1 + secs=$(step_secs "$READ_SECS") || return 1 + # shellcheck disable=SC2016 + missing=$(fm_run_timed "$secs" bash -c ' + . "$1/fm-pr-lib.sh" + PR_OWNER=$2 PR_REPO=$3 + github_read_required_contexts "$4" && github_read_required_producers "$5" \ + && github_required_checks_missing "$6" "$FM_PR_GITHUB_REQUIRED" "$FM_PR_GITHUB_PRODUCERS"' \ + _ "$SCRIPT_DIR" "$FM_PR_OWNER" "$FM_PR_REPO" "$base" "$head" "$rollup" 2>/dev/null) || return 1 + printf '%s' "$missing" | jq -Rsc 'split("\n") | map(select(length > 0))' +} + +# snapshot <pr-url> [--required]: --required adds required_missing, the GitHub +# required checks not yet reported on the head (null when unreadable), which +# only the gate reads. +snapshot() { # <pr-url> [--required] + local url=$1 raw mr discussions enc secs snap missing complete + fm_pr_url_parse "$url" || return 1 + case "$FM_PR_PROVIDER" in + github) + command -v gh >/dev/null 2>&1 || return 1 + fm_pr_github_host_export "$FM_PR_HOST" + secs=$(step_secs "$READ_SECS") || return 1 + raw=$(fm_run_timed "$secs" gh api graphql --paginate --slurp -f query="$GITHUB_QUERY" \ + -F "owner=${FM_PR_PATH%%/*}" -F "repo=${FM_PR_PATH#*/}" -F "number=$FM_PR_NUMBER" 2>/dev/null) || return 1 + raw=$(printf '%s' "$raw" | jq -ce ' + select(type == "array" and length > 0) + | select(all(.[]; .errors == null and .data.repository.pullRequest != null)) + | map(.data.repository.pullRequest) as $pages + | select(all($pages[]; .headRefOid == $pages[0].headRefOid)) + | {data:{repository:{pullRequest:($pages[0] + {comments:($pages[0].comments + { + totalCount:$pages[-1].comments.totalCount, + pageInfo:$pages[-1].comments.pageInfo, + nodes:[$pages[] | .comments.nodes[]?]})})}}}' 2>/dev/null) || return 1 + snap=$(printf '%s' "$raw" | jq -c "$GITHUB_NORMALIZE" 2>/dev/null) || return 1 + [ "${2-}" = --required ] || { printf '%s\n' "$snap"; return 0; } + missing=$(github_required_unreported "$raw") || missing=null + printf '%s' "$snap" | jq -c --argjson m "$missing" '. + {required_missing: $m}' + ;; + gitlab) + command -v glab >/dev/null 2>&1 || return 1 + enc=$(jq -rn --arg p "$FM_PR_PATH" '$p | @uri') + secs=$(step_secs "$READ_SECS") || return 1 + mr=$(fm_run_timed "$secs" glab api --hostname "$FM_PR_HOST" \ + "projects/$enc/merge_requests/$FM_PR_NUMBER" 2>/dev/null) || return 1 + secs=$(step_secs "$READ_SECS") || return 1 + # A failed merge request read above means the forge is unreadable and + # retries; a failed discussions read here reports incomplete threads. + # --paginate prints one JSON array per page; together they are one list. + if discussions=$(fm_run_timed "$secs" glab api --paginate --hostname "$FM_PR_HOST" \ + "projects/$enc/merge_requests/$FM_PR_NUMBER/discussions?per_page=100" 2>/dev/null) \ + && [ -n "$discussions" ] \ + && discussions=$(printf '%s\n' "$discussions" \ + | jq -cs '[.[] | if type == "array" then .[] else error("invalid discussions page") end]' 2>/dev/null); then + complete=true + else + discussions='[]' + complete=false + fi + printf '%s\n%s\n%s\n' "$mr" "$discussions" "$complete" | jq -cs "$GITLAB_NORMALIZE" 2>/dev/null + ;; + *) return 1 ;; + esac +} + +JQ_TIME='def ts: if . == null then null else (try (sub("\\.[0-9]+"; "") | sub("\\+00:00$"; "Z") | fromdateiso8601) catch null) end;' + +GATE_PROGRAM="$JQ_TIME"' + def isbot: .bot or (.author | test("\\[bot\\]$")) or (.author | test("^(group|project)_[0-9]+_bot_")) + or (.author as $a | any($bots[]; . == $a)); + . as $s + | ($s.pushed_at | ts) as $pushed + | ([($s.ci_done_at | ts), $pushed] | map(select(. != null)) | max) as $anchor + | ([$s.activity[] | select(isbot) | .author] | unique) as $seen + | (if $anchor == null then null else ($anchor + $quiet - $now) end) as $left + | ($anchor != null and ($left <= 0 or (($seen | length) > 0 + and all($seen[]; . as $b | any($s.activity[]; .author == $b and .on_head))))) as $settled + | [ + (if $s.state != "open" then "the PR is \($s.state)" else empty end), + (if $s.draft then "the PR is a draft" else empty end), + (if $expect != "" and $s.head != $expect then "the head moved from \($expect[0:12]) to \($s.head[0:12])" else empty end), + (if $s.conflict == "conflict" then "the PR has merge conflicts" + elif $s.conflict == "behind" then "the branch is behind its base" + elif $s.conflict == "unknown" then "mergeability is not computed yet (\($s.merge_status))" else empty end), + (if $s.blocked_other != null then "the forge reports merge status \($s.blocked_other)" else empty end), + (if $s.ci != "success" then "CI is \($s.ci) on head \($s.head[0:12])" else empty end), + (if ($s | has("required_missing") | not) then empty + elif $s.required_missing == null then "the required checks of the base branch could not be read" + elif ($s.required_missing | length) > 0 then "required check(s) \($s.required_missing | join(", ")) have not reported on head \($s.head[0:12])" + else empty end), + (if $s.changes_requested then "changes are requested" else empty end), + (if $s.threads_unresolved > 0 then "\($s.threads_unresolved) review thread(s) unresolved" else empty end), + (if ($s.threads_complete | not) then "review threads could not all be read" else empty end), + (if ($s.comments_complete | not) then "review comments could not all be read" else empty end), + (if $settled or $s.ci != "success" then empty + elif ($seen | length) > 0 then "automated reviewers (\($seen | join(", "))) have not submitted a review of this head; quiet window \($left)s left" + else "no automated reviewer has submitted a review of this head; quiet window \($left)s left" end) + ] + | {clean: (length == 0), head: $s.head, reasons: .}' + +TRIGGER_PROGRAM="$JQ_TIME"' + . as $s + | { + state: $s.state, + triggers: [ + (if $s.head != $head then "the head moved to \($s.head[0:12])" else empty end), + (if $s.ci == "failure" then "checks are failing on \($s.head[0:12])" else empty end), + (if $s.changes_requested then "changes were requested" else empty end), + (if $s.conflict == "conflict" then "the PR has merge conflicts" else empty end), + (if $s.threads_unresolved > 0 then "\($s.threads_unresolved) review thread(s) unresolved" else empty end), + (if ($s.threads_complete | not) then "review threads could not all be read" else empty end), + (if ($s.comments_complete | not) then "review comments could not all be read" else empty end), + ([$s.activity[] | select(([.at, .touched] | map(ts // 0) | max) >= $parked and (.approval_only | not)) | .author] | unique + | if length > 0 then "new comments from \(join(", "))" else empty end) + ] + }' + +# gate_eval <id> <expect-head> prints the gate result JSON; returns 1 when the +# forge cannot be read. +gate_eval() { # <id> <expect-head> + local id=$1 expect=$2 pr snap bots + pr=$(fm_meta_get "$STATE/$id.meta" pr) + [ -n "$pr" ] || return 1 + snap=$(snapshot "$pr" --required) || return 1 + [ -n "$snap" ] || return 1 + bots=$(review_bots_json) || return 1 + printf '%s' "$snap" | jq -c --argjson now "$(date +%s)" --argjson quiet "$QUIET_SECS" \ + --arg expect "$expect" --argjson bots "$bots" "$GATE_PROGRAM" +} + +gate_print() { # <gate-json>; returns 0 clean, 3 unclean + if [ "$(printf '%s' "$1" | jq -r .clean)" = true ]; then + printf 'clean head=%s\n' "$(printf '%s' "$1" | jq -r .head)" + return 0 + fi + printf '%s' "$1" | jq -r '.reasons[] | "unclean: " + .' + return 3 +} + +cmd_gate() { + local id=${1-} expect='' result + task_id_ok "$id" || die "usage: fm-park.sh gate <task-id> [--expect-head <sha>]" 2 + shift + if [ "${1-}" = --expect-head ]; then + expect=${2-} + fm_pr_head_valid "$expect" || die "--expect-head needs a full commit sha" 2 + fi + [ -f "$STATE/$id.meta" ] || die "no task record for $id" + result=$(gate_eval "$id" "$expect") || die "could not read the PR for $id from its forge" + gate_print "$result" +} + +# --- park ------------------------------------------------------------------ + +# Local preconditions that make a park reversible: an armed merge poll, a clean +# copy at the forge head, a worker that finished its turn and reported, and no +# instruction waiting for it. Prints the first refusal and returns 1. +park_preflight() { # <id> + local id=$1 meta kind pr wt crew secs backend + meta="$STATE/$id.meta" + [ -f "$meta" ] || { echo "no task record"; return 1; } + backend=$(fm_backend_of_meta "$meta") + if ! fm_control_backend_state_verified "$backend" || + ! fm_control_backend_exit_supported "$backend" || + ! fm_control_backend_relaunch_supported "$backend"; then + echo "the ${backend:-unknown} backend cannot stop and relaunch the agent"; return 1 + fi + kind=$(fm_meta_get "$meta" kind) + [ "${kind:-ship}" = ship ] || { echo "only a ship lane can park (kind=$kind)"; return 1; } + pr=$(fm_meta_get "$meta" pr) + [ -n "$pr" ] || { echo "no PR is recorded (pr=)"; return 1; } + if ! fm_pr_url_parse "$pr" || { [ "$FM_PR_PROVIDER" != github ] && [ "$FM_PR_PROVIDER" != gitlab ]; }; then + echo "parking supports GitHub pull requests and GitLab merge requests only"; return 1 + fi + [ -f "$STATE/$id.check.sh" ] || { echo "no merge poll is armed (run bin/fm-pr-check.sh first)"; return 1; } + [ ! -e "$(record_path "$id")" ] || { echo "already parked"; return 1; } + wt=$(fm_meta_get "$meta" worktree) + [ -n "$wt" ] && [ -d "$wt" ] || { echo "the local copy is missing"; return 1; } + [ -z "$(git -C "$wt" status --porcelain 2>/dev/null)" ] || { echo "the local copy has uncommitted changes"; return 1; } + agent_quiet "$id" || return 1 + secs=$(step_secs 60) || { echo "the tick ran out of time"; return 1; } + crew=$(FM_HOME=$FM_HOME FM_STATE_OVERRIDE=$STATE fm_run_timed "$secs" "$CREW_STATE_BIN" "$id" 2>/dev/null | head -1 || true) + case "$crew" in + 'state: done '*|'state: paused '*) ;; + *) echo "the worker has not finished at a ready report (${crew:-crew state unreadable})"; return 1 ;; + esac +} + +# Durable Treehouse lease for a pool slot; prints yes when taken, no when the +# copy is not a Treehouse slot, and fails when a slot cannot be leased. +slot_lease() { # <id> + local id=$1 meta project wt slot_dir pool slot leased holder secs + local -a root=() + meta="$STATE/$id.meta" + project=$(fm_meta_get "$meta" project) + wt=$(fm_meta_get "$meta" worktree) + if [ -z "$project" ] || ! fm_treehouse_pool_slot "$project" "$wt"; then + printf 'no\n' + return 0 + fi + # treehouse lease names a slot, resolved against the pool of the repository + # it runs in, so a pool under a non-default root (<root>/.treehouse/<pool>) + # is addressed through that root; the printed path is verified either way. + slot_dir=$(dirname "$(CDPATH='' cd -- "$wt" && pwd -P)") + slot=$(basename "$slot_dir") + pool=$(dirname "$slot_dir") + [ "$(basename "$(dirname "$pool")")" != .treehouse ] || root=(--root "$(dirname "$(dirname "$pool")")") + # A lane parked before and relaunched since still holds its lease, which + # treehouse lease refuses to take again, so this task's own lease counts. + secs=$(step_secs 60) || return 1 + holder=$( (cd "$project" && fm_run_timed "$secs" treehouse status --json "${root[@]}") 2>/dev/null \ + | jq -r --arg p "$(CDPATH='' cd -- "$wt" && pwd -P)" \ + '[.[] | select(.status == "leased" and .path == $p) | .lease_holder] | first // empty' 2>/dev/null) || holder= + if [ "$holder" = "$id" ]; then + printf 'yes\n' + return 0 + fi + secs=$(step_secs 60) || return 1 + leased=$( (cd "$project" && fm_run_timed "$secs" treehouse lease "$slot" --lease-holder "$id" "${root[@]}") 2>/dev/null) || return 1 + [ "$(CDPATH='' cd -- "$leased" 2>/dev/null && pwd -P)" = "$(CDPATH='' cd -- "$wt" && pwd -P)" ] || return 1 + printf 'yes\n' +} + +cmd_park() { + local id=${1-} why result head local_head pr lease parked_at out rc=0 line inbox_lock + task_id_ok "$id" || die "usage: fm-park.sh park <task-id>" 2 + lock_try "$(lane_lock "$id")" || die "another park action holds $id" + why=$(park_preflight "$id") || die "refusing to park $id: $why" + result=$(gate_eval "$id" "") || die "refusing to park $id: its PR could not be read from the forge" + if ! gate_print "$result" > "$PARK_TMP/gate" ; then + die "refusing to park $id: $(sed 's/^unclean: //' "$PARK_TMP/gate" | paste -sd ';' - | sed 's/;/; /g')" 3 + fi + head=$(printf '%s' "$result" | jq -r .head) + local_head=$(git -C "$(fm_meta_get "$STATE/$id.meta" worktree)" rev-parse HEAD 2>/dev/null || true) + [ "$local_head" = "$head" ] || die "refusing to park $id: the local copy is at ${local_head:0:12}, the PR head is ${head:0:12}" + pr=$(fm_meta_get "$STATE/$id.meta" pr) + lease=$(slot_lease "$id") || die "refusing to park $id: its pool slot could not be leased durably" + # Review activity after this moment relaunches the lane once it is parked. + parked_at=$(date +%s) + # The verdict above is already history: read the forge once more, bound to + # the head just verified, immediately before anything is stopped. + if ! result=$(gate_eval "$id" "$head") || ! gate_print "$result" > "$PARK_TMP/gate"; then + die "refusing to park $id: the re-check before stopping was not clean ($(sed 's/^unclean: //' "$PARK_TMP/gate" 2>/dev/null | paste -sd ';' - | sed 's/;/; /g'))" 3 + fi + inbox_lock=$(fm_task_inbox_delivery_lock_path "$STATE" "$id") + fm_task_inbox_lock_acquire "$inbox_lock" || die "refusing to park $id: its inbox delivery lock could not be acquired" + LOCKS_HELD="$LOCKS_HELD $inbox_lock" + # fm-send takes the same lock before writing, so no new instruction can + # land between this inbox check and the stop. + if ! why=$(agent_quiet "$id"); then + die "refusing to park $id: $why" + fi + # Written before the stop and marked pending, so a park that dies after the + # stop still leaves the lane watched; the next tick settles it. + record_write "$id" "pr=$pr" "head=$head" "parked_at=$parked_at" "slot_leased=$lease" "pending=1" \ + || die "refusing to park $id: its park record could not be written" + out=$(FM_HOME=$FM_HOME fm_run_timed "$ACTION_SECS" "$CONTROL_BIN" "$id" exit 2>&1) || rc=$? + if [ "$rc" -ne 0 ]; then + die "refusing to park $id: the agent could not be stopped: $(printf '%s\n' "$out" | tail -1); the next tick settles its pending park record" + fi + record_write "$id" "pr=$pr" "head=$head" "parked_at=$parked_at" "slot_leased=$lease" \ + || die "$id: the agent was stopped but its park record could not be marked parked; the next tick settles it" + line="paused [at=$(date +%s)]: parked on review: PR $pr clean at ${head:0:12}; agent stopped, relaunches on review activity" + status_append "$id" "$line" + fm_parent_channel_report "$FM_HOME" "$STATE" "$line" >/dev/null 2>&1 || true + printf 'parked %s: PR %s clean at %s; agent stopped\n' "$id" "$pr" "${head:0:12}" +} + +# --- resume ---------------------------------------------------------------- + +cmd_resume() { + local id=${1-} trigger pr head note out rc=0 rec + task_id_ok "$id" || die "usage: fm-park.sh resume <task-id> --trigger <text>" 2 + shift + [ "${1-}" = --trigger ] && [ -n "${2-}" ] || die "usage: fm-park.sh resume <task-id> --trigger <text>" 2 + trigger=$2 + rec=$(record_path "$id") + [ -f "$rec" ] || die "$id is not parked" + lock_try "$(lane_lock "$id")" || die "another park action holds $id" + pr=$(record_get "$id" pr) + head=$(record_get "$id" head) + note="$PARK_TMP/note" + { + printf 'This lane was parked on review: PR %s was clean and ready to merge at head %s, so its previous agent was stopped.\n' "$pr" "$head" + printf 'You are a fresh agent relaunched in the same local copy on the same branch because: %s.\n' "$trigger" + printf 'Read the PR and its new review activity, address it on this branch, then report again exactly as your instructions define done.\n' + } > "$note" + # Written before the launch, so the fresh worker's own reports always follow + # it; while the record stays, crew state still reads the record, not this. + status_append "$id" "working [at=$(date +%s)]: relaunched from review park: $trigger" + out=$(FM_HOME=$FM_HOME fm_run_timed "$ACTION_SECS" "$CONTROL_BIN" "$id" relaunch --note-file "$note" 2>&1) || rc=$? + if [ "$rc" -ne 0 ]; then + record_write "$id" "pr=$pr" "head=$head" "parked_at=$(record_get "$id" parked_at)" \ + "slot_leased=$(record_get "$id" slot_leased)" "resume_error=$(date +%s)" || true + printf 'relaunch-failed %s: PR %s (%s): %s\n' "$id" "$pr" "$trigger" "$(printf '%s\n' "$out" | tail -1)" + return 1 + fi + rm -f -- "$rec" + printf 'relaunched %s: PR %s - %s\n' "$id" "$pr" "$trigger" +} + +# --- land ------------------------------------------------------------------ + +# Parked lanes always clean up; other lanes require explicit home opt-in. +land_eligible() { # <id> + [ -e "$(record_path "$1")" ] || [ -e "$CONFIG/auto-land" ] +} + +cmd_land() { + local id=${1-} eligible=0 detach=0 queue=0 pr out rc=0 why tmp + task_id_ok "$id" || die "usage: fm-park.sh land <task-id> [--if-eligible] [--detach] [--queue]" 2 + shift + while [ "$#" -gt 0 ]; do + case "$1" in + --if-eligible) eligible=1 ;; + --detach) detach=1 ;; + --queue) queue=1 ;; + *) die "usage: fm-park.sh land <task-id> [--if-eligible] [--detach] [--queue]" 2 ;; + esac + shift + done + [ "$eligible" = 0 ] || land_eligible "$id" || return 0 + [ -f "$STATE/$id.meta" ] || return 0 + if [ "$queue" = 1 ] || [ "$detach" = 1 ]; then + tmp=$(mktemp "$STATE/.review-park.XXXXXX") || die "cannot create cleanup request" + if ! { printf 'pr=%s\n' "$(fm_meta_get "$STATE/$id.meta" pr)" > "$tmp" && chmod 0600 "$tmp" \ + && mv -f -- "$tmp" "$STATE/$id.review-park-land"; }; then + rm -f -- "$tmp" + die "cannot publish cleanup request" + fi + [ "$detach" = 1 ] || return 0 + action_start "$id" land + printf 'automatic cleanup started\n' + return 0 + fi + pr=$(fm_meta_get "$STATE/$id.meta" pr) + # A park or relaunch under way finishes first; each is bounded by the same + # backstop, so this wait is too. + if ! fm_lock_acquire_wait_max "$(lane_lock "$id")" "$ACTION_SECS"; then + printf 'cleanup-refused %s: PR %s merged; another park action still holds the lane\n' "$id" "$pr" + return 1 + fi + LOCKS_HELD="$LOCKS_HELD $(lane_lock "$id")" + [ -f "$STATE/$id.meta" ] || return 0 + pr=$(fm_meta_get "$STATE/$id.meta" pr) + if [ -e "$STATE/$id.review-park-land" ] \ + && [ "$(fm_meta_get "$STATE/$id.review-park-land" pr)" != "$pr" ]; then + rm -f -- "$STATE/$id.review-park-land" + return 0 + fi + out=$(FM_HOME=$FM_HOME fm_run_timed "$ACTION_SECS" "$TEARDOWN_BIN" "$id" 2>&1) || rc=$? + case "$rc" in + 0) printf 'cleaned up %s: PR %s merged; lane torn down and its working copy returned\n' "$id" "$pr" ;; + 4) + printf 'cleaned up %s: PR %s merged; lane torn down, but its returned pool slot was lost\n' "$id" "$pr" + printf '%s\n' "$out" | grep -E '^(warning: treehouse return left pool slot | )' || true + ;; + *) + why=$(printf '%s\n' "$out" | grep -m1 -E 'REFUSED|^error|refus' || printf '%s\n' "$out" | tail -1) + printf 'cleanup-refused %s: PR %s merged; automatic cleanup refused: %s\n' "$id" "$pr" "$why" + return 1 + ;; + esac +} + +# --- detached action ------------------------------------------------------- + +cmd_run() { + local id=${1-} verb=${2-} out + task_id_ok "$id" || die "usage: fm-park.sh run <task-id> park|resume|land [<args>...]" 2 + case "$verb" in park|resume|land) ;; *) die "usage: fm-park.sh run <task-id> park|resume|land [<args>...]" 2 ;; esac + shift 2 + out=$( LOCKS_HELD=; trap release_locks EXIT; "cmd_$verb" "$id" "$@" 2>/dev/null ) || true + if [ -n "$out" ]; then + case "$out" in *"another lifecycle action is already running"*|*"another park action still holds"*) return 0 ;; esac + report_write "$id" "$out" || return 1 + [ "$verb" != land ] || rm -f -- "$STATE/$id.review-park-land" + elif [ "$verb" = land ] && [ ! -e "$STATE/$id.meta" ]; then + rm -f -- "$STATE/$id.review-park-land" + fi +} + +# --- tick ------------------------------------------------------------------ + +# Prints a close once; the tick writes the marker after recording the wake. +closed_once() { # <id> <pr> + [ ! -e "$STATE/.review-park-closed-$1" ] || return 0 + printf 'closed %s: PR %s was closed without merging; the lane needs a decision\n' "$1" "$2" +} + +watch_parked() { # <id> + local id=$1 pr head parked snap result state triggers + ! lane_busy "$id" || return 0 + if [ -n "$(record_get "$id" pending)" ]; then + ( LOCKS_HELD=; trap release_locks EXIT; settle_pending "$id" ) || true + # Relaunching replaces the agent, so wait until the stop is confirmed. + [ -e "$(record_path "$id")" ] && [ -z "$(record_get "$id" pending)" ] || return 0 + fi + [ -z "$(record_get "$id" resume_error)" ] || return 0 + pr=$(record_get "$id" pr) + head=$(record_get "$id" head) + parked=$(record_get "$id" parked_at) + case "$parked" in ''|*[!0-9]*) return 0 ;; esac + snap=$(snapshot "$pr") || return 0 + result=$(printf '%s' "$snap" | jq -c --arg head "$head" --argjson parked "$parked" "$TRIGGER_PROGRAM") || return 0 + state=$(printf '%s' "$result" | jq -r .state) + case "$state" in + merged) return 0 ;; + closed) closed_once "$id" "$pr"; return 0 ;; + esac + triggers=$(printf '%s' "$result" | jq -r '.triggers | join("; ")') + [ -n "$triggers" ] || return 0 + action_start "$id" resume --trigger "$triggers" +} + +watch_unparked() { # <id> + local id=$1 meta pr snap state result + meta="$STATE/$id.meta" + [ "$(fm_meta_get "$meta" kind)" != secondmate ] || return 0 + pr=$(fm_meta_get "$meta" pr) + [ -n "$pr" ] && [ -f "$STATE/$id.check.sh" ] || return 0 + [ ! -e "$(record_path "$id")" ] || return 0 + snap=$(snapshot "$pr") || return 0 + state=$(printf '%s' "$snap" | jq -r .state) + [ "$state" != closed ] || { closed_once "$id" "$pr"; return 0; } + [ "$state" = open ] && [ -e "$CONFIG/park-on-review" ] || return 0 + ! lane_busy "$id" || return 0 + local idle_secs + idle_secs=$(step_secs "$READ_SECS") || return 0 + fm_run_timed "$idle_secs" "$SCRIPT_DIR/fm-park.sh" idle-check "$id" >/dev/null 2>&1 || return 0 + result=$(printf '%s' "$snap" | jq -c --argjson now "$(date +%s)" --argjson quiet "$QUIET_SECS" \ + --arg expect "" --argjson bots "$(review_bots_json)" "$GATE_PROGRAM") || return 0 + [ "$(printf '%s' "$result" | jq -r .clean)" = true ] || return 0 + action_start "$id" park +} + +# A pending record stays watched while its agent stop may still be in flight. +# Confirmed dead is promoted to parked; confirmed alive after the stop bound is dropped. +settle_pending() { # <id> + local id=$1 meta secs alive parked_at + lock_try "$(lane_lock "$id")" || return 0 + [ -n "$(record_get "$id" pending)" ] || return 0 + meta="$STATE/$id.meta" + secs=$(step_secs "$READ_SECS") || return 0 + # shellcheck disable=SC2016 # The inner script expands after bash -c receives positional args. + alive=$(fm_run_timed "$secs" bash -c '. "$1"; fm_backend_agent_alive "$2" "$3"' _ \ + "$SCRIPT_DIR/fm-backend.sh" "$(fm_backend_of_meta "$meta")" "$(fm_backend_target_of_meta "$meta")" 2>/dev/null) || true + case "$alive" in + dead) + record_write "$id" "pr=$(record_get "$id" pr)" "head=$(record_get "$id" head)" \ + "parked_at=$(record_get "$id" parked_at)" "slot_leased=$(record_get "$id" slot_leased)" ;; + alive) + parked_at=$(record_get "$id" parked_at) + [[ "$parked_at" =~ ^[0-9]+$ ]] || return 0 + [ "$(( $(date +%s) - parked_at ))" -ge "$ACTION_SECS" ] && rm -f -- "$(record_path "$id")" ;; + esac +} + +# visit_lanes <cursor> <visit> <id>... runs <visit> on each lane, starting at +# the lane saved in <cursor> and wrapping around. When the tick runs out of time +# it saves the next lane there, so a slow early lane cannot starve later ones. +visit_lanes() { # <cursor> <visit> <id>... + local cursor=$1 visit=$2 start id + local -a first=() rest=() + shift 2 + start=$(cat "$cursor" 2>/dev/null || true) + for id in "$@"; do + if [[ "$id" < "$start" ]]; then rest+=("$id"); else first+=("$id"); fi + done + for id in ${first[@]+"${first[@]}"} ${rest[@]+"${rest[@]}"}; do + if [ "$(date +%s)" -ge "$DEADLINE" ]; then + printf '%s\n' "$id" > "$cursor" + return 1 + fi + "$visit" "$id" + done + rm -f -- "$cursor" +} + +cmd_tick() { + local rec id meta any=0 + local -a parked=() unparked=() + for rec in "$STATE"/*.review-park "$STATE"/*.review-park-land "$STATE"/.review-park-report-*; do [ -e "$rec" ] && any=1 && break; done + [ "$any" = 1 ] || [ -e "$CONFIG/park-on-review" ] || [ -e "$CONFIG/auto-land" ] || return 0 + command -v jq >/dev/null 2>&1 || return 0 + lock_try "$STATE/.review-park-tick.lock" || return 0 + DEADLINE=$(( $(date +%s) + TICK_SECS )) + local lines='' line + local -a reports=() + for rec in "$STATE"/.review-park-report-*; do + [ -f "$rec" ] || continue + reports+=("$rec") + lines+=$(cat -- "$rec")$'\n' + done + for rec in "$STATE"/*.review-park-land; do + [ -e "$rec" ] || continue + id=$(basename "$rec" .review-park-land) + fm_pr_task_id_valid "$id" && [ -f "$STATE/$id.meta" ] || { rm -f -- "$rec"; continue; } + lane_busy "$id" || action_start "$id" land + done + for rec in "$STATE"/*.review-park; do + [ -e "$rec" ] || continue + id=$(basename "$rec" .review-park) + fm_pr_task_id_valid "$id" && [ -f "$STATE/$id.meta" ] || continue + parked+=("$id") + done + # Reserve half the tick for unparked lanes; parked forge reads cannot starve them. + local full_deadline=$DEADLINE + DEADLINE=$(( $(date +%s) + (TICK_SECS + 1) / 2 )) + lines+=$(visit_lanes "$STATE/.review-park-tick-next" watch_parked ${parked[@]+"${parked[@]}"})$'\n' || true + DEADLINE=$full_deadline + if [ -e "$CONFIG/park-on-review" ] || [ -e "$CONFIG/auto-land" ]; then + for meta in "$STATE"/*.meta; do + [ -e "$meta" ] || continue + id=$(basename "$meta" .meta) + fm_pr_task_id_valid "$id" || continue + unparked+=("$id") + done + lines+=$(visit_lanes "$STATE/.review-park-tick-next-unparked" watch_unparked ${unparked[@]+"${unparked[@]}"})$'\n' || true + fi + lines=$(printf '%s' "$lines" | grep -v '^$') || return 0 + fm_wake_append check park-on-review "check: park-on-review: $(printf '%s\n' "$lines" | paste -sd ';' - | sed 's/;/; /g')" || return 0 + rm -f -- ${reports[@]+"${reports[@]}"} + while IFS= read -r line; do + case "$line" in "closed "*:*) line=${line#closed }; : > "$STATE/.review-park-closed-${line%%:*}" ;; esac + done <<< "$lines" + printf '%s\n' "$lines" +} + +cmd_list() { + local rec id + for rec in "$STATE"/*.review-park; do + [ -e "$rec" ] || continue + id=$(basename "$rec" .review-park) + printf '%s\tPR %s\thead %s\tparked %s%s\n' "$id" "$(record_get "$id" pr)" \ + "$(record_get "$id" head | cut -c1-12)" \ + "$(date -u -r "$(record_get "$id" parked_at)" +%Y-%m-%dT%H:%MZ 2>/dev/null || date -u -d "@$(record_get "$id" parked_at)" +%Y-%m-%dT%H:%MZ 2>/dev/null || record_get "$id" parked_at)" \ + "$([ -z "$(record_get "$id" pending)" ] || printf '\tparking; stop not yet confirmed')$([ -z "$(record_get "$id" resume_error)" ] || printf '\trelaunch failed; run fm-park.sh resume by hand')" + done +} + +case "${1:-}" in + idle-check) shift; agent_idle "${1:?}" ;; + -h|--help) usage ;; + gate) shift; cmd_gate "$@" ;; + park) shift; cmd_park "$@" ;; + resume) shift; cmd_resume "$@" ;; + land) shift; cmd_land "$@" ;; + tick) cmd_tick ;; + list) cmd_list ;; + run) shift; cmd_run "$@" ;; + *) usage >&2; exit 2 ;; +esac diff --git a/bin/fm-pr-check.sh b/bin/fm-pr-check.sh index afe77ef25ae..17f641f3d3c 100755 --- a/bin/fm-pr-check.sh +++ b/bin/fm-pr-check.sh @@ -9,7 +9,13 @@ # to fix it; the merged wake itself is unchanged. # Refuses when bin/fm-dod-lib.sh will not accept the named head as reachable # outside the worker's disposable copy; in no-mistakes mode a forge-reported -# head is that named head and is already stored on the forge. +# head is that named head and is already stored on the forge. Registration is +# the gate caller that refreshes the copy's remote-tracking refs before it +# refuses (FM_DOD_REFRESH_REMOTES), because a no-mistakes pipeline pushes from +# its own gate clone and nothing else fetches that push into the copy; the +# fetch also leaves the refs that the gate's other callers read. The +# merge-time re-record (FM_PR_CHECK_MERGE=1) refreshes the same way, while +# bin/fm-pr-merge.sh holds that one task's control lock. # The published check is a byte-for-byte copy of bin/fm-pr-poll.sh bound by its # own registration, so a later repository update does not invalidate it; task and # PR data live only in a private sidecar and are never interpolated into shell @@ -33,21 +39,35 @@ # # scan records every task whose status reports a PR that nobody recorded, so # merge monitoring never depends on the supervisor remembering this command. -# bin/fm-watch.sh runs it at the start of each check sweep. A candidate is a -# task record that is not a secondmate or scout and has no pr= line, or has the -# reported pr= but no valid published poll; its PR is -# the first "PR <url>" token in the newest done: line that names a supported PR -# URL, whatever the rest of the line says. A URL without the "PR " prefix is -# prose and never counts. Each candidate is recorded through the ordinary -# path above, so every refusal still applies. A refusal prints one line naming -# the task, the URL, and the reason, once per task and URL -# (state/<task>.pr-register-refused), and the scan retries silently on later -# sweeps, so a draft later marked ready is recorded without help. Each sweep +# bin/fm-watch.sh runs it at the start of each check sweep. A task reports a PR +# only when its newest crew-declared status line (last_crew_status_line in +# bin/fm-classify-lib.sh) is a done: line in a ready shape +# (fm_dod_ready_note_url in bin/fm-dod-lib.sh) naming a supported PR URL, so +# prose around the URL, a line saying the PR is not ready, and any later crew +# line each leave nothing to record. A candidate is a task record that is not a +# secondmate or scout and has no pr= line, has the reported pr= but no valid +# published poll, or has a pr= that its status log named and the reported PR now +# replaces; a pr= the log never named is the supervisor's own choice and stays, +# and a merged or retired pr= is never replaced. Each candidate is recorded +# through the ordinary path above, so every refusal still applies. A refusal +# queues a check wake and prints one line naming the task, the URL, and the +# reason, once per task and URL (state/<task>.pr-register-refused), and the scan +# retries silently on later sweeps, so a draft later marked ready is recorded +# without help. The wake is queued before the once-only marker is written, so a +# stop between the two repeats a notice rather than losing it. Each sweep # starts just after the last task the previous sweep attempted # (state/.pr-check-scan-cursor), so a registration that hangs or is refused # cannot take the whole budget on every sweep. Registrations # stop FM_CHECK_TIMEOUT (default 30) minus 5 seconds after the scan starts; a -# registration cut at that bound prints nothing and is retried. +# registration cut at that bound reports the same way that the PR merge watch +# could not be armed, once per task and URL (state/<task>.pr-register-timed-out), +# and is retried. +# A registration's remote-ref refresh ends 5 seconds before that bound +# (FM_DOD_REFRESH_DEADLINE), so a registration still refused after it reports +# that refusal instead of being cut. Once that time has passed, the sweep +# starts no further registration, so the cursor stays on the last task that +# had a full turn; the first registration of a sweep still runs, and with no +# time left to fetch it decides on the last-known refs. # Usage: fm-pr-check.sh <task-id> <pr-url> # fm-pr-check.sh scan set -eu @@ -71,26 +91,13 @@ DEGRADED_WATCH_ENABLED=0 . "$SCRIPT_DIR/fm-timeout-lib.sh" pr_check_reported_url() { # <status-file> - local line token found= + local line url [ -f "$1" ] && [ ! -L "$1" ] || return 1 - while IFS= read -r line; do - [ "$(status_line_verb "$line")" = "done" ] || continue - while IFS= read -r token; do - token=${token#*PR } - while :; do - case "$token" in - *[\).,\;:\'\"]) token=${token%?} ;; - *) break ;; - esac - done - if fm_pr_url_parse "$token"; then - found=$FM_PR_URL - break - fi - done < <(status_line_note "$line" | grep -oE '(^|[^[:alnum:]_])PR https://[^[:space:]]+' || true) - done < <(grep -E '^[[:space:]]*done' "$1" | grep -F 'PR https://' || true) - [ -n "$found" ] || return 1 - printf '%s\n' "$found" + line=$(last_crew_status_line "$1") + [ "$(status_line_verb "$line")" = "done" ] || return 1 + url=$(fm_dod_ready_note_url "$(status_line_note "$line")") || return 1 + fm_pr_url_parse "$url" || return 1 + printf '%s\n' "$FM_PR_URL" } pr_check_rotate() { # <last-id>: start the lines on stdin just after the line for last-id, wrapping @@ -104,7 +111,8 @@ pr_check_pr_retired() { # <id> <recorded-url> } pr_check_scan() { - local budget start meta id kind url recorded left marker err rc tmp candidates='' cursor_file cursor + local budget start meta id kind url recorded replaces left refused timed_out marker note line err rc tmp candidates='' cursor_file cursor + local refresh_deadline attempted now budget=${FM_CHECK_TIMEOUT:-30} case "$budget" in ''|*[!0-9]*) budget=30 ;; esac budget=$((budget - 5)) @@ -120,35 +128,56 @@ pr_check_scan() { url=$(pr_check_reported_url "$STATE/$id.status") || continue recorded=$(grep '^pr=' "$meta" | tail -1 | cut -d= -f2- || true) pr_check_pr_retired "$id" "$recorded" && continue - [ -z "$recorded" ] || { [ "$recorded" = "$url" ] && ! fm_pr_poll_artifacts_valid "$STATE" "$id" "$SCRIPT_DIR/fm-pr-poll.sh"; } || continue - candidates="$candidates$id $url"$'\n' + replaces= + if [ "$recorded" = "$url" ]; then + ! fm_pr_poll_artifacts_valid "$STATE" "$id" "$SCRIPT_DIR/fm-pr-poll.sh" || continue + elif [ -n "$recorded" ]; then + fm_dod_status_names_pr "$STATE/$id.status" "$recorded" || continue + replaces=$recorded + fi + candidates="$candidates$id $url $replaces"$'\n' done cursor_file="$STATE/.pr-check-scan-cursor" cursor=$(cat "$cursor_file" 2>/dev/null || true) candidates=$(printf '%s' "$candidates" | pr_check_rotate "$cursor") - while IFS=' ' read -r id url; do + refresh_deadline=$((start + budget - 5)) + attempted=0 + while IFS=' ' read -r id url replaces; do [ -n "$id" ] || continue - left=$((start + budget - $(date +%s))) + now=$(date +%s) + left=$((start + budget - now)) [ "$left" -gt 0 ] || break - marker="$STATE/$id.pr-register-refused" + [ "$attempted" = 0 ] || [ "$now" -lt "$refresh_deadline" ] || break + attempted=1 + refused="$STATE/$id.pr-register-refused" + timed_out="$STATE/$id.pr-register-timed-out" printf '%s\n' "$id" > "$cursor_file" 2>/dev/null || true rc=0 - err=$(fm_run_timed "$left" env FM_PR_CHECK_SCAN=1 bash "$SCRIPT_DIR/fm-pr-check.sh" "$id" "$url" 2>&1 >/dev/null) || rc=$? + # The grace lets a registration cut at the bound run its cleanup trap. + err=$( (fm_exec_timed "$left" 2 env FM_PR_CHECK_SCAN=1 FM_PR_CHECK_SCAN_REPLACES="$replaces" \ + FM_DOD_REFRESH_DEADLINE="$refresh_deadline" \ + bash "$SCRIPT_DIR/fm-pr-check.sh" "$id" "$url") 2>&1 >/dev/null) || rc=$? if [ "$rc" -eq 0 ]; then - rm -f -- "$marker" + rm -f -- "$refused" "$timed_out" continue fi - [ "$rc" -ne 124 ] || continue + if fm_timed_out "$rc"; then + marker=$timed_out + note="its PR merge watch could not be armed: registration timed out after ${left}s, and later sweeps keep retrying" + else + marker=$refused + err=$(printf '%s\n' "$err" | sed -n 's/^error: //p' | head -1) + note="it was not recorded: ${err:-registration failed}" + fi if [ -f "$marker" ] && [ ! -L "$marker" ] && [ "$(cat "$marker")" = "$url" ]; then continue fi - tmp=$(mktemp "$STATE/.pr-register-refused.XXXXXX") || continue - if ! { printf '%s\n' "$url" > "$tmp" && mv -f -- "$tmp" "$marker"; }; then - rm -f -- "$tmp" - continue + line="$id reported $url in a done line, but $note" + fm_wake_append check pr-register "check: pr-register: $line" || continue + if tmp=$(mktemp "$STATE/.pr-register-refused.XXXXXX"); then + { printf '%s\n' "$url" > "$tmp" && mv -f -- "$tmp" "$marker"; } || rm -f -- "$tmp" fi - err=$(printf '%s\n' "$err" | sed -n 's/^error: //p' | head -1) - printf '%s reported %s in a done line, but it was not recorded: %s\n' "$id" "$url" "${err:-registration failed}" + printf '%s\n' "$line" done <<EOF_SCAN $candidates EOF_SCAN @@ -319,6 +348,7 @@ case "$PROVIDER:$MODE" in *:no-mistakes|*:) DONE_LINE="done: PR $URL checks green" ;; *) DONE_LINE="done: PR $URL" ;; esac +FM_DOD_REFRESH_REMOTES=1 if { [ -z "$PR_HEAD" ] || ! fm_dod_forge_head_is_named_head "$MODE"; } \ && ! GATE_REASON=$(fm_dod_accept_ship_done "${KIND:-ship}" "$MODE" "$WT" "$PROJECT" "$DONE_LINE" "$STATE" "$ID" "$META"); then echo "error: $GATE_REASON" >&2 @@ -330,6 +360,8 @@ BASELINE_BACKUP= BASELINE_BACKUP_COMPLETE=0 META_LOCK= META_LOCK_HELD=0 +MERGE_NOTICE_LOCK= +MERGE_NOTICE_LOCK_HELD=0 PR_POLL_PUBLISH_LOCK= PR_POLL_PUBLISH_LOCK_HELD=0 CONTROL_LOCK= @@ -354,6 +386,10 @@ pr_check_cleanup() { fm_lock_release "$CONTROL_LOCK" || true CONTROL_LOCK_HELD=0 fi + if [ "$MERGE_NOTICE_LOCK_HELD" = 1 ]; then + fm_lock_release "$MERGE_NOTICE_LOCK" || true + MERGE_NOTICE_LOCK_HELD=0 + fi if [ "$META_LOCK_HELD" = 1 ]; then fm_lock_release "$META_LOCK" || true META_LOCK_HELD=0 @@ -374,6 +410,8 @@ fm_pr_poll_prepare "$STATE" "$ID" "$PROVIDER" "$URL" "$HOST" "$PROJECT_PATH" "$N # (bin/fm-pr-merge.sh already holds it) nor reads or rewrites the baseline: it # records before any forge call, and the marker it leaves is keyed to the PR # identity, so it can only ever match the PR it was written for. +BASELINE_TRAIN=0 +BASELINE_DEGRADED= if [ "${FM_PR_CHECK_MERGE:-}" != 1 ] && [ "$DEGRADED_WATCH_ENABLED" = 1 ]; then CONTROL_LOCK="$STATE/.control-$ID.lock" fm_lock_acquire_wait "$CONTROL_LOCK" || exit 1 @@ -390,9 +428,7 @@ if [ "${FM_PR_CHECK_MERGE:-}" != 1 ] && [ "$DEGRADED_WATCH_ENABLED" = 1 ]; then fi exit 1 fi - BASELINE_TRAIN=0 BASELINE_TRAIN_UNKNOWN=0 - BASELINE_DEGRADED= while IFS= read -r baseline_line; do case $baseline_line in ontrain) BASELINE_TRAIN=1 ;; @@ -417,10 +453,25 @@ META_LOCK_HELD=1 [ -f "$META" ] && [ ! -L "$META" ] && [ "$(fm_pr_file_link_count "$META")" = 1 ] \ || { echo "error: task metadata is unavailable" >&2; exit 1; } # A scan gathers its candidates before it registers them, so the supervisor may -# have recorded another PR meanwhile; that choice is never overwritten. +# have recorded another PR meanwhile; that choice is never overwritten. Only the +# exact pr= the scan chose to replace gives way, and only while its merge is +# unrecorded: the merge-notification lock bin/fm-merge-outcome-lib.sh records a +# merge under is held from that recheck through the pr= write, so a replacement +# never interleaves with the record of the PR it replaces. if [ "${FM_PR_CHECK_SCAN:-}" = 1 ]; then RECORDED_PR=$(grep '^pr=' "$META" | tail -1 | cut -d= -f2- || true) - [ -z "$RECORDED_PR" ] || [ "$RECORDED_PR" = "$URL" ] || exit 0 + if [ -n "${FM_PR_CHECK_SCAN_REPLACES:-}" ] && [ "$RECORDED_PR" = "$FM_PR_CHECK_SCAN_REPLACES" ]; then + MERGE_NOTICE_LOCK="$STATE/$ID.pr-poll-merge-notified.lock" + fm_lock_acquire_wait "$MERGE_NOTICE_LOCK" + MERGE_NOTICE_LOCK_HELD=1 + pr_check_pr_retired "$ID" "$RECORDED_PR" && exit 0 + if [ "$RECORDED_PR" = "${FM_PR_CHECK_SCAN_REPLACES:-}" ]; then + reported=$(pr_check_reported_url "$STATE/$ID.status") || exit 0 + [ "$reported" = "$URL" ] || exit 0 + fi + fi + [ -z "$RECORDED_PR" ] || [ "$RECORDED_PR" = "$URL" ] \ + || [ "$RECORDED_PR" = "${FM_PR_CHECK_SCAN_REPLACES:-}" ] || exit 0 fi META_DEVICE=$(fm_pr_file_device "$META") || exit 1 STATE_DEVICE=$(fm_pr_file_device "$STATE") || exit 1 @@ -443,7 +494,7 @@ fm_pr_metadata_identity_parse "$META_TMP" || exit 1 fm_pr_regular_destination_on_device_or_absent "$META" "$STATE_DEVICE" || exit 1 # Keep the current registration recoverable until all replacement files commit. -if [ "${FM_PR_CHECK_MERGE:-}" != 1 ] && [ "$DEGRADED_WATCH_ENABLED" = 1 ]; then +if [ "${FM_PR_CHECK_MERGE:-}" != 1 ] && { [ "$DEGRADED_WATCH_ENABLED" = 1 ] || { [ "${FM_PR_CHECK_SCAN:-}" = 1 ] && [ "${RECORDED_PR:-}" = "${FM_PR_CHECK_SCAN_REPLACES:-}" ] && [ -n "${RECORDED_PR:-}" ]; }; }; then BASELINE_BACKUP=$(mktemp -d "$STATE/.fm-pr-baseline.XXXXXX") || exit 1 restore_baselines() { local artifact backup @@ -480,6 +531,11 @@ fm_pr_metadata_identity_parse "$META" || { [ -z "${BASELINE_BACKUP:-}" ] || base [ "$FM_PR_META_PROVIDER" = "$PROVIDER" ] && [ "$FM_PR_META_URL" = "$URL" ] \ && [ "$FM_PR_META_HOST" = "$HOST" ] && [ "$FM_PR_META_PATH" = "$PROJECT_PATH" ] \ && [ "$FM_PR_META_NUMBER" = "$NUMBER" ] || { [ -z "${BASELINE_BACKUP:-}" ] || baseline_failure; exit 1; } +if [ "$MERGE_NOTICE_LOCK_HELD" = 1 ]; then + fm_lock_release "$MERGE_NOTICE_LOCK" + MERGE_NOTICE_LOCK_HELD=0 +fi + PR_POLL_PUBLISH_LOCK="$STATE/.pr-poll-publish-$ID.lock" fm_lock_acquire_wait "$PR_POLL_PUBLISH_LOCK" || { [ -z "${BASELINE_BACKUP:-}" ] || baseline_failure; exit 1; } PR_POLL_PUBLISH_LOCK_HELD=1 diff --git a/bin/fm-pr-lib.sh b/bin/fm-pr-lib.sh index 20ccf994a64..59a147cc810 100755 --- a/bin/fm-pr-lib.sh +++ b/bin/fm-pr-lib.sh @@ -1143,11 +1143,154 @@ fm_pr_github_read_record() { # <host> <owner> <repo> <number> fm_pr_github_read_record_with_gh_axi "$@" } +# GitHub required-check reading shared by bin/fm-pr-merge.sh and +# bin/fm-park.sh. These read the repository named by PR_OWNER and PR_REPO, +# which the caller sets. +github_urlencode_path_segment() { + local LC_ALL=C input=$1 encoded='' char octet hex + while [ -n "$input" ]; do + char=${input%"${input#?}"} + input=${input#?} + case "$char" in + [-._~a-zA-Z0-9]) encoded=$encoded$char ;; + *) + printf -v octet '%d' "'$char" + [ "$octet" -ge 0 ] || octet=$((octet + 256)) + printf -v hex '%02X' "$octet" + encoded=$encoded%$hex + ;; + esac + done + printf '%s' "$encoded" +} + +# Whether a failed branch-rules read (the gh stderr given) is GitHub's +# plan-gated 403 ("Upgrade to GitHub Pro or make this repository public"), +# which means the repository's plan cannot expose branch rules at all, on +# GitHub or GitHub Enterprise Server - not that this script failed to read +# them, and not that the token lacks a permission. Such a repository has no +# active ruleset rule of any kind. Any other failure (auth, rate limit, +# network, a 404, an unrelated 403) is not this and stays unreadable. +github_branch_rules_unavailable_on_plan() { + case "$1" in + *"Upgrade to GitHub Pro or make this repository public"*) return 0 ;; + esac + return 1 +} + +FM_PR_GITHUB_REQUIRED= +FM_PR_GITHUB_REQUIRED_ERROR= +github_read_required_contexts() { + local base=$1 branch_path branch_json rules_json classic='' ruleset='' api_err api_err_text + FM_PR_GITHUB_REQUIRED='[]' + FM_PR_GITHUB_REQUIRED_ERROR= + branch_path=$(github_urlencode_path_segment "$base") + + if ! branch_json=$(gh api "repos/$PR_OWNER/$PR_REPO/branches/$branch_path" 2>/dev/null) \ + || [ -z "$branch_json" ] \ + || ! classic=$(printf '%s' "$branch_json" | jq -c ' + if type != "object" or (.protected | type) != "boolean" then + error("branch payload is unreadable") + elif .protected == false then + empty + elif (.protection.required_status_checks | type) != "object" then + error("branch protection summary is unreadable") + else + .protection.required_status_checks + | ((.checks // []) | if type == "array" then .[] else error("invalid checks") end + | {context, app_id}), + ((.contexts // []) | if type == "array" then .[] else error("invalid contexts") end + | {context: ., app_id: null}) + | if (.context | type) == "string" and (.context | length) > 0 + and (.app_id == null or (.app_id | type) == "number") + then . else error("invalid required check") end + | if .app_id == -1 then .app_id = null else . end + end' 2>/dev/null); then + classic='' + FM_PR_GITHUB_REQUIRED_ERROR="the branch protection summary for base branch $base could not be read" + fi + + if ! api_err=$(mktemp "${TMPDIR:-/tmp}/fm-pr-merge-required-rules.XXXXXX"); then + FM_PR_GITHUB_REQUIRED_ERROR="${FM_PR_GITHUB_REQUIRED_ERROR:+$FM_PR_GITHUB_REQUIRED_ERROR +}the branch rules for base branch $base could not be read" + else + if ! rules_json=$(gh api --paginate "repos/$PR_OWNER/$PR_REPO/rules/branches/$branch_path" 2>"$api_err"); then + api_err_text=$(cat "$api_err" 2>/dev/null) + if ! github_branch_rules_unavailable_on_plan "$api_err_text"; then + FM_PR_GITHUB_REQUIRED_ERROR="${FM_PR_GITHUB_REQUIRED_ERROR:+$FM_PR_GITHUB_REQUIRED_ERROR +}the branch rules for base branch $base could not be read" + fi + elif [ -z "$rules_json" ] || ! ruleset=$(printf '%s' "$rules_json" | jq -c ' + if type != "array" then error("rules payload is unreadable") else .[] end + | select(type != "object" or .type == "required_status_checks") + | if type == "object" and (.parameters.required_status_checks | type) == "array" + then .parameters.required_status_checks[] else error("invalid required check rule") end + | if type == "object" and (.context | type) == "string" and (.context | length) > 0 + and (.integration_id == null or (.integration_id | type) == "number") + then {context, app_id: .integration_id} else error("invalid required check rule") end + | if .app_id == -1 then .app_id = null else . end' 2>/dev/null); then + ruleset='' + FM_PR_GITHUB_REQUIRED_ERROR="${FM_PR_GITHUB_REQUIRED_ERROR:+$FM_PR_GITHUB_REQUIRED_ERROR +}the branch rules for base branch $base could not be read" + fi + rm -f "$api_err" + fi + + FM_PR_GITHUB_REQUIRED=$(printf '%s\n%s\n' "$classic" "$ruleset" | jq -sc ' + unique_by([.context, .app_id]) | group_by(.context) + | map(if any(.[]; .app_id != null) then map(select(.app_id != null)) else . end) | add // []') + [ -z "$FM_PR_GITHUB_REQUIRED_ERROR" ] +} + +github_required_checks_missing() { + local json=$1 required=$2 producers=$3 + printf '%s' "$json" | jq -r --argjson required "$required" --argjson producers "$producers" ' + if (.statusCheckRollup | type) != "array" then error("no check rollup") else . end + | .statusCheckRollup as $reported + | $required + | map(. as $requirement + | select(any($reported[]; + if $requirement.app_id == null then + (if .__typename == "CheckRun" then .name else .context end) == $requirement.context + elif .__typename == "CheckRun" then + .name == $requirement.context + and any($producers[]; .name == $requirement.context and .app.id == $requirement.app_id) + else + .context == $requirement.context + end) | not) + | .context) | unique[] + ' 2>/dev/null || return 1 +} + +# Reads the check runs at <head> with their producing app into +# FM_PR_GITHUB_PRODUCERS, only when a required check in FM_PR_GITHUB_REQUIRED +# names its app; '[]' otherwise or when the read fails, which returns 1. +FM_PR_GITHUB_PRODUCERS='[]' +github_read_required_producers() { # <head> + local head=$1 runs + FM_PR_GITHUB_PRODUCERS='[]' + printf '%s' "$FM_PR_GITHUB_REQUIRED" | jq -e 'any(.[]; .app_id != null)' >/dev/null || return 0 + if ! runs=$(gh api --paginate "repos/$PR_OWNER/$PR_REPO/commits/$head/check-runs" 2>/dev/null) \ + || [ -z "$runs" ] \ + || ! FM_PR_GITHUB_PRODUCERS=$(printf '%s' "$runs" | jq -sc --arg head "$head" ' + [ .[] | if (.check_runs | type) == "array" then .check_runs[] else error("invalid check runs") end + | if (.name | type) == "string" and (.app.id | type) == "number" and .head_sha == $head + then . else error("invalid check producer") end ]' 2>/dev/null); then + # shellcheck disable=SC2034 # read by the callers + FM_PR_GITHUB_PRODUCERS='[]' + return 1 + fi +} + +# Reads one merge request's state and its own head commit (the MR's sha, never +# a merge, squash, or merge-train commit, which is what lets bin/fm-teardown.sh +# prove a squash-merged or merge-train-merged copy landed). fm_pr_gitlab_read_record() { # <host> <path> <number> local host=$1 path=$2 number=$3 project_url json fields line - local total=0 named=0 state='' merged='' + local total=0 named=0 state='' merged='' head='' FM_PR_RECORD_STATE= FM_PR_RECORD_MERGED= + FM_PR_RECORD_HEAD= command -v glab >/dev/null 2>&1 || return 1 command -v jq >/dev/null 2>&1 || return 1 project_url="https://$host/$path" @@ -1159,7 +1302,8 @@ fm_pr_gitlab_read_record() { # <host> <path> <number> if ! fields=$(printf '%s' "$json" | jq -r ' if type == "object" and (.state | type == "string") and .state != "" then "state=" + .state, - "merged=" + (if .state == "merged" then "true" else "false" end) + "merged=" + (if .state == "merged" then "true" else "false" end), + "head=" + (if (.sha | type) == "string" then .sha else "" end) else error("invalid merge request state") end' 2>/dev/null); then @@ -1170,13 +1314,14 @@ fm_pr_gitlab_read_record() { # <host> <path> <number> case "$line" in state=*) state=${line#state=} ;; merged=*) merged=${line#merged=} ;; + head=*) head=${line#head=} ;; *) continue ;; esac named=$((named + 1)) done <<FIELDS $fields FIELDS - if [ "$named" -ne 2 ] || [ "$total" -ne 2 ] || [ -z "$state" ] \ + if [ "$named" -ne 3 ] || [ "$total" -ne 3 ] || [ -z "$state" ] \ || { [ "$merged" != true ] && [ "$merged" != false ]; }; then return 1 fi @@ -1184,9 +1329,12 @@ FIELDS # Consumed by bin/fm-crew-state.sh passed_pr_detail. # shellcheck disable=SC2034 FM_PR_RECORD_STATE=$state - # Consumed by bin/fm-crew-state.sh passed_pr_detail. + # Consumed by bin/fm-crew-state.sh passed_pr_detail and bin/fm-teardown.sh. # shellcheck disable=SC2034 FM_PR_RECORD_MERGED=$merged + # Consumed by bin/fm-teardown.sh pr_is_merged; empty when the forge omitted it. + # shellcheck disable=SC2034 + FM_PR_RECORD_HEAD=$head } # tea addresses a repo by slug only and takes the host from a named login, so diff --git a/bin/fm-pr-merge.sh b/bin/fm-pr-merge.sh index 3a229cbfd7c..b793d50f181 100755 --- a/bin/fm-pr-merge.sh +++ b/bin/fm-pr-merge.sh @@ -21,6 +21,9 @@ # # Merge method on GitHub defaults to --squash when the caller passes none of # --squash, --merge, --rebase, or --method after the optional -- separator. +# With the opt-in config/pr-squash-summary presence flag, a GitHub squash merge +# whose caller passed no subject or body takes the PR title and the body's +# "## What Changed" section as its commit text; github_add_squash_summary owns it. # On Forgejo the merge style defaults to the repository's own # default_merge_style, read live, and a merge is refused when it cannot be read. # A GitHub merge is refused unless every pre-merge condition holds, each read @@ -35,7 +38,7 @@ # once that bound is spent it reports mergeability still pending rather than # unmergeable, with the same nonzero exit as any other refusal. # A required check that never reported is absent from the checks -# list rather than red, so github_read_required_contexts below reads the +# list rather than red, so github_read_required_contexts (bin/fm-pr-lib.sh) reads the # required set from classic branch protection and active rulesets. Check-run # requirements retain their producer app binding: a same-named check run from another app cannot # satisfy them, and a duplicate name-only entry cannot weaken that binding. @@ -467,6 +470,93 @@ caller_requested_auto_merge() { return "$requested" } +# Whether the caller's own extra arguments set the commit subject or body, +# including gh's -t, -b, and -F short options inside a short-option cluster. +caller_has_commit_text() { + local arg + for arg in "$@"; do + case "$arg" in + --subject|--subject=*|--body|--body=*|--body-file|--body-file=*) return 0 ;; + --*) ;; + -*[tbF]*) return 0 ;; + esac + done + return 1 +} + +# The "## What Changed" section of the PR body on stdin, without its heading +# or surrounding blank lines. It ends at the next level-two heading or the +# no-mistakes attestation comment; nothing is printed when it is absent. +# Lines inside a fenced code block (``` or ~~~, closed by a run of the same +# character at least as long) are never read as a heading or that comment; +# a ``` line whose info string holds a backtick is inline code, not a fence. +what_changed_section() { + tr -d '\r' | awk ' + # The fence run a line opens or closes with: up to three spaces of + # indent, then three or more backticks or tildes; empty when none. A + # backtick run followed by another backtick is inline code, not a fence. + function fence_run(line, s, c, n) { + s = line + sub(/^ ? ? ?/, "", s) + c = substr(s, 1, 1) + if (c != "`" && c != "~") return "" + n = 0 + while (substr(s, n + 1, 1) == c) n++ + if (n < 3) return "" + if (c == "`" && index(substr(s, n + 1), "`")) return "" + return substr(s, 1, n) + } + { + run = fence_run($0) + if (fence != "") { + rest = $0 + sub(/^ ? ? ?/, "", rest) + rest = substr(rest, length(run) + 1) + if (run != "" && substr(run, 1, 1) == substr(fence, 1, 1) \ + && length(run) >= length(fence) && rest ~ /^[[:space:]]*$/) fence = "" + if (inside) lines[++n] = $0 + next + } + if (run != "") { + fence = run + if (inside) lines[++n] = $0 + next + } + } + /^## / { if (inside) exit; if ($0 ~ /^## What Changed[[:space:]]*$/) { inside = 1; next } } + inside && /^<!-- no-mistakes-/ { exit } + inside { lines[++n] = $0 } + END { + first = 1; last = n + while (first <= last && lines[first] ~ /^[[:space:]]*$/) first++ + while (last >= first && lines[last] ~ /^[[:space:]]*$/) last-- + for (i = first; i <= last; i++) print lines[i] + }' +} + +# Opt-in config/pr-squash-summary: a GitHub squash merge whose caller passed no +# commit text of its own gets the PR title plus " (#N)" as its subject and the +# body's What Changed section as its body. Without that section, or when the PR +# text cannot be read, the repository's own squash text stands. +github_add_squash_summary() { + local json title body + case "$(caller_merge_method "$@")" in + ''|[sS][qQ][uU][aA][sS][hH]) ;; + *) return 0 ;; + esac + [ -e "${FM_CONFIG_OVERRIDE:-$FM_HOME/config}/pr-squash-summary" ] || return 0 + caller_has_commit_text "$@" && return 0 + if ! json=$(gh pr view "$URL" --json title,body 2>/dev/null) \ + || ! title=$(printf '%s' "$json" | jq -er '.title | strings | select(length > 0)' 2>/dev/null) \ + || ! body=$(printf '%s' "$json" | jq -r '.body // "" | strings' 2>/dev/null); then + printf 'note: could not read the title and body of %s; the squash commit keeps the repository default text\n' "$URL" >&2 + return 0 + fi + body=$(printf '%s\n' "$body" | what_changed_section) + [ -n "$body" ] || return 0 + merge_args+=(--subject "$title (#$PR_NUMBER)" --body "$body") +} + reject_repo_overrides() { local arg for arg in "$@"; do @@ -1042,96 +1132,12 @@ github_checks_not_green() { ' 2>/dev/null || return 1 } -FM_PR_GITHUB_REQUIRED= -FM_PR_GITHUB_REQUIRED_ERROR= -github_read_required_contexts() { - local base=$1 branch_path branch_json rules_json classic='' ruleset='' api_err api_err_text - FM_PR_GITHUB_REQUIRED='[]' - FM_PR_GITHUB_REQUIRED_ERROR= - branch_path=$(github_urlencode_path_segment "$base") - - if ! branch_json=$(gh api "repos/$PR_OWNER/$PR_REPO/branches/$branch_path" 2>/dev/null) \ - || [ -z "$branch_json" ] \ - || ! classic=$(printf '%s' "$branch_json" | jq -c ' - if type != "object" or (.protected | type) != "boolean" then - error("branch payload is unreadable") - elif .protected == false then - empty - elif (.protection.required_status_checks | type) != "object" then - error("branch protection summary is unreadable") - else - .protection.required_status_checks - | ((.checks // []) | if type == "array" then .[] else error("invalid checks") end - | {context, app_id}), - ((.contexts // []) | if type == "array" then .[] else error("invalid contexts") end - | {context: ., app_id: null}) - | if (.context | type) == "string" and (.context | length) > 0 - and (.app_id == null or (.app_id | type) == "number") - then . else error("invalid required check") end - | if .app_id == -1 then .app_id = null else . end - end' 2>/dev/null); then - classic='' - FM_PR_GITHUB_REQUIRED_ERROR="the branch protection summary for base branch $base could not be read" - fi - - if ! api_err=$(mktemp "${TMPDIR:-/tmp}/fm-pr-merge-required-rules.XXXXXX"); then - FM_PR_GITHUB_REQUIRED_ERROR="${FM_PR_GITHUB_REQUIRED_ERROR:+$FM_PR_GITHUB_REQUIRED_ERROR -}the branch rules for base branch $base could not be read" - else - if ! rules_json=$(gh api --paginate "repos/$PR_OWNER/$PR_REPO/rules/branches/$branch_path" 2>"$api_err"); then - api_err_text=$(cat "$api_err" 2>/dev/null) - if ! github_branch_rules_unavailable_on_plan "$api_err_text"; then - FM_PR_GITHUB_REQUIRED_ERROR="${FM_PR_GITHUB_REQUIRED_ERROR:+$FM_PR_GITHUB_REQUIRED_ERROR -}the branch rules for base branch $base could not be read" - fi - elif [ -z "$rules_json" ] || ! ruleset=$(printf '%s' "$rules_json" | jq -c ' - if type != "array" then error("rules payload is unreadable") else .[] end - | select(type != "object" or .type == "required_status_checks") - | if type == "object" and (.parameters.required_status_checks | type) == "array" - then .parameters.required_status_checks[] else error("invalid required check rule") end - | if type == "object" and (.context | type) == "string" and (.context | length) > 0 - and (.integration_id == null or (.integration_id | type) == "number") - then {context, app_id: .integration_id} else error("invalid required check rule") end - | if .app_id == -1 then .app_id = null else . end' 2>/dev/null); then - ruleset='' - FM_PR_GITHUB_REQUIRED_ERROR="${FM_PR_GITHUB_REQUIRED_ERROR:+$FM_PR_GITHUB_REQUIRED_ERROR -}the branch rules for base branch $base could not be read" - fi - rm -f "$api_err" - fi - - FM_PR_GITHUB_REQUIRED=$(printf '%s\n%s\n' "$classic" "$ruleset" | jq -sc ' - unique_by([.context, .app_id]) | group_by(.context) - | map(if any(.[]; .app_id != null) then map(select(.app_id != null)) else . end) | add // []') - [ -z "$FM_PR_GITHUB_REQUIRED_ERROR" ] -} - -github_required_checks_missing() { - local json=$1 required=$2 producers=$3 - printf '%s' "$json" | jq -r --argjson required "$required" --argjson producers "$producers" ' - if (.statusCheckRollup | type) != "array" then error("no check rollup") else . end - | .statusCheckRollup as $reported - | $required - | map(. as $requirement - | select(any($reported[]; - if $requirement.app_id == null then - (if .__typename == "CheckRun" then .name else .context end) == $requirement.context - elif .__typename == "CheckRun" then - .name == $requirement.context - and any($producers[]; .name == $requirement.context and .app.id == $requirement.app_id) - else - .context == $requirement.context - end) | not) - | .context) | unique[] - ' 2>/dev/null || return 1 -} - # Pre-merge conditions from a live PR view, base requirements, and head producers. # Sets FM_PR_MERGE_HEAD to the verified head on success. Returns 3, rather than # the usual 1, when mergeable=UNKNOWN is the only failing condition, so the # caller can retry a still-computing mergeability read instead of refusing. github_verify_mergeable() { - local json fields line red failed name covered missing unreported producers runs pending + local json fields line red failed name covered missing unreported producers pending local total=0 named=0 refusals='' mergeable_refusal='' local state='' draft='' mergeable='' merge_state='' live_head='' base='' FM_PR_GITHUB_BEHIND=false @@ -1259,20 +1265,12 @@ EOF $FM_PR_GITHUB_REQUIRED_ERROR EOF fi - producers='[]' - if printf '%s' "$FM_PR_GITHUB_REQUIRED" | jq -e 'any(.[]; .app_id != null)' >/dev/null; then - if ! runs=$(gh api --paginate "repos/$PR_OWNER/$PR_REPO/commits/$live_head/check-runs" 2>/dev/null) \ - || [ -z "$runs" ] \ - || ! producers=$(printf '%s' "$runs" | jq -sc --arg head "$live_head" ' - [ .[] | if (.check_runs | type) == "array" then .check_runs[] else error("invalid check runs") end - | if (.name | type) == "string" and (.app.id | type) == "number" and .head_sha == $head - then . else error("invalid check producer") end ]' 2>/dev/null); then - producers='[]' - FM_PR_GITHUB_OTHER_REFUSAL=true - refusals="$refusals - required check producers at head $live_head could not be read + if ! github_read_required_producers "$live_head"; then + FM_PR_GITHUB_OTHER_REFUSAL=true + refusals="$refusals - required check producers at head $live_head could not be read " - fi fi + producers=$FM_PR_GITHUB_PRODUCERS if ! missing=$(github_required_checks_missing "$json" "$FM_PR_GITHUB_REQUIRED" "$producers"); then FM_PR_GITHUB_OTHER_REFUSAL=true refusals="$refusals - the GitHub pull request check rollup could not be read @@ -1477,38 +1475,6 @@ github_read_outcome() { return 1 } -github_urlencode_path_segment() { - local LC_ALL=C input=$1 encoded='' char octet hex - while [ -n "$input" ]; do - char=${input%"${input#?}"} - input=${input#?} - case "$char" in - [-._~a-zA-Z0-9]) encoded=$encoded$char ;; - *) - printf -v octet '%d' "'$char" - [ "$octet" -ge 0 ] || octet=$((octet + 256)) - printf -v hex '%02X' "$octet" - encoded=$encoded%$hex - ;; - esac - done - printf '%s' "$encoded" -} - -# Whether a failed branch-rules read (the gh stderr given) is GitHub's -# plan-gated 403 ("Upgrade to GitHub Pro or make this repository public"), -# which means the repository's plan cannot expose branch rules at all, on -# GitHub or GitHub Enterprise Server - not that this script failed to read -# them, and not that the token lacks a permission. Such a repository has no -# active ruleset rule of any kind. Any other failure (auth, rate limit, -# network, a 404, an unrelated 403) is not this and stays unreadable. -github_branch_rules_unavailable_on_plan() { - case "$1" in - *"Upgrade to GitHub Pro or make this repository public"*) return 0 ;; - esac - return 1 -} - # Read the effective merge-queue method for the observed base branch. The four # situations the refusal has to keep apart - no queue rule, a rules response # that could not be read, several rules that disagree, and a rule whose method @@ -1889,6 +1855,7 @@ case "$PROVIDER" in if ! caller_has_merge_method "$@"; then merge_args=(--squash) fi + github_add_squash_summary "$@" FM_PR_GITHUB_CALLER_METHOD=$(caller_merge_method "$@") # mergeable reads UNKNOWN for a short while after a push or base-branch # change while GitHub recomputes it; retry a bounded number of times, diff --git a/bin/fm-procevent-extension-capture.pl b/bin/fm-procevent-extension-capture.pl index c4e5b90fb4e..f5e16054350 100644 --- a/bin/fm-procevent-extension-capture.pl +++ b/bin/fm-procevent-extension-capture.pl @@ -143,18 +143,41 @@ sub publish_new { # can contain the same bytes on success, so only an explicit host-error sidecar # written for exit 70 can identify a reusable, still-unhandled capture. sub private_file_matches { - my ($name, $expected) = @_; + my ($name, $expected, $identity) = @_; sysopen(my $fh, $name, O_RDONLY | O_NOFOLLOW | O_NONBLOCK) or return 0; my @st = stat($fh); return 0 unless @st && -f $fh && $st[4] == $< && ($st[2] & 07777) == 0600 - && $st[3] == 1 && $st[7] == length($expected); + && $st[3] == 1 && ($identity ? $st[7] > 0 && $st[7] <= 1024 : $st[7] == length($expected)); my $bytes = ''; - while (length($bytes) < length($expected)) { - my $read = sysread($fh, my $buffer, length($expected) - length($bytes)); + while (length($bytes) < $st[7]) { + my $read = sysread($fh, my $buffer, $st[7] - length($bytes)); return 0 unless defined $read && $read > 0; $bytes .= $buffer; } - return $bytes eq $expected; + return $identity ? ($identity->($bytes) // '') eq $expected : $bytes eq $expected; +} +# Timing samples are diagnostic, not a different failure. Retain the first +# complete capture while comparing its bound and operation on subsequent polls. +sub host_error_identity { + my ($bytes) = @_; + my $error = eval { decode_json($bytes) }; + return undef unless ref($error) eq 'HASH' + && ($error->{schema} // '') eq 'firstmate.process-event-extension-error.v1' + && ($error->{extension_id} // '') eq $extension_id + && ($error->{operation} // '') eq 'source.poll' + && !ref($error->{code}) && ($error->{code} // '') =~ /\A[a-z0-9-]{1,64}\z/; + return $bytes if keys(%$error) == 4; + return undef unless keys(%$error) == 5 && $error->{code} eq 'timeout'; + my $deadline = $error->{deadline}; + return undef unless ref($deadline) eq 'HASH' && keys(%$deadline) == 6; + for my $field ('bound', 'operation') { + return undef if ref($deadline->{$field}) || ($deadline->{$field} // '') !~ /\A[a-z][a-z0-9.-]{0,63}\z/; + } + for my $field ('timeout_ms', 'elapsed_ms', 'overrun_ms', 'grace_ms') { + return undef if ref($deadline->{$field}) || ($deadline->{$field} // '') !~ /\A[0-9]{1,16}\z/; + } + delete @{$deadline}{'elapsed_ms', 'overrun_ms'}; + return JSON::PP->new->canonical->encode($error); } sub random_token { open(my $random, '<', '/dev/urandom') or fail('cannot create capture reservation'); @@ -256,7 +279,7 @@ sub write_reservation { exit 0; } my $extension_owner = join("\n", "schema=fm-procevent-extension-owner.v1", "extension_id=$extension_id", "extension_version=$extension_version", "capability_version=$capability_version", "package_digest=$package_digest", "binding_digest=$binding_digest", ""); -my ($host_error, $error_bytes); +my ($host_error, $error_bytes, $error_identity); if ($rc == 70 && !$truncated && $written > 0 && $written <= 1024) { seek($stage, 0, 0) or fail("cannot rewind host error"); $error_bytes = ''; @@ -265,14 +288,10 @@ sub write_reservation { defined $read && $read > 0 or fail("cannot read host error"); $error_bytes .= $buffer; } - my $error = eval { decode_json($error_bytes) }; - if (ref($error) eq 'HASH' && keys(%$error) == 4 - && ($error->{schema} // '') eq 'firstmate.process-event-extension-error.v1' - && ($error->{extension_id} // '') eq $extension_id - && ($error->{operation} // '') eq 'source.poll' - && !ref($error->{code}) && ($error->{code} // '') =~ /\A[a-z0-9-]{1,64}\z/) { + $error_identity = host_error_identity($error_bytes); + if (defined $error_identity) { $host_error = "schema=fm-procevent-host-error.v1\nidentity=sha256:" - . sha256_hex(join("\0", $adapter, $extension_owner, $error_bytes)) . "\n"; + . sha256_hex(join("\0", $adapter, $extension_owner, $error_identity)) . "\n"; } } chdir($inbox_dir) or fail("cannot enter inbox directory"); @@ -282,7 +301,7 @@ sub write_reservation { my $prefix = "$id.$1"; next if -e "$prefix.handled" || -l "$prefix.handled"; next unless private_file_matches($marker, $host_error) - && private_file_matches("$prefix.result", $error_bytes) + && private_file_matches("$prefix.result", $error_identity, \&host_error_identity) && private_file_matches("$prefix.adapter", "$adapter\n") && private_file_matches("$prefix.extension", $extension_owner); close($stage) or fail("cannot close staged output"); diff --git a/bin/fm-record-hygiene-lib.sh b/bin/fm-record-hygiene-lib.sh new file mode 100644 index 00000000000..7550639bbcf --- /dev/null +++ b/bin/fm-record-hygiene-lib.sh @@ -0,0 +1,117 @@ +# shellcheck shell=bash +record_hygiene_report() { + local file id output now cutoff archive dir pr seen_prs prior n file_mtime read_timeout read_status exclude + read_timeout=${FM_RECORD_HYGIENE_READ_TIMEOUT:-10} + case "$read_timeout" in ''|*[!0-9]*|0) read_timeout=10 ;; esac + seen_prs= + now=$(date +%s) + cutoff=$((now - 86400)) + [ -d "$STATE" ] || return 0 + exclude=$(status_scan_parent_channel_exclude "$STATE") + + for file in "$STATE"/*.status; do + [ -f "$file" ] && [ ! -L "$file" ] || continue + [ "$file" = "$exclude" ] && continue + id=${file##*/} + id=${id%.status} + if [ ! -e "$STATE/$id.meta" ]; then + file_mtime=$(_fm_status_file_mtime "$file" 2>/dev/null) || file_mtime= + case "$file_mtime" in ''|*[!0-9]*) continue ;; esac + if [ "$file_mtime" -lt "$cutoff" ]; then + echo "RECORD_HYGIENE: status log without metadata older than 24 hours: $id" + # Archive only in the locked, mutating bootstrap pass. Detect-only startup + # must remain read-only. + if [ "${FM_BOOTSTRAP_DETECT_ONLY:-0}" != 1 ]; then + case "$id" in ''|.*|*[!A-Za-z0-9._-]*) continue ;; esac + # shellcheck source=bin/fm-wake-lib.sh + command -v fm_meta_lock_path >/dev/null 2>&1 || . "$SCRIPT_DIR/fm-wake-lib.sh" + lock=$(fm_meta_lock_path "$STATE/$id.meta") || continue + fm_lock_try_acquire "$lock" || continue + if { [ ! -e "$STATE/$id.meta" ] && [ ! -L "$STATE/$id.meta" ]; } \ + && [ -f "$file" ] && [ ! -L "$file" ]; then + file_mtime=$(_fm_status_file_mtime "$file" 2>/dev/null) || file_mtime= + case "$file_mtime" in ''|*[!0-9]*) file_mtime= ;; esac + if [ -n "$file_mtime" ] && [ "$file_mtime" -lt "$cutoff" ]; then + # shellcheck disable=SC2153 # DATA is set by the sourcing script (fm-bootstrap.sh). + mkdir -p -- "$DATA" || { fm_lock_release "$lock"; continue; } + dir=$(CDPATH='' cd -- "$DATA" && pwd -P) || { fm_lock_release "$lock"; continue; } + dir="$dir/$id" + if { [ ! -e "$dir" ] && [ ! -L "$dir" ]; } || { [ -d "$dir" ] && [ ! -L "$dir" ]; }; then + mkdir -p -- "$dir" || { fm_lock_release "$lock"; continue; } + archive="$dir/status.log" + n=0 + while [ -e "$archive" ] || [ -L "$archive" ]; do + n=$((n + 1)) + archive="$dir/status.$n.log" + done + if mv -- "$file" "$archive"; then + # Start status-archive retention at archive time, not at the + # orphan's last write, so teardown's 30-day prune keeps it. + touch -- "$archive" || echo "RECORD_HYGIENE: could not refresh archived status log time: $id" >&2 + else + echo "RECORD_HYGIENE: could not archive orphan status log: $id" >&2 + fi + fi + fi + fi + fm_lock_release "$lock" + fi + fi + fi + done + + for file in "$STATE"/*.meta; do + [ -f "$file" ] && [ ! -L "$file" ] || continue + id=${file##*/} + id=${id%.meta} + [ "$(fm_meta_get "$file" kind)" = secondmate ] && continue + [ -n "$(fm_meta_get "$file" remote_host)" ] && continue + if [ -z "$(fm_meta_get "$file" window)" ]; then + echo "RECORD_HYGIENE: metadata has no tab: $id" + fi + pr=$(sed -n 's/^pr=//p' "$file" | head -1) + [ -n "$pr" ] || continue + prior=$(printf '%s' "$seen_prs" | awk -F '\t' -v pr="$pr" '$1 == pr { print $2; exit }') + if [ -n "$prior" ]; then + echo "RECORD_HYGIENE: duplicate PR $pr in records: $prior, $id" + else + seen_prs="$(printf '%s\t%s' "$pr" "$id")"$'\n'"$seen_prs" + fi + done + + if command -v tasks-axi >/dev/null 2>&1 && [ -d "$DATA" ]; then + output=$(FM_TASKS_AXI_TIMEOUT="$read_timeout" fm_backlog_row_list "$DATA" --state in_flight 2>&1) + read_status=$? + if fm_timed_out "$read_status"; then + echo "RECORD_HYGIENE: In-flight backlog read timed out after ${read_timeout}s" + return 0 + elif [ "$read_status" -ne 0 ]; then + echo "RECORD_HYGIENE: In-flight backlog read failed (exit=$read_status): $(printf '%s' "$output" | head -1)" + return 0 + fi + while IFS= read -r id; do + [ -n "$id" ] || continue + case "$id" in + */*) echo "RECORD_HYGIENE: In-flight backlog row without metadata: $id" ;; + *) [ -f "$STATE/$id.meta" ] && [ ! -L "$STATE/$id.meta" ] || echo "RECORD_HYGIENE: In-flight backlog row without metadata: $id" ;; + esac + done < <(printf '%s\n' "$output" | awk ' + /^tasks\[[0-9][0-9]*\]\{/ { rows=1; next } + rows && /^[[:space:]]/ { + sub(/^[[:space:]]+/, "") + if (substr($0, 1, 1) == "\"") { + value = substr($0, 2) + gsub(/""/, "\034", value) + sub(/".*/, "", value) + gsub(/\034/, "\"", value) + } else { + sub(/,.*/, "") + value = $0 + } + if (value != "") print value + next + } + { rows=0 } + ') + fi +} diff --git a/bin/fm-remote-job-lib.sh b/bin/fm-remote-job-lib.sh index c4dc9de6caa..2b4afd0eda1 100755 --- a/bin/fm-remote-job-lib.sh +++ b/bin/fm-remote-job-lib.sh @@ -71,7 +71,8 @@ # # The worker accepts only a tracked, non-symlink executable named fm-*.sh below # its configured FM_ROOT/bin. Every child receives env -i with the composed -# PATH, HOME, FM_HOME, FM_ROOT_OVERRIDE, and FM_REMOTE_JOB_ACTIVE=1. The PATH +# PATH, HOME, FM_HOME, FM_ROOT_OVERRIDE, and FM_REMOTE_JOB_ACTIVE=1, plus USER +# and LOGNAME when they are nonempty in the worker's environment. The PATH # is intentionally filesystem-discovered rather than login-shell-derived: # ~/.local/bin; nvm, asdf, and mise shims/install bins; Nix; Homebrew; and the # system tail. No shell startup files are evaluated. Each discovered set is diff --git a/bin/fm-remote-job-worker.sh b/bin/fm-remote-job-worker.sh index 474500aac30..4a84f7d7b4f 100755 --- a/bin/fm-remote-job-worker.sh +++ b/bin/fm-remote-job-worker.sh @@ -7,10 +7,10 @@ # account queue, refuses symlinks and malformed records, and executes only a # tracked non-symlink fm-*.sh under this worker's configured FM_ROOT/bin. # -# Each child runs under env -i with the shared filesystem-composed PATH, HOME, -# FM_HOME, FM_ROOT_OVERRIDE, and FM_REMOTE_JOB_ACTIVE=1. Commands receive their -# captured stdin and have a 360-second default timeout. Their stdout and stderr -# are independently constrained to the job library's 1048576-byte bound. A +# The shared library header owns the child environment contract. Commands +# receive their captured stdin and have a 360-second default timeout. Their +# stdout and stderr are independently constrained to the job library's +# 1048576-byte bound. A # record is marked done only after its bounded outputs and numeric exit status # have been committed. The library header owns the exact record fields and # lifecycle. @@ -945,6 +945,9 @@ worker_run_job() { # <account-home> <job-dir> if [ -n "${FM_REMOTE_JOB_PLATFORM_OVERRIDE:-}" ]; then child_env+=("FM_REMOTE_JOB_PLATFORM_OVERRIDE=$FM_REMOTE_JOB_PLATFORM_OVERRIDE") fi + # macOS credential lookups (claude auth status) key on the login identity. + [ -z "${USER:-}" ] || child_env+=("USER=$USER") + [ -z "${LOGNAME:-}" ] || child_env+=("LOGNAME=$LOGNAME") remaining=$((deadline - $(date +%s))) [ "$remaining" -gt 0 ] || { worker_cleanup_output_capture "$job" "$stdout_reader" "$stderr_reader" diff --git a/bin/fm-review-check.sh b/bin/fm-review-check.sh index f118bcd8f8f..18a9f051492 100755 --- a/bin/fm-review-check.sh +++ b/bin/fm-review-check.sh @@ -68,6 +68,9 @@ MODE=$(fm_dod_meta_value "$META" mode) WT=$(fm_dod_meta_value "$META" worktree) PROJECT=$(fm_dod_meta_value "$META" project) DONE_LINE="done: review $URL uploaded for review" +# No remote-ref refresh here: a forge=external done is decided by +# fm_dod_accept_external_review_done, whose optional backup-remote check reads +# a push the worker made from this same copy. if ! GATE_REASON=$(fm_dod_accept_ship_done "${KIND:-ship}" "$MODE" "$WT" "$PROJECT" "$DONE_LINE" "$STATE" "$ID" "$META"); then echo "error: $GATE_REASON" >&2 exit 1 diff --git a/bin/fm-send.sh b/bin/fm-send.sh index 2384b24e49f..1198106fb93 100755 --- a/bin/fm-send.sh +++ b/bin/fm-send.sh @@ -1136,8 +1136,20 @@ else fi if [ "$INBOX_PLANE" = 1 ]; then INBOX_TASK_ID=$(fm_send_id_from_meta "$TARGET_META") - INBOX_META_LOCK=$(fm_meta_lock_path "$TARGET_META") || exit 1 + INBOX_DELIVERY_LOCK=$(fm_task_inbox_delivery_lock_path "$STATE" "$INBOX_TASK_ID") + if ! fm_task_inbox_lock_acquire "$INBOX_DELIVERY_LOCK"; then + if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then + fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true + fi + echo "error: steer not sent to $INBOX_TASK_ID: its inbox delivery lock could not be acquired" >&2 + exit 1 + fi + INBOX_META_LOCK=$(fm_meta_lock_path "$TARGET_META") || { + fm_lock_release "$INBOX_DELIVERY_LOCK" + exit 1 + } if ! fm_task_inbox_lock_acquire "$INBOX_META_LOCK"; then + fm_lock_release "$INBOX_DELIVERY_LOCK" if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true fi @@ -1158,6 +1170,7 @@ else [ "$CURRENT_INBOX_SPAWN_GEN" != "$FM_SEND_EXPECTED_SPAWN_GEN" ]; } || [ -n "$(fm_meta_get "$TARGET_META" remote_host)" ]; then fm_lock_release "$INBOX_META_LOCK" + fm_lock_release "$INBOX_DELIVERY_LOCK" if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true fi @@ -1173,6 +1186,7 @@ else fi if [ "${inbox_write_rc:-0}" -ne 0 ]; then fm_lock_release "$INBOX_META_LOCK" + fm_lock_release "$INBOX_DELIVERY_LOCK" if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true fi @@ -1180,6 +1194,7 @@ else exit 1 fi fm_lock_release "$INBOX_META_LOCK" + fm_lock_release "$INBOX_DELIVERY_LOCK" # Enqueue IS durable delivery to the task's record: mark the pending # expectation delivered now, without resolving it - only a correlated # parent report acknowledges the request. diff --git a/bin/fm-session-lock-lib.sh b/bin/fm-session-lock-lib.sh index 391496faf21..339da9da434 100644 --- a/bin/fm-session-lock-lib.sh +++ b/bin/fm-session-lock-lib.sh @@ -25,6 +25,8 @@ _FM_SESSION_LOCK_LIB_DIR=${BASH_SOURCE[0]%/*} [ "$_FM_SESSION_LOCK_LIB_DIR" != "${BASH_SOURCE[0]}" ] || _FM_SESSION_LOCK_LIB_DIR=. # shellcheck source=bin/fm-cursor-lib.sh . "${_FM_SESSION_LOCK_LIB_DIR:-/}/fm-cursor-lib.sh" +# shellcheck source=bin/fm-omp-process-lib.sh +. "${_FM_SESSION_LOCK_LIB_DIR:-/}/fm-omp-process-lib.sh" unset _FM_SESSION_LOCK_LIB_DIR # Known harness command names; extend when a new adapter is verified. omp and @@ -106,6 +108,8 @@ fm_ps_ppid() { # <pid> -> parent pid # is identified by its install path on macOS and by argv[0] on Linux. # 3. a bare interpreter (node, python) running a harness script path. # 4. Cursor's own structural identity, owned by bin/fm-cursor-lib.sh. +# 5. omp's structural identity, owned by bin/fm-omp-process-lib.sh, which +# also covers the bun-package install whose process name is `bun`. FM_HARNESS_IS_CLAUDE=0 fm_harness_process_matches() { # <comm> <args> local comm=$1 args=$2 base argv0 name @@ -134,6 +138,7 @@ fm_harness_process_matches() { # <comm> <args> # locate its own harness in the ancestry, so every session start refuses the # fleet lock as read-only and the park can never arm. fm_cursor_process_matches "$comm" "$args" "$argv0" && return 0 + fm_omp_process_matches "$comm" "$args" && return 0 return 1 } diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 05857172275..2457ca4b9a3 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -497,7 +497,8 @@ # Worker tool exclusions: # docs/configuration.md "Worker tool exclusions" owns config/crew-exclude-tools # and its operator contract. Resolve it with bin/fm-exclude-tools-lib.sh -# before provisioning; __PIEXCLUDE__ below owns the Pi launch substitution. +# before provisioning; __PIEXCLUDE__ and __CLAUDEEXCLUDE__ below own the Pi and +# Claude launch substitutions. # Worker account pin (config/claude-account, config/pi-account): # Opt-in. docs/configuration.md's "Worker account pin" section owns unpinned # account selection. A present file pins every supported launch of @@ -546,6 +547,12 @@ # __PIEXCLUDE__ optional ` --exclude-tools '<comma-joined names>'` from # config/crew-exclude-tools on Pi/pi-signed ship and scout # launches (supplies its own leading space, empty otherwise) +# __CLAUDEEXCLUDE__ optional `--disallowedTools '<comma-joined names>' ` from +# config/crew-exclude-tools on Claude ship and scout launches +# (supplies its own trailing space, empty otherwise). Claude +# Code drops a disallowed tool's schema from every request, and +# a later flag always follows so the variadic option cannot +# swallow the launch prompt. # __PIRESUME__ optional relaunch-only `--session <reference>` that keeps a # Pi replacement on the session the endpoint's runtime already # reports (relaunch_resume_args below owns it; it supplies its @@ -2919,6 +2926,7 @@ launch_template() { claude) printf '%s' 'CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false CLAUDE_CODE_SEND_FEEDBACK=0 claude __CLAUDEPERMFLAG__ __CLAUDEADDDIRS__--settings __CLAUDESETTINGS__ ' if [ "$kind" != secondmate ]; then + printf '%s' '__CLAUDEEXCLUDE__' printf '%s' "--append-system-prompt '$(spawn_claude_task_channel_statement)' " fi # Claude Code strips invisible characters, U+2063 included, from the @@ -5994,8 +6002,9 @@ EOF } # Rewrite <meta> into <out> without the <owned keys>, placing <line>... ahead -# of the first pr= line so no key fm_pr_metadata_identity_parse refuses lands -# after the PR region. +# of the first pr= line so the PR region stays the record's tail that +# fm_pr_metadata_identity_parse requires. Preserved PR metadata must remain +# the record tail or the armed merge poll stops validating relaunched lanes. spawn_meta_rewrite_before_pr() { # <meta> <out> <owned keys> <line>... local meta=$1 out=$2 keys=$3 shift 3 @@ -7423,8 +7432,9 @@ preserve_relaunch_meta() { if [ "$SPAWN_CONTROL_PARENT" = 1 ] && [ -n "${FM_CONTROL_RELAUNCH_TX:-}" ]; then echo "control_relaunch_tx=$FM_CONTROL_RELAUNCH_TX" fi - # Preserved lines go last so nothing fm_pr_metadata_identity_parse - # (bin/fm-pr-lib.sh) refuses is written after an armed pr= identity region. + # Preserved lines go last so an armed pr= identity block stays last, as + # fm_pr_metadata_identity_parse (bin/fm-pr-lib.sh) requires; otherwise the + # armed merge poll stops validating relaunched lanes. if [ "$RELAUNCH" -eq 1 ]; then preserve_relaunch_meta fi @@ -7546,6 +7556,9 @@ if [ "$RELAUNCH" -eq 1 ]; then fi LAUNCH=${LAUNCH//__PIRESUME__/$RESUME_ARGS} LAUNCH=${LAUNCH//__CLAUDEPERMFLAG__/$CLAUDE_PERM_FLAG} +CLAUDE_EXCLUDE= +[ -z "$EXCLUDE_TOOLS" ] || [ "$HARNESS" != claude ] || CLAUDE_EXCLUDE="--disallowedTools $(shell_quote "$EXCLUDE_TOOLS") " +LAUNCH=${LAUNCH//__CLAUDEEXCLUDE__/$CLAUDE_EXCLUDE} case "$LAUNCH" in *__CLAUDESETTINGS__*) if [ "$KIND" = secondmate ]; then diff --git a/bin/fm-task-inbox-lib.sh b/bin/fm-task-inbox-lib.sh index 7d4368bce1f..8a39d47ea88 100644 --- a/bin/fm-task-inbox-lib.sh +++ b/bin/fm-task-inbox-lib.sh @@ -159,6 +159,11 @@ fm_task_inbox_dir() { # <state-dir> <task-id> printf '%s/%s.inbox' "$1" "$2" } +# Serializes a local inbox enqueue with parking the same task. +fm_task_inbox_delivery_lock_path() { # <state-dir> <task-id> + printf '%s/.%s.inbox-delivery.lock' "$1" "$2" +} + fm_task_inbox_handled_dir() { # <state-dir> <task-id> printf '%s/%s.inbox/handled' "$1" "$2" } diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index 273ffb9dab1..a4673b4ef46 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -49,7 +49,9 @@ # refresh_remote_refs has updated those refs from each remote when the local ones # do not show the work, OR - for a # normal ship task whose commits are not so reachable - when its PR is merged and -# GitHub reports a PR head that contains the current local work, or its content is +# the forge (GitHub, Forgejo, or - only when config/park-on-review or +# config/auto-land opts this home in - a GitLab merge request) reports a PR head +# that contains the current local work, or its content is # already present in the up-to-date default branch. This recognizes the common # squash-merge-then-delete-branch flow, where the branch's own commits live nowhere # on a remote yet the change is fully in main. A task whose meta records @@ -402,6 +404,7 @@ for _teardown_source in \ fm-composer-lib.sh \ fm-cursor-lib.sh \ fm-nm-run-lib.sh \ + fm-dod-lib.sh \ fm-wake-lib.sh \ fm-path-lib.sh \ fm-lease-lib.sh @@ -433,6 +436,9 @@ unset _teardown_source . "$SCRIPT_DIR/fm-secondmate-parent-lib.sh" # shellcheck source=bin/fm-nm-run-lib.sh . "$SCRIPT_DIR/fm-nm-run-lib.sh" +# The bounded remote-ref refresh is shared with the done-claim gate. +# shellcheck source=bin/fm-dod-lib.sh +. "$SCRIPT_DIR/fm-dod-lib.sh" # Archive paths live only in teardown; presentation/wake readers stay unaware. teardown_archive_directory() { # <directory> if [ -L "$1" ] || { [ -e "$1" ] && [ ! -d "$1" ]; }; then @@ -1668,12 +1674,25 @@ pr_number_from_target() { printf '%s' "$n" } +# GitLab serves a merge request's head as refs/merge-requests/<n>/head; GitHub +# and Forgejo serve a pull request's as refs/pull/<n>/head. ensure_commit_object() { - local target=$1 commit=$2 n + local target=$1 commit=$2 n ref git -C "$WT" cat-file -e "$commit^{commit}" 2>/dev/null && return 0 - n=$(pr_number_from_target "$target") || return 1 + case "$target" in + *"/-/merge_requests/"*) + n=${target##*/-/merge_requests/} + n=${n%%[!0-9]*} + [ -n "$n" ] || return 1 + ref="refs/merge-requests/$n/head" + ;; + *) + n=$(pr_number_from_target "$target") || return 1 + ref="refs/pull/$n/head" + ;; + esac git -C "$WT" remote get-url origin >/dev/null 2>&1 || return 1 - git -C "$WT" fetch --quiet origin "refs/pull/$n/head" >/dev/null 2>&1 || return 1 + git -C "$WT" fetch --quiet origin "$ref" >/dev/null 2>&1 || return 1 git -C "$WT" cat-file -e "$commit^{commit}" 2>/dev/null } @@ -1714,10 +1733,20 @@ EOF # Is the worktree's PR merged for local work contained in that PR? Resolves the # PR from the recorded pr= URL first, then from the branch name, and asks the # forge for both the PR state and head: a recorded Forgejo URL is read through -# tea (fm_pr_forgejo_read_record), and everything else through GitHub. Returns +# tea (fm_pr_forgejo_read_record), a recorded GitLab merge request through glab +# (fm_pr_gitlab_read_record, whose head is the MR's own head rather than the +# squash or merge-train commit main received) when gitlab_mr_landing_opted_in, +# and everything else through GitHub. Returns # non-zero when the PR is not merged, the current work is not contained in the # PR head, no PR is found, or any forge read error occurs - the caller then # falls back to the content check. +# The GitLab merged-MR read is opt-in with the park-on-review flags +# (bin/fm-park.sh, docs/configuration.md "Park on review"); without either a +# GitLab MR goes through the GitHub read like any other target. +gitlab_mr_landing_opted_in() { + [ -e "$CONFIG/park-on-review" ] || [ -e "$CONFIG/auto-land" ] +} + pr_is_merged() { local branch=$1 target view state remainder head resolved_url current landed=0 if [ -n "$PR_URL" ]; then @@ -1732,6 +1761,12 @@ pr_is_merged() { head=$FM_PR_RECORD_HEAD fm_pr_head_valid "$head" || return 1 resolved_url=$FM_PR_URL + elif gitlab_mr_landing_opted_in && fm_pr_url_parse "$target" && [ "$FM_PR_PROVIDER" = gitlab ]; then + fm_pr_gitlab_read_record "$FM_PR_HOST" "$FM_PR_PATH" "$FM_PR_NUMBER" || return 1 + [ "$FM_PR_RECORD_MERGED" = true ] || return 1 + head=$FM_PR_RECORD_HEAD + fm_pr_head_valid "$head" || return 1 + resolved_url=$FM_PR_URL else view=$(cd "$WT" && gh pr view "$target" --json state,headRefOid,url -q '.state + "\t" + .headRefOid + "\t" + .url' 2>/dev/null) || return 1 state=${view%%$'\t'*} @@ -2131,33 +2166,22 @@ teardown_note_dirty_return() { # A worker may have pushed from another checkout, so tracking refs that do not # show the work prove nothing until they are refreshed. Runs at most once per -# teardown. Each fetch is bounded to 30 seconds and non-interactive, and -# --no-prune keeps a branch a remote has dropped visible, whatever fetch.prune -# says. The explicit -# refspec covers every branch even when a clone tracks only some of them. A failed -# fetch only leaves the last-known refs in place, so the unchanged checks below -# still refuse work those refs do not show; REMOTE_REFRESH_FAILED names it for the -# refusal. The no-mistakes gate is skipped: it is a local validation copy, and -# fetching it would count work as landed that no real remote holds. +# teardown, through bin/fm-dod-lib.sh's fm_dod_refresh_remote_refs, which owns +# the bounded, no-prune fetch and skips the no-mistakes gate. A failed fetch +# only leaves the last-known refs in place, so the unchanged checks below still +# refuse work those refs do not show; FM_DOD_REFRESH_FAILED names it for the +# refusal. REMOTE_REFS_REFRESHED=0 -REMOTE_REFRESH_FAILED= refresh_remote_refs() { - local remote [ "$REMOTE_REFS_REFRESHED" = 0 ] || return 0 REMOTE_REFS_REFRESHED=1 - while IFS= read -r remote; do - case "$remote" in ''|no-mistakes) continue ;; esac - GIT_TERMINAL_PROMPT=0 fm_run_timed 30 git -C "$WT" fetch --quiet --no-prune --no-tags --no-recurse-submodules \ - "$remote" "+refs/heads/*:refs/remotes/$remote/*" </dev/null >/dev/null 2>&1 \ - || REMOTE_REFRESH_FAILED="$REMOTE_REFRESH_FAILED${REMOTE_REFRESH_FAILED:+, }$remote" - done <<EOF -$(git -C "$WT" remote 2>/dev/null) -EOF + FM_DOD_REFRESH_FAILED= + fm_dod_refresh_remote_refs "$WT" 30 } report_remote_refresh_failure() { - [ -n "$REMOTE_REFRESH_FAILED" ] || return 0 - echo "Could not refresh remote: $REMOTE_REFRESH_FAILED. This verdict used the last-known refs, so the work may be there; retry once the remote is reachable." >&2 + [ "$REMOTE_REFS_REFRESHED" = 1 ] && [ -n "$FM_DOD_REFRESH_FAILED" ] || return 0 + echo "Could not refresh remote: $FM_DOD_REFRESH_FAILED. This verdict used the last-known refs, so the work may be there; retry once the remote is reachable." >&2 } # Sets the caller's $dirty from a fresh status read, or refuses when the copy @@ -4195,6 +4219,8 @@ rm -f "$STATE/$ID.turn-ended" "$STATE/$ID.progress" \ "$STATE/$ID.reconcile-nudged" "$STATE/$ID.gemini-settings.json" \ "$STATE/$ID.claude-settings.json" "$STATE/$ID.devin-config.json" \ "$STATE/$ID.polytoken-busy.sh" "$STATE/$ID.pr-register-refused" \ + "$STATE/$ID.review-park" "$STATE/.review-park-closed-$ID" \ + "$STATE/$ID.pr-register-timed-out" \ "$STATE/.$ID.branch-outcome-index" \ "$STATE/.secondmate-relaunch-$ID" "$STATE/.secondmate-relaunch-bound-$ID" # kiro's per-task home is a directory (its relocated KIRO_HOME holds the hook diff --git a/bin/fm-test-isolation-proof.sh b/bin/fm-test-isolation-proof.sh index cffbb47f21b..25ad763e73a 100755 --- a/bin/fm-test-isolation-proof.sh +++ b/bin/fm-test-isolation-proof.sh @@ -181,6 +181,7 @@ tests/fm-ensure-agents-md.test.sh tests/fm-grok-harness.test.sh tests/fm-herdr-lab.test.sh tests/fm-lint.test.sh +tests/fm-park.test.sh tests/fm-pi-primary-types.test.sh tests/fm-pr-merge.test.sh tests/fm-review-diff.test.sh diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 3560120928a..fe318e73ba4 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -391,7 +391,8 @@ family_for_basename() { fm-calm-pi-queue-retention-live-e2e.test.sh|\ fm-herdr-submit-confirm-live-e2e.test.sh|\ fm-backend-rex-composer-live-e2e.test.sh|\ - fm-watch-wedge-cap-live.test.sh|fm-read-repetition-live.test.sh) + fm-watch-wedge-cap-live.test.sh|fm-read-repetition-live.test.sh|\ + fm-claude-tool-exclude-live.test.sh) printf '%s\n' live-harness-optin ;; fm-backend-herdr.test.sh|fm-backend-tmux-smoke.test.sh|fm-backend.test.sh|\ @@ -409,7 +410,7 @@ family_for_basename() { fm-teardown-endpoint-safety.test.sh) printf '%s\n' backend-dispatch ;; - fm-check-unregister.test.sh|fm-pipeline-spend.test.sh|fm-pr-check-security.test.sh|\ + fm-check-unregister.test.sh|fm-park.test.sh|fm-pipeline-spend.test.sh|fm-pr-check-security.test.sh|\ fm-pr-health.test.sh|fm-pr-merge.test.sh|fm-pr-reviewers.test.sh|fm-pr-state.test.sh|\ fm-review-diff.test.sh|fm-teardown.test.sh|fm-x-mode.test.sh) printf '%s\n' pr-forge @@ -535,6 +536,7 @@ tests/fm-ensure-agents-md.test.sh tests/fm-grok-harness.test.sh tests/fm-herdr-lab.test.sh tests/fm-lint.test.sh +tests/fm-park.test.sh tests/fm-pi-primary-types.test.sh tests/fm-pr-merge.test.sh tests/fm-review-diff.test.sh @@ -569,6 +571,7 @@ tests/fm-ensure-agents-md.test.sh 895 tests/fm-grok-harness.test.sh 7666 tests/fm-herdr-lab.test.sh 18325 tests/fm-lint.test.sh 252498 +tests/fm-park.test.sh 70085 tests/fm-pi-primary-types.test.sh 5426 tests/fm-pr-merge.test.sh 300199 tests/fm-review-diff.test.sh 4134 @@ -606,8 +609,8 @@ list_portable_parallel_1() { tests/fm-captain-hold-lifecycle.test.sh tests/fm-backend-t3code.test.sh tests/fm-crew-state.test.sh +tests/fm-park.test.sh tests/fm-t3-mcp.test.sh -tests/fm-arm-pretool-check.test.sh tests/fm-herdr-lab.test.sh tests/fm-cd-pretool-check.test.sh tests/fm-grok-harness.test.sh @@ -626,6 +629,7 @@ tests/fm-pr-merge.test.sh tests/fm-lint.test.sh tests/fm-test-run.test.sh tests/fm-backend-herdr.test.sh +tests/fm-arm-pretool-check.test.sh tests/fm-x-mode.test.sh tests/fm-brief.test.sh tests/fm-composer-lib.test.sh @@ -783,6 +787,7 @@ tests/fm-classify-decision-key.test.sh 4362 tests/fm-claude-account-quota.test.sh 2492 tests/fm-claude-stop-autoarm-live-e2e.test.sh 73 tests/fm-claude-stop-autoarm.test.sh 61189 +tests/fm-claude-tool-exclude-live.test.sh 75 tests/fm-claude-trust.test.sh 12010 tests/fm-cmux-claude-composer-live-e2e.test.sh 77 tests/fm-codex-continuity-live-e2e.test.sh 108 @@ -1736,6 +1741,9 @@ families_for_changed_path() { printf '%s\n' __script__:fm-read-repetition-live.test.sh printf '%s\n' __script__:fm-pi-primary-types.test.sh ;; + tests/fixtures/fm-claude-request-capture.py) + printf '%s\n' __script__:fm-claude-tool-exclude-live.test.sh + ;; .pi/extensions/fm-branch-supervision.ts|.pi/extensions/lib/fm-async-exec.ts|\ .pi/extensions/lib/fm-branch-dispatch.ts|.pi/extensions/lib/fm-native-contract.ts) # The portable suites that actually load these files, named one by one. @@ -1794,7 +1802,7 @@ families_for_changed_path() { printf '%s\n' session-bootstrap printf '%s\n' live-harness-optin ;; - bin/fm-extension.mjs|bin/fm-extension.sh|docs/examples/process-event-extension/*) + bin/fm-extension*.mjs|bin/fm-extension.sh|docs/examples/process-event-extension/*) printf '%s\n' __script__:fm-extension-binding.test.sh printf '%s\n' __script__:fm-procevent-extension-errors.test.sh ;; diff --git a/bin/fm-wake-drain.sh b/bin/fm-wake-drain.sh index ff6a2ac8ffe..19d27f08469 100755 --- a/bin/fm-wake-drain.sh +++ b/bin/fm-wake-drain.sh @@ -91,6 +91,9 @@ rotate_scratch_locked() { done } +# Main reclaims stale grants. A branch drain uses the grant just published for +# its turn, and its sandbox may not be able to inspect the host's PID via ps. +# Reclaiming from that actor would delete its own snapshot before it can drain. reclaim_stale_branch_grant_locked() { [ -e "$ELIGIBLE_ROWS_FILE" ] || [ -L "$ELIGIBLE_ROWS_FILE" ] || return 0 if ! fm_wake_branch_grant_live "$ELIGIBLE_ROWS_FILE" "$ELIGIBLE_OWNER_FILE"; then @@ -851,7 +854,7 @@ else fi DRAIN_LOCK_HELD=true rotate_scratch_locked -reclaim_stale_branch_grant_locked || exit 1 +[ "$ACTOR" = branch ] || reclaim_stale_branch_grant_locked || exit 1 [ "$ACTOR" != main ] || retire_unconsumable_rows_locked [ "$ACTOR" != branch ] || require_branch_eligible_rows || exit 1 diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index a93f692b361..7ccb0ecca69 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -627,7 +627,7 @@ fm_lock_holder_alive() { local lockdir=$1 pid=$2 fm_pid_alive "$pid" || return 1 _fm_lock_record_matches "$lockdir" "$pid" pid-start fm_pid_start_time || return 1 - [ "$lockdir" = "$STATE/.watch.lock" ] || return 0 + [ "$lockdir" -ef "$STATE/.watch.lock" ] || return 0 _fm_lock_record_matches "$lockdir" "$pid" pid-identity fm_pid_identity } @@ -754,7 +754,7 @@ fm_lock_wait_pending_mutation() { } fm_lock_try_create() { - local lockdir=$1 allowed_steal_owner=${2:-} ownerdir + local lockdir=$1 allowed_steal_owner=${2:-} ownerdir= local FM_LOCK_PENDING_PREVIOUS_PID FM_LOCK_PENDING_PID= FM_LOCK_OWNER_DIR= if [ "${BASH_VERSINFO[0]}" -lt 4 ]; then @@ -762,14 +762,23 @@ fm_lock_try_create() { FM_LOCK_PENDING_PREVIOUS_PID=$(set +u; printf '%s' "$!") # Bash 3.2 defers fatal signals until command substitutions finish, and # does not expose a process substitution's pid through $!. - ownerdir=$(fm_lock_owner_dir "$lockdir") || return 1 + if ! ownerdir=$(fm_lock_owner_dir "$lockdir"); then + fm_lock_discard_owner "$ownerdir" + return 1 + fi else FM_LOCK_PENDING_PREVIOUS_PID=${!:-} # Newer bash can enter EXIT before the child finishes. A process # substitution exposes its pid even before read returns the owner path. - IFS= read -r ownerdir < <(fm_lock_owner_dir "$lockdir") || return 1 + if ! IFS= read -r ownerdir < <(fm_lock_owner_dir "$lockdir"); then + fm_lock_discard_owner "$ownerdir" + return 1 + fi FM_LOCK_PENDING_PID=$! - wait "$FM_LOCK_PENDING_PID" 2>/dev/null || return 1 + if ! wait "$FM_LOCK_PENDING_PID" 2>/dev/null; then + fm_lock_discard_owner "$ownerdir" + return 1 + fi FM_LOCK_PENDING_PREVIOUS_PID=$FM_LOCK_PENDING_PID FM_LOCK_PENDING_PID= fi @@ -800,6 +809,10 @@ fm_lock_try_create() { fm_lock_remove_path() { local lockdir=$1 ownerdir + if [ "${2:-}" = stale ] && [ -d "$lockdir" ] && [ ! -L "$lockdir" ]; then + fm_lock_reap_dead_directory "$lockdir" + return + fi if [ -L "$lockdir" ]; then ownerdir=$(fm_lock_link_owner "$lockdir" 2>/dev/null || true) rm -f "$lockdir" 2>/dev/null || return 1 @@ -823,6 +836,64 @@ fm_lock_mid_acquire_is_fresh() { return 1 } +# A real directory can contain owner records left inside it instead of being +# the symlink to one owner record. Age never overrides a live nested holder. +fm_lock_nested_owners_stale() { + local lockdir=$1 base=${2:-${1##*/}} owner pid + for owner in "$lockdir/$base.owner."*; do + [ -d "$owner" ] && [ ! -L "$owner" ] || continue + pid=$(cat "$owner/pid" 2>/dev/null || true) + fm_lock_holder_alive "$owner" "$pid" && return 1 + fm_lock_mid_acquire_is_fresh "$owner" "$pid" && return 1 + done + return 0 +} + +# Pin cleanup to the real directory through cwd. Competing reapers may remove +# its name and publish a successor symlink, but cleanup cannot follow that +# successor and rmdir cannot unlink it. Unknown contents remain untouched. +fm_lock_reap_dead_directory() { + local lockdir=$1 lock_abs base rc directory_pid nested_owner + lock_abs=$(fm_lock_abs_path "$lockdir") || return 1 + base=${lock_abs##*/} + if ( + # A failed cd is permanent only while the same real directory remains. + # A concurrent reaper's removal or replacement must stay retriable. + if [ "$_FM_UNAME" = Darwin ]; then + identity_stat=(/usr/bin/stat -f '%d:%i') + else + identity_stat=(stat -c '%d:%i') + fi + directory_identity=$("${identity_stat[@]}" "$lock_abs" 2>/dev/null) || exit 1 + if ! cd -- "$lock_abs" 2>/dev/null; then + [ ! -L "$lock_abs" ] && [ -d "$lock_abs" ] \ + && [ "$directory_identity" = "$("${identity_stat[@]}" "$lock_abs" 2>/dev/null)" ] && exit 2 + exit 1 + fi + [ ! -L "$lock_abs" ] && [ . -ef "$lock_abs" ] || exit 1 + directory_pid=$(cat ./pid 2>/dev/null || true) + fm_lock_holder_alive "$lock_abs" "$directory_pid" && exit 1 + fm_lock_mid_acquire_is_fresh . "$directory_pid" && exit 1 + fm_lock_nested_owners_stale . "$base" || exit 1 + for nested_owner in "./$base.owner."*; do + [ -d "$nested_owner" ] && [ ! -L "$nested_owner" ] || continue + fm_lock_discard_owner "$nested_owner" + done + fm_lock_clean_known_files . + rmdir -- "$lock_abs" 2>/dev/null && exit 0 + [ . -ef "$lock_abs" ] || exit 1 + exit 2 + ); then + return 0 + else + rc=$? + fi + if [ "$rc" -eq 2 ]; then + FM_LOCK_RECLAIM_FAILED_PATH=$lockdir + fi + return 1 +} + fm_lock_recheck_stale_owner() { local lockdir=$1 expected_owner=$2 expected_pid=$3 actual_pid if [ -n "$expected_owner" ]; then @@ -838,6 +909,9 @@ fm_lock_recheck_stale_owner() { if fm_lock_mid_acquire_is_fresh "$lockdir" "$actual_pid"; then return 1 fi + if [ -d "$lockdir" ] && [ ! -L "$lockdir" ]; then + fm_lock_nested_owners_stale "$lockdir" || return 1 + fi return 0 } @@ -1376,7 +1450,8 @@ fm_recovery_marker_handover_restore() { # <marker> <snapshot-token> <snapshot-s } # fm_lock_reap_dead_link <lockdir> -# Remove a link lock whose owner is dead without a nested mutex. Renaming the +# Remove a dead real-directory lock, or a link lock without a nested mutex. +# For links, renaming the # dead owner directory to this process's tombstone elects exactly one reaper, # so a competing reaper that verified the same dead owner cannot remove a # successor's link. A reaper that died after winning leaves its tombstone; a @@ -1388,6 +1463,10 @@ fm_recovery_marker_handover_restore() { # <marker> <snapshot-token> <snapshot-s # probes keep the PID-only answer; only a proven mismatch permits takeover. fm_lock_reap_dead_link() { local lockdir=$1 owner pid token tomb current reaper recorded start + if [ -d "$lockdir" ] && [ ! -L "$lockdir" ]; then + fm_lock_remove_path "$lockdir" stale + return + fi [ -L "$lockdir" ] || return 1 owner=$(fm_lock_link_owner "$lockdir" 2>/dev/null) || return 1 fm_current_pid current || return 1 @@ -1456,6 +1535,7 @@ fm_lock_try_acquire() { FM_LOCK_HELD_PATH= FM_LOCK_OWNER_DIR= FM_LOCK_RECOVERED_PID= + FM_LOCK_RECLAIM_FAILED_PATH= if fm_lock_try_create "$lockdir"; then return 0 @@ -1543,7 +1623,7 @@ fm_lock_try_acquire() { FM_LOCK_OWNER_DIR= return 1 fi - fm_lock_remove_path "$lockdir" || true + fm_lock_remove_path "$lockdir" stale || true rc=1 if fm_lock_try_create "$lockdir" "$steal_owner"; then rc=0 @@ -1575,14 +1655,20 @@ fm_lock_parent_gone() { # <lockdir> [ ! -d "$parent" ] } -# Wait for a lock. The 0.1s retry serves live contention on short critical -# sections, where a wider interval would delay every handoff between the -# watcher and its writers. The wait is unbounded only while the lock can still -# be taken: it returns 1 within 0.2s of its parent directory's removal +# Wait for a lock without a contention deadline. The 0.1s retry serves short +# critical sections, where a wider interval would delay every handoff between +# the watcher and its writers. It returns 1 within 0.2s of its parent's removal # (fm_lock_parent_gone) instead of polling for a lock that can never exist. +# Callers that handle failure may opt into --refuse-unreapable to refuse a +# stale real directory that cannot be removed, naming its exact path. The +# ordinary wait and live contention retain their behavior without a deadline. fm_lock_acquire_wait() { - local lockdir=$1 + local lockdir=$1 refuse_unreapable=${2:-} while ! fm_lock_try_acquire "$lockdir"; do + if [ "$refuse_unreapable" = --refuse-unreapable ] && [ -n "${FM_LOCK_RECLAIM_FAILED_PATH:-}" ]; then + echo "error: cannot reclaim lock directory $FM_LOCK_RECLAIM_FAILED_PATH; inspect its contents and permissions" >&2 + return 1 + fi ! fm_lock_parent_gone "$lockdir" || return 1 sleep 0.1 done @@ -2080,7 +2166,9 @@ fm_failure_episode_reset() { # also records "session_pid=S recovery_generation=G", binding that # handling turn to its live session-lock owner and watcher recovery episode. # A same-session re-anchor of state/.lock moves S with lock line 1 -# (fm_autoarm_rekey_session). +# (fm_autoarm_rekey_session). A claim taken by a Stop firing whose payload +# named its Claude session also records "claude_session=C", which every +# later owned write of that generation keeps. # Line 2 is the claiming process's pid-identity, the same identity every other # supervision lock in this repo records (fm_pid_identity above). The # identity is MANDATORY: a claimant that cannot record it does not claim @@ -2120,11 +2208,19 @@ fm_failure_episode_reset() { # whichever id the sidecar carries after lock recovery. An entry written by # a build before these fields existed records neither and falls back to the # descent signal alone. +# For a firing that names its Claude session, a claim recorded under a +# different claude_session is not open either: Claude delivers a hook's +# exit-2 rewake to the session that launched it, so a claim from a session +# Claude has since continued under a new id (a backgrounded session's +# --fork-session resume, still under the lock-holding front end) can only +# wake a conversation nobody runs. A claim or a caller without a session +# is judged as before. # - Every firing DEFERS (exits 0) to an open claim; anything else - a # terminal outcome, a dead or identity-mismatched owner, a stuck owner, an -# identityless entry, or no claim at all - lets the next firing take -# generation N+1 (fm_autoarm_claim_next). Taking a newer generation IS the -# reclaim: a steady-state predecessor is never signalled or revoked. +# identityless entry, an earlier session's claim, or no claim at all - lets +# the next firing take generation N+1 (fm_autoarm_claim_next). Taking a +# newer generation IS the reclaim: a steady-state predecessor is never +# signalled or revoked. # - NO mutex is ever held across a blocking step. The owner lock # state/.claude-autoarm.lock survives only as a micro-mutex serializing # individual ledger reads-then-writes (a few non-blocking file @@ -2189,7 +2285,8 @@ _fm_autoarm_epoch_field() { # <epoch-file> <field> # Parse the current ledger claim. Sets FM_AUTOARM_GEN, FM_AUTOARM_OWNER, # FM_AUTOARM_OUTCOME, FM_AUTOARM_SESSION, FM_AUTOARM_RECOVERY, -# FM_AUTOARM_OWNER_SESSION_ID, FM_AUTOARM_OWNER_LOCK_IDENTITY (line 3 of the +# FM_AUTOARM_OWNER_SESSION_ID, FM_AUTOARM_CLAUDE_SESSION (empty for an entry +# that recorded none), FM_AUTOARM_OWNER_LOCK_IDENTITY (line 3 of the # entry: the session-lock identity of the holder as the claim was taken), and # FM_AUTOARM_IDENTITY (line 2 of the entry, and ONLY # line 2 - identity is never substituted from a lock, so a transient @@ -2203,6 +2300,7 @@ fm_autoarm_ledger_read() { # <state-dir> FM_AUTOARM_SESSION= FM_AUTOARM_RECOVERY= FM_AUTOARM_OWNER_SESSION_ID= + FM_AUTOARM_CLAUDE_SESSION= FM_AUTOARM_OWNER_LOCK_IDENTITY= FM_AUTOARM_IDENTITY= FM_AUTOARM_GEN=$(_fm_autoarm_epoch_field "$epoch" epoch) || return 1 @@ -2211,6 +2309,7 @@ fm_autoarm_ledger_read() { # <state-dir> FM_AUTOARM_SESSION=$(_fm_autoarm_epoch_field "$epoch" session_pid 2>/dev/null || true) FM_AUTOARM_RECOVERY=$(_fm_autoarm_epoch_field "$epoch" recovery_generation 2>/dev/null || true) FM_AUTOARM_OWNER_SESSION_ID=$(_fm_autoarm_epoch_field "$epoch" owner_session_id 2>/dev/null || true) + FM_AUTOARM_CLAUDE_SESSION=$(_fm_autoarm_epoch_field "$epoch" claude_session 2>/dev/null || true) case "$FM_AUTOARM_GEN" in ''|*[!0-9]*) return 1 ;; esac @@ -2239,15 +2338,21 @@ fm_pid_descends_from() { # <pid> <ancestor> # window without producing a first beat is the same hang. An identityless # entry is never open: real generation claims always record identity, a legacy # build's entry gets its deference from its held role-carrying lock through -# the legacy shim, and anything else must not defer. -fm_autoarm_claim_open() { # <state-dir> [grace] - local state=$1 grace=${2:-${FM_GUARD_GRACE:-300}} epoch current lock_pid lock_identity recorded +# the legacy shim, and anything else must not defer. A caller that names its +# Claude session (the Stop payload's session_id) also finds a claim recorded +# under another session not open. +fm_autoarm_claim_open() { # <state-dir> [grace] [claude-session] + local state=$1 grace=${2:-${FM_GUARD_GRACE:-300}} session=${3:-} epoch current lock_pid lock_identity recorded epoch="$state/.claude-autoarm-epoch" case "$grace" in ''|*[!0-9]*|0) grace=300 ;; esac fm_autoarm_ledger_read "$state" || return 1 [ "$FM_AUTOARM_OUTCOME" = arming ] || return 1 + if [ -n "$session" ] && [ -n "$FM_AUTOARM_CLAUDE_SESSION" ] \ + && [ "$session" != "$FM_AUTOARM_CLAUDE_SESSION" ]; then + return 1 + fi fm_pid_alive "$FM_AUTOARM_OWNER" || return 1 [ -n "$FM_AUTOARM_IDENTITY" ] || return 1 current=$(fm_pid_identity "$FM_AUTOARM_OWNER" 2>/dev/null) || return 1 @@ -2323,15 +2428,19 @@ fm_autoarm_session_matches() { # <state-dir> <identity> [recorded-session-id] } # Atomically publish this process as the owner of generation N+1, under one -# short micro-mutex hold. Returns 0 with FM_AUTOARM_MY_GEN set on success, 2 -# when a competing claimant won the race (the ledger holds an open claim), and -# 1 when the micro-mutex is contended, the mandatory identity cannot be -# computed, or the write failed. -fm_autoarm_claim_next() { # <state-dir> [grace] - local state=$1 grace=${2:-${FM_GUARD_GRACE:-300}} lock epoch pid gen identity session_id lock_pid lock_identity tmp interrupted_identity interrupted_session +# short micro-mutex hold, recording the claimant's Claude session when it names +# one (an id outside [A-Za-z0-9._-] is recorded as none). Returns 0 with +# FM_AUTOARM_MY_GEN set on success, 2 when a competing claimant won the race +# (the ledger holds a claim open to this session), and 1 when the micro-mutex +# is contended, the mandatory identity cannot be computed, or the write failed. +fm_autoarm_claim_next() { # <state-dir> [grace] [claude-session] + local state=$1 grace=${2:-${FM_GUARD_GRACE:-300}} claude_session=${3:-} lock epoch pid gen identity session_id lock_pid lock_identity tmp interrupted_identity interrupted_session lock="$state/.claude-autoarm.lock" epoch="$state/.claude-autoarm-epoch" FM_AUTOARM_MY_GEN= + case "$claude_session" in + *[!A-Za-z0-9._-]*) claude_session= ;; + esac # Resolve the pid into a variable FIRST: expanding ${BASHPID:-$$} inside a # command substitution would resolve it in that subshell, recording the # identity of a process that exits immediately. @@ -2347,7 +2456,7 @@ fm_autoarm_claim_next() { # <state-dir> [grace] lock_identity=$(fm_session_pid_identity "$lock_pid" 2>/dev/null || true) [ -n "$lock_identity" ] || session_id= fm_lock_try_acquire "$lock" || return 1 - if fm_autoarm_claim_open "$state" "$grace"; then + if fm_autoarm_claim_open "$state" "$grace" "$claude_session"; then fm_lock_release "$lock" return 2 fi @@ -2373,6 +2482,7 @@ fm_autoarm_claim_next() { # <state-dir> [grace] printf 'epoch=%s owner_pid=%s outcome=arming updated_at=%s' \ "$gen" "$pid" "$(date +%s)" [ -z "$session_id" ] || printf ' owner_session_id=%s' "$session_id" + [ -z "$claude_session" ] || printf ' claude_session=%s' "$claude_session" printf '\n%s\n' "$identity" [ -z "$session_id" ] || printf '%s\n' "$lock_identity" } > "$tmp" 2>/dev/null || ! mv -f "$tmp" "$epoch" 2>/dev/null; then @@ -2435,6 +2545,7 @@ _fm_autoarm_write_owned_locked() { # <state-dir> <gen> <outcome> [marker-file] "$gen" "$pid" "$outcome" "$(date +%s)" [ -z "$session" ] || printf ' session_pid=%s' "$session" [ -z "$recovery" ] || printf ' recovery_generation=%s' "$recovery" + [ -z "$FM_AUTOARM_CLAUDE_SESSION" ] || printf ' claude_session=%s' "$FM_AUTOARM_CLAUDE_SESSION" printf '\n' [ -z "$identity" ] || printf '%s\n' "$identity" } > "$tmp" 2>/dev/null || ! mv -f "$tmp" "$epoch" 2>/dev/null; then diff --git a/bin/fm-watch.sh b/bin/fm-watch.sh index ecbba7e9bf3..4d547d09ee4 100755 --- a/bin/fm-watch.sh +++ b/bin/fm-watch.sh @@ -151,8 +151,14 @@ # check: pr-register: <lines> # with config/pr-register-scan present, a task's done # line reported a PR that the check sweep's -# fm-pr-check.sh scan could not record; the line -# names the task, the PR, and the refusal +# fm-pr-check.sh scan could not record, or whose +# merge watch it could not arm in time; the line +# names the task, the PR, and the refusal or timeout +# check: record hygiene: <lines> +# with config/record-hygiene present, the heartbeat's +# background record-hygiene scan found a new set of +# RECORD_HYGIENE findings (fm-record-hygiene-lib.sh); +# an unchanged set does not wake again # check: rejected unauthenticated state checks: <paths> # unsafe state checks were refused without execution # check: rejected unauthenticated PR poll retirement receipts: <paths> @@ -221,6 +227,7 @@ STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" PR_DEGRADED_WATCH_ENABLED=0 [ ! -e "$CONFIG/pr-degraded-watch" ] || PR_DEGRADED_WATCH_ENABLED=1 +DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" mkdir -p "$STATE" # Hold this state directory open for the main loop's teardown check: a # directory recreated at the same path is then a different directory, and the @@ -248,6 +255,8 @@ WATCH_HOME_EXISTED=0 . "$SCRIPT_DIR/fm-push-transition-lib.sh" # shellcheck source=bin/fm-pr-lib.sh . "$SCRIPT_DIR/fm-pr-lib.sh" +# shellcheck source=bin/fm-record-hygiene-lib.sh +. "$SCRIPT_DIR/fm-record-hygiene-lib.sh" # Only for the arm-time check on FM_PROCEVENT_LAUNCH_CONFIRM_SECONDS below; # the per-cycle reconcile itself runs as a separate process. # shellcheck source=bin/fm-procevent-lib.sh @@ -2430,12 +2439,72 @@ busy_stream_snapshot_current() { # <task> <observed snapshot> [ "$(( $(date +%s) - ts ))" -ge "$STREAM_SILENT_MAX_SECS" ] } +# A declared wait no longer explains quiet when its run is parked at a human-owed +# gate nobody has been told about, or when the run is terminal. Surface those +# authoritative changes once per declaration and run, then use the same bounded +# recheck cadence for repeats. The shared predicates bind the gate to this run. +# OFF unless the home creates config/paused-wait-supersession: absent the flag it +# returns before any read, so the watcher spends no crew-state read here and +# keeps the unchanged declared-wait cadence and wording. This applies to the +# foreground watcher only: while the away-mode flag exists it also returns before +# any read, so away mode keeps today's declared-wait pause cadence on every path. +paused_wait_superseded() { # <window> <task> + local win=$1 task=$2 line statusf key last run state reason rest part + [ -e "$CONFIG/paused-wait-supersession" ] && ! afk_present || return 1 + statusf="$STATE/$task.status" + last=$(status_declared_wait_line "$statusf") + status_is_paused "$last" || return 1 + line=$("$FM_CREW_STATE_BIN" "$task" 2>/dev/null) || line='' + if run=$(crew_gate_awaits_human_decision "$task" "$line") \ + && ! status_has_open_needs_decision "$statusf" "$run"; then + reason="stale: $win (paused wait superseded: run $run is parked at a human-owed gate with no open decision)" + state="gate:$run" + else + state=${line#state: }; state=${state%% *} + case "$state" in done|failed) ;; *) return 1 ;; esac + run='' + rest="$line · " + while [ -n "$rest" ]; do + part=${rest%% · *} + rest=${rest#* · } + case "$part" in "run: "?*) run=${part#run: } ;; esac + done + case "$run" in *[[:space:]]*) run='' ;; esac + reason="stale: $win (paused wait superseded: current run${run:+ $run} is terminal ($state))" + state="terminal${run:+:$run}" + fi + key=$(window_key "$win") + resurface_absorbed "$win" "$STATE/.paused-resurfaced-$key" 0 "$reason" \ + "declared:$(fm_wake_signal_sig "$statusf" || true):$state" 0 + triage_log "checked declared wait supersession ($state): $win" + return 0 +} + +# 0 when an unchanged paused declaration with no usable declared time is due its +# recheck and config/paused-recheck-silent says to run it without a wake: the +# agent is confirmed alive and no decision is open. Advances the throttle marker. +# The cheap age and marker reads run first, so the backend probe and the status +# fold happen only when silence would replace a wake. +paused_recheck_silent() { # <window> <statusf> <key> <declaration> <status-age> + local win=$1 statusf=$2 key=$3 declaration=$4 age=$5 marker + marker="$STATE/.paused-resurfaced-$key" + [ -e "$CONFIG/paused-recheck-silent" ] \ + && [ "$age" -ge "$PAUSE_RESURFACE_SECS" ] \ + && [ "$(cat "$marker" 2>/dev/null || true)" = "$declaration" ] \ + && [ "$(age_of "$marker")" -ge "$PAUSE_RESURFACE_SECS" ] \ + && { [ "$(window_kind "$win")" = secondmate ] || [ "$(fm_backend_agent_alive "$(window_backend "$win")" "$win" 2>/dev/null || true)" = alive ]; } \ + && [ -z "$(status_own_open_decisions "$statusf")" ] \ + || return 1 + printf '%s' "$declaration" > "$marker" +} + # Absorb a stale pane under a declared external-wait pause (paused:) or a # dead-agent captain-held transfer. Non-overdue waits re-surface every # PAUSE_RESURFACE_SECS; an overdue pause opens a keyed blocked decision and stays # quiet until the worker writes another status event. Called on any # stale poll once pause_state_class permits the bounded cadence, so it must be -# cheap: it NEVER re-reads crew state. The re-surface age is anchored on the +# cheap: it never re-reads crew state except through the opt-in superseded-wait +# check (paused_wait_superseded) that runs first. The re-surface age is anchored on the # status file mtime, not a per-hash marker, so a churny idle pane (a ticking # clock, a token counter) cannot keep resetting the cadence the way a hash-tied # timer would. The bounded re-surface itself is the shared resurface_absorbed @@ -2448,7 +2517,8 @@ busy_stream_snapshot_current() { # <task> <observed snapshot> # wording; a caller that reached the bounded cadence off pause tracking alone, with # no declaring verb left on the log, keeps the external-wait wording it always had. handle_paused_stale() { # <window> <task> <hash> - local win=$1 task=$2 h=$3 key statusf mtime age detail reason declaration last until now min_age rc snapshot pause_age + local win=$1 task=$2 h=$3 key statusf mtime age detail reason declaration last until now min_age rc snapshot pause_age silent=0 + if paused_wait_superseded "$win" "$task"; then return 0; fi key=$(window_key "$win") pause_overdue_reconcile "$task" "$STATE/$task.status"; rc=$? if [ "$rc" -eq 0 ]; then @@ -2468,6 +2538,12 @@ handle_paused_stale() { # <window> <task> <hash> last=$(status_declared_wait_line "$statusf") min_age=$PAUSE_RESURFACE_SECS declaration="declared:$(fm_wake_signal_sig "$statusf" || true)" + # A lane parked on review stopped its agent on purpose; bin/fm-park.sh's tick + # watches its PR and owns every wake for it. + if [ -f "$STATE/$task.review-park" ]; then + triage_log "absorbed stale (parked on review, watched by fm-park.sh tick): $win" + return 0 + fi pause_age=$(status_pause_age "$last" "$age" "$now") if status_is_paused "$last" && status_pause_is_overdue "$last" "$pause_age" "$now"; then detail="paused, wait overdue" @@ -2487,6 +2563,7 @@ handle_paused_stale() { # <window> <task> <hash> return 0 elif [ "$now" -lt "$until" ]; then detail="paused, declared time beyond recheck cadence" + silent=1 reason="paused ${age}s, awaiting external - the declared time is beyond the recheck cadence; confirm the wait still holds" else # The declared time has passed: recheck now, once per declaration, then @@ -2498,8 +2575,13 @@ handle_paused_stale() { # <window> <task> <hash> fi else detail="paused, awaiting external" + silent=1 reason="paused ${age}s, awaiting external - declared pause, rechecked on a long cadence not a wedge; confirm the wait still holds" fi + if [ "$silent" = 1 ] && paused_recheck_silent "$win" "$statusf" "$key" "$declaration" "$age"; then + triage_log "absorbed stale (${detail}, unchanged declaration rechecked without a wake, age ${age}s): $win" + return 0 + fi if [ "$detail" = "paused, wait overdue" ]; then resurface_absorbed "$win" "$STATE/.paused-resurfaced-$key" "$age" "stale: $win ($reason)" "$declaration" "$min_age" "$statusf" "blocked [key=paused-overdue-$task]: paused ${pause_age}s, wait overdue - decide or release" "$STATE/.paused-overdue-$task" "$snapshot" else @@ -2648,8 +2730,14 @@ stale_reason_naming_dialog() { # <reason> [dialog-name] } clear_pause_tracking() { # <window-key> - local key=$1 - clear_pause_state "$key" + local key=$1 throttle="$STATE/.paused-resurfaced-$1" scope + scope=$(cat "$throttle" 2>/dev/null || true) + case "$scope" in + declared:*:gate:*|declared:*:terminal:*) + rm -f "$STATE/.paused-$key" "$STATE/.paused-rechecked-$key" + ;; + *) clear_pause_state "$key" ;; + esac clear_stale_hash_tracking "$key" } @@ -3012,6 +3100,7 @@ surface_nonterminal_stale() { # <window> <hash> [dialog-name] key=$(window_key "$win") task=$(window_to_task "$win" "$STATE") last=$(status_declared_wait_line "$STATE/$task.status") + if paused_wait_superseded "$win" "$task"; then return 0; fi STALE_WAIT_DECLARATION= if status_is_paused "$last"; then declared=0 @@ -3025,8 +3114,10 @@ surface_nonterminal_stale() { # <window> <hash> [dialog-name] STALE_WAIT_DECLARATION="$STALE_WAIT_DECLARATION:due" stale_wait_throttled "$key" "$STALE_WAIT_DECLARATION" && throttled=0 fi - else - stale_wait_throttled "$key" "$STALE_WAIT_DECLARATION" && throttled=0 + elif stale_wait_throttled "$key" "$STALE_WAIT_DECLARATION" \ + || paused_recheck_silent "$win" "$STATE/$task.status" "$key" "$STALE_WAIT_DECLARATION" \ + "$(( $(date +%s) - $(stat_mtime "$STATE/$task.status") ))"; then + throttled=0 fi elif status_is_captain_held "$last"; then declared=0 @@ -3310,6 +3401,7 @@ run_check_capture() { local pgid rc=0 fm_check_output_cleanup FM_CHECK_RESULT= + FM_CHECK_CUT= FM_CHECK_OUTPUT=$(mktemp "$STATE/.fm-check-output.XXXXXX") || return 1 chmod 0600 "$FM_CHECK_OUTPUT" || { fm_check_output_cleanup; return 1; } FM_CHECK_SIGNAL_PENDING= @@ -3336,7 +3428,10 @@ run_check_capture() { FM_CHECK_RESULT=$(cat "$FM_CHECK_OUTPUT" 2>/dev/null || true) fm_check_output_cleanup # Keep what the check printed before the cut, then report the cut itself. - [ "$rc" -ne 124 ] || FM_CHECK_RESULT="${FM_CHECK_RESULT:+$FM_CHECK_RESULT$'\n'}timed out after ${CHECK_TIMEOUT}s" + [ "$rc" -ne 124 ] || { + FM_CHECK_CUT="timed out after ${CHECK_TIMEOUT}s" + FM_CHECK_RESULT="${FM_CHECK_RESULT:+$FM_CHECK_RESULT$'\n'}$FM_CHECK_CUT" + } } FM_CAPTURE_OUTPUT= @@ -4121,10 +4216,10 @@ while :; do pr_register_reason= if [ -e "$CONFIG/pr-register-scan" ]; then FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" run_check_capture "$SCRIPT_DIR/fm-pr-check.sh" scan || exit 1 - if [ -n "$FM_CHECK_RESULT" ]; then - pr_register_reason="check: pr-register: $FM_CHECK_RESULT" - fm_wake_append check pr-register "$pr_register_reason" || exit 1 - fi + # The scan queued each notice itself, before marking it as sent; only a + # scan cut at the check bound, the line run_check_capture writes, is queued here. + [ -z "$FM_CHECK_RESULT" ] || pr_register_reason="check: pr-register: $FM_CHECK_RESULT" + [ -z "$FM_CHECK_CUT" ] || fm_wake_append check pr-register "check: pr-register: $FM_CHECK_CUT" || exit 1 fi for c in "$STATE"/*.check.sh; do [ -e "$c" ] || continue @@ -4225,9 +4320,21 @@ EOF triage_log "published merge outcome for $id but could not retire its authority record" exit 1 fi + # Persist before retiring the poll; start only after releasing its control lock. + FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" FM_CONFIG_OVERRIDE="$CONFIG" \ + "$SCRIPT_DIR/fm-park.sh" land "$id" --if-eligible --queue >/dev/null 2>&1 || { + triage_log "could not queue merged-lane cleanup for $id" + exit 1 + } retire_merged_pr_poll "$id" pr_poll_control_release || exit 1 + FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" FM_CONFIG_OVERRIDE="$CONFIG" \ + "$SCRIPT_DIR/fm-park.sh" land "$id" --if-eligible --detach >/dev/null 2>&1 || true touch "$STATE/.last-check" + # A merged lane eligible for merge-time cleanup, also one + # bin/fm-pr-merge.sh already recorded, is torn down by a detached + # bin/fm-park.sh land that only starts here, so this iteration never + # waits on a teardown; the tick reports its outcome. if [ "$FM_MERGE_OUTCOME_ALREADY_RECORDED" = true ]; then triage_log "absorbed duplicate merged PR poll result for $id" continue @@ -4314,6 +4421,17 @@ EOF touch "$STATE/.last-check" wake "$reason" fi + # Parked-on-review lanes, close detection, opt-in automatic parking, and + # the outcomes of detached park, relaunch, and cleanup actions + # (bin/fm-park.sh owns all of them, bounds the whole tick, and records its + # wake before printing it, so the watcher only rings). + park_out=$(FM_HOME="$FM_HOME" FM_STATE_OVERRIDE="$STATE" FM_CONFIG_OVERRIDE="$CONFIG" \ + "$SCRIPT_DIR/fm-park.sh" tick 2>/dev/null) || true + if [ -n "$park_out" ]; then + reason="check: park-on-review: $(printf '%s' "$park_out" | paste -sd ';' - | sed 's/;/; /g')" + touch "$STATE/.last-check" + wake "$reason" + fi touch "$STATE/.last-check" [ -z "$pr_register_reason" ] || wake "$pr_register_reason" if [ -n "$contribution_check_output" ]; then @@ -4654,6 +4772,7 @@ EOF # A pane that leaves the stale path ends the wedge episode it was in, so # the cap that episode fired goes with the rest of its bookkeeping. paused_bound=1 + [ "$busy_now" -ne 0 ] || paused_wait_superseded "$w" "$task" || true if [ "$busy_now" -eq 0 ] && busy_turn_over_age "$task"; then busy_turn_bound_check "$w" "$task" "$h" "$ssf" "$ewf" "$dialog" && paused_bound=0 else @@ -4673,6 +4792,7 @@ EOF printf '%s' "$h" > "$hf" echo 0 > "$cf" paused_bound=1 + [ "$busy_now" -ne 0 ] || paused_wait_superseded "$w" "$task" || true if [ "$busy_now" -eq 0 ] && busy_turn_over_age "$task"; then busy_turn_bound_check "$w" "$task" "$h" "$ssf" "$ewf" "$dialog" && paused_bound=0 else @@ -4703,6 +4823,29 @@ EOF fi done < <(recorded_windows) + # Collect a completed hygiene scan before launching another one. The report + # child keeps endpoint/backlog reads out of the polling path. Only a home + # with config/record-hygiene present scans; read the flag once per cycle so + # collection and launch agree. + hygiene_enabled=0 + [ -e "$CONFIG/record-hygiene" ] && hygiene_enabled=1 + hygiene_result="$STATE/.record-hygiene-result" + hygiene_running="$STATE/.record-hygiene-running" + if [ "$hygiene_enabled" = 1 ] && [ -f "$hygiene_result" ]; then + hygiene_report=$(cat "$hygiene_result") + if [ -n "$hygiene_report" ]; then + if ! cmp -s <(printf '%s\n' "$hygiene_report") "$STATE/.record-hygiene-last"; then + hygiene_reason="check: record hygiene: $hygiene_report" + fm_wake_append check record-hygiene "$hygiene_reason" || exit 1 + printf '%s\n' "$hygiene_report" > "$STATE/.record-hygiene-last" + wake "$hygiene_reason" + fi + else + rm -f "$STATE/.record-hygiene-last" + fi + rm -f "$hygiene_result" "$hygiene_running" + fi + # Heartbeat: the watcher runs a cheap fleet-scan at a regular cadence no matter # what. Time-based via .last-heartbeat mtime; interval doubles per consecutive # no-change heartbeat (idle fleet) up to HEARTBEAT_MAX, and resets on any @@ -4718,6 +4861,45 @@ EOF if [ -f "$CONFIG/jira-status-transitions" ]; then FM_HOME=$FM_HOME FM_STATE_OVERRIDE=$STATE "$SCRIPT_DIR/fm-jira-transition.sh" reconcile >/dev/null 2>&1 </dev/null & fi + if [ "$hygiene_enabled" = 1 ] && [ -e "$hygiene_running" ]; then + hygiene_owner=$(cat "$hygiene_running" 2>/dev/null || true) + IFS=$'\t' read -r hygiene_pid hygiene_started <<< "$hygiene_owner" + hygiene_live_start='' + case "$hygiene_pid" in ''|*[!0-9]*) ;; *) + kill -0 "$hygiene_pid" 2>/dev/null && hygiene_live_start=$(ps -o lstart= -p "$hygiene_pid" 2>/dev/null | sed 's/^[[:space:]]*//') ;; + esac + hygiene_age=$(age_of "$hygiene_running") + report_timeout=${FM_RECORD_HYGIENE_TIMEOUT:-60} + case "$report_timeout" in ''|*[!0-9]*|0) report_timeout=60 ;; esac + if [ -z "$hygiene_live_start" ] || [ "$hygiene_live_start" != "$hygiene_started" ] || [ "$hygiene_age" -gt $(( report_timeout * 3 )) ]; then + rm -f "$hygiene_running" + fi + fi + if [ "$hygiene_enabled" = 1 ] && [ ! -e "$hygiene_running" ] && [ ! -e "$hygiene_result" ]; then + ( + report_timeout=${FM_RECORD_HYGIENE_TIMEOUT:-60} + case "$report_timeout" in ''|*[!0-9]*|0) report_timeout=60 ;; esac + FM_BOOTSTRAP_DETECT_ONLY=1 record_hygiene_report > "$hygiene_result.tmp" & + report_pid=$! + ( + sleep "$report_timeout" + : > "$hygiene_result.timeout" + kill "$report_pid" 2>/dev/null || true + ) & + watchdog_pid=$! + wait "$report_pid" || true + kill "$watchdog_pid" 2>/dev/null || true + wait "$watchdog_pid" 2>/dev/null || true + if [ -e "$hygiene_result.timeout" ]; then + printf 'RECORD_HYGIENE: report timed out after %ss\n' "$report_timeout" > "$hygiene_result.tmp" + rm -f "$hygiene_result.timeout" + fi + mv -f "$hygiene_result.tmp" "$hygiene_result" + ) >/dev/null 2>&1 & + hygiene_worker=$! + hygiene_started=$(ps -o lstart= -p "$hygiene_worker" 2>/dev/null | sed 's/^[[:space:]]*//') + printf '%s\t%s\n' "$hygiene_worker" "$hygiene_started" > "$hygiene_running" + fi # Triage: in always-on mode a heartbeat is benign unless the cheap fleet-scan # turns up a captain-relevant status the per-wake path missed. Absorb the # no-change case (advance the schedule and back off exactly as wake() would, diff --git a/docs/architecture.md b/docs/architecture.md index e848ccbe32a..5eb48a825d7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -11,7 +11,7 @@ firstmate's supervisor contract and routing index for conditional procedures is The declared-wait vocabulary, including the legacy "external wait" label, is owned by [`bin/fm-classify-lib.sh`](../bin/fm-classify-lib.sh); worker declaration instructions and the ship worker's scratch-location and clean-worktree contract are owned by [`bin/fm-brief.sh`](../bin/fm-brief.sh). A zero-token bash watcher (`bin/fm-watch.sh`) sleeps on the fleet, classifies detected wakes in bash, and wakes the first mate only when something is actionable. -Actionable wakes include captain-relevant status signals, no-verb signals without positive evidence that their crew is still executing, authenticated check output such as PR merge polling or a Relay mention (a check cut off at `FM_CHECK_TIMEOUT` reports `timed out after <seconds>s` after anything it printed first), stale panes whose crew is not provably working whether their status log looks terminal or non-terminal, provably-working stale panes that persist past `FM_STALE_ESCALATE_SECS` with no wait their own worker declared, no backlog captain hold on their task, no writes to their own task worktree, and - in a home that armed `config/wedge-defer-worker-wait`, `config/wedge-defer-parked-gate` or `config/wedge-defer-validating` - no still-open `blocked:` or `needs-decision:` line of their own worker's newest, or no validation gate of their own awaiting an unanswered supervisor decision, or no attributed validation run that is running or fixing, declared external waits and attended captain-held transfers that remain declared past `FM_PAUSE_RESURFACE_SECS`, busy-state-unknown panes whose endpoint remains proven alive and whose own status log or task worktree was written within `FM_STALE_ESCALATE_SECS`, after their first alarm and bounded recheck (without that progress evidence each new hash still surfaces immediately), and heartbeat backstop hits. +Actionable wakes include captain-relevant status signals, no-verb signals without positive evidence that their crew is still executing, authenticated check output such as PR merge polling or a Relay mention (a check cut off at `FM_CHECK_TIMEOUT` reports `timed out after <seconds>s` after anything it printed first), stale panes whose crew is not provably working whether their status log looks terminal or non-terminal, provably-working stale panes that persist past `FM_STALE_ESCALATE_SECS` with no wait their own worker declared, no backlog captain hold on their task, no writes to their own task worktree, and - in a home that armed `config/wedge-defer-worker-wait`, `config/wedge-defer-parked-gate` or `config/wedge-defer-validating` - no still-open `blocked:` or `needs-decision:` line of their own worker's newest, or no validation gate of their own awaiting an unanswered supervisor decision, or no attributed validation run that is running or fixing, declared external waits (an unchanged one is quiet in a home that armed [`config/paused-recheck-silent`](configuration.md#silent-paused-rechecks-configpaused-recheck-silent)) and attended captain-held transfers that remain declared past `FM_PAUSE_RESURFACE_SECS`, busy-state-unknown panes whose endpoint remains proven alive and whose own status log or task worktree was written within `FM_STALE_ESCALATE_SECS`, after their first alarm and bounded recheck (without that progress evidence each new hash still surfaces immediately), and heartbeat backstop hits. For an ordinary crew task, a wait is read from both of its records: the status line a worker declared, and the backlog hold `bin/fm-captain-hold.sh` recorded once firstmate handed the work to the captain. So a delivered ordinary crew task whose last line stays a `done` PR-ready line bounds repeated alarms from new pane hashes to the `FM_PAUSE_RESURFACE_SECS` cadence for the length of the captain's decision. The first hash still alarms, each new hash inside that window is absorbed, and a new hash after the window re-surfaces the hold; a terminal pane hash that never changes stays inert after its first alarm exactly as it did before this bound. @@ -22,7 +22,7 @@ Repeated provably-working stale escalations on the same unchanged pane add an es A plain stale wake, or the final wedge escalation, for a pane that shows a recognised dialog also ends its reason with `blocked-on-prompt:` and the dialog's name, on the unchanged triage and schedule; `bin/fm-watch.sh`'s header owns the wording. The deferral, dead-record, and declared-wait recheck reasons described below do not carry the name. In the same branch that is about to escalate, the pane's own account of its quiet is consulted first: the worker's declared `paused:` or verified `captain-held` status line. -That declaration defers the escalation to the `FM_PAUSE_RESURFACE_SECS` recheck cadence instead, because a lane waiting on something it named is silent for a reason the escalation would misreport, and the ladder would otherwise climb for as long as the wait lasts. +That declaration defers the escalation to the `FM_PAUSE_RESURFACE_SECS` recheck cadence while the wait still explains the authoritative current state; the optional foreground-watcher supersession is documented in [`configuration.md`](configuration.md#superseded-declared-wait-wake-configpaused-wait-supersession). A declared clearing time (`paused: ... until <UTC ISO 8601>`) that has already passed stops counting as that account, so a lane whose own wait is over is judged by the records below exactly as a lane that never declared one, and either keeps the unchanged escalation schedule, reason and `demand-deep-inspection` wording unless one of them holds. When the status line accounts for nothing, the task's backlog captain hold recorded by `bin/fm-captain-hold.sh` is read next, because the worker's line never shows that hold; it defers to the same recheck naming the captain, and publishes no wait age because the line predates the hold. In a home that armed the default-off `config/wedge-defer-worker-wait` flag, the worker's own still-open newest `blocked:` or `needs-decision:` line is also that account, because that line already woke firstmate and the lane is owed firstmate's answer; its recheck names the verb's clearing action, and any newer status line or a newer acknowledged inbox record ends it (`status_worker_wait_verb` in `bin/fm-classify-lib.sh` owns which lines count). @@ -263,6 +263,10 @@ Text for a worker to read and commands that drive a worker's process are separat ### Task status archives When [status archival is enabled](configuration.md#task-status-archive-configstatus-archive), teardown preserves the complete status stream in `data/<task>/status.log`, using a separate `status.<N>.log` for reused IDs without overwriting earlier histories. +When [record hygiene is enabled](configuration.md#record-hygiene-report-configrecord-hygiene), session-start bootstrap and the watcher heartbeat report home-local record inconsistencies: status logs without metadata older than 24 hours, in-flight rows without metadata, metadata without a tab, and duplicate `pr=` references across records. +Only the mutating, locked session-start bootstrap archives orphan status logs older than 24 hours to `data/<id>/status.log`; a detect-only bootstrap and the heartbeat do not archive them. +The scan is limited to the current home's records and is a report, not a general cleanup sweep. +The shared report implementation is [`fm-record-hygiene-lib.sh`](../bin/fm-record-hygiene-lib.sh). Forced secondmate retirement stores descendant logs under the surviving parent's `data/<secondmate>/children/<child>/` tree. These archives are cold evidence: presentation, wake classification, and fleet snapshots consume live `state/` records and never read archived logs. Ordinary child teardown uses its own home's configuration; retiring a whole home still removes older archives stored inside that home. diff --git a/docs/captain-hold-lifecycle.md b/docs/captain-hold-lifecycle.md index a267cf93cbf..14633c7b29b 100644 --- a/docs/captain-hold-lifecycle.md +++ b/docs/captain-hold-lifecycle.md @@ -43,6 +43,7 @@ It never reads report bodies, review artifacts, terminal output, or chat. | `reconcile-requests` | Internal intake that records a reconcile request from a board selection. | [Reconcile](#reconcile-re-check-reality-never-a-blind-close) | | `reconcile close`, `reconcile note`, `reconcile list` | Retire or list pending reconcile requests. | [Verifying and retiring a request](#verifying-and-retiring-a-request) | | `diverged` | Read-only report of a call whose two records disagree. | [Record divergence](#record-divergence) | +| `pr-policy`, `settle-pr` | Declare and apply opt-in forge settlement. | [PR settlement](#pr-settlement) | ### Creating a hold (`hold`) @@ -203,7 +204,7 @@ The refusal names the record and the validation reason, so the captain can repai ### What `--force` does not lift `--force` does not lift the deferral, because it authorizes discarding unlanded work, never the captain's question. -Only `answer` with the captain's words or evidence-backed `reconcile close` resolves the call, by either closing the question or releasing the gated work. +The [lifecycle policy](../.agents/skills/captain-hold-lifecycle/SKILL.md#policy) owns the allowed answer and evidence-backed settlement paths. `bin/fm-backlog-transition-lib.sh` owns the transition and its record, and `bin/fm-captain-hold.sh --help` owns the predicate's contract. ## Answer-time resolution @@ -327,7 +328,22 @@ A successful normal answer also retires any pending request, because an answered Every retirement is checked. If request removal fails after an answer, close, or note is already durable, the durable outcome stands, but the command fails and leaves the pending request visible for retry. -No path here closes a captain call without either the captain's words through `answer` or the evidence through `reconcile close`. +Board-requested reconciliation remains separate from the opt-in forge path below. + +## PR settlement + +[PR hold settlement](configuration.md#pr-hold-settlement-confighold-pr-settle) connects the contribution observer to the hold lifecycle without changing cleanup or the merge poll. +`bin/fm-captain-hold.sh --help` owns the declaration schema, supported events, and settlement commands. +The observer invokes settlement for both fresh and retained terminal observations. +It records evidence before applying the declared task transition, under the task's control lock. +A retry uses that evidence to finish an interrupted transition without writing a second resolution. +The existing parent channel receives the resolution before the declaration is retired. +A `release` declaration follows the attended-only rule of [Answering a call](#answering-a-call-answer): while the home is away or the away-posture record is locked the hold stays unchanged, and a later poll applies the declaration after the captain returns. +When another command holds the task, for example a captain answer in progress, settlement leaves the declaration unchanged and a later poll applies it. +Linking a follow-up PR to the held task keeps its declaration valid. +A declaration that can no longer settle is removed, and the observer prints one diagnostic line when the question changed or the PR ended in the other terminal state. +A successful settlement prints nothing, so it causes no wake. +`tests/fm-contributions.test.sh` exercises merged and unmerged closure, default-off behavior, unrelated work, superseded declarations, away deferral, busy-task deferral, follow-up links, closure reports for a task with two closed PRs, and interrupted settlement. ## Card hygiene: a landed subject is not a live call @@ -441,7 +457,7 @@ The away-posture restriction on queued merges and its residual limits are owned A captain call can have two records, and closing one does not close the other. The status-log fold is the open-decision set `bin/fm-classify-lib.sh` reads from a task's status log, where a keyed `needs-decision` or `blocked` line opens a decision. A `resolved [key=...]` line closes the status-log fold. -The structured captain-held task closes only through `answer`. +The structured captain-held task settles only through the [lifecycle policy's answer or evidence paths](../.agents/skills/captain-hold-lifecycle/SKILL.md#policy). Until this guard existed, closing on the status side alone left no trace of the disagreement. The fold went quiet, the durable record kept saying the captain owed an answer, and nothing warned. diff --git a/docs/configuration.md b/docs/configuration.md index 00f31749be2..b78d73e3865 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -15,6 +15,7 @@ Start with the directory layout, then use the setting reference for the behavior | Supervision and presentation | [Pi supervision branch](#pi-supervision-branch), [supervision host](#supervision-host-configsupervision-host), [Codex native Stop park](#codex-native-stop-park-configcodex-stop-park), [warm-cache compaction](#warm-cache-compaction-configwarm-compact), and [Calm preference](#calm-preference-configcalm) | | Persistent secondmates | [Secondmate routes](#secondmate-routes-datasecondmatesmd) | | Review bots on owned GitHub pull requests | [Contribution review bots](#contribution-review-bots-configcontribution-review-bots) | +| Stopping idle workers while a PR awaits review, and cleanup on merge | [Park on review](#park-on-review) | | Per-run overrides and tuning | [Environment variables](#environment-variables) | ## FM_HOME @@ -872,13 +873,39 @@ With the file absent, nothing is armed and an already armed check prints nothing Creating the flag takes effect at the next session start, or immediately after `bin/fm-pr-health.sh arm`. The flag is a home-local preference and is not inherited by secondmate homes. +## PR hold settlement (config/hold-pr-settle) + +The optional local, gitignored `config/hold-pr-settle` presence flag enables settlement of explicitly declared PR-dependent captain holds. +Create the empty flag in the owning home and use `bin/fm-captain-hold.sh pr-policy` to declare each hold's settlement policy. +The command's `--help` owns its arguments and durable record format. +The existing contribution observer applies matching GitHub merge or closure evidence, including observations retained after the worker stops. +It also reports an unmerged closure through the contribution notification channel, once for each owning task that is still an open captain hold after settlement. +A task without a captain hold gets no closure notification, so enabling the flag does not replay historical closed PRs. +No captain answer is invented, and unrelated holds remain open. +With the flag absent, no settlement or new closure notification occurs. +The flag takes effect on the next contribution poll and is not inherited by secondmate homes. +The observer's existing `bin/fm-contributions.sh arm` command enables polling when it is not already armed. +The [captain-hold policy](../.agents/skills/captain-hold-lifecycle/SKILL.md#policy) owns when a forge event may settle a question. + +## Park on review + +The optional local, gitignored `config/park-on-review` presence flag opts this home into stopping the idle agent of a ship lane whose PR or merge request only awaits human review. +`config/auto-land` opts into merge-time cleanup of unparked ship lanes. +Either flag also lets `bin/fm-teardown.sh` accept a merged GitLab merge request, including a squash or merge-train merge, as landed work when the merge request's head contains the lane's work; with neither flag, teardown does not read GitLab merge requests. +Both flags are home-local and not inherited by secondmate homes. +The full behavior contract and command details are owned by [`bin/fm-park.sh`](../bin/fm-park.sh); the watcher calls its `tick` command on each check sweep. + ## PR registration scan (config/pr-register-scan) The optional local, gitignored `config/pr-register-scan` presence flag opts this home into recording, without the supervisor, each PR a task reports. With the flag present, the watcher's check sweep runs `bin/fm-pr-check.sh scan` before the merge polls. -The scan reads each ship task's newest `done:` line that names a `PR <url>`, and records that PR through the ordinary registration path when the task has no `pr=`, or has that same `pr=` but no published merge poll. -It skips persistent secondmate and scout records, a `pr=` that names another PR, and a PR whose merge was already reported or whose poll was retired. -A refused registration, such as a draft, prints one `check: pr-register:` wake for each task and PR, and later sweeps retry without a new wake, so a PR marked ready later is recorded. +The scan reads a ship task's PR only from a ready report: the task's newest crew-declared status line must be a `done:` line that is exactly `PR <url>`, `PR <url> checks green`, or `PR <url> published for review`, so prose around the URL and any later crew line report nothing. +Later `resolved:` and `captain-held:` lines are supervisor bookkeeping and do not withdraw a ready report. +It records that PR through the ordinary registration path when the task has no `pr=`, or has that same `pr=` but no published merge poll. +When the status log named the recorded `pr=` and the ready report names a different PR, that PR replaces the recorded one, so a lane that closes a PR and opens another moves its merge poll. +The flag also covers the PR a secondmate home's parent-channel report carries for a child's ready report (`bin/fm-inactive-reconcile.sh`): with it present, that report applies the same ready-report spellings, accepts any recognized forge URL, and lets the same replacement override the recorded `pr=`, while with it absent a recorded `pr=` always stands and only a GitHub-shaped `PR <url>` or `PR <url> checks green` line fills in a missing one. +It skips persistent secondmate and scout records, a `pr=` the status log never named because that is the supervisor's own choice, and a PR whose merge was already reported or whose poll was retired. +A refused registration, such as a draft, or one cut at the scan's time bound, which reports that the PR merge watch could not be armed, queues one `check: pr-register:` wake for each task, PR and reason before marking it sent, and later sweeps retry without a new wake, so a PR marked ready later is recorded. With the file absent, the check sweep does not run the scan, and `pr=` and the merge poll come only from a manual `bin/fm-pr-check.sh <id> <PR url>` run. The flag takes effect at the next check sweep, and is a home-local preference that is not inherited by secondmate homes. @@ -889,6 +916,24 @@ With the flag present, registration records a degraded/train baseline and each p Without the flag, registration performs no baseline read, polls make no GitHub rollup or GitLab JSON, pipeline, or train reads, and the watcher ignores `degraded` and `ontrain` results while retaining the existing merged-PR wake. The flag is home-local, is not inherited by secondmate homes, and takes effect on the next check sweep. +## Treehouse pool report (config/treehouse-pool-report) + +The optional local, gitignored `config/treehouse-pool-report` presence flag opts this home into a read-only Treehouse pool hygiene report at bootstrap. +With the flag present and the Treehouse state directory and `jq` available, bootstrap prints `TREEHOUSE_POOL_REPORT:` findings for pools with unavailable project paths, unowned slots older than `FM_POOL_ORPHAN_DAYS`, duplicate-origin project clones, and `project-*` or `scratch-project-*` pools without an active owner or lease; the `treehouse` executable is not required. +The report never changes a pool or slot; pruning or other pool cleanup remains an explicit operator action. +With the file absent, bootstrap reads no Treehouse state and prints no pool report. +Only the file's presence is read, at each bootstrap run. +The flag is a home-local preference and is not inherited by secondmate homes. + +## Squash commit summary (config/pr-squash-summary) + +The optional local, gitignored `config/pr-squash-summary` presence flag opts this home into writing the commit text of the GitHub squash merges `bin/fm-pr-merge.sh` performs. +With the flag present, a squash merge whose caller passed no subject or body of its own uses the PR title plus ` (#N)` as the commit subject and the PR body's `## What Changed` section as the commit body, so the commit on the base branch describes the change rather than listing every branch commit. +The section is copied verbatim up to the next level-two heading, which leaves out the operator intent before it and the no-mistakes appendix and attestation after it. +A PR body without that section, a PR whose text cannot be read, a merge method other than squash, and GitLab and Forgejo merges keep the repository's own squash text. +Only while the flag is present does it override the repository's own squash commit message setting; it is off by default, and with the file absent `bin/fm-pr-merge.sh` reads no PR text and sends the same merge command as before. +The flag is a home-local preference, is not inherited by secondmate homes, and takes effect on the next merge. + ## No-mistakes pipeline spend (config/pipeline-spend) The optional local, gitignored `config/pipeline-spend` presence flag opts this home into recording per-task no-mistakes pipeline spend in `data/pipeline-spend.jsonl` during teardown. @@ -989,6 +1034,15 @@ Each opted-in archival prunes owned logs older than 30 completed days by mtime; Age-based retention works independently of the selected backlog backend and keeps histories available for a predictable audit period. [`architecture.md`](architecture.md#task-status-archives) describes archive locations and isolation from live state; [`bin/fm-teardown.sh`](../bin/fm-teardown.sh) owns the mechanics. +## Record hygiene report (config/record-hygiene) + +The optional local, gitignored `config/record-hygiene` presence flag opts this home into reporting its own inconsistent task records. +Create it with `touch "$FM_HOME/config/record-hygiene"`; remove it to stop the report. +With the flag present, session-start bootstrap prints a `RECORD_HYGIENE:` line for each inconsistency, and the watcher heartbeat runs the scan in the background and raises one `check: record hygiene:` notification when its findings change. +It is off by default: without the flag, session start and the heartbeat print no report, move no files, and perform no record-hygiene reads. +The flag takes effect at the next session start or heartbeat, and is a home-local preference that is not inherited by secondmate homes. +[`architecture.md`](architecture.md#task-status-archives) owns scan scope and status-log archiving; [`bin/fm-record-hygiene-lib.sh`](../bin/fm-record-hygiene-lib.sh) owns the mechanics. + ## Record export (config/record-export-root) The optional local, gitignored `config/record-export-root` file contains exactly one absolute directory path and opts this home into a private copy of a fixed record allowlist. @@ -1112,6 +1166,29 @@ With the flag absent the wedge timer spends no read for it and keeps the unchang The flag is a home-local supervision-noise preference and is not inherited by secondmate homes. [`architecture.md`](architecture.md) owns the wait-evidence contract and `bin/fm-classify-lib.sh`'s `status_worker_wait_verb` owns which lines count. +## Superseded declared-wait wake (config/paused-wait-supersession) + +The optional local, gitignored `config/paused-wait-supersession` presence flag opts this home into a default-off check in the watcher's declared-wait handling. +With it present, a pane whose worker's newest wait line is a `paused:` declaration is surfaced ahead of the `FM_PAUSE_RESURFACE_SECS` recheck cadence when its crew's own current state shows the wait no longer explains the quiet: the run is parked at a human-owed gate with no open `needs-decision` for that run, or the run is terminal. +Each superseding state surfaces once per declaration and run on both the idle and busy paths, and repeats use the same bounded recheck cadence. +It stays opt-in because it adds a wake reason that fires ahead of that cadence and spends a current-state read on each paused poll, so which lanes pay that read for earlier notice is a home's choice. + +With the flag absent the watcher spends no current-state read for it and keeps the unchanged declared-wait cadence and wording. The flag affects the foreground watcher only; away mode keeps today's declared-wait pause cadence. +The flag is a home-local supervision-noise preference and is not inherited by secondmate homes. +[`architecture.md`](architecture.md) owns the declared-wait contract and `bin/fm-watch.sh`'s `paused_wait_superseded` owns the exact derivation, reasons, and throttle. + +## Silent paused rechecks (config/paused-recheck-silent) + +The optional local, gitignored `config/paused-recheck-silent` presence flag opts this home into a default-off quiet form of the declared-wait recheck. +With it present, the watcher's `FM_PAUSE_RESURFACE_SECS` recheck of a worker's `paused:` declaration queues no wake when the declaration is unchanged since its first recheck wake, the agent is confirmed alive (a secondmate's liveness is never read), its declared `until` time is not yet due, and the task has no open decision. +The watcher records the recheck in the declaration's existing throttle marker, so the recheck cadence is unchanged. +A first sight, a changed status line, a due declared time, and an open decision still wake. +Captain-held transfers are not affected. + +With the flag absent, every recheck wakes as before. +The flag is a home-local supervision-noise preference and is not inherited by secondmate homes. +`bin/fm-watch.sh`'s `handle_paused_stale` owns the exact condition. + ## Open-decision reminders (config/open-decision-reminders) The optional local, gitignored `config/open-decision-reminders` presence flag enables a default-off watcher backstop for unanswered keyed `needs-decision` and `blocked` records. @@ -1602,8 +1679,8 @@ Neither fallback is unrestricted execution, and Firstmate does not change vendor ## Worker tool exclusions (config/crew-exclude-tools) -The optional local, gitignored `config/crew-exclude-tools` hides named tools from this home's ship and scout workers, for example to keep an MCP server's write tools out of reach while its read tools stay available. -The contract is runtime-neutral: a runtime must support hiding the listed tool names or refuse the launch, and a non-empty list is never silently ignored. +The optional local, gitignored `config/crew-exclude-tools` hides named tools from this home's ship and scout workers, for example to keep an MCP server's write tools out of reach while its read tools stay available, or to stop sending the schemas of tools a worker never calls with every model request. +The contract is runtime-neutral: a runtime must support hiding the listed tool names or refuse the launch, and a non-empty list is never silently ignored by the launch (Claude name typos are the exception, see [Validation](#validation)). With no file, or a file with no entries, every launch on every runtime is unchanged. Create the file with one tool name per line, such as `mcp__<server>__<tool>` for an MCP tool. @@ -1616,11 +1693,20 @@ It does not apply to a secondmate's own agent, which neither reads nor refuses o | Runtime | With a non-empty list | | --- | --- | +| `claude` | Launches with `--disallowedTools '<names>'`, so each listed tool's schema is absent from every model request, not only denied at call time. An entry may also be an MCP server name such as `mcp__<server>`, which hides all of that server's tools. Applies to every ship and scout spawn and relaunch. | | `pi`, `pi-signed` | Hides listed tool names, MCP tools included, on every ship and scout spawn and relaunch. | | Every other runtime, and a raw launch command | The launch refuses with an error naming `config/crew-exclude-tools`, because that runtime has no verified way to hide tools. | A relaunch validates the list and the replacement runtime's support before stopping the running worker, so an exclusion-list refusal preserves the running agent. +### Trimming Claude tool schemas + +Claude Code sends every enabled tool's schema with each model call, and a long-lived worker resends them on every turn. +List a tool the workers do not use to remove its schema from that fixed cost, for example `Workflow` or the artifact tools; the measured examples are in `docs/verification/runtime-backends.md`. +Claude Code can supply a built-in `Artifact` tool when the artifacts MCP server is disallowed, so hide both `mcp__<artifacts-server>` and `Artifact` to remove every artifact schema. +Only a name that matches a tool present in the worker's session has any effect; [Validation](#validation) explains why Claude names go unchecked, so confirm the saving from a worker's first request or `/context`. +The measured effect on a Claude Code 2.1.319 worker launch is recorded in [`docs/verification/runtime-backends.md`](verification/runtime-backends.md). + ### Validation An entry may use only `A-Z`, `a-z`, `0-9`, `_`, `.`, and `-`. @@ -1629,6 +1715,7 @@ An unreadable or nonregular file, or a path inspection error, also refuses and n For a new worker, these checks run before its endpoint, local copy, or task record is created; Firstmate never launches with a partial list. Only exact tool names are accepted, not wildcard patterns. Firstmate checks syntax and runtime support before launch but never runs `pi mcp list` or otherwise connects to servers to validate names. +Claude workers get no post-launch name check, so a misspelled Claude entry is silently ineffective. When its first agent run starts, after Pi's startup tool-loading boundary, the worker extension compares the launch's exclusion list with its own loaded-tool registry and appends a timestamped warning note to `state/<task-id>.status` naming the configuration file and every unmatched entry for the supervisor. The check runs before worker actions so it cannot supersede a terminal status emitted during the turn. An unmatched entry is reported as **unverified**, not valid: Pi versions that omit excluded tools from the registry cannot distinguish a correct exclusion from a typo, and a server that has not connected cannot verify its tools either. @@ -2382,6 +2469,8 @@ An absent required tool with an automated install path reports `MISSING:` with i Bootstrap also reports a `TANGLE:` line when `FM_ROOT` is on a named non-default branch; follow the printed checkout remediation rather than treating it as an installable tool problem. In a read-only session that did not get the fleet lock, the same line is advisory and omits the checkout command. +With the opt-in [Treehouse pool report](#treehouse-pool-report-configtreehouse-pool-report), bootstrap also prints `TREEHOUSE_POOL_REPORT:` findings. + **Project refresh at startup** The locked session-start deferred network stage runs bootstrap's best-effort project clone refresh through `fm-fleet-sync.sh`; [`fm-bootstrap.sh`'s header](../bin/fm-bootstrap.sh) owns the exact clone-refresh overlap, liveness-before-convergence, per-mate concurrency, ordered diagnostic replay, and sequential-fallback contract. @@ -3513,7 +3602,7 @@ FM_HOME_SUMMARY_TIMEOUT=60 # seconds bounding the complete best-effort home- FM_HOME_SUMMARY_ERROR_LOG_MAX_BYTES=65536 # approximate size cap for state/.home-summary-refresh.log before it is trimmed to the newest 200 lines; invalid or zero values use 65536 FM_HOME_SUMMARY_FAILURE_REPORT=2 # recorded publication failures since the ledger's own last publication before session start reports a HOME_SUMMARY line; invalid or zero values use 2 FM_SNAPSHOT_CREW_STATE_TIMEOUT=10 # seconds bounding each local per-task current-state read inside bin/fm-fleet-snapshot.sh; remote endpoint liveness is not probed on the snapshot path -FM_SNAPSHOT_LOCAL_READ_CONCURRENCY=8 # maximum local tasks whose current-state and endpoint observations are collected concurrently during snapshot composition +FM_SNAPSHOT_LOCAL_READ_CONCURRENCY=8 # maximum local tasks whose current-state and endpoint observations, status-history fold, and row composition run concurrently during snapshot composition FM_SNAPSHOT_BUDGET=5 # one total seconds budget for all concurrent remote home-ledger reads FM_SNAPSHOT_CACHE_DIR=$FM_HOME/state/secondmate-summary-cache # private parent-side cache of successfully fetched remote home ledgers FM_SNAPSHOT_UNDATED_HOLD_AGE_DAYS=14 # floored elapsed-day threshold at which an undated captain hold (no hold-until; age from its UTC hold-set timestamp, falling back to since for legacy unstamped holds) is projected as a Charted Next gate instead of a live Captain's Call; 0 applies once the computed age is non-negative @@ -3624,6 +3713,7 @@ FM_FLEET_PRUNE=1 # set to 0 to skip pruning local branches whose upstream FM_STALE_WORKTREE_LOCK_AGE_SECS=30 # min mtime age before fm-teardown.sh treats a leftover worktree git index.lock as provably stale FM_TREEHOUSE_RETURN_LOCK_RETRIES=3 # retries after a treehouse return fails on the transient git index.lock signature FM_TREEHOUSE_RETURN_LOCK_RETRY_WAIT_SECS=1 # seconds fm-teardown.sh waits before each retry after that signature +FM_POOL_ORPHAN_DAYS=30 # age in days before the opt-in Treehouse pool report flags an unowned slot; invalid values use 30 FM_STALE_WORKTREE_LOCK_RETRY_WAIT_SECS= # legacy alias for FM_TREEHOUSE_RETURN_LOCK_RETRY_WAIT_SECS when the new variable is unset FM_FLEET_SYNC_PACKED_REFS_LOCK_RETRIES=3 # fetch retries after fm-fleet-sync.sh hits the orphaned .git/packed-refs.lock signature FM_FLEET_SYNC_PACKED_REFS_LOCK_RETRY_WAIT_SECS=1 # seconds fm-fleet-sync.sh waits before each of those retries diff --git a/docs/extension-bindings.md b/docs/extension-bindings.md index e3c6e4d271c..5144a60e9e5 100644 --- a/docs/extension-bindings.md +++ b/docs/extension-bindings.md @@ -88,6 +88,9 @@ The mode-`0600` document has schema `firstmate.extension-binding.v1` and exactly - `consents` records `trusted_same_user_code` plus every supported consent fact as an explicit boolean. - `timeout_ms` bounds one invocation between 100 and 3,600,000 milliseconds. +Invocation, handshake, launch-readiness, and launch-release deadlines each allow one 1,000 millisecond grace when their expiry check arrives more than 2,000 milliseconds late, as after host suspension. +An on-time deadline retains its original bound, and a second suspension cannot renew the grace. + The host supports at most 128 binding records and refuses malformed, unsafe, duplicate-id, or duplicate-adapter registries rather than selecting around them. Binding publication is atomic and does not replace a concurrent file. `list`, `inspect`, and `verify` expose the resulting identity and live compatibility without creating state when no registry exists. @@ -215,7 +218,11 @@ An unknown enabled version refuses rather than downgrading. A missing or changed package never executes. A malformed binding, integrity mismatch, failed handshake, crash, nonzero exit, timeout, oversized stream, wrong request id, or invalid response never selects another adapter. A source invocation failure becomes a bounded host-produced `firstmate.process-event-extension-error.v1` result and remains unhandled. -For the same source, a repeated complete host-error envelope reuses an existing unhandled capture only when its host-error sidecar, exact result bytes, adapter, and immutable extension identity all match. +Host timeout evidence includes a `deadline` object identifying the bound, actual operation, configured `timeout_ms`, measured `elapsed_ms` and `overrun_ms`, and granted `grace_ms`. +The launch barrier reports its release-deadline diagnostics on its bounded stderr. +For the same source, a repeated complete host-error envelope reuses an existing unhandled capture only when its host-error sidecar, result identity, adapter, and immutable extension identity all match. +For timeout envelopes with deadline evidence, result identity compares parsed fields with `elapsed_ms` and `overrun_ms` excluded, retaining the first complete diagnostic capture for that bound. +Other host-error envelopes require exact result bytes. A matching retry allocates no new result sequence and records no handled acknowledgement. Ordinary reconciliation continues to publish the retained evidence and retry polling, so old failure evidence does not block recovery. A changed error or extension identity remains distinct, and a failure after its matching evidence was handled receives a fresh capture. diff --git a/docs/fm-test-isolation-proof.md b/docs/fm-test-isolation-proof.md index 43d279b6754..32d33229766 100644 --- a/docs/fm-test-isolation-proof.md +++ b/docs/fm-test-isolation-proof.md @@ -35,6 +35,7 @@ This record owns concurrent isolation evidence for the portable parallel candida - `tests/fm-grok-harness.test.sh` - `tests/fm-herdr-lab.test.sh` - `tests/fm-lint.test.sh` +- `tests/fm-park.test.sh` - `tests/fm-pi-primary-types.test.sh` - `tests/fm-pr-merge.test.sh` - `tests/fm-review-diff.test.sh` @@ -91,6 +92,14 @@ The backend suite spawns real tasks, and `fm-spawn.sh` creates each task's `/tmp `docs/fm-test-isolation-proof.json` still holds the 2026-08-20 portable-pool artifact; a clean 26-candidate artifact needs a Linux run with Bash 5 and tmux. +## 2026-11-10: Review parking suite added + +`tests/fm-park.test.sh` keeps all state under `fm_test_tmproot` and drives a fake tmux binary, so it shares no global state. + +- Command: `bin/fm-test-isolation-proof.sh --jobs 4 --json <path>` on macOS arm64 with Bash 5.3 and tmux, run under `caffeinate -i` +- Result: `FM_ISOLATION_SUMMARY total=27 failed=0 concurrency=4 duration_ms=1372646` +- `tests/fm-park.test.sh` exit 0 (188248 ms). + ## Family concurrency proofs `bin/fm-test-isolation-proof.sh --pool <family>` runs the same concurrent proof over a whole `bin/fm-test-run.sh` family, for a stateful family that stays serial on CI but can earn bounded local concurrency. diff --git a/docs/fm-test-portable-shards.md b/docs/fm-test-portable-shards.md index d1f7bb4e651..2193aa8bcf0 100644 --- a/docs/fm-test-portable-shards.md +++ b/docs/fm-test-portable-shards.md @@ -36,6 +36,7 @@ The 2026-11-04 `tests/fm-install-herdr.test.sh` serial hint is a provisional loc The 2026-10-26 `tests/fm-test-tap-verify.test.sh` serial hint is a provisional local measurement of 1000 ms, above a 238 ms `bin/fm-test-run.sh` sample on this Linux host, until the next CI refresh replaces it with an Ubuntu timing-artifact maximum. The 2026-11-08 `tests/fm-lavish-url.test.sh` serial hint is a provisional local measurement of 30000 ms, above the slowest of three `bin/fm-test-run.sh` samples (25148-26650 ms) on a Linux host, until the next CI refresh replaces it with an Ubuntu timing-artifact maximum. The 2026-10-27 read-repetition suites carry provisional Linux x86_64 local runner maxima: 602 ms for `tests/fm-read-repetition.test.sh` (six successful samples, 528-602 ms) and 1110 ms for `tests/fm-read-repetition-live.test.sh` (five successful samples, 949-1110 ms). +`tests/fm-park.test.sh` is now a portable parallel member; its 70085 ms parallel hint is its completed passing Ubuntu `FM_TEST_END` sample from CI run [42730709201](https://github.com/kunchenguid/firstmate/actions/runs/42730709201), and `tests/fm-arm-pretool-check.test.sh` moved to parallel lane 2 to keep the lanes within 5%. The live samples exercised installed Pi 1.4.0 RPC without capability skips; they are not CI timing evidence or timings for unavailable harnesses. Both suites retain serial placement and full coverage; replace these provisional hints with Ubuntu timing-artifact maxima at the next CI refresh. The 2026-10-29 `tests/fm-live-lab-isolation.test.sh` serial hint is a provisional local measurement of 6459 ms, the slowest of six `bin/fm-test-run.sh` samples (4308-6459 ms) taken inside a private `unshare -Urm` /tmp on this Linux host, until the next CI refresh replaces it with an Ubuntu timing-artifact maximum. diff --git a/docs/remote-secondmates.md b/docs/remote-secondmates.md index cf104557288..483685bc5d7 100644 --- a/docs/remote-secondmates.md +++ b/docs/remote-secondmates.md @@ -142,7 +142,7 @@ It never touches a worker whose checkout still exists. Remote job execution never runs a login or interactive shell. So `~/.profile`, `~/.bashrc`, and `~/.zshrc` never contribute to the job worker's runtime `PATH`. -`bin/fm-remote-job-lib.sh` is the single owner of the worker `PATH`. +The [`bin/fm-remote-job-lib.sh` header](../bin/fm-remote-job-lib.sh) owns the child environment contract and worker `PATH`. It builds the `PATH` by filesystem discovery rather than by evaluating shell startup files. ### Worker PATH order diff --git a/docs/scripts.md b/docs/scripts.md index ca269e26a91..8a884142b2a 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -11,7 +11,7 @@ The shared no-mistakes gate lifecycle boundary is summarized in [architecture.md | `fm-sessionstart-nudge.sh` | Print the native session-start hook nudge when the primary has not already run the digest | | `fm-sessionstart-run.sh` | Route a native session-open hook to the full digest, a context re-emit, or the nudge | | `fm-operational-input.sh` | Construct and parse the canonical cross-language operational-input protocol | -| `fm-bootstrap.sh` | Detect toolchain and fleet problems, run the locked session-start sweeps, and install approved tools | +| `fm-bootstrap.sh` | Detect toolchain and fleet problems, report home-local record inconsistencies when `config/record-hygiene` is present, run the locked session-start sweeps, and install approved tools | | `fm-startup-network.sh` | Run session start's network checks and inactive-outcome scan off its blocking path, retaining reports and durable findings | | `fm-fleet-sync.sh` | Refresh project clones with safe fast-forwards, self-heals, `STUCK:` reports, branch pruning, and bounded recovery from an orphaned `.git/packed-refs.lock` | | `fm-fleet-snapshot.sh` | Print structured fleet snapshot JSON and refresh only its parent-side remote-ledger cache (schema `fm-fleet-snapshot.v1`) | @@ -32,7 +32,7 @@ The shared no-mistakes gate lifecycle boundary is summarized in [architecture.md | `fm-remote-doctor.sh` | Check, and with `--fix` repair, one remote account's [second-mate readiness](remote-secondmates.md#readiness-repair-and-the-human-steps) | | [`fm-backlog-handoff.sh`](../bin/fm-backlog-handoff.sh) | Move queued backlog items into a secondmate home; its header owns route-specific wake outcomes and retries | | `fm-backlog-receive.sh` | Idempotently ingest one confined remote handoff outbox through tasks-axi | -| `fm-captain-hold.sh` | Hold tasks for the captain, record the captain's answers, gate investigation completion, and report record divergence between the status log and the backlog | +| `fm-captain-hold.sh` | Hold tasks for the captain, record the captain's answers, gate investigation completion, settle declared holds from forge PR evidence, and report record divergence between the status log and the backlog | | `fm-decision-hold.sh` | One-release compatibility shim mapping the retired decision commands onto fm-captain-hold.sh | | `fm-brief.sh` | Scaffold ship (explicit `--mode`, plus the project's registered `--forge` and `--pr`), scout, secondmate-charter, and Herdr-lab briefs, with Captain's intent and Firstmate spec subsections on ship/scout | | [`fm-dod-lib.sh`](../bin/fm-dod-lib.sh) | Own ship/scout worker role scope, ship definitions of done, the named-head reachability gate on ship `done:` acceptance, and the no-mistakes `--intent` contract | @@ -162,7 +162,8 @@ The shared no-mistakes gate lifecycle boundary is summarized in [architecture.md | `fm-pr-poll.sh` | Provide the byte-static watcher program for validated pull-request, merge-request, Bitbucket pull-request, and Gerrit-change poll sidecars; reports merged or post-ready degradation for GitHub checks, GitLab head pipelines, and merge-train drops (see [GitLab merge watch](gitlab-merge-watch.md)) | | `fm-contributions.sh` | Observe owned publications, retain exact-head judgments, measure required actors, and wake on maintainer or configured review-bot signals | | `fm-pr-health.sh` | Opt-in: enroll owned PRs; wake once per latest commit for conflicts, failing checks, and author actions; record maintainer-gated CI and missing required reviews without waking the task, using GitHub notifications plus a fallback sweep; for a Bitbucket Cloud pull request a task recorded, wake on conflicts, failed merge checks, failed builds, and unresolved review threads read through `twg` (see [configuration.md](configuration.md#pr-health-monitor-configpr-health-monitor)) | -| `fm-pr-check.sh` | Record validated task `pr=` and `pr_head=` values, then atomically arm a static merge poll that also wakes on degraded ready PR/MR states (GitHub checks, GitLab pipelines, and merge-train drops) when `config/pr-degraded-watch` is present; refuses GitHub drafts and persistent secondmate records (see [architecture.md](architecture.md)); `scan`, run by the watcher's check sweep when `config/pr-register-scan` is present, records each PR a task's `done:` line reports when no `pr=` exists | +| `fm-park.sh` | Park a ship lane whose PR or MR only awaits human review from a verified clean state, relaunch a fresh worker on review activity, and clean it up on merge (see [configuration.md](configuration.md#park-on-review)) | +| `fm-pr-check.sh` | Record validated task `pr=` and `pr_head=` values, then atomically arm a static merge poll that also wakes on degraded ready PR/MR states (GitHub checks, GitLab pipelines, and merge-train drops) when `config/pr-degraded-watch` is present; refuses GitHub drafts and persistent secondmate records (see [architecture.md](architecture.md)); `scan`, run by the watcher's check sweep when `config/pr-register-scan` is present, records a task's ready `done:` PR when no `pr=` exists, its matching poll is missing, or it replaces one its status log named | | `fm-review-check.sh` | Record a forge=external task's validated `review=` and `review_head=` without arming a merge watch ([design](external-review-forge.md)) | | `fm-external-review-lib.sh` | Shared readers for the forge=external review tool adapter, ready-URL pattern, backup remote, and remote URLs | | `fm-pr-merge.sh` | Record PR metadata, merge a task's canonical full GitHub, GitLab, or Forgejo URL, refuse a Gerrit change because firstmate never submits one, then refuse an outcome it cannot prove landed or queued | diff --git a/docs/sessionstart-nudge.md b/docs/sessionstart-nudge.md index b21eb2b60d0..6b19c709ac1 100644 --- a/docs/sessionstart-nudge.md +++ b/docs/sessionstart-nudge.md @@ -103,6 +103,9 @@ So `clear` or `compact` cannot skip startup sweeps after a truncated run. So a proven `clear` or `compact` re-emit re-verifies ownership and proceeds. A lock another live session took meanwhile still produces the ordinary read-only digest. +The [`bin/fm-lock.sh` header](../bin/fm-lock.sh) owns claim-lock recovery and refusal behavior. +If startup names an unreclaimable lock directory, inspect the reported path's contents and permissions before retrying. + ### Nudge wrapper on a run-tier harness On a run-tier harness, only `resume`, `reload`, and `fork` are routed to the nudge wrapper. diff --git a/docs/t3code-backend.md b/docs/t3code-backend.md index 040292762d3..dc18fa5684d 100644 --- a/docs/t3code-backend.md +++ b/docs/t3code-backend.md @@ -115,7 +115,7 @@ A secondmate additionally receives the launch prefix every other backend types ( A `claude` ship or scout worker also receives the task-worker channel statement that a pane launch appends to the system prompt, written as a `CLAUDE.local.md` in its worktree because T3 owns the system prompt; a secondmate does not, as on every backend. Without it a Claude worker can refuse the launch brief as prompt injection, which happened live. The statement's pull-request-tool workaround follows the [verified version boundary](verification/runtime-backends.md#claude-pull-request-tools); an older or unreadable server keeps the clause, while a server at or past the verified build gets none. -Firstmate records the PR from the worker's `done: PR <url>` status line either way. +Firstmate records the PR from a worker's ready `done:` report; [`config/pr-register-scan`](configuration.md#pr-registration-scan-configpr-register-scan) describes the scan's accepted forms. A `claude` task checks the leased worktree and refuses a tracked, existing, or symlinked `CLAUDE.local.md`, because that file is the backend's channel and teardown removes it. ## Current lifecycle and safety @@ -227,9 +227,9 @@ Resolve ambiguity by explicitly setting both `FM_SUPERVISOR_BACKEND=t3code` and Injection is a `t3_thread_send` only into a settled supervisor thread, one the [thread status table](#restart-and-liveness-behavior) reads idle/alive with no pending request; any other row, including `blocked` and an unreadable thread, defers the digest exactly as a busy pane does on every other backend. An unknown native busy verdict defers new-turn delivery but does not prove that work resumed. Away housekeeping keeps an overdue stale alert pending and buffers one possible-wedge report while the verdict remains unknown, including a failed run or a transient thread-read failure. -Delivered warnings stay suppressed while the same unknown condition persists. -If a fresh daemon entry discards a queued, undelivered warning, it clears that warning's reported marker and preserves its stale timer, so the next housekeeping pass queues exactly one replacement. -Refreshing a live daemon preserves the queue; a failed start restores the previous queue and reported markers. +Within the same away session, delivered warnings stay suppressed while the same unknown condition persists. +The [stale-artifact lifecycle](../.agents/skills/afk/SKILL.md#stale-artifact-lifecycle) owns session-boundary cleanup and recheck-window resets. +Refreshing a live daemon preserves the queue and episode markers; a failed start restores the previous queue and episode markers. Confirmed busy activity, a missing thread, or an idle verdict after the unknown condition was reported clears the stale and reported markers without another report, so a later unknown condition can report again. Declared external-wait rechecks also survive native uncertainty; the worker's declaration still controls their cadence. `stale_window_is_busy` and `housekeeping` in `bin/fm-supervise-daemon.sh` own stale tracking; `fm_afk_clear_stale_artifacts` in `bin/fm-afk-start.sh` and the launcher in `bin/fm-afk-launch.sh` own queue discard and startup rollback. diff --git a/docs/turnend-guard.md b/docs/turnend-guard.md index 0d5b99e6f12..ec1a58da815 100644 --- a/docs/turnend-guard.md +++ b/docs/turnend-guard.md @@ -364,8 +364,12 @@ Pinning that signal to the session-lock identity recorded at claim time is what An orphan fails both signals: it no longer descends from the replacement's lock pid, and its recorded lock-holder identity is the dead session's. An entry from a build before these fields existed records neither and falls back to descent alone. +The auto-arm also records the Stop payload's `session_id` on line 1 as `claude_session`, which later writes of that generation keep, and a firing from another Claude session does not find that claim open. +Claude delivers a hook's exit-2 rewake to the session that launched the hook, so a claim taken before Claude continued the conversation under a new session id, as a backgrounded session's `--fork-session` resume does, can only wake a conversation that no longer runs, even while its owner still descends from the lock-holding front end. +A claim or a payload with no session id is judged as before, and so is every check the guard itself makes, because it names no session. + Anything else lets the next Stop-owned firing take the next generation and arm. -That covers a finished outcome, a dead or identity-mismatched owner, an owner orphaned by its session's exit, a stuck owner, an identityless entry, or no entry. +That covers a finished outcome, a dead or identity-mismatched owner, an owner orphaned by its session's exit, a stuck owner, an identityless entry, an earlier session's claim, or no entry. Taking a newer generation is the reclaim, and a steady-state predecessor is never signalled or revoked. No mutex is held across arming or output. diff --git a/docs/verification/runtime-backends.md b/docs/verification/runtime-backends.md index fd9471d6a1c..9f0aa04db37 100644 --- a/docs/verification/runtime-backends.md +++ b/docs/verification/runtime-backends.md @@ -688,6 +688,32 @@ skip-runner: pi-signed is not installed, so its pin check was not exercised The guard submits no prompt and spends no tokens, so it runs by default wherever a runner is installed; rerun it after every Claude or Pi upgrade. +## Claude worker tool exclusions + +`config/crew-exclude-tools` reaches a Claude ship or scout launch as `--disallowedTools '<names>'` placed directly before `--append-system-prompt`, because the option is variadic and would otherwise swallow the launch prompt. +[Worker tool exclusions](../configuration.md#worker-tool-exclusions-configcrew-exclude-tools) owns the contract; this record owns the measurement. +Verified on 2026-11-06 with Claude Code 2.1.319 on Linux, interactive launches in a tmux pane with tool search on, against a capture-only local server (`ANTHROPIC_BASE_URL`) that records each `/v1/messages` body and answers 529, so no model is called. +The arms differ only in the `--disallowedTools` value; the figures are the bytes of the JSON `tools` array and of the whole request for the main-agent request. The two repeats of the `Workflow,Artifact,mcp__claude-artifacts` arm matched on tool schema bytes and differed by 12 request bytes. + +| Launch | Tools sent | Tool schema bytes | Request bytes | +| --- | --- | --- | --- | +| no list, artifacts MCP server connected | 20 | 52064 | 188189 | +| `Workflow,Artifact,mcp__claude-artifacts` | 12 | 28416 | 161748 | +| `mcp__claude-artifacts` only (built-in `Artifact` takes its place) | 14 | 70987 | 206684 | +| `Workflow,Artifact,mcp__claude-artifacts` | 12 | 28416 | 161736 | + +Findings that shape the contract: + +- A disallowed name is removed from the request's `tools` array, and a disallowed deferred MCP tool's name also leaves the deferred-tool list in the messages, so the saving is not only the schema. +- `--tools` is not a substitute: naming an allow-list made Claude Code send every connected MCP tool eagerly (65 tools, 139760 schema bytes in the same environment), the opposite of a saving. +- A server-level entry `mcp__<server>` hides every tool of that server, and with the artifacts server hidden Claude Code supplies a 37098-byte built-in `Artifact` tool, so both names are needed to drop every artifact schema. + +The token-free live guard refreshes the per-version evidence and fails naming the Claude Code version if a release stops honoring the flag: + +```sh +bin/fm-test-run.sh tests/fm-claude-tool-exclude-live.test.sh +``` + ## Secondmate busy contract Verified 2026-09-22 on this host with Claude Code 2.1.278, OpenCode 1.18.31, Pi 0.86.1, and Grok 1.0.40. @@ -929,6 +955,30 @@ Live validation ran 2026-10-01 on Grok 1.0.46 (2765805b9442) [stable], with foll The same run's `FM_COMPOSER_MATRIX_LIVE=1 tests/fm-composer-matrix-live-e2e.test.sh` proved the real idle grok 1.0.46 composer `empty` through tmux and checked that the cursorless re-read was not `pending`. The portable owner of the shape is `test_matrix_grok_144_approval_titled_bottom_border` in `tests/fm-composer-lib.test.sh`, which pins the `Grok 4.7 (high)` approval title's idle `empty` on tmux (with the cursor on either the content row or bottom border), herdr, and cmux/orca capability profiles, and typed `pending` on tmux and herdr only. It also pins idle `empty` for `Grok 4.6` at all four efforts (`low`, `medium`, `high`, `xhigh`) on tmux only, `unknown` for the tested malformed titles on tmux, rejection of light, heavy, double, and ASCII approval-title boxes on tmux, herdr, and cmux/orca profiles, and Grok 1.0.5's three-column overhang still proving `empty` for a title without the suffix on tmux. + +Grok 1.0.77 (50ca579fa996), verified 2026-11-12 on Rex at 80x24, keeps this aligned approval title for `grok-4.7`, `grok-4.6`, and `grok-4.5`, but titles `grok-4.7-build-fast` with a two-word model name: `╰── … ── Grok 4.7 Fast (medium) · always-approve ─╯`. +The space in `4.7 Fast` failed the typed-title model check, so that idle composer read `unknown` and `exit` and `relaunch` refused it (#8388). +The accepted delta is that one ` Fast` word after the model version, and only on the ` · always-approve` title; a second word, another word, and the `Fast` title at Grok 1.0.5's overhang stay `unknown`. +The live check launched `grok --always-approve --model grok-4.7-build-fast --reasoning-effort medium` in a scratch Rex block and read it through `fm_backend_rex_composer_state` from `bin/backends/rex.sh`, with the cursor (row 19) and cursorless. +Before the change, idle, after one turn, and after a Ctrl+C interrupt each read `unknown`, and a typed draft read `pending-unproven`. +After the change the same states read: + +```text +idle: cursor_y=19 state=empty cursorless=empty +after Ctrl+C: cursor_y=19 state=empty cursorless=empty +typed `draft text not sent`: cursor_y=19 state=pending cursorless=pending +cleared with Ctrl+U: cursor_y=19 state=empty cursorless=empty +``` + +`test_matrix_grok_1077_fast_model_title` in `tests/fm-composer-lib.test.sh` feeds the real styled Rex captures in `tests/fixtures/rex-grok-1.0.77/` through the classifier and pins idle `empty`, typed `pending`, and the unobserved titles `unknown`. +The live guard launches only the default model, so it does not exercise the `Fast` title; its 1.0.77 result is: + +```text +$ FM_REX_COMPOSER_LIVE=1 bash tests/fm-backend-rex-composer-live-e2e.test.sh +ok - grok (grok 1.0.77 (50ca579fa996)): real idle composer classifies empty on rex +ok - grok (grok 1.0.77 (50ca579fa996)): the same idle screen read cursorless is not pending (verdict: empty) +``` + After a Grok upgrade, rerun that live guard and refresh both this record and the 1.0.5 note above: each accepted width is pinned to an observed release, the 1.0.5 overhang has never been live-verified, and neither the approval wording nor either width carries a guarantee across a release. ### 2026-09-20 claude 2.1.236 statusLine footer through Herdr @@ -3406,6 +3456,21 @@ The evidence below was produced on 2026-09-05 against omp 18.1.11 (`~/.local/bin ### Process identity and markers `ps -o comm=` reports the bare name `omp` for the agent process, from both its `!` bash path and the model's bash tool, so identity is the anchored name; `ompd` and `comp` never match. +The bun global package (`bun install -g @oh-my-pi/pi-coding-agent`) installs `~/.bun/bin/omp` as a symlink to `dist/cli.js` with a `#!/usr/bin/env bun` shebang, so its process is named `bun` and only its arguments (`bun <path>/omp`) name omp. +`bin/fm-omp-process-lib.sh` owns both shapes: the exact name `omp`, or `bun` whose argument right after bun is a script named exactly `omp`, including an install path that contains spaces. +`ompd`, `comp`, and another script followed by an `omp` argument never match. +`tests/fm-omp-harness.test.sh` and `tests/fm-session-lock-ancestry.test.sh` pin these shapes with real processes; the former also drives `bin/fm-harness.sh` and `bin/fm-lock.sh` from a bun-run script. + +The token-free guard refreshes the installed-omp evidence and spends no model tokens: it starts omp in JSON-RPC mode with no prompt, and a loaded extension runs `bin/fm-harness.sh` and `bin/fm-lock.sh` from inside that process; the guard passes only when the saved lock names that process's pid. + +```text +FM_OMP_IDENTITY_LIVE=1 bash tests/fm-omp-primary-live-e2e.test.sh +ok - omp omp/18.12.3 (bun-package): detected as omp and acquired the session lock with no model tokens +``` + +Run on 2026-11-10 against omp 18.12.3, the bun package at `~/.bun/bin/omp`, on macOS 26 arm64. +Before the change the same install detected as `unknown` and the lock refused with `cannot locate harness process in ancestry`. +The portable cases ran on macOS only; the Linux procps shape (`comm` is the exec name `bun`) is covered by the fake-ps cases in `tests/fm-session-lock-ancestry.test.sh`, not by a Linux run. omp publishes no harness marker: `PI_CODING_AGENT` is absent from the binary, and the default profile sets neither `PI_CODING_AGENT_DIR` nor `OMP_PROFILE` in the process environment. `FM_OMP_HARNESS=omp` is Firstmate's own launch marker and wins over an inherited `CLAUDECODE` only under a real omp ancestor; `tests/fm-omp-harness.test.sh` pins both directions with real processes. diff --git a/docs/watcher-continuity.md b/docs/watcher-continuity.md index d02ae4d7718..349c710da50 100644 --- a/docs/watcher-continuity.md +++ b/docs/watcher-continuity.md @@ -293,6 +293,7 @@ Every presented row is claimed to exactly one actor under the durable queue lock Publication is refused if main already claimed any requested row. Activation, publication, release, and deactivation exit nonzero if queue-lock acquisition fails, preserving both grant records and leaving another owner's lock intact. - A main drain validates that owner evidence under the queue lock and reclaims the grant when its process is gone or its identity no longer matches. + A branch drain never reclaims: it drains the grant just published for its turn, and its sandbox may be unable to inspect the host pid. - A main drain claims every currently unclaimed row and excludes an active branch grant from both presentation and acknowledgement. ### Lock deadlines during presentation @@ -559,7 +560,7 @@ It checks that a newly appended keyed decision is classified without rereading e - An unconfirmed successor reported in the banner without withholding the wake. - The [session-bound HUP/TERM/INT failure handoff](turnend-guard.md#failure-progression-and-block-budget) and replacement-session notice isolation. -It also covers generation-claim single-flight, stuck-claim supersession, superseded-owner silence, notice-marker refusal and retry, ownership-atomic episode reset, and the legacy upgrade shim. +It also covers generation-claim single-flight, stuck-claim and earlier-session supersession, superseded-owner silence, notice-marker refusal and retry, ownership-atomic episode reset, and the legacy upgrade shim. [`turnend-guard.md`](turnend-guard.md) owns those behavior contracts. `FM_CLAUDE_LIVE_E2E=1 tests/fm-claude-stop-autoarm-live-e2e.test.sh`: diff --git a/tests/fixtures/fm-claude-request-capture.py b/tests/fixtures/fm-claude-request-capture.py new file mode 100755 index 00000000000..2b346dbe7a5 --- /dev/null +++ b/tests/fixtures/fm-claude-request-capture.py @@ -0,0 +1,37 @@ +#!/usr/bin/env python3 +"""Capture-only stand-in for the Claude API: record each /v1/messages request body, answer 529. + +usage: fm-claude-request-capture.py <port-file> <capture-file> + +Binds an ephemeral loopback port, writes it to <port-file>, appends one JSON request body per +line to <capture-file>, and never contacts a model, so a run costs no tokens. +""" +import http.server +import sys + +port_file, capture_file = sys.argv[1], sys.argv[2] + + +class Handler(http.server.BaseHTTPRequestHandler): + def log_message(self, *args): + pass + + def do_POST(self): + body = self.rfile.read(int(self.headers.get("content-length", 0))) + if "/v1/messages" in self.path and "count_tokens" not in self.path: + with open(capture_file, "ab") as out: + out.write(body.replace(b"\n", b" ") + b"\n") + reply = b'{"type":"error","error":{"type":"overloaded_error","message":"capture only"}}' + self.send_response(529) + self.send_header("content-type", "application/json") + self.send_header("content-length", str(len(reply))) + self.end_headers() + self.wfile.write(reply) + + do_GET = do_POST + + +server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) +with open(port_file, "w") as out: + out.write(str(server.server_address[1])) +server.serve_forever() diff --git a/tests/fixtures/rex-grok-1.0.77/grok-1.0.77-fast-idle-after-interrupt.vt b/tests/fixtures/rex-grok-1.0.77/grok-1.0.77-fast-idle-after-interrupt.vt new file mode 100644 index 00000000000..713f5a5a188 --- /dev/null +++ b/tests/fixtures/rex-grok-1.0.77/grok-1.0.77-fast-idle-after-interrupt.vt @@ -0,0 +1,24 @@ +  + /p/t/c/-/3/scratchpad/grokdir 28K / 256K │ [Dashboard]  +  +     +  ❯ Reply with just the word ok.  9:02 PM  █ +   █ + █ + ok  9:02 PM █ + █ + Worked for 2.1s █ + █ +  + Help improve Grok [Opt out] [Opt in]  + Off by default. Opt-in to allow SpaceXAI to retain coding  + data, e.g., prompts, traces, & metrics, for training and  + debugging purposes. Change anytime via settings.  + Read Terms and Privacy Policy.  +  + ╭──────────────────────────────────────────────────────────────────────────╮  + │ ❯  │  + ╰──────────────────────────────── Grok 4.7 Fast (medium) · always-approve ─╯  +  + Shift+Tab:mode │ Ctrl+.:shortcuts   +  diff --git a/tests/fixtures/rex-grok-1.0.77/grok-1.0.77-fast-typed.vt b/tests/fixtures/rex-grok-1.0.77/grok-1.0.77-fast-typed.vt new file mode 100644 index 00000000000..f364ecff065 --- /dev/null +++ b/tests/fixtures/rex-grok-1.0.77/grok-1.0.77-fast-typed.vt @@ -0,0 +1,24 @@ +  + /p/t/c/-/3/scratchpad/grokdir 28K / 256K │ [Dashboard]  +  +     +  ❯ Reply with just the word ok.  9:02 PM  █ +   █ + █ + ok  9:02 PM █ + █ + Worked for 2.1s █ + █ +  + Help improve Grok [Opt out] [Opt in]  + Off by default. Opt-in to allow SpaceXAI to retain coding  + data, e.g., prompts, traces, & metrics, for training and  + debugging purposes. Change anytime via settings.  + Read Terms and Privacy Policy.  +  + ╭──────────────────────────────────────────────────────────────────────────╮  + │ ❯ draft text not sent │  + ╰──────────────────────────────── Grok 4.7 Fast (medium) · always-approve ─╯  +  + Enter:send │ Shift+Enter/Opt+Enter:newline │ Shift+Tab:mode   +  diff --git a/tests/fm-afk-launch.test.sh b/tests/fm-afk-launch.test.sh index d045a093a1e..6f08377e1d5 100755 --- a/tests/fm-afk-launch.test.sh +++ b/tests/fm-afk-launch.test.sh @@ -256,7 +256,8 @@ unit_stop_archives_the_record_last() { } # --------------------------------------------------------------------------- -# UNIT 1: fm_afk_clear_stale_artifacts removes exactly the four stale artifacts. +# UNIT 1: fm_afk_clear_stale_artifacts removes exactly the four delivery +# artifacts plus the prior session's five per-task marker families. # --------------------------------------------------------------------------- unit_clear_stale() { local st @@ -266,6 +267,12 @@ unit_clear_stale() { : > "$st/state/.subsuper-escalations.since" : > "$st/state/.subsuper-inject-wedged" : > "$st/state/.subsuper-unknown-acked" + : > "$st/state/.subsuper-stale-task1" + : > "$st/state/.subsuper-reported-stale-task1" + : > "$st/state/.subsuper-dialog-task1" + : > "$st/state/.subsuper-paused-task1" + : > "$st/state/.subsuper-pause-until-due-task1" + : > "$st/state/.subsuper-seen-status-task1" # classification position must be kept : > "$st/state/.wake-queue" # durable queue must be untouched # Source fm-afk-start.sh inside a child bash (it sets `set -eu` and would # otherwise leak that into this test shell) and call the clear helper. @@ -279,6 +286,20 @@ unit_clear_stale() { else fail "clear-stale: stale artifacts survived" fi + if ! compgen -G "$st/state/.subsuper-stale-*" >/dev/null \ + && ! compgen -G "$st/state/.subsuper-reported-stale-*" >/dev/null \ + && ! compgen -G "$st/state/.subsuper-dialog-*" >/dev/null \ + && ! compgen -G "$st/state/.subsuper-paused-*" >/dev/null \ + && ! compgen -G "$st/state/.subsuper-pause-until-due-*" >/dev/null; then + pass "clear-stale: removes the prior session's five per-task marker families" + else + fail "clear-stale: per-task session markers survived" + fi + if [ -e "$st/state/.subsuper-seen-status-task1" ]; then + pass "clear-stale: keeps the durable classification position" + else + fail "clear-stale: removed the classification position" + fi if [ -e "$st/state/.wake-queue" ]; then pass "clear-stale: leaves the durable wake-queue intact (no pending work dropped)" else @@ -287,6 +308,95 @@ unit_clear_stale() { rm -rf "$st" } +# --------------------------------------------------------------------------- +# The per-task away markers are session-scoped episode state: the return path +# (fm_afk_launch_stop) and the fresh-entry cleanup must both drop all five +# families, or the next session's first housekeeping reads a dead session's +# idle clock as its own and reports a wedge whose duration never happened. +# The last pass replays the implicit-state-machines audit's reproduction: a +# hand-written prior-session clock, an idle pane read, delivery stubbed off. +# --------------------------------------------------------------------------- +unit_per_task_markers_session_scoped() { + local st state now + st=$(mktemp -d "${TMPDIR:-/tmp}/fm-afk-marker-scope.XXXXXX") + state="$st/state" + mkdir -p "$state" + now=$(date +%s) + # One finished ship task, as the audit's scratch home: a terminal status and + # a meta that still maps its window. + printf 'done [at=%s]: PR https://example.test/o/r/pull/1 checks green\n' "$now" > "$state/demo.status" + printf 'window=%s\nbackend=tmux\nharness=claude\n' 'fmaudit-none:fm-demo' > "$state/demo.meta" + printf '0:fixture\n' > "$state/.subsuper-seen-status-demo" + printf '%s\n' "$now" > "$state/.subsuper-last-scan" + seed_markers() { + # Same path and content as the daemon's stale_marker_record: the epoch the + # pane was first seen idle, mid-episode when the session ended. + printf '%s\n' $(( now - 172801 )) > "$state/.subsuper-stale-demo" + : > "$state/.subsuper-reported-stale-demo" + : > "$state/.subsuper-dialog-demo" + : > "$state/.subsuper-paused-demo" + : > "$state/.subsuper-pause-until-due-demo" + } + markers_gone() { + ! compgen -G "$state/.subsuper-stale-*" >/dev/null \ + && ! compgen -G "$state/.subsuper-reported-stale-*" >/dev/null \ + && ! compgen -G "$state/.subsuper-dialog-*" >/dev/null \ + && ! compgen -G "$state/.subsuper-paused-*" >/dev/null \ + && ! compgen -G "$state/.subsuper-pause-until-due-*" >/dev/null + } + + # The return path drops the session's markers with the away flag. + seed_markers + : > "$state/.afk" + FM_HOME="$st" FM_STATE_OVERRIDE="$state" "$LAUNCH" stop >/dev/null 2>&1 + if markers_gone && [ ! -e "$state/.afk" ] && [ -e "$state/.subsuper-seen-status-demo" ]; then + pass "return: stop drops all five per-task marker families, keeps the classification position" + else + fail "return: per-task markers survived stop ($(compgen -G "$state/.subsuper-*" | tr '\n' ' '))" + fi + + # The fresh-entry cleanup drops them alongside the delivery artifacts. + seed_markers + FM_HOME="$st" FM_STATE_OVERRIDE="$state" \ + bash -c '. "$1"; fm_afk_clear_stale_artifacts "$2"' _ "$START" "$state" + if markers_gone; then + pass "entry: fresh-session cleanup drops all five per-task marker families" + else + fail "entry: per-task markers survived fresh-session cleanup ($(compgen -G "$state/.subsuper-*" | tr '\n' ' '))" + fi + + # The audit narrative end to end: session one's clock goes through stop and + # the next entry's cleanup, then the new daemon's first housekeeping pass. + # Reset to the audit's repro shape first: ONLY the stale clock, because + # housekeeping's pause migration retires the episode before the wedge check + # whenever a paused marker is also present. + rm -f "$state"/.subsuper-stale-* "$state"/.subsuper-reported-stale-* \ + "$state"/.subsuper-dialog-* "$state"/.subsuper-paused-* \ + "$state"/.subsuper-pause-until-due-* + printf '%s\n' $(( now - 172801 )) > "$state/.subsuper-stale-demo" + : > "$state/.afk" + FM_HOME="$st" FM_STATE_OVERRIDE="$state" "$LAUNCH" stop >/dev/null 2>&1 + : > "$state/.afk" # a fresh entry, two days later in the audit's telling + printf '%s\n' "$(date +%s)" > "$state/.subsuper-last-scan" + FM_HOME="$st" FM_STATE_OVERRIDE="$state" \ + bash -c '. "$1"; fm_afk_clear_stale_artifacts "$2"' _ "$START" "$state" + FM_HOME="$st" FM_STATE_OVERRIDE="$state" FM_DAEMON_PRIMARY_HARNESS=claude \ + bash -c ' + . "$1" + stale_window_is_busy() { return 1; } + escalate_flush() { return 0; } + inject_msg() { return 0; } + inject_wedge_alarm() { return 0; } + housekeeping "$2" + ' _ "$ROOT/bin/fm-supervise-daemon.sh" "$state" + if ! grep -q 'possible wedge' "$state/.subsuper-escalations" 2>/dev/null; then + pass "session scope: a dead session's idle clock cannot report 'possible wedge' in the next session" + else + fail "session scope: first housekeeping reported a wedge that never happened ($(cat "$state/.subsuper-escalations"))" + fi + rm -rf "$st" +} + unit_relative_paths_are_absolute_before_daemon_launch() { local root home state out status linked_home root=$(mktemp -d "${TMPDIR:-/tmp}/fm-afk-relative-home.XXXXXX") @@ -1787,6 +1897,7 @@ e2e_rex() { } unit_clear_stale +unit_per_task_markers_session_scoped unit_enter_records_the_posture_in_one_step_without_a_daemon unit_retired_two_step_entry_is_refused unit_pi_never_launches_the_daemon diff --git a/tests/fm-afk-return.test.sh b/tests/fm-afk-return.test.sh index ee04099376b..1faf467bb1b 100755 --- a/tests/fm-afk-return.test.sh +++ b/tests/fm-afk-return.test.sh @@ -484,7 +484,7 @@ test_return_brief_points_at_the_drain_on_a_host_home_only() { for harness in claude pi; do dir="$TMP_ROOT/window-pointer-$harness" install_runner "$dir" - for f in fm-supervision-engine-lib.sh fm-session-lock-lib.sh fm-harness.sh fm-cursor-lib.sh fm-gemini-lib.sh fm-hermes-process-lib.sh; do + for f in fm-supervision-engine-lib.sh fm-session-lock-lib.sh fm-omp-process-lib.sh fm-harness.sh fm-cursor-lib.sh fm-gemini-lib.sh fm-hermes-process-lib.sh; do cp "$ROOT/bin/$f" "$dir/bin/" done : > "$dir/home/config/supervision-host" @@ -521,7 +521,7 @@ test_return_brief_all_silent_window_does_not_point_at_drain() { local dir fakebin out f dir="$TMP_ROOT/window-pointer-silent" install_runner "$dir" - for f in fm-supervision-engine-lib.sh fm-session-lock-lib.sh fm-harness.sh fm-cursor-lib.sh fm-gemini-lib.sh fm-hermes-process-lib.sh; do + for f in fm-supervision-engine-lib.sh fm-session-lock-lib.sh fm-omp-process-lib.sh fm-harness.sh fm-cursor-lib.sh fm-gemini-lib.sh fm-hermes-process-lib.sh; do cp "$ROOT/bin/$f" "$dir/bin/" done : > "$dir/home/config/supervision-host" @@ -961,7 +961,7 @@ test_return_brief_reports_only_an_open_downtime_episode_as_a_gap() { # main-session lock so the latch record's key is the current one. seed_host_latch() { # <case-dir> <errors> <cooldown> <retry-after> <log-lines> local dir=$1 key f - for f in fm-supervision-engine-lib.sh fm-session-lock-lib.sh fm-harness.sh fm-cursor-lib.sh fm-gemini-lib.sh fm-hermes-process-lib.sh; do + for f in fm-supervision-engine-lib.sh fm-session-lock-lib.sh fm-omp-process-lib.sh fm-harness.sh fm-cursor-lib.sh fm-gemini-lib.sh fm-hermes-process-lib.sh; do cp "$ROOT/bin/$f" "$dir/bin/" done printf 'claude sonnet\n' > "$dir/home/config/supervision-host" diff --git a/tests/fm-agy-primary.test.sh b/tests/fm-agy-primary.test.sh index c70bc08f0e3..a4f8a7d0537 100755 --- a/tests/fm-agy-primary.test.sh +++ b/tests/fm-agy-primary.test.sh @@ -59,7 +59,7 @@ install_scripts() { fm-arm-pretool-check.sh fm-cd-pretool-check.sh fm-subagent-pretool-check.sh \ fm-claude-stop-autoarm.sh fm-hook-host-lib.sh \ fm-primary-scope-lib.sh fm-supervision-lib.sh fm-wake-lib.sh fm-path-lib.sh \ - fm-session-lock-lib.sh fm-cursor-lib.sh fm-gemini-lib.sh fm-hermes-process-lib.sh fm-operational-input.sh \ + fm-session-lock-lib.sh fm-omp-process-lib.sh fm-cursor-lib.sh fm-gemini-lib.sh fm-hermes-process-lib.sh fm-operational-input.sh \ fm-supervision-instructions.sh fm-harness.sh fm-lock.sh \ fm-gate-refuse-lib.sh fm-afk-contract.sh fm-classify-lib.sh fm-timeout-lib.sh \ fm-supervision-engine-lib.sh fm-lease-lib.sh; do diff --git a/tests/fm-backend-herdr-focus-flash-e2e.test.sh b/tests/fm-backend-herdr-focus-flash-e2e.test.sh index d8e2c64a6c2..99b270ba38f 100755 --- a/tests/fm-backend-herdr-focus-flash-e2e.test.sh +++ b/tests/fm-backend-herdr-focus-flash-e2e.test.sh @@ -268,6 +268,7 @@ done C_CALL_LOG="$TMP_ROOT/call-c.log" C_FOCUS_SAMPLES="$TMP_ROOT/focus-c.samples" +C_CLOSE_FOCUS="$TMP_ROOT/focus-c-close.snapshot" C_OPERATION_ACTIVE="$TMP_ROOT/operation-c.active" C_SAMPLER_READY="$TMP_ROOT/sampler-c.ready" SAMPLER_STOP="$TMP_ROOT/sampler-c.stop" @@ -296,14 +297,20 @@ done # A short proof budget keeps the exhausted-proof path fast; the count below is # what proves the proof was exhausted rather than skipped. C_PROOF_POLLS=3 +export -f lab focus_snapshot C_OUT=$(PATH="$FAKEBIN:$HERDR_ORIGINAL_PATH" FM_FLASH_CALL_LOG="$C_CALL_LOG" \ + FM_FLASH_CLOSE_FOCUS="$C_CLOSE_FOCUS" \ FM_BACKEND_HERDR_IDLE_SHELL_PROOF_POLLS="$C_PROOF_POLLS" bash -c ' . "$1/bin/backends/herdr.sh" fm_backend_herdr_cli() { local session=$1 shift printf "%s\n" "$*" >> "$FM_FLASH_CALL_LOG" - HERDR_SESSION="$session" herdr "$@" --session "$session" + HERDR_SESSION="$session" herdr "$@" --session "$session" || return $? + # Observe the real close before the restore can erase its focus change. + if [ "$1 $2" = "pane close" ]; then + focus_snapshot > "$FM_FLASH_CLOSE_FOCUS" || return 1 + fi } fm_backend_herdr_projection_close_pane_focus_preserving "$2" "$3" ' _ "$ROOT" "$HERDR_LAB_SESSION" "$C_DOOMED_PANE" 2>&1) @@ -333,6 +340,8 @@ pass 'fallback: a doomed pane holding a persistent child exhausts the proof and C_AFTER=$(focus_snapshot) || fail 'could not capture the Part C post-close focus' [ "$C_AFTER" = "$C_BEFORE" ] \ || fail "the fallback close left focus off the anchor ($C_BEFORE -> $C_AFTER)" +[ -s "$C_CLOSE_FOCUS" ] || fail 'Part C did not capture focus immediately after the explicit close' +C_AT_CLOSE=$(< "$C_CLOSE_FOCUS") C_WRONG=$(grep -Fvxc -- "$C_BEFORE" "$C_FOCUS_SAMPLES" || true) if [ "$STEAL_LIVE" = 1 ]; then # A defective release cannot make this path focus-safe, which is precisely why @@ -340,10 +349,12 @@ if [ "$STEAL_LIVE" = 1 ]; then # explicitly accepted here, but only as a BOUNDED one: the restore backstop # must have put the anchor back exactly, and the whole exposure must end with # the operation rather than parking the captain somewhere else. - [ "$C_WRONG" -ge 1 ] \ - || fail 'Part C reached the fallback on a defective release but observed no wrong-focus sample at all, so the sampler proved nothing' - pass "fallback on a defective release: a bounded wrong-focus window of $C_WRONG samples was fully restored to the anchor" + [ "$C_AT_CLOSE" != "$C_BEFORE" ] \ + || fail 'Part C reached the fallback on a defective release but the explicit close did not change focus' + pass 'fallback on a defective release: the explicit close changed focus and the backstop restored the exact anchor' else + [ "$C_AT_CLOSE" = "$C_BEFORE" ] \ + || fail "a focus-preserving release changed focus at the explicit close ($C_BEFORE -> $C_AT_CLOSE)" [ "$C_WRONG" -eq 0 ] \ || fail "a focus-preserving release exposed $C_WRONG wrong-focus samples on the fallback path" pass 'fallback on a focus-preserving release: the plain explicit close preserved exact focus throughout' diff --git a/tests/fm-backend-t3code.test.sh b/tests/fm-backend-t3code.test.sh index 03d1cef8634..45ad94912c7 100755 --- a/tests/fm-backend-t3code.test.sh +++ b/tests/fm-backend-t3code.test.sh @@ -909,42 +909,51 @@ test_native_restart_rearms_undelivered_stale_warning() { housekeeping "$1" ' "$state" || exit 1 [ -s "$state/.subsuper-escalations" ] || exit 1 - # The native daemon has stopped without delivering its buffer. + # The native daemon has stopped without delivering its buffer. Stop keeps + # the delivery buffer for the return/catch-up path but ends the session's + # per-task episode state: the stale clock and its report marker are gone. "$ROOT/bin/fm-afk-launch.sh" stop || exit 1 [ -s "$state/.subsuper-escalations" ] || exit 1 - [ -f "$state/.subsuper-reported-stale-worker" ] || exit 1 - # A warning already delivered for another condition must stay suppressed. + [ ! -e "$state/.subsuper-reported-stale-worker" ] || exit 1 + [ ! -e "$marker" ] || exit 1 + # A warning already delivered for another condition is the same + # session-scoped episode state; the next entry clears it too. printf 900 > "$state/.subsuper-reported-stale-delivered" "$ROOT/bin/fm-afk-launch.sh" enter --words 'supervise pending work' >/dev/null || exit 1 first=$(cat "$state/.subsuper-escalations") - # Failed startup must restore both the queue and its suppression markers. + # Failed startup must restore both the queue and the markers it captured. bash -c ' . "$0/bin/fm-afk-launch.sh" fm_afk_launch_record_write() { return 1; } ! fm_afk_launch_main start-native ' "$ROOT" || exit 1 [ "$(cat "$state/.subsuper-escalations")" = "$first" ] || exit 1 - [ -f "$state/.subsuper-reported-stale-worker" ] || exit 1 + [ -f "$state/.subsuper-reported-stale-delivered" ] || exit 1 "$ROOT/bin/fm-afk-launch.sh" start-native || exit 1 assert_absent "$state/.subsuper-escalations" "fresh startup must discard the old queue" assert_absent "$state/.subsuper-reported-stale-worker" "discarding an undelivered warning must re-arm reporting" - assert_present "$state/.subsuper-reported-stale-delivered" "startup must preserve suppression of delivered warnings" - [ "$(cat "$marker")" = 1 ] || fail "restart must preserve the stale condition's age" + assert_absent "$state/.subsuper-reported-stale-delivered" "startup must drop the prior session's suppression markers" + [ ! -e "$marker" ] || fail "restart must drop the prior session's stale clock" out=$(t3_run 'fm_backend_t3code_busy_state thread-live') [ "$out" = unknown ] || fail "the restarted thread must remain unknown" + # The still-unknown thread is re-derived on THIS session's fresh clock: a + # new stale wake opens a new episode at 1000, so the recheck reports its + # own 100s, never the prior session's 999s. t3_run ' . "$0/bin/fm-supervise-daemon.sh" _now() { printf 1000; } LOG="$1/daemon.log" + handle_wake "stale: thread-live" "$1" housekeeping "$1" _now() { printf 1100; } housekeeping "$1" housekeeping "$1" ' "$state" || exit 1 + [ "$(cat "$marker")" = 1000 ] || fail "the new session must restart the stale clock" [ "$(wc -l < "$state/.subsuper-escalations" | tr -d "[:space:]")" = 1 ] \ || fail "persistent uncertainty must queue exactly one replacement warning" assert_contains "$(cat "$state/.subsuper-escalations")" \ - 'stale persisted 999s (possible wedge): thread-live' "restart must report the retained stale condition" + 'stale persisted 100s (possible wedge): thread-live' "restart must re-derive the stale condition on a fresh clock" # Once that replacement is delivered, uncertainty must not report again. t3_run ' . "$0/bin/fm-supervise-daemon.sh" @@ -959,7 +968,7 @@ test_native_restart_rearms_undelivered_stale_warning() { [ "$(wc -l < "$state/delivered.log" | tr -d "[:space:]")" = 1 ] || exit 1 ) || fail "$session restart lost or duplicated the stale warning" done - pass "native restart re-arms discarded T3 stale warnings exactly once and preserves delivered suppression" + pass "native restart clears session episode state and re-derives T3 stale warnings once on a fresh clock" } test_housekeeping_preserves_unknown_stale_recheck() { diff --git a/tests/fm-bootstrap.test.sh b/tests/fm-bootstrap.test.sh index 64f17761dc6..d02fff17098 100755 --- a/tests/fm-bootstrap.test.sh +++ b/tests/fm-bootstrap.test.sh @@ -261,6 +261,183 @@ assert_timeout_report() { [ "$elapsed" -ge "$timeout" ] || fail "expected elapsed >= timeout, got elapsed=${elapsed}s timeout=${timeout}s" } +# seed_record_hygiene_home <case_dir>: build <case_dir>/home with every record +# inconsistency the hygiene report detects, plus a fake tasks-axi whose list +# calls are appended to <case_dir>/tasks-axi-list.log. Prints the fake bin dir. +seed_record_hygiene_home() { + local case_dir=$1 fakebin + mkdir -p "$case_dir/home/config" "$case_dir/home/state" "$case_dir/home/data" + printf '%s\n' manual > "$case_dir/home/config/backlog-backend" + fakebin=$(make_fake_toolchain "$case_dir") + cat > "$fakebin/tasks-axi" <<'SH' +#!/usr/bin/env bash +case "${1:-}" in + --version) echo 0.2.6 ;; + update) echo 'usage: tasks-axi update <id> --body-file <path> --archive-body' ;; + mv) echo 'usage: tasks-axi mv <id> [<id>...] --to <path-or-dir>' ;; + list) + printf '%s\n' "$*" >> "${0%/*}/../tasks-axi-list.log" + case " $* " in *' --fields id '*) echo 'unsupported extra field: id' >&2; exit 2 ;; esac + printf 'count: 3\ntasks[1]{id}:\n missing-record\ntasks[2]{id}:\n "missing,record",extra\ntasks[3]{id}:\n nested/task\n' + ;; +esac +SH + chmod +x "$fakebin/tasks-axi" + mkdir -p "$case_dir/home/state/nested" + printf '%s\n' 'kind=ship' > "$case_dir/home/state/nested/task.meta" + printf '%s\n' 'failed [at=1]: fixture orphan' > "$case_dir/home/state/orphan.status" + touch -t 202001010000 "$case_dir/home/state/orphan.status" + printf '%s\n' 'pr=https://example.test/pull/42' > "$case_dir/home/state/first.meta" + printf '%s\n' 'pr=https://example.test/pull/99' > "$case_dir/home/state/middle.meta" + printf '%s\n' 'pr=https://example.test/pull/42' > "$case_dir/home/state/second.meta" + printf 'kind=ship\nbackend=tmux\n' > "$case_dir/home/state/no-window.meta" + printf 'backend=tmux\nwindow=outside:window\npr=https://example.test/pull/42\n' > "$case_dir/outside.meta" + ln -s "$case_dir/outside.meta" "$case_dir/home/state/linked.meta" + printf '%s\n' "$fakebin" +} + +test_record_hygiene_archives_old_orphan_status() { + local case_dir fakebin out + case_dir="$TMP_ROOT/record-hygiene" + fakebin=$(seed_record_hygiene_home "$case_dir") + : > "$case_dir/home/config/record-hygiene" + out=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + FM_BOOTSTRAP_NETWORK=skip FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + assert_contains "$out" 'RECORD_HYGIENE: status log without metadata older than 24 hours: orphan' \ + "bootstrap did not report an old orphan status log" + assert_contains "$out" 'RECORD_HYGIENE: In-flight backlog row without metadata: missing-record' \ + "bootstrap did not report an In-flight row without a task record" + assert_contains "$out" 'RECORD_HYGIENE: In-flight backlog row without metadata: missing,record' \ + "bootstrap truncated a comma-containing in-flight ID" + assert_contains "$out" 'RECORD_HYGIENE: In-flight backlog row without metadata: nested/task' \ + "bootstrap matched a nested metadata path as a top-level record" + assert_contains "$out" 'RECORD_HYGIENE: metadata has no tab: no-window' \ + "bootstrap did not report metadata without a tab independently of backend state" + assert_contains "$out" 'RECORD_HYGIENE: duplicate PR https://example.test/pull/42 in records: first, second' \ + "bootstrap did not report duplicate PR references" + assert_not_contains "$out" 'linked' "bootstrap read symlinked metadata" + [ -f "$case_dir/home/data/orphan/status.log" ] || fail "bootstrap did not archive the orphan status log" + [ ! -e "$case_dir/home/state/orphan.status" ] || fail "archived orphan status log remained in state" + [ ! -e "$case_dir/home/state/.meta-orphan.lock" ] && [ ! -L "$case_dir/home/state/.meta-orphan.lock" ] \ + || fail "successful archive left its metadata lock behind" + assert_contains "$out" 'RECORD_HYGIENE: duplicate PR https://example.test/pull/42 in records: first, second' \ + "bootstrap lost duplicate PR detection across an intervening metadata record" + pass "bootstrap archives an old orphan status log and reports interleaved duplicates" +} + +test_record_hygiene_absent_flag_is_inert() { + local case_dir fakebin out_off out_on before after + case_dir="$TMP_ROOT/record-hygiene-off" + fakebin=$(seed_record_hygiene_home "$case_dir") + before=$(cd "$case_dir/home" && ls -laR state data) + out_off=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + FM_BOOTSTRAP_NETWORK=skip FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + after=$(cd "$case_dir/home" && ls -laR state data) + assert_not_contains "$out_off" 'RECORD_HYGIENE' "bootstrap reported record hygiene without config/record-hygiene" + [ "$before" = "$after" ] || fail "bootstrap changed state or data records without config/record-hygiene" + [ -f "$case_dir/home/state/orphan.status" ] || fail "bootstrap archived an orphan status log without config/record-hygiene" + ! grep -q in_flight "$case_dir/tasks-axi-list.log" 2>/dev/null \ + || fail "bootstrap read the In-flight backlog for record hygiene without config/record-hygiene" + + # Opting in on the same home changes only the RECORD_HYGIENE lines. + : > "$case_dir/home/config/record-hygiene" + out_on=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + FM_BOOTSTRAP_NETWORK=skip FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + assert_contains "$out_on" 'RECORD_HYGIENE: status log without metadata older than 24 hours: orphan' \ + "bootstrap did not report record hygiene once config/record-hygiene was created" + [ "$out_off" = "$(printf '%s\n' "$out_on" | grep -v '^RECORD_HYGIENE: ' || true)" ] \ + || fail "bootstrap output without config/record-hygiene differs beyond the RECORD_HYGIENE lines" + pass "bootstrap without config/record-hygiene reports nothing, moves nothing, and reads no backlog for it" +} + +test_record_hygiene_watcher_absent_flag_is_inert() { + local case_dir status + case_dir="$TMP_ROOT/record-hygiene-watcher-off" + mkdir -p "$case_dir/home"/{state,data,config} + printf 'kind=ship\nbackend=tmux\n' > "$case_dir/home/state/hygiene.meta" + status=0 + FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$ROOT" FM_LIVE=0 \ + FM_POLL=1 FM_HEARTBEAT=1 FM_HEARTBEAT_MAX=1 FM_CHECK_INTERVAL=999999 \ + FM_RECORD_HYGIENE_TIMEOUT=1 timeout 5 "$ROOT/bin/fm-watch.sh" >"$case_dir/out" 2>"$case_dir/err" || status=$? + { [ "$status" -eq 0 ] || [ "$status" -eq 124 ]; } \ + || fail "watcher failed without config/record-hygiene (exit=$status)" + sleep 2 + [ -e "$case_dir/home/state/.last-heartbeat" ] || fail "watcher never reached a heartbeat" + for marker in .record-hygiene-running .record-hygiene-result .record-hygiene-result.tmp .record-hygiene-last; do + [ ! -e "$case_dir/home/state/$marker" ] || fail "watcher heartbeat ran record hygiene without config/record-hygiene ($marker)" + done + ! grep -q $'\tcheck\trecord-hygiene\t' "$case_dir/home/state/.wake-queue" 2>/dev/null \ + || fail "watcher queued a record hygiene wake without config/record-hygiene" + pass "watcher heartbeat without config/record-hygiene runs no hygiene scan" +} + +test_record_hygiene_reclaims_stale_running_marker() { + local case_dir status tries + case_dir="$TMP_ROOT/record-hygiene-stale-marker" + mkdir -p "$case_dir/home"/{state,data,config} + : > "$case_dir/home/config/record-hygiene" + printf '99999999 nonexistent-process-start-time\n' > "$case_dir/home/state/.record-hygiene-running" + printf 'kind=ship\nbackend=tmux\n' > "$case_dir/home/state/hygiene.meta" + status=0 + FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$ROOT" FM_LIVE=0 \ + FM_POLL=1 FM_HEARTBEAT=1 FM_HEARTBEAT_MAX=1 FM_CHECK_INTERVAL=999999 \ + FM_RECORD_HYGIENE_TIMEOUT=1 timeout 30 "$ROOT/bin/fm-watch.sh" >"$case_dir/out" 2>"$case_dir/err" || status=$? + { [ "$status" -eq 0 ] || [ "$status" -eq 124 ]; } \ + || fail "watcher failed while reclaiming stale hygiene marker (exit=$status)" + tries=0 + while [ ! -s "$case_dir/home/state/.record-hygiene-last" ] \ + && ! grep -q $'\tcheck\trecord-hygiene\t' "$case_dir/home/state/.wake-queue" 2>/dev/null \ + && [ "$tries" -lt 30 ]; do + sleep 0.1 + tries=$((tries + 1)) + done + [ -s "$case_dir/home/state/.record-hygiene-last" ] \ + || grep -q $'\tcheck\trecord-hygiene\t' "$case_dir/home/state/.wake-queue" 2>/dev/null \ + || fail "replacement hygiene worker did not complete within 3s" + pass "watcher reclaims a stale hygiene marker and starts a worker" +} + +test_record_hygiene_watcher_data_is_defined() { + local case_dir status + case_dir="$TMP_ROOT/record-hygiene-watcher-data" + mkdir -p "$case_dir/home"/{state,data,config} + : > "$case_dir/home/config/record-hygiene" + status=0 + FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$ROOT" FM_LIVE=0 \ + FM_POLL=1 FM_HEARTBEAT=1 FM_HEARTBEAT_MAX=1 FM_CHECK_INTERVAL=999999 \ + timeout 5 "$ROOT/bin/fm-watch.sh" >"$case_dir/out" 2>"$case_dir/err" || status=$? + { [ "$status" -eq 0 ] || [ "$status" -eq 124 ]; } \ + || fail "watcher failed during hygiene check (exit=$status)" + assert_not_contains "$(cat "$case_dir/err")" 'DATA: unbound variable' \ + "watcher heartbeat called record hygiene without its data directory" + pass "watcher heartbeat supplies the home data directory to record hygiene" +} + +test_record_hygiene_bounds_backlog_read() { + local case_dir fakebin out + case_dir="$TMP_ROOT/record-hygiene-timeout" + mkdir -p "$case_dir/home/config" "$case_dir/home/state" "$case_dir/home/data" + : > "$case_dir/home/config/record-hygiene" + printf '%s\n' manual > "$case_dir/home/config/backlog-backend" + fakebin=$(make_fake_toolchain "$case_dir") + cat > "$fakebin/tasks-axi" <<'SH' +#!/usr/bin/env bash +case "${1:-}" in + --version) echo 0.2.6 ;; + update) echo 'usage: tasks-axi update <id> --body-file <path> --archive-body' ;; + mv) echo 'usage: tasks-axi mv <id> [<id>...] --to <path-or-dir>' ;; + list) sleep 5 ;; +esac +SH + chmod +x "$fakebin/tasks-axi" + out=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + FM_RECORD_HYGIENE_READ_TIMEOUT=1 FM_BOOTSTRAP_NETWORK=skip \ + FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + assert_contains "$out" 'RECORD_HYGIENE: In-flight backlog read timed out after 1s' \ + "bootstrap did not report a bounded backlog read timeout" + pass "bootstrap bounds and reports backlog read timeout" +} + # Each row (fields are '^'-separated; the install URL contains a literal '|'): # <label>^<lease 1/0>^<tasks-axi version or ->^<quota 1/0>^<backend or ->^<mode>^<expect>^<notcontains> # mode=empty -> output must be empty (expect/notcontains ignored) @@ -1539,7 +1716,212 @@ SH pass "bootstrap gates resolver fields and additive harnesses on the typed key or a declared typesafe source" } +test_treehouse_pool_report_is_read_only_and_classifies_pools() { + local case_dir home root fakebin registered unregistered other_home_clone missing_date fractional_repo leased_repo many_repo clone_one clone_two bare_repo nested_slot jq_count_file real_jq jq_calls i out + case_dir="$TMP_ROOT/pool-report" + home="$case_dir/home" + root="$case_dir/pools" + fakebin="$case_dir/bin" + registered="$home/projects/registered" + unregistered="$case_dir/unregistered" + missing_date="$case_dir/missing-date" + other_home_clone="$case_dir/other-home/projects/repo" + fractional_repo="$home/projects/fractional" + leased_repo="$home/projects/leased" + many_repo="$home/projects/many" + clone_one="$home/projects/clone-one" + clone_two="$home/projects/clone-two" + bare_repo="$case_dir/origin.git" + nested_slot="$home/projects/parent/nested-slot" + mkdir -p "$home/projects" "$root/one" "$root/two" "$root/nested" "$root/orphan" "$root/other-home" "$root/project-leftover" "$root/empty-path" "$root/timestamp" "$root/leased" "$root/many" "$other_home_clone" "$missing_date" "$fractional_repo" "$leased_repo" "$many_repo" "$home/projects/parent" "$nested_slot" + fakebin=$(make_fake_toolchain "$case_dir") + git init -q "$registered" + git init -q "$other_home_clone" + git init -q "$fractional_repo" + git init -q "$leased_repo" + git init -q "$many_repo" + git init -q "$home/projects/parent" + git -C "$registered" config user.email pool-test@example.invalid + git -C "$registered" config user.name pool-test + printf 'fixture\n' > "$registered/README" + git -C "$registered" add README + git -C "$registered" commit -qm fixture + git init -q --bare "$bare_repo" + git -C "$registered" remote add origin "$bare_repo" + git -C "$registered" push -q -u origin HEAD + git clone -q "$bare_repo" "$clone_one" + git clone -q "$bare_repo" "$clone_two" + git -C "$clone_one" remote set-url origin 'https://pool-user:pool-password@example.invalid/repo.git' + git -C "$clone_two" remote set-url origin 'https://pool-user:pool-password@example.invalid/repo.git' + real_jq=$(command -v jq) + jq_count_file="$case_dir/jq-count" + rm -f "$fakebin/treehouse" + cat > "$fakebin/jq" <<'SH' +#!/usr/bin/env bash +printf '.\n' >> "$FM_TEST_JQ_COUNT" +exec "$FM_TEST_REAL_JQ" "$@" +SH + chmod +x "$fakebin/jq" + { + printf '{"worktrees":[' + for ((i = 0; i < 64; i++)); do + [ "$i" -eq 0 ] || printf ',' + printf '{"path":"%s"}' "$many_repo" + done + printf ']}\n' + } > "$root/many/treehouse-state.json" + cat > "$root/one/treehouse-state.json" <<JSON +{"worktrees":[{"path":"$clone_one","created_at":"2020-01-01T00:00:00Z"},{"path":"$clone_one","created_at":"2020-01-01T00:00:00Z"},{"path":"$missing_date"}]} +JSON + cat > "$root/two/treehouse-state.json" <<JSON +{"worktrees":[{"path":"$clone_two","created_at":"2020-01-01T00:00:00Z"}]} +JSON + cat > "$root/nested/treehouse-state.json" <<JSON +{"worktrees":[{"path":"$nested_slot","created_at":"2020-01-01T00:00:00Z"}]} +JSON + cat > "$root/orphan/treehouse-state.json" <<JSON +{"worktrees":[{"path":"$unregistered","created_at":"2020-01-01T00:00:00Z"}]} +JSON + cat > "$root/other-home/treehouse-state.json" <<JSON +{"worktrees":[{"path":"$other_home_clone","owner_started_at":"2020-01-01T00:00:00Z"}]} +JSON + cat > "$root/project-leftover/treehouse-state.json" <<'JSON' +{"worktrees":[]} +JSON + cat > "$root/empty-path/treehouse-state.json" <<'JSON' +{"worktrees":[{"created_at":"2020-01-01T00:00:00Z"}]} +JSON + cat > "$root/timestamp/treehouse-state.json" <<JSON +{"worktrees":[{"path":"$fractional_repo","created_at":"2026-07-30T23:29:43.458891-06:00"}]} +JSON + cat > "$root/leased/treehouse-state.json" <<JSON +{"worktrees":[{"path":"$leased_repo","created_at":"2020-01-01T00:00:00Z","leased":true,"lease_id":"lease-1","lease_holder":"secondmate-home","leased_at":"2026-10-01T00:00:00Z"}]} +JSON + mkdir -p "$home/config" + : > "$home/config/treehouse-pool-report" + before=$(find "$root" "$registered" "$bare_repo" "$clone_one" "$clone_two" "$other_home_clone" "$fractional_repo" "$leased_repo" "$many_repo" "$home/projects/parent" -type f -exec shasum -a 256 {} \; | sort) + : > "$jq_count_file" + out=$(PATH="$fakebin:$(fm_test_base_path_sans "$BASE_PATH" treehouse)" FM_TEST_JQ_COUNT="$jq_count_file" FM_TEST_REAL_JQ="$real_jq" TREEHOUSE_ROOT="$root" FM_POOL_ORPHAN_DAYS=30 \ + FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" FM_BOOTSTRAP_DETECT_ONLY=1 FM_BOOTSTRAP_NETWORK=skip \ + FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + jq_calls=$(wc -l < "$jq_count_file") + after=$(find "$root" "$registered" "$bare_repo" "$clone_one" "$clone_two" "$other_home_clone" "$fractional_repo" "$leased_repo" "$many_repo" "$home/projects/parent" -type f -exec shasum -a 256 {} \; | sort) + [ "$before" = "$after" ] || fail "pool report modified pool or repository files" + assert_contains "$out" "unregistered pool $root/orphan" "pool with no existing project repository was not reported" + assert_not_contains "$out" "unregistered pool $root/other-home" "another home's existing project clone was falsely reported orphaned" + assert_contains "$out" "unregistered pool $root/nested" "a slot inside a parent repository was mistaken for its own checkout" + assert_contains "$out" "duplicate-clone pool $root/two (project $clone_two)" "separate clones with one origin were not reported with a local project path" + assert_not_contains "$out" "pool-user" "duplicate-pool report exposed the origin username" + assert_not_contains "$out" "pool-password" "duplicate-pool report exposed the origin password" + assert_not_contains "$out" "duplicate-clone pool $root/many" "sibling slots in one pool were reported as duplicate pools" + [ "$jq_calls" -le $((64 * 3)) ] || fail "pool scan used $jq_calls jq invocations for 64 slots; expected a linear bound" + assert_not_contains "$out" "duplicate-clone pool $root/one" "sibling slots in one pool were reported as duplicate pools" + assert_contains "$out" "test-leftover pool $root/project-leftover" "test leftover pool was not reported" + assert_not_contains "$out" "unregistered pool $root/project-leftover" "empty worktrees array created a synthetic slot" + assert_not_contains "$out" "project path unavailable: )" "missing path was treated as a project" + assert_not_contains "$out" "duplicate-clone pool $root/empty-path" "missing path was resolved from bootstrap's working directory" + assert_contains "$out" "unowned slot older than 30d: $clone_one" "old created_at Z timestamp was not reported" + assert_contains "$out" "unowned slot older than 30d: $fractional_repo" "fractional timestamp with offset was not reported" + assert_not_contains "$out" "unowned slot older than 30d: $missing_date" "missing created_at slot was reported stale" + assert_not_contains "$out" "unowned slot older than 30d: $leased_repo" "durably leased slot was reported stale" + pass "bootstrap reports pool orphans, duplicate clones, old unowned slots, and test pools" +} + +test_treehouse_pool_report_needs_its_flag() { + local case_dir home root fakebin real_jq jq_log out baseline + case_dir="$TMP_ROOT/pool-report-flag-absent" + home="$case_dir/home" + root="$case_dir/pools" + mkdir -p "$home/config" "$root/orphan" "$root/project-leftover" + printf '%s\n' manual > "$home/config/backlog-backend" + fakebin=$(make_fake_toolchain "$case_dir") + real_jq=$(command -v jq) + jq_log="$case_dir/jq-args" + cat > "$fakebin/jq" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "$*" >> "$FM_TEST_JQ_LOG" +exec "$FM_TEST_REAL_JQ" "$@" +SH + chmod +x "$fakebin/jq" + cat > "$root/orphan/treehouse-state.json" <<JSON +{"worktrees":[{"path":"$case_dir/missing","created_at":"2020-01-01T00:00:00Z"}]} +JSON + printf '%s\n' '{"worktrees":[]}' > "$root/project-leftover/treehouse-state.json" + : > "$jq_log" + out=$(PATH="$fakebin:$BASE_PATH" FM_TEST_JQ_LOG="$jq_log" FM_TEST_REAL_JQ="$real_jq" TREEHOUSE_ROOT="$root" \ + FM_HOME="$home" FM_ROOT_OVERRIDE="$home" FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + ! grep -qF "$root" "$jq_log" || fail "bootstrap without config/treehouse-pool-report read Treehouse state: $(cat "$jq_log")" + baseline=$(PATH="$fakebin:$BASE_PATH" FM_TEST_JQ_LOG="$jq_log" FM_TEST_REAL_JQ="$real_jq" TREEHOUSE_ROOT="$case_dir/no-pools" \ + FM_HOME="$home" FM_ROOT_OVERRIDE="$home" FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + assert_not_contains "$out" "TREEHOUSE_POOL_REPORT" "bootstrap printed a pool report without its flag" + assert_equals "$baseline" "$out" "bootstrap output with Treehouse pools differs from a home with none while the flag is absent" + : > "$home/config/treehouse-pool-report" + out=$(PATH="$fakebin:$BASE_PATH" FM_TEST_JQ_LOG="$jq_log" FM_TEST_REAL_JQ="$real_jq" TREEHOUSE_ROOT="$root" \ + FM_HOME="$home" FM_ROOT_OVERRIDE="$home" FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + assert_contains "$out" "TREEHOUSE_POOL_REPORT: test-leftover pool $root/project-leftover" "the same fixture did not report once the flag exists" + pass "bootstrap reads and reports Treehouse pools only when config/treehouse-pool-report exists" +} + +test_record_hygiene_archives_old_orphan_status +test_record_hygiene_old_orphan_skips_held_metadata_lock() { + local case_dir fakebin lock out + case_dir="$TMP_ROOT/record-hygiene-lock" + fakebin=$(seed_record_hygiene_home "$case_dir") + : > "$case_dir/home/config/record-hygiene" + lock=$(FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$ROOT" bash -c '. "$1/bin/fm-wake-lib.sh"; fm_meta_lock_path "$2"' _ "$ROOT" "$case_dir/home/state/orphan.meta") + mkdir -p "$lock" + printf '%s\n' "$$" > "$lock/pid" + out=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + FM_BOOTSTRAP_NETWORK=skip FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + assert_contains "$out" 'RECORD_HYGIENE: status log without metadata older than 24 hours: orphan' \ + "bootstrap did not scan the orphan status log while its metadata lock was held" + [ -f "$case_dir/home/state/orphan.status" ] || fail "bootstrap moved orphan status while metadata lock was held" + [ ! -e "$case_dir/home/data/orphan" ] || fail "bootstrap archived orphan status while metadata lock was held" + [ -d "$lock" ] || fail "bootstrap removed another holder's metadata lock" + pass "bootstrap skips archiving while the task metadata lock is held" +} +test_record_hygiene_old_orphan_survives_spawn_publication() { + local case_dir fakebin out + case_dir="$TMP_ROOT/record-hygiene-spawn-race" + fakebin=$(seed_record_hygiene_home "$case_dir") + # The mtime read follows the scan's first metadata-absence check, so a + # metadata write there is a spawn publishing between that check and the move. + out=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_TEST_LIB_SOURCED=1 bash -c ' + SCRIPT_DIR="$1/bin"; STATE="$2/state"; DATA="$2/data" + # shellcheck source=bin/fm-classify-lib.sh + . "$SCRIPT_DIR/fm-classify-lib.sh" + # shellcheck source=bin/fm-tasks-axi-lib.sh + . "$SCRIPT_DIR/fm-tasks-axi-lib.sh" + # shellcheck source=bin/fm-backlog-transition-lib.sh + . "$SCRIPT_DIR/fm-backlog-transition-lib.sh" + # shellcheck source=bin/fm-backend.sh + . "$SCRIPT_DIR/fm-backend.sh" + # shellcheck source=bin/fm-timeout-lib.sh + . "$SCRIPT_DIR/fm-timeout-lib.sh" + # shellcheck source=bin/fm-record-hygiene-lib.sh + . "$SCRIPT_DIR/fm-record-hygiene-lib.sh" + _fm_status_file_mtime() { + [ "${1##*/}" != orphan.status ] || [ -e "$STATE/orphan.meta" ] \ + || printf "kind=ship\nwindow=fm:orphan\n" > "$STATE/orphan.meta" + echo 1577836800 + } + record_hygiene_report + ' _ "$ROOT" "$case_dir/home" 2>&1) || fail "record hygiene pass failed: $out" + [ -f "$case_dir/home/state/orphan.meta" ] || fail "fixture did not publish metadata during the scan" + [ -f "$case_dir/home/state/orphan.status" ] || fail "record hygiene archived a status log whose metadata a spawn just published" + [ ! -e "$case_dir/home/data/orphan" ] || fail "record hygiene created an archive for a just-published worker" + pass "record hygiene keeps a status log whose metadata is published before the move" +} +test_record_hygiene_old_orphan_skips_held_metadata_lock +test_record_hygiene_old_orphan_survives_spawn_publication +test_record_hygiene_watcher_data_is_defined +test_record_hygiene_reclaims_stale_running_marker +test_record_hygiene_bounds_backlog_read +test_record_hygiene_absent_flag_is_inert +test_record_hygiene_watcher_absent_flag_is_inert test_bootstrap_reporting +test_treehouse_pool_report_is_read_only_and_classifies_pools +test_treehouse_pool_report_needs_its_flag test_absent_tool_cases_hide_a_real_install_in_the_base_path test_no_mistakes_min_version test_gh_axi_min_version diff --git a/tests/fm-calm-pi-extension.test.sh b/tests/fm-calm-pi-extension.test.sh index 080ab2435af..eec9488aad2 100755 --- a/tests/fm-calm-pi-extension.test.sh +++ b/tests/fm-calm-pi-extension.test.sh @@ -587,6 +587,7 @@ function makeHost(session) { Object.assign(host, { // Pi reads its session through runtimeHost, which a session replacement swaps. runtimeHost: { session }, + pendingBashComponents: [], compactionQueuedMessages: [], statuses: [], warnings: [], diff --git a/tests/fm-claude-stop-autoarm-live-e2e.test.sh b/tests/fm-claude-stop-autoarm-live-e2e.test.sh index 0cfaf2f563f..56b9d07b35f 100755 --- a/tests/fm-claude-stop-autoarm-live-e2e.test.sh +++ b/tests/fm-claude-stop-autoarm-live-e2e.test.sh @@ -159,6 +159,13 @@ fi || fail "cooperative guard consumed a forced continuation while the auto-arm launch was healthy" [ "$(sed -n 's/^.*outcome=\([a-z][a-z]*\) .*$/\1/p' "$HOME_DIR/state/.claude-autoarm-epoch" 2>/dev/null)" = rewake ] \ || fail "auto-arm epoch ledger must record the rewake outcome" +# The claim is bound to the Claude session its Stop payload names, the session +# that receives the hook's rewake (docs/turnend-guard.md "Auto-arm generation +# claim"), so the real payload must carry this session's own id. +SESSION_ID=$(jq -r 'select(.type == "system" and .subtype == "init") | .session_id' "$TRANSCRIPT" 2>/dev/null | head -n 1) +[ -n "$SESSION_ID" ] || fail "the stream-json transcript names no session id" +grep -q "^epoch=.* claude_session=$SESSION_ID\$" "$HOME_DIR/state/.claude-autoarm-epoch" \ + || fail "auto-arm epoch ledger must record the Stop payload's session $SESSION_ID: $(sed -n '1p' "$HOME_DIR/state/.claude-autoarm-epoch")" [ ! -e "$HOME_DIR/state/.claude-autoarm.lock" ] || fail "auto-arm owner lock was left behind" # Live-owner negative control: a separate supported-harness process owns a diff --git a/tests/fm-claude-stop-autoarm.test.sh b/tests/fm-claude-stop-autoarm.test.sh index 0d7a7bf0c94..d7457e79327 100755 --- a/tests/fm-claude-stop-autoarm.test.sh +++ b/tests/fm-claude-stop-autoarm.test.sh @@ -32,6 +32,7 @@ install_autoarm_scripts() { cp "$ROOT/bin/fm-wake-lib.sh" "$dir/bin/fm-wake-lib.sh" cp "$ROOT/bin/fm-path-lib.sh" "$dir/bin/fm-path-lib.sh" cp "$ROOT/bin/fm-session-lock-lib.sh" "$dir/bin/fm-session-lock-lib.sh" + cp "$ROOT/bin/fm-omp-process-lib.sh" "$dir/bin/fm-omp-process-lib.sh" cp "$ROOT/bin/fm-cursor-lib.sh" "$dir/bin/fm-cursor-lib.sh" cp "$ROOT/bin/fm-hook-host-lib.sh" "$dir/bin/fm-hook-host-lib.sh" cp "$ROOT/bin/fm-lock.sh" "$dir/bin/fm-lock.sh" @@ -1966,6 +1967,59 @@ test_superseded_owner_goes_silent_and_never_double_translates() { pass "auto-arm: a superseded owner goes silent - one supersession episode, one translation, no held mutex" } +# Claude delivers a hook's exit-2 rewake to the session that launched it. When +# Claude continues the conversation under a new session id (a backgrounded +# session resumes with --fork-session), the conversation keeps running under +# the same front-end process that holds the session lock, so an owner claimed +# by the earlier session stays alive, open, and reachable by descent. The +# continued session's firings must not defer to it, because its close could +# only wake the abandoned session. A firing from the claim's own session still +# defers, and the earlier owner goes silent. +test_open_claim_from_an_earlier_session_is_superseded() { + local dir a_out b_out b_status c_out c_status a_status count + dir=$(make_primary_dir "$TMP_ROOT/earlier-session-claim") + : > "$dir/state/task1.meta" + write_arm_fixture "$dir" blocking-actionable + a_out="$dir/state/a.out" + # One fake harness holds the lock for the whole scenario, as the front-end + # does across the fork; each firing names its Claude session in the payload. + FM_HOME="$dir" "$FAKE_CLAUDE" -c ' + printf "%s\n" "$$" > "$FM_HOME/state/.lock" + fire() { printf "%s\n" "{\"session_id\":\"$2\",\"stop_hook_active\":false}" | "$FM_HOME/bin/fm-claude-stop-autoarm.sh" > "$FM_HOME/state/$1.out" 2>&1; echo $? > "$FM_HOME/state/$1.rc"; } + fire a sess-before-fork & + a_pid=$! + i=0 + while ! grep -q "outcome=arming" "$FM_HOME/state/.claude-autoarm-epoch" 2>/dev/null || [ ! -e "$FM_HOME/state/arm-ran" ]; do + [ "$i" -lt 50 ] || { echo "owner A never published its arming claim" > "$FM_HOME/state/scenario-fail"; exit 1; } + sleep 0.1; i=$((i + 1)) + done + fire b sess-before-fork + cp "$FM_HOME/state/arm-ran" "$FM_HOME/state/arm-ran.after-b" + kill -0 "$a_pid" 2>/dev/null || { echo "owner A finished before the session change could be exercised" > "$FM_HOME/state/scenario-fail"; exit 1; } + fire c sess-after-fork + wait "$a_pid" + ' + [ ! -e "$dir/state/scenario-fail" ] || fail "$(cat "$dir/state/scenario-fail")" + b_status=$(cat "$dir/state/b.rc"); b_out=$(cat "$dir/state/b.out") + c_status=$(cat "$dir/state/c.rc"); c_out=$(cat "$dir/state/c.out") + a_status=$(cat "$dir/state/a.rc") + expect_code 0 "$b_status" "a firing from the claim's own session must still defer to its open claim" + [ -z "$b_out" ] || fail "same-session deferring firing produced output: $b_out" + count=$(wc -l < "$dir/state/arm-ran.after-b" | tr -d ' ') + [ "$count" -eq 1 ] || fail "same-session deferring firing must not arm, saw $count arms" + expect_code 2 "$c_status" "a firing from the continued session must take over and translate its own close" + assert_contains "$c_out" "firstmate watcher wake" "the continued session's generation must carry the rewake banner" + expect_code 0 "$a_status" "the earlier session's owner must exit 0 instead of translating into the abandoned session" + [ ! -s "$a_out" ] || fail "the earlier session's owner emitted output after losing its generation: $(cat "$a_out")" + [ "$(epoch_field "$dir" epoch)" = 2 ] || fail "expected the continued session's generation on the ledger: $(epoch_field "$dir" epoch)" + [ "$(epoch_outcome "$dir")" = rewake ] || fail "the continued session's outcome was overwritten: $(epoch_outcome "$dir")" + [ "$(epoch_field "$dir" claude_session)" = sess-after-fork ] \ + || fail "the rewake entry must name the continued session it wakes: $(sed -n '1p' "$dir/state/.claude-autoarm-epoch")" + count=$(wc -l < "$dir/state/arm-ran" | tr -d ' ') + [ "$count" -eq 2 ] || fail "expected exactly the earlier owner's and the continued session's arms, saw $count" + pass "auto-arm: a claim from an earlier Claude session no longer defers the continued session's firings" +} + test_need_vanished_mid_cycle_closes_quietly() { local dir out status dir=$(make_primary_dir "$TMP_ROOT/vanished") @@ -2387,6 +2441,7 @@ test_recycled_bridge_claim_stays_open_on_session_id_match test_identityless_ledger_never_defers test_superseded_owner_never_reinvokes_the_arm test_superseded_owner_goes_silent_and_never_double_translates +test_open_claim_from_an_earlier_session_is_superseded test_need_vanished_mid_cycle_closes_quietly test_afk_mid_cycle_suppresses_rewake test_active_in_marked_secondmate_home diff --git a/tests/fm-claude-tool-exclude-live.test.sh b/tests/fm-claude-tool-exclude-live.test.sh new file mode 100755 index 00000000000..12e23726d0a --- /dev/null +++ b/tests/fm-claude-tool-exclude-live.test.sh @@ -0,0 +1,100 @@ +#!/usr/bin/env bash +# Token-free installed Claude Code guard for config/crew-exclude-tools. +# +# The hermetic suites pin the launch flag fm-spawn emits; only the real binary can +# prove that flag removes a tool's schema from the model request, which is the whole +# point of the exclusion. This guard launches the installed `claude` twice against a +# capture-only local server (tests/fixtures/fm-claude-request-capture.py) that records +# the request body and answers 529, so no model is called and no tokens are spent. +# The arms differ only in --disallowedTools, and the request's tool list must lose +# exactly the excluded tool's schema. +# +# The launch gets a clean environment, a private config directory and a synthetic API +# key, so it needs no Claude Code login and cannot reach any endpoint but the capture +# server, whatever provider variables the host sets. Enabled by default wherever claude +# and python3 are installed; FM_CLAUDE_TOOL_EXCLUDE_LIVE=1 makes an absent tool or a claude +# that sends no request a failure, and =0 skips. +set -eu + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +fm_live_gate default-on FM_CLAUDE_TOOL_EXCLUDE_LIVE claude python3 + +TMP_ROOT=$(fm_test_tmproot fm-claude-tool-exclude) +CLAUDE_VERSION=$(claude --version 2>/dev/null | head -n 1) +CAPTURE_PID='' +stop_capture() { + [ -z "$CAPTURE_PID" ] || kill "$CAPTURE_PID" 2>/dev/null || true + CAPTURE_PID='' +} +# lib.sh's own EXIT trap is fm_test_cleanup; this one replaces it, so it must run it too. +trap 'stop_capture; fm_test_cleanup' EXIT + +# unusable <reason>: skip when the host cannot run the comparison, fail when it was requested. +unusable() { + if [ "${FM_CLAUDE_TOOL_EXCLUDE_LIVE:-}" = 1 ] || [ "${FM_LIVE:-}" = 1 ]; then + fail "$1" + fi + printf 'skip: live: %s\n' "$1" + exit 0 +} + +# capture_tools <label> [claude args...]: write the main request's tool names, one per line, +# to $TMP_ROOT/<label>.tools. Runs in the main shell so unusable can end the script. +capture_tools() { + local label=$1 port_file capture port + shift + port_file=$TMP_ROOT/$label.port + capture=$TMP_ROOT/$label.jsonl + python3 "$ROOT/tests/fixtures/fm-claude-request-capture.py" "$port_file" "$capture" & + CAPTURE_PID=$! + for _ in $(seq 1 50); do + [ -s "$port_file" ] && break + sleep 0.1 + done + [ -s "$port_file" ] || fail "the capture server did not start for $label" + port=$(cat "$port_file") + # A clean environment, a private config directory and a synthetic key leave the + # capture server as the only endpoint this launch can reach: no inherited + # Bedrock, Vertex, proxy or login setting can send "say hi" to a real model. + ( + cd "$TMP_ROOT" || exit 1 + fm_run_timed 40 env -i PATH="$PATH" HOME="$TMP_ROOT/home" TERM=dumb \ + CLAUDE_CONFIG_DIR="$TMP_ROOT/claude-config" \ + ANTHROPIC_API_KEY=sk-ant-synthetic-capture-only \ + ANTHROPIC_BASE_URL="http://127.0.0.1:$port" \ + DISABLE_AUTOUPDATER=1 CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 \ + claude -p "say hi" --max-turns 1 "$@" >/dev/null 2>&1 || true + ) + stop_capture + [ -s "$capture" ] || unusable "claude $CLAUDE_VERSION sent no model request for $label" + python3 -I - "$capture" >"$TMP_ROOT/$label.tools" <<'PY' +import json, sys +best = [] +for line in open(sys.argv[1]): + try: + names = [t["name"] for t in json.loads(line).get("tools", [])] + except ValueError: + continue + if len(names) > len(best): + best = names +print("\n".join(best)) +PY +} + +# shellcheck source=bin/fm-timeout-lib.sh +. "$ROOT/bin/fm-timeout-lib.sh" + +capture_tools baseline +capture_tools trimmed --disallowedTools Workflow +baseline=$(cat "$TMP_ROOT/baseline.tools") +trimmed=$(cat "$TMP_ROOT/trimmed.tools") + +printf '%s\n' "$baseline" | grep -qx Workflow \ + || unusable "claude $CLAUDE_VERSION baseline request does not carry Workflow, so the comparison proves nothing" +printf '%s\n' "$trimmed" | grep -qx Workflow \ + && fail "claude $CLAUDE_VERSION still sends the Workflow schema under --disallowedTools" +[ "$(printf '%s\n' "$baseline" | grep -vx Workflow | sort | tr '\n' ' ')" = "$(printf '%s\n' "$trimmed" | sort | tr '\n' ' ')" ] \ + || fail "claude $CLAUDE_VERSION changed more than the excluded tool's schema" +pass "Claude Code $CLAUDE_VERSION: --disallowedTools removes exactly the excluded tool's schema from the model request" diff --git a/tests/fm-codex-hook-layer-live-e2e.test.sh b/tests/fm-codex-hook-layer-live-e2e.test.sh index fae10df0ecb..2910a677436 100755 --- a/tests/fm-codex-hook-layer-live-e2e.test.sh +++ b/tests/fm-codex-hook-layer-live-e2e.test.sh @@ -144,7 +144,7 @@ for name in ['bin', 'state', 'docs', '.codex']: for name in ['fm-turnend-guard.sh', 'fm-operational-input.sh', 'fm-supervision-instructions.sh', 'fm-harness.sh', 'fm-primary-scope-lib.sh', 'fm-supervision-lib.sh', 'fm-supervision-engine-lib.sh', 'fm-wake-lib.sh', 'fm-path-lib.sh', - 'fm-hook-host-lib.sh', 'fm-session-lock-lib.sh', 'fm-cursor-lib.sh', + 'fm-hook-host-lib.sh', 'fm-session-lock-lib.sh', 'fm-omp-process-lib.sh', 'fm-cursor-lib.sh', 'fm-hermes-process-lib.sh', 'fm-timeout-lib.sh']: shutil.copy2(root / 'bin' / name, primary / 'bin' / name) shutil.copytree(root / 'docs/supervision-protocols', primary / 'docs/supervision-protocols') diff --git a/tests/fm-composer-lib.test.sh b/tests/fm-composer-lib.test.sh index 9b2c19be34f..00589625423 100755 --- a/tests/fm-composer-lib.test.sh +++ b/tests/fm-composer-lib.test.sh @@ -1303,6 +1303,33 @@ test_matrix_grok_144_approval_titled_bottom_border() { pass "matrix: grok 1.0.44's approval-titled equal-width bottom is empty while typed and unknown titles stay safe" } +test_matrix_grok_1077_fast_model_title() { + # Grok 1.0.77 (50ca579fa996) launched with --always-approve --model + # grok-4.7-build-fast, captured live on 2026-11-12 through the Rex adapter's + # styled viewport read (24 rows, cursor row 19) after one turn and a Ctrl+C + # interrupt. Its bottom title `Grok 4.7 Fast (medium) · always-approve` + # carries a two-word model name, which read unknown, so exit and relaunch + # refused the idle pane (#8388). + local fixtures idle typed top empty_row bottom + fixtures="$ROOT/tests/fixtures/rex-grok-1.0.77" + idle=$(<"$fixtures/grok-1.0.77-fast-idle-after-interrupt.vt") + typed=$(<"$fixtures/grok-1.0.77-fast-typed.vt") + assert_screen "grok 1.0.77 fast idle with cursor" empty "$CAPS_TMUX" "$idle" 19 + assert_screen "grok 1.0.77 fast idle cursorless" empty "$CAPS_STYLED" "$idle" + assert_screen "grok 1.0.77 fast idle plain" empty "$CAPS_PLAIN" "$(printf '%s' "$idle" | fm_composer_strip_ansi)" + assert_screen "grok 1.0.77 fast typed with cursor" pending "$CAPS_TMUX" "$typed" 19 + assert_screen "grok 1.0.77 fast typed cursorless" pending "$CAPS_STYLED" "$typed" + top=' ╭'$(_grok_rule 74)'╮' + empty_row=' │ ❯'"$(printf '%*s' 72 '')"'│' + bottom=$(_grok_titled_bottom 74 'Grok 4.7 Fast Mini (medium) · always-approve') + assert_screen "grok second model word after Fast" unknown "$CAPS_TMUX" "$top"$'\n'"$empty_row"$'\n'"$bottom" 1 + bottom=$(_grok_titled_bottom 74 'Grok 4.7 Turbo (medium) · always-approve') + assert_screen "grok model word other than Fast" unknown "$CAPS_TMUX" "$top"$'\n'"$empty_row"$'\n'"$bottom" 1 + bottom=$(_grok_titled_bottom 77 'Grok 4.7 Fast (medium)') + assert_screen "grok Fast title with 1.0.5 overhang" unknown "$CAPS_TMUX" "$top"$'\n'"$empty_row"$'\n'"$bottom" 1 + pass "matrix: grok 1.0.77's Fast-model approval title is empty while typed and unobserved titles stay safe" +} + # <count> box-drawing rule glyphs, one glyph per iteration so the emitted width # never depends on whether ${#} counts bytes or characters in the ambient locale. _grok_rule() { # <count> @@ -1705,6 +1732,7 @@ test_matrix_opencode_leftbar_abs_cwd_furniture test_matrix_grok_titled_bottom_border test_matrix_claude_titled_top_rule test_matrix_grok_144_approval_titled_bottom_border +test_matrix_grok_1077_fast_model_title test_matrix_kimi_bordered_shell_glyph_box test_matrix_claude_inside_zellij_ansi_dump test_strict_blank_row_divergence diff --git a/tests/fm-contributions.test.sh b/tests/fm-contributions.test.sh index 6ed86567b24..9f3c72ba72e 100755 --- a/tests/fm-contributions.test.sh +++ b/tests/fm-contributions.test.sh @@ -2082,8 +2082,300 @@ test_ignored_commenters_are_not_tracked() { pass 'commenters in config/contributions-ignore-commenters are never raised as unanswered' } +test_forge_settles_only_declared_holds() { + local home mode action id state body out + for mode in merged closed; do + home=$(new_home "hold-terminal-$mode") + forge_home "$home" + rm "$home/state/delivery.meta" + touch "$home/config/hold-pr-settle" + for action in close release; do + id="question-$action" + with_home "$home" "$ROOT/bin/fm-tasks-axi.sh" add "$id" 'Decide this PR' --pr https://github.com/o/r/pull/8 >/dev/null || fail 'add failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold "$id" --reason 'Decide this PR' >/dev/null || fail 'hold failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" pr-policy "$id" https://github.com/o/r/pull/8 terminal "$action" >/dev/null || fail 'policy failed' + done + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold delivery --reason 'Unrelated unfinished work' >/dev/null || fail 'unrelated hold failed' + with_home "$home" "$ROOT/bin/fm-tasks-axi.sh" add bystander 'Work without a captain hold' --pr https://github.com/o/r/pull/8 >/dev/null || fail 'bystander add failed' + printf '%s\n' "$mode" > "$home/forge/state" + out=$(with_home "$home" "$ROOT/bin/fm-contributions.sh" poll) || fail 'terminal poll failed' + # Settlement must not cost a model turn: the watcher wakes on any poll line. + [ "$mode" = closed ] || [ -z "$out" ] || fail "a successful settlement produced poll output: $out" + for action in close release; do + id="question-$action" + if with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open "$id"; then fail "$mode left $id held"; fi + body=$(with_home "$home" "$ROOT/bin/fm-tasks-axi.sh" show "$id" --full) + printf '%s\n' "$body" | grep -F 'Reconciliation evidence:' >/dev/null || fail 'forge evidence missing' + printf '%s\n' "$body" | grep -F "Forge PR $mode: https://github.com/o/r/pull/8" >/dev/null || fail 'terminal event missing' + printf '%s\n' "$body" | grep -F 'Captain decision:' >/dev/null && fail 'forge event claimed a captain answer' + state=$(printf '%s\n' "$body" | sed -n 's/^ state: //p') + if [ "$action" = close ]; then [ "$state" = "done" ] || fail 'decision-only row stayed unfinished' + else [ "$state" != "done" ] || fail 'forge event completed unfinished work'; fi + done + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery || fail 'unrelated question was settled' + if [ "$mode" = closed ]; then + with_home "$home" "$ROOT/bin/fm-contributions.sh" pending | jq -e '[.[] | select(.type == "closed" and .url == "https://github.com/o/r/pull/8") | .task] == ["delivery"]' >/dev/null \ + || fail 'unmerged closure was not reported to exactly the task that is still held' + fi + cp "$home/data/backlog.md" "$home/settled.md" + with_home "$home" "$ROOT/bin/fm-contributions.sh" poll >/dev/null || fail 'retry failed' + cmp -s "$home/data/backlog.md" "$home/settled.md" || fail 'retry changed the settled task' + done + pass 'forge events settle only declared questions and preserve unfinished work' +} + +test_forge_hold_policy_guards() { + local home mode out + for mode in disabled unmatched reheld; do + home=$(new_home "hold-guard-$mode") + forge_home "$home" + touch "$home/config/hold-pr-settle" + with_home "$home" "$ROOT/bin/fm-tasks-axi.sh" update delivery --pr https://github.com/o/r/pull/8 >/dev/null || fail 'link failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold delivery --reason 'Decide the PR' >/dev/null || fail 'hold failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" pr-policy delivery https://github.com/o/r/pull/8 merged close >/dev/null || fail 'policy failed' + case "$mode" in + disabled) rm "$home/config/hold-pr-settle" ;; + reheld) with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold delivery --reason 'Another question about this work' >/dev/null || fail 'rehold failed' ;; + esac + if [ "$mode" = unmatched ]; then printf 'closed\n'; else printf 'merged\n'; fi > "$home/forge/state" + out=$(with_home "$home" "$ROOT/bin/fm-contributions.sh" poll) || fail 'guard poll failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery || fail "$mode policy settled the question" + if [ "$mode" = unmatched ]; then + printf '%s\n' "$out" | grep -F 'contributions: PR hold settlement failed for delivery https://github.com/o/r/pull/8' >/dev/null \ + || fail 'a policy that can never match was dropped without a diagnostic' + out=$(with_home "$home" "$ROOT/bin/fm-contributions.sh" poll) || fail 'unmatched repeat poll failed' + [ -z "$out" ] || fail "a dropped policy was reported again: $out" + fi + if [ "$mode" = disabled ]; then + [ -z "$out" ] || fail 'disabled feature emitted a terminal notification' + touch "$home/config/hold-pr-settle" + with_home "$home" "$ROOT/bin/fm-contributions.sh" poll >/dev/null || fail 'cached terminal poll failed' + if with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery; then fail 'cached forge event did not settle the hold'; fi + fi + done + pass 'disabled, unmatched, and superseded policies cannot settle a hold' +} + +test_forge_hold_settlement_retry() { + local home real body + home=$(new_home hold-settlement-retry) + forge_home "$home" + touch "$home/config/hold-pr-settle" + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold delivery --reason 'Merge this PR?' >/dev/null || fail 'hold failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" pr-policy delivery https://github.com/o/r/pull/8 merged close >/dev/null || fail 'policy failed' + # Retaining a finished worker's deliverable must not invalidate its question. + with_home "$home" "$ROOT/bin/fm-tasks-axi.sh" show delivery --full | sed -n 's/^ body: //p' | jq -r . > "$home/body.txt" + printf '\nDeliverable of the finished work: https://github.com/o/r/pull/8\n' >> "$home/body.txt" + with_home "$home" "$ROOT/bin/fm-tasks-axi.sh" update delivery --body-file "$home/body.txt" >/dev/null || fail 'deliverable update failed' + with_home "$home" "$ROOT/bin/fm-tasks-axi.sh" update delivery --report data/delivery/report.md >/dev/null || fail 'report link failed' + real=$(command -v tasks-axi) + cat > "$home/fakebin/tasks-axi" <<'SH' +#!/usr/bin/env bash +if [ "$1:$2" = done:delivery ] && [ ! -e "$FORGE/close-failed" ]; then + touch "$FORGE/close-failed" + exit 1 +fi +exec "$REAL_TASKS_AXI" "$@" +SH + chmod +x "$home/fakebin/tasks-axi" + printf 'merged\n' > "$home/forge/state" + with_home "$home" env REAL_TASKS_AXI="$real" "$ROOT/bin/fm-contributions.sh" poll >/dev/null || fail 'interrupted poll failed' + with_home "$home" env REAL_TASKS_AXI="$real" "$ROOT/bin/fm-captain-hold.sh" open delivery || fail 'failed close lost the hold' + body=$(with_home "$home" env REAL_TASKS_AXI="$real" "$ROOT/bin/fm-tasks-axi.sh" show delivery --full) + printf '%s\n' "$body" | grep -F 'Reconciliation evidence:' >/dev/null || fail 'no durable evidence before the failed close' + with_home "$home" env REAL_TASKS_AXI="$real" "$ROOT/bin/fm-contributions.sh" poll >/dev/null || fail 'retry poll failed' + if with_home "$home" env REAL_TASKS_AXI="$real" "$ROOT/bin/fm-captain-hold.sh" open delivery; then fail 'retry did not settle the hold'; fi + body=$(with_home "$home" env REAL_TASKS_AXI="$real" "$ROOT/bin/fm-tasks-axi.sh" show delivery --full) + [ "$(printf '%s\n' "$body" | grep -o 'Resolution recorded by fm-captain-hold.' | wc -l | tr -d ' ')" = 1 ] || fail 'retry duplicated the resolution' + pass 'retained delivery keeps its PR policy and interrupted settlement replays once' +} + +test_forge_hold_settlement_rejects_uncertain_evidence() { + local home + home=$(new_home hold-uncertain) + forge_home "$home" + touch "$home/config/hold-pr-settle" + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold delivery --reason 'Merge this PR?' >/dev/null || fail 'hold failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" pr-policy delivery https://github.com/o/r/pull/8 terminal close >/dev/null || fail 'policy failed' + mutate_record "$home" delivery '.records[0].observation.state="merged" | .records[0].error="forge unavailable"' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" settle-pr delivery https://github.com/o/r/pull/8 || fail 'uncertain evidence command failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery || fail 'errored evidence settled a hold' + mutate_record "$home" delivery '.records[0].error=null' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" settle-pr delivery https://github.com/o/r/pull/9 || fail 'different URL command failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery || fail 'another PR settled the hold' + with_home "$home" "$ROOT/bin/fm-tasks-axi.sh" update delivery --title 'A different question' >/dev/null || fail 'title update failed' + if with_home "$home" "$ROOT/bin/fm-captain-hold.sh" settle-pr delivery https://github.com/o/r/pull/8 2>/dev/null; then + fail 'a policy for a changed question was dropped without a failure' + fi + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery || fail 'a stale policy settled the hold' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" settle-pr delivery https://github.com/o/r/pull/8 || fail 'a dropped policy was retried' + pass 'errored evidence, a different PR, and a changed question cannot settle a hold' +} + +test_forge_release_waits_for_an_attended_session() { + local home out + home=$(new_home hold-away-release) + forge_home "$home" + touch "$home/config/hold-pr-settle" + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold delivery --reason 'Merge this PR?' >/dev/null || fail 'hold failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" pr-policy delivery https://github.com/o/r/pull/8 merged release >/dev/null || fail 'policy failed' + with_home "$home" "$ROOT/bin/fm-afk-contract.sh" enter --words 'watch the fleet' >/dev/null || fail 'could not enter away posture' + printf 'merged\n' > "$home/forge/state" + out=$(with_home "$home" "$ROOT/bin/fm-contributions.sh" poll) || fail 'away poll failed' + [ -z "$out" ] || fail "a deferred release produced poll output: $out" + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery || fail 'a merged PR released a captain hold while the home was away' + with_home "$home" "$ROOT/bin/fm-afk-contract.sh" archive >/dev/null || fail 'could not archive the away record' + with_home "$home" "$ROOT/bin/fm-contributions.sh" poll >/dev/null || fail 'attended poll failed' + if with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery; then fail 'the hold stayed after the captain returned'; fi + pass 'a declared release waits while the home is away and applies on the next attended poll' +} + +hold_lock() { # home lock: hold the lock until $home/release exists; sets HOLDER_PID + local home=$1 lock=$2 i=0 + rm -f "$home/holder.ready" "$home/release" + FM_STATE_OVERRIDE="$home/state" bash -c ' + . "$1" + fm_lock_acquire_wait "$2" || exit 10 + printf "ready\n" > "$3" + while [ ! -e "$4" ]; do sleep 0.05; done + fm_lock_release "$2" + ' _ "$ROOT/bin/fm-wake-lib.sh" "$lock" "$home/holder.ready" "$home/release" & + HOLDER_PID=$! + while [ "$i" -lt 100 ] && [ ! -s "$home/holder.ready" ]; do sleep 0.05; i=$((i + 1)); done + [ -s "$home/holder.ready" ] || { kill "$HOLDER_PID" 2>/dev/null || true; fail "the fixture never took $lock"; } +} + +test_forge_release_waits_for_the_away_record_lock() { + local home out holder_pid i started + home=$(new_home hold-away-lock) + forge_home "$home" + touch "$home/config/hold-pr-settle" + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold delivery --reason 'Merge this PR?' >/dev/null || fail 'hold failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" pr-policy delivery https://github.com/o/r/pull/8 merged release >/dev/null || fail 'policy failed' + # An away entry in progress holds this lock before its record exists. + hold_lock "$home" "$home/state/.afk-contract.lock" + holder_pid=$HOLDER_PID + printf 'merged\n' > "$home/forge/state" + started=$SECONDS + out=$(with_home "$home" "$ROOT/bin/fm-contributions.sh" poll) + i=$? + touch "$home/release" + wait "$holder_pid" || fail 'the lock holder failed' + [ "$i" -eq 0 ] || fail 'locked poll failed' + # The watcher cuts a check after 30 seconds; a merge may hold this lock longer. + [ $((SECONDS - started)) -lt 20 ] || fail "a busy away-record lock held the poll for $((SECONDS - started))s" + [ -z "$out" ] || fail "a release deferred by the lock produced poll output: $out" + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery || fail 'a merged PR released a captain hold while the away record was locked' + with_home "$home" "$ROOT/bin/fm-contributions.sh" poll >/dev/null || fail 'unlocked poll failed' + if with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery; then fail 'the hold stayed after the lock was released'; fi + pass 'a declared release waits while another process holds the away-record lock' +} + +test_forge_policy_survives_a_follow_up_pr_link() { + local home + home=$(new_home hold-follow-up-link) + forge_home "$home" + touch "$home/config/hold-pr-settle" + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold delivery --reason 'Merge this PR?' >/dev/null || fail 'hold failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" pr-policy delivery https://github.com/o/r/pull/8 merged close >/dev/null || fail 'policy failed' + with_home "$home" "$ROOT/bin/fm-tasks-axi.sh" update delivery --pr https://github.com/o/r/pull/10 >/dev/null || fail 'follow-up link failed' + printf 'merged\n' > "$home/forge/state" + with_home "$home" "$ROOT/bin/fm-contributions.sh" poll >/dev/null || fail 'poll failed' + if with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery; then fail 'linking a follow-up PR left the merged PR hold open'; fi + pass 'linking a follow-up PR keeps the declared policy valid' +} + +closed_event_tasks() { # home + with_home "$1" "$ROOT/bin/fm-contributions.sh" pending | jq -c '[.[] | select(.type == "closed") | .task] | sort' +} + +test_closed_pr_is_reported_only_to_open_captain_holds() { + local home out + home=$(new_home hold-closed-scope) + forge_home "$home" + rm "$home/state/delivery.meta" + printf 'closed\n' > "$home/forge/state" + with_home "$home" "$ROOT/bin/fm-contributions.sh" poll >/dev/null || fail 'closing poll failed' + with_home "$home" "$ROOT/bin/fm-tasks-axi.sh" "done" delivery >/dev/null || fail 'could not finish the task' + touch "$home/config/hold-pr-settle" + out=$(with_home "$home" "$ROOT/bin/fm-contributions.sh" poll) || fail 'first enabled poll failed' + [ -z "$out" ] || fail "enabling the flag reported a historical closed PR: $out" + [ ! -s "$home/state/.wake-queue" ] || fail 'enabling the flag queued a wake for a task without a hold' + [ "$(closed_event_tasks "$home")" = '[]' ] || fail 'a task without a captain hold got a closed event' + pass 'a closed PR of a task without a captain hold raises no event' +} + +test_late_linked_hold_gets_its_own_closed_event_once() { + local home out + home=$(new_home hold-closed-late) + forge_home "$home" + rm "$home/state/delivery.meta" + touch "$home/config/hold-pr-settle" + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold delivery --reason 'Rework or drop?' >/dev/null || fail 'hold failed' + printf 'closed\n' > "$home/forge/state" + with_home "$home" "$ROOT/bin/fm-contributions.sh" poll >/dev/null || fail 'closing poll failed' + [ "$(closed_event_tasks "$home")" = '["delivery"]' ] || fail 'the first held owner was not told about the closure' + with_home "$home" "$ROOT/bin/fm-tasks-axi.sh" add late 'Second question on the PR' --pr https://github.com/o/r/pull/8 >/dev/null || fail 'late add failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold late --reason 'Reopen it?' >/dev/null || fail 'late hold failed' + with_home "$home" "$ROOT/bin/fm-contributions.sh" poll >/dev/null || fail 'late-owner poll failed' + [ "$(closed_event_tasks "$home")" = '["delivery","late"]' ] || fail 'a hold linked after the closure inherited the report of the first owner' + out=$(with_home "$home" "$ROOT/bin/fm-contributions.sh" poll) || fail 'repeat poll failed' + [ -z "$out" ] || fail "the closure was reported again: $out" + [ "$(closed_event_tasks "$home")" = '["delivery","late"]' ] || fail 'a repeat poll changed the closed events' + pass 'a hold linked after the closure gets its own closed event once' +} + +test_forge_settlement_defers_while_the_task_is_busy() { + local home out rc started + home=$(new_home hold-busy-task) + forge_home "$home" + touch "$home/config/hold-pr-settle" + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold delivery --reason 'Merge this PR?' >/dev/null || fail 'hold failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" pr-policy delivery https://github.com/o/r/pull/8 merged close >/dev/null || fail 'policy failed' + # A captain answer in progress holds the task control lock. + hold_lock "$home" "$home/state/.control-delivery.lock" + printf 'merged\n' > "$home/forge/state" + started=$SECONDS + rc=0 + out=$(with_home "$home" "$ROOT/bin/fm-contributions.sh" poll) || rc=$? + [ "$rc" -ne 0 ] || [ -n "$out" ] || { out=$(with_home "$home" "$ROOT/bin/fm-captain-hold.sh" settle-pr delivery https://github.com/o/r/pull/8 2>&1) || rc=$?; } + touch "$home/release" + wait "$HOLDER_PID" || fail 'the lock holder failed' + [ "$rc" -eq 0 ] || fail "a busy task failed the settlement: $out" + # The watcher cuts a check after 30 seconds; an answer may hold this lock longer. + [ $((SECONDS - started)) -lt 20 ] || fail "a busy task control lock held the settlement for $((SECONDS - started))s" + [ -z "$out" ] || fail "a settlement deferred by a busy task produced output: $out" + [ -f "$home/data/delivery/hold-pr-policy.json" ] || fail 'a busy task lost its declared policy' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery || fail 'a busy task was settled under its lock holder' + with_home "$home" "$ROOT/bin/fm-contributions.sh" poll >/dev/null || fail 'unlocked poll failed' + if with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery; then fail 'the hold stayed after the task was free'; fi + pass 'a declared settlement defers quietly while another command holds the task' +} + +test_settled_task_gets_no_closed_event_for_its_other_pr() { + local home out + home=$(new_home hold-two-terminal) + forge_home "$home" + rm "$home/state/delivery.meta" + touch "$home/config/hold-pr-settle" + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" hold delivery --reason 'Decide this PR' >/dev/null || fail 'hold failed' + with_home "$home" "$ROOT/bin/fm-captain-hold.sh" pr-policy delivery https://github.com/o/r/pull/8 terminal close >/dev/null || fail 'policy failed' + with_home "$home" "$ROOT/bin/fm-tasks-axi.sh" update delivery --pr https://github.com/o/r/pull/9 >/dev/null || fail 'second link failed' + # Retain both closures first, so the enabled poll settles them in URL order. + rm "$home/config/hold-pr-settle" + printf 'closed\n' > "$home/forge/state" + with_home "$home" "$ROOT/bin/fm-contributions.sh" poll >/dev/null || fail 'closing poll failed' + touch "$home/config/hold-pr-settle" + out=$(with_home "$home" "$ROOT/bin/fm-contributions.sh" poll) || fail 'settling poll failed' + if with_home "$home" "$ROOT/bin/fm-captain-hold.sh" open delivery; then fail 'the first closed PR did not settle the hold'; fi + [ -z "$out" ] || fail "the second closed PR woke a task settled in the same poll: $out" + [ ! -s "$home/state/.wake-queue" ] || fail 'the second closed PR queued a wake for a settled task' + [ "$(closed_event_tasks "$home")" = '[]' ] || fail 'a task settled in the same poll got a closed event' + pass 'a task settled by its first PR gets no closed event for its second closed PR' +} + failures=0 -for test_name in test_actor_coverage test_stale_verdict test_unchecked_is_not_silence test_newest_check_has_no_verdict test_comment_wake test_review_wake test_inline_wake test_review_bot_inline_wake test_review_bot_watching_is_opt_in test_review_bot_allowlist_is_configurable test_ready_issue_wake test_fresh_issue_requires_maintainer test_missing_lane_remains_missing test_partial_freshness_keeps_measured_rows test_malformed_record_cannot_prove_silence test_issue_timeline_and_exact_ack test_verdict_retains_judged_head test_verdict_actor_values_are_discoverable test_observed_replacement_refreshes_verdict test_unobserved_head_leaves_verdict_unknown test_away_yolo_is_fleet_work test_away_yolo_cross_home_is_fleet_work test_retired_and_unsupported_coverage test_unsupported_forge_is_not_fleet_work test_held_unsupported_forge_is_not_captain_work test_unsupported_poll_is_silent_and_clears_legacy_errors test_legacy_cleanup_preserves_supported_failure test_shared_contribution_signal_wakes_once test_watcher_keeps_diagnostics_separate_from_contribution_wakes test_expired_child_unsupported_forge_stays_unmeasured test_watcher_surfaces_new_contribution_once test_home_summary_coverage test_unreadable_pending_is_not_empty test_record_task_identity_matches_dirname_basename test_read_only_views_create_no_state test_budget_refusal_between_calls test_budget_bounded_call_timeout test_genuine_failure_near_deadline_is_unavailable test_http_500_once_retries_without_waking test_http_500_twice_records_the_failed_read test_http_404_is_not_retried test_real_gh_errors_drive_retry_and_summary test_next_url_does_not_inherit_a_forge_diagnostic test_shared_url_observed_once test_terminal_contribution_settles test_late_owner_inherits_terminal_observation test_interrupted_multi_owner_poll_settles_every_owner test_done_task_open_pr_still_observed test_many_settled_records_keep_the_poll_inside_the_check_bound test_many_settled_records_beside_a_malformed_one_keep_the_poll_inside_the_check_bound test_record_without_trailing_newline_does_not_join_its_neighbour test_records_beyond_one_read_batch_are_all_read test_reservation_defers_later_url_when_fifteen_seconds_do_not_remain test_three_second_pr_reads_complete_fresh_in_one_cycle test_slow_read_deadline_kill_is_budget_refusal test_unmeasured_url_does_not_starve_the_tail test_budget_is_cut_down_to_the_watcher_check_bound test_arm_plumbs_a_configured_budget_into_the_check_shim test_unavailable_forge_records_error_and_wakes_once_per_episode test_late_owner_keeps_failure_episode_suppressed test_retire_ends_observation_of_a_gone_contribution test_late_owner_of_a_retired_final_contribution_is_not_retired test_retire_is_idempotent_and_refuses_unknown_pairs test_unanswered_thread_is_raised_again_under_hold test_resolved_thread_is_answered test_reply_rules test_reply_age_is_configurable_and_validated test_closed_pull_request_drops_its_threads test_unanswered_reraise_is_off_unless_configured test_arm_plumbs_the_reply_age_into_the_check_shim test_many_reviews_stay_readable test_reply_age_config_file test_ignored_commenters_are_not_tracked; do +for test_name in test_forge_settlement_defers_while_the_task_is_busy test_settled_task_gets_no_closed_event_for_its_other_pr test_forge_release_waits_for_an_attended_session test_forge_release_waits_for_the_away_record_lock test_forge_policy_survives_a_follow_up_pr_link test_closed_pr_is_reported_only_to_open_captain_holds test_late_linked_hold_gets_its_own_closed_event_once test_forge_hold_settlement_retry test_forge_hold_settlement_rejects_uncertain_evidence test_forge_settles_only_declared_holds test_forge_hold_policy_guards test_actor_coverage test_stale_verdict test_unchecked_is_not_silence test_newest_check_has_no_verdict test_comment_wake test_review_wake test_inline_wake test_review_bot_inline_wake test_review_bot_watching_is_opt_in test_review_bot_allowlist_is_configurable test_ready_issue_wake test_fresh_issue_requires_maintainer test_missing_lane_remains_missing test_partial_freshness_keeps_measured_rows test_malformed_record_cannot_prove_silence test_issue_timeline_and_exact_ack test_verdict_retains_judged_head test_verdict_actor_values_are_discoverable test_observed_replacement_refreshes_verdict test_unobserved_head_leaves_verdict_unknown test_away_yolo_is_fleet_work test_away_yolo_cross_home_is_fleet_work test_retired_and_unsupported_coverage test_unsupported_forge_is_not_fleet_work test_held_unsupported_forge_is_not_captain_work test_unsupported_poll_is_silent_and_clears_legacy_errors test_legacy_cleanup_preserves_supported_failure test_shared_contribution_signal_wakes_once test_watcher_keeps_diagnostics_separate_from_contribution_wakes test_expired_child_unsupported_forge_stays_unmeasured test_watcher_surfaces_new_contribution_once test_home_summary_coverage test_unreadable_pending_is_not_empty test_record_task_identity_matches_dirname_basename test_read_only_views_create_no_state test_budget_refusal_between_calls test_budget_bounded_call_timeout test_genuine_failure_near_deadline_is_unavailable test_http_500_once_retries_without_waking test_http_500_twice_records_the_failed_read test_http_404_is_not_retried test_real_gh_errors_drive_retry_and_summary test_next_url_does_not_inherit_a_forge_diagnostic test_shared_url_observed_once test_terminal_contribution_settles test_late_owner_inherits_terminal_observation test_interrupted_multi_owner_poll_settles_every_owner test_done_task_open_pr_still_observed test_many_settled_records_keep_the_poll_inside_the_check_bound test_many_settled_records_beside_a_malformed_one_keep_the_poll_inside_the_check_bound test_record_without_trailing_newline_does_not_join_its_neighbour test_records_beyond_one_read_batch_are_all_read test_reservation_defers_later_url_when_fifteen_seconds_do_not_remain test_three_second_pr_reads_complete_fresh_in_one_cycle test_slow_read_deadline_kill_is_budget_refusal test_unmeasured_url_does_not_starve_the_tail test_budget_is_cut_down_to_the_watcher_check_bound test_arm_plumbs_a_configured_budget_into_the_check_shim test_unavailable_forge_records_error_and_wakes_once_per_episode test_late_owner_keeps_failure_episode_suppressed test_retire_ends_observation_of_a_gone_contribution test_late_owner_of_a_retired_final_contribution_is_not_retired test_retire_is_idempotent_and_refuses_unknown_pairs test_unanswered_thread_is_raised_again_under_hold test_resolved_thread_is_answered test_reply_rules test_reply_age_is_configurable_and_validated test_closed_pull_request_drops_its_threads test_unanswered_reraise_is_off_unless_configured test_arm_plumbs_the_reply_age_into_the_check_shim test_many_reviews_stay_readable test_reply_age_config_file test_ignored_commenters_are_not_tracked; do ( "$test_name" ) || failures=$((failures + 1)) done [ "$failures" -eq 0 ] || fail "$failures contribution regressions" diff --git a/tests/fm-control-relaunch.test.sh b/tests/fm-control-relaunch.test.sh index 6f6d5f9c89a..41e72e1e0f4 100755 --- a/tests/fm-control-relaunch.test.sh +++ b/tests/fm-control-relaunch.test.sh @@ -392,6 +392,36 @@ test_same_harness_relaunch_keeps_identity_and_reuses_the_endpoint() { pass "fm-control relaunch: a same-harness relaunch replaces the agent in the same endpoint and worktree" } +# A lane relaunched after its PR was recorded keeps that PR as its record's +# tail, so the armed merge poll still validates against it. +test_relaunch_keeps_a_recorded_pr_valid_for_the_merge_poll() { + local dir out rc url=https://github.com/example/repo/pull/7 check + dir=$(new_case keeps-pr rl44) + add_ship_task "$dir" rl44 claude + cat > "$dir/fakebin/gh" <<'SH' +#!/usr/bin/env bash +case "$*" in + *headRefOid*) printf '%s\n' 0123456789abcdef0123456789abcdef01234567 ;; + *isDraft*) printf '%s\n' '{"isDraft":false}' ;; + *state*) printf '%s\n' MERGED ;; + *) exit 2 ;; +esac +SH + chmod +x "$dir/fakebin/gh" + out=$(env PATH="$dir/fakebin:$PATH" FM_HOME="$dir/home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_STATE_OVERRIDE="$dir/home/state" "$ROOT/bin/fm-pr-check.sh" rl44 "$url" 2>&1); rc=$? + expect_code 0 "$rc" "the executable merge-poll registration should succeed"$'\n'"$out" + check="$dir/home/state/rl44.check.sh" + [ -f "$check" ] || fail "fm-pr-check did not arm the merge poll: $out" + + out=$(run_control "$dir" rl44 relaunch --note "address review"); rc=$? + expect_code 0 "$rc" "a relaunch of a lane with a recorded PR should succeed"$'\\n'"$out" + [ -n "$(meta_field "$dir" rl44 control_relaunch_tx)" ] || fail "the relaunch transaction id was not recorded" + out=$(env PATH="$dir/fakebin:$PATH" FM_HOME="$dir/home" FM_STATE_OVERRIDE="$dir/home/state" bash "$check") + [ "$out" = merged ] || fail "the armed merge poll stopped accepting the relaunched record: '$out'" + pass "fm-control relaunch: a recorded PR stays valid to the armed merge poll" +} + test_relaunch_refuses_before_exit_when_the_composer_holds_pending_text() { local dir out rc dir=$(new_case pending-exit rl43) @@ -1237,6 +1267,28 @@ test_pi_exclude_tools_follow_the_relaunch() { pass "fm-control relaunch: a Pi replacement keeps the home's tool exclusions" } +test_claude_exclude_tools_follow_the_relaunch() { + local dir out rc id=rl-claude-excl + dir=$(new_case claude-exclude "$id") + add_ship_task "$dir" "$id" claude + printf claude > "$dir/fake/command" + printf claude > "$dir/fake/becomes" + mkdir -p "$dir/home/config" + printf '%s\n' '# hide unused schemas' 'Workflow' 'mcp__srv' > "$dir/home/config/crew-exclude-tools" + out=$(run_control "$dir" "$id" relaunch --note "keep exclusions"); rc=$? + expect_code 0 "$rc" "a Claude relaunch with exclusions should succeed"$'\n'"$out" + assert_contains "$(cat "$dir/fake/literal")" "--disallowedTools 'Workflow,mcp__srv' --append-system-prompt " \ + "the relaunched Claude worker must keep the home's tool exclusions" + rm "$dir/home/config/crew-exclude-tools" + : > "$dir/fake/literal" + printf claude > "$dir/fake/command" + out=$(run_control "$dir" "$id" relaunch --note "exclusions removed"); rc=$? + expect_code 0 "$rc" "a Claude relaunch after the file is removed should succeed"$'\n'"$out" + assert_not_contains "$(cat "$dir/fake/literal")" "--disallowedTools" \ + "a relaunch without the file must launch with no exclusions" + pass "fm-control relaunch: a Claude replacement keeps the home's tool exclusions" +} + test_exclude_tools_refusals_happen_before_the_agent_stops() { local dir out rc id=rl-excl-refuse dir=$(new_case excl-refuse "$id") @@ -3202,6 +3254,7 @@ test_relaunch_off_opencode_removes_tracked_wiring test_exit_and_relaunch_remove_the_dialog_file test_exit_removes_the_dialog_file_before_releasing_the_lock test_same_harness_relaunch_keeps_identity_and_reuses_the_endpoint +test_relaunch_keeps_a_recorded_pr_valid_for_the_merge_poll test_relaunch_refuses_before_exit_when_the_composer_holds_pending_text test_relaunch_refuses_before_exit_when_the_composer_state_is_unproven test_relaunch_from_linked_home_preserves_recorded_worktree @@ -3230,6 +3283,7 @@ test_account_list_relaunch_moves_only_on_explicit_account test_account_list_relaunch_refusals_happen_before_stop test_secondmate_account_pin_follows_relaunch_and_refuses_before_stop test_pi_exclude_tools_follow_the_relaunch +test_claude_exclude_tools_follow_the_relaunch test_exclude_tools_refusals_happen_before_the_agent_stops test_explicit_model_wins_over_the_recorded_one test_relaunch_onto_an_unverified_harness_is_refused diff --git a/tests/fm-cursor-primary.test.sh b/tests/fm-cursor-primary.test.sh index d15cbbff088..d5ad0f84778 100755 --- a/tests/fm-cursor-primary.test.sh +++ b/tests/fm-cursor-primary.test.sh @@ -73,7 +73,7 @@ install_scripts() { fm-sessionstart-run.sh fm-sessionstart-nudge.sh fm-arm-pretool-check.sh \ fm-cd-pretool-check.sh fm-claude-stop-autoarm.sh fm-hook-host-lib.sh \ fm-primary-scope-lib.sh fm-supervision-lib.sh fm-wake-lib.sh fm-path-lib.sh \ - fm-session-lock-lib.sh fm-cursor-lib.sh fm-operational-input.sh \ + fm-session-lock-lib.sh fm-omp-process-lib.sh fm-cursor-lib.sh fm-operational-input.sh \ fm-supervision-instructions.sh fm-harness.sh fm-lock.sh \ fm-gate-refuse-lib.sh fm-afk-contract.sh fm-classify-lib.sh fm-timeout-lib.sh \ fm-supervision-engine-lib.sh fm-lease-lib.sh; do diff --git a/tests/fm-dispatch-resolve.test.sh b/tests/fm-dispatch-resolve.test.sh index 0768d833c86..0bf2ef71291 100755 --- a/tests/fm-dispatch-resolve.test.sh +++ b/tests/fm-dispatch-resolve.test.sh @@ -1465,10 +1465,11 @@ assert_not_contains "$out" ' fallback:' "a picked rule that clears its own floo assert_contains "$out" " profile: --harness 'cursor' --model 'cursor-grok-4.6-medium'" "the picked rule resolves at probability 0.35 over floor 0.3" reset_log -write_floor_response "$RESPONSE" rule_2 0.95 0.05 0.55 0.05 0.30 0.05 +# 0.3 prints the same on jq 1.6 through 1.8; a trailing zero in 0.30 does not. +write_floor_response "$RESPONSE" rule_2 0.95 0.05 0.55 0.05 0.3 0.05 TYPESAFE_API_KEY=$KEY run code out err "$BRIEF" assert_contains "$out" ' status: clear' "a high answer confidence does not lift a picked rule over its own floor" -assert_contains "$out" ' fallback: rule_4 (A simple bug fix with a stated root cause.) probability 0.30 clears its floor 0.3; rule_2 probability 0.55 is below its floor 0.9' "the runner-up clears the same floor it would need as the pick" +assert_contains "$out" ' fallback: rule_4 (A simple bug fix with a stated root cause.) probability 0.3 clears its floor 0.3; rule_2 probability 0.55 is below its floor 0.9' "the runner-up clears the same floor it would need as the pick" reset_log write_floor_response "$RESPONSE" rule_2 0.55 0.05 0.55 0.05 0.25 0.10 diff --git a/tests/fm-dod-lib.test.sh b/tests/fm-dod-lib.test.sh index cdd70086e7a..1e330f03b1f 100644 --- a/tests/fm-dod-lib.test.sh +++ b/tests/fm-dod-lib.test.sh @@ -62,6 +62,231 @@ test_remote_containing_named_head_is_accepted() { pass "named head on a remote-tracking ref is accepted" } +test_gate_mirror_only_named_head_is_refused() { + local repo wt sha reason rc + repo="$TMP_ROOT/mirror-repo" + wt="$TMP_ROOT/mirror-wt" + fm_git_worktree "$repo" "$wt" fm/mirror + git -C "$wt" commit -q --allow-empty -m 'fix pushed only to the local gate' + sha=$(git -C "$wt" rev-parse HEAD) + # The local no-mistakes gate mirror lives on this machine, so a head that + # only reached it has not left the machine. + git -C "$wt" update-ref refs/remotes/no-mistakes/fm/mirror "$sha" + reason=$(accept_done ship no-mistakes "$wt" "$repo" "done: PR https://example.test/o/r/pull/4 checks green") + rc=$? + [ "$rc" -eq 1 ] || fail "named head only on the no-mistakes gate mirror was accepted (exit $rc)" + case "$reason" in + *"named head $sha is unreachable outside the worker copy") ;; + *) fail "gate-mirror refusal did not name the commit: $reason" ;; + esac + # A refresh fetches origin, which does not hold the head, and never counts + # the mirror, so the refusal stands with no refresh failure to report. + reason=$(FM_DOD_REFRESH_REMOTES=1 accept_done ship no-mistakes "$wt" "$repo" "done: PR https://example.test/o/r/pull/4 checks green") + rc=$? + [ "$rc" -eq 1 ] || fail "named head only on the gate mirror was accepted after a refresh (exit $rc)" + case "$reason" in + *"named head $sha is unreachable outside the worker copy") ;; + *) fail "gate-mirror refusal after a refresh changed its reason: $reason" ;; + esac + git -C "$wt" update-ref refs/remotes/origin/fm/mirror "$sha" + accept_done ship no-mistakes "$wt" "$repo" "done: PR https://example.test/o/r/pull/4 checks green" >/dev/null \ + || fail "named head on origin beside the gate mirror was refused" + pass "named head only on the no-mistakes gate mirror is refused, even after a refresh; on origin it is accepted" +} + +test_head_pushed_from_gate_clone_is_accepted_after_refresh() { + local repo wt origin sha reason rc + repo="$TMP_ROOT/gate-push-repo" + wt="$TMP_ROOT/gate-push-wt" + origin="$repo.origin.git" + fm_git_worktree "$repo" "$wt" fm/gate-push + git -C "$wt" commit -q --allow-empty -m 'fix the pipeline pushed from its gate clone' + sha=$(git -C "$wt" rev-parse HEAD) + git -C "$wt" update-ref refs/remotes/no-mistakes/fm/gate-push "$sha" + # The pipeline pushes from its own gate clone, so origin holds the head while + # neither the copy nor the project clone has fetched it. + git -C "$origin" fetch -q "$wt" "fm/gate-push:refs/heads/fm/gate-push" + [ "$(git -C "$origin" rev-parse refs/heads/fm/gate-push)" = "$sha" ] \ + || fail "fixture: origin does not hold the gate clone's push" + ! git -C "$wt" rev-parse -q --verify refs/remotes/origin/fm/gate-push >/dev/null \ + || fail "fixture: the copy already tracks origin's branch" + reason=$(accept_done ship no-mistakes "$wt" "$repo" "done: PR https://gitlab.example.test/g/r/-/merge_requests/5 checks green") + rc=$? + [ "$rc" -eq 1 ] || fail "a caller that does not refresh accepted an unfetched head (exit $rc)" + case "$reason" in + *"named head $sha is unreachable outside the worker copy") ;; + *) fail "unfetched refusal without a refresh did not name the commit: $reason" ;; + esac + ! git -C "$wt" rev-parse -q --verify refs/remotes/origin/fm/gate-push >/dev/null \ + || fail "a caller that does not refresh fetched anyway" + FM_DOD_REFRESH_REMOTES=1 accept_done ship no-mistakes "$wt" "$repo" "done: PR https://gitlab.example.test/g/r/-/merge_requests/5 checks green" >/dev/null \ + || fail "head origin holds through the gate clone was refused after a refresh" + [ "$(git -C "$wt" rev-parse refs/remotes/origin/fm/gate-push)" = "$sha" ] \ + || fail "the refresh did not leave origin's head in the copy's refs" + accept_done ship no-mistakes "$wt" "$repo" "done: PR https://gitlab.example.test/g/r/-/merge_requests/5 checks green" >/dev/null \ + || fail "a caller that does not refresh refused the head a refresh already fetched" + pass "head origin holds through the gate clone is accepted after a refresh, and later reads see it" +} + +test_passed_refresh_deadline_refuses_with_last_known_refs() { + local repo wt origin sha reason rc + repo="$TMP_ROOT/deadline-passed-repo" + wt="$TMP_ROOT/deadline-passed-wt" + origin="$repo.origin.git" + fm_git_worktree "$repo" "$wt" fm/deadline-passed + git -C "$wt" commit -q --allow-empty -m 'fix origin holds while the copy has not fetched it' + sha=$(git -C "$wt" rev-parse HEAD) + git -C "$origin" fetch -q "$wt" "fm/deadline-passed:refs/heads/fm/deadline-passed" + reason=$(FM_DOD_REFRESH_REMOTES=1 FM_DOD_REFRESH_DEADLINE=1 accept_done ship no-mistakes "$wt" "$repo" "done: PR https://gitlab.example.test/g/r/-/merge_requests/8 checks green") + rc=$? + [ "$rc" -eq 1 ] || fail "a head was accepted although the refresh deadline had passed (exit $rc)" + case "$reason" in + *"named head $sha is unreachable outside the worker copy; could not refresh remote: origin, so this verdict used the last-known refs"*) ;; + *) fail "a refusal past the refresh deadline did not say it used the last-known refs: $reason" ;; + esac + ! git -C "$wt" rev-parse -q --verify refs/remotes/origin/fm/deadline-passed >/dev/null \ + || fail "a refresh past its deadline fetched anyway" + pass "past the refresh deadline nothing is fetched and the refusal says it used the last-known refs" +} + +test_separate_project_clone_is_refreshed() { + local repo wt origin sha reason rc + repo="$TMP_ROOT/separate-project-repo" + wt="$TMP_ROOT/separate-project-wt" + origin="$repo.origin.git" + fm_git_init_commit "$repo" + fm_git_add_origin "$repo" "$origin" + # The copy is a repository of its own whose only remote is the project clone, + # so only a refresh of the project clone reaches the origin holding the head. + git clone -q "$repo" "$wt" + git -C "$wt" checkout -q -b fm/separate + git -C "$wt" commit -q --allow-empty -m 'fix the pipeline pushed from its gate clone' + sha=$(git -C "$wt" rev-parse HEAD) + git -C "$origin" fetch -q "$wt" "fm/separate:refs/heads/fm/separate" + [ "$(git -C "$wt" rev-parse --path-format=absolute --git-common-dir)" != \ + "$(git -C "$repo" rev-parse --path-format=absolute --git-common-dir)" ] \ + || fail "fixture: the copy shares the project clone's git directory" + reason=$(accept_done ship no-mistakes "$wt" "$repo" "done: PR https://gitlab.example.test/g/r/-/merge_requests/7 checks green") + rc=$? + [ "$rc" -eq 1 ] || fail "a caller that does not refresh accepted an unfetched head (exit $rc)" + case "$reason" in + *"named head $sha is unreachable outside the worker copy") ;; + *) fail "unfetched refusal without a refresh did not name the commit: $reason" ;; + esac + FM_DOD_REFRESH_REMOTES=1 accept_done ship no-mistakes "$wt" "$repo" "done: PR https://gitlab.example.test/g/r/-/merge_requests/7 checks green" >/dev/null \ + || fail "head the project clone's origin holds was refused after a refresh" + [ "$(git -C "$repo" rev-parse refs/remotes/origin/fm/separate)" = "$sha" ] \ + || fail "the refresh did not fetch the project clone's origin" + ! git -C "$wt" rev-parse -q --verify refs/remotes/origin/fm/separate >/dev/null \ + || fail "fixture: the copy's own remote held the head" + pass "a project clone with its own git directory is refreshed too, and its origin's head is accepted" +} + +test_failed_refresh_refuses_with_last_known_refs() { + local repo wt sha reason rc + repo="$TMP_ROOT/refresh-fail-repo" + wt="$TMP_ROOT/refresh-fail-wt" + fm_git_worktree "$repo" "$wt" fm/refresh-fail + git -C "$wt" commit -q --allow-empty -m 'fix behind an unreachable remote' + sha=$(git -C "$wt" rev-parse HEAD) + git -C "$repo" remote set-url origin "file://$TMP_ROOT/refresh-fail-missing.git" + reason=$(FM_DOD_REFRESH_REMOTES=1 accept_done ship no-mistakes "$wt" "$repo" "done: PR https://example.test/o/r/pull/6 checks green") + rc=$? + [ "$rc" -eq 1 ] || fail "named head was accepted although the refresh failed (exit $rc)" + case "$reason" in + *"named head $sha is unreachable outside the worker copy; could not refresh remote: origin, so this verdict used the last-known refs"*) ;; + *) fail "failed-refresh refusal did not say it used the last-known refs: $reason" ;; + esac + [ "$(printf '%s\n' "$reason" | wc -l | tr -d ' ')" = 1 ] \ + || fail "failed-refresh refusal is not one line: $reason" + pass "a failed refresh still refuses and says the verdict used the last-known refs" +} + +test_remote_hanging_before_origin_leaves_origin_its_refresh() { + local repo wt origin fakebin sha start + repo="$TMP_ROOT/hang-first-repo" + wt="$TMP_ROOT/hang-first-wt" + origin="$repo.origin.git" + fakebin="$TMP_ROOT/hang-first-bin" + fm_git_worktree "$repo" "$wt" fm/hang-first + git -C "$wt" commit -q --allow-empty -m 'fix the pipeline pushed from its gate clone' + sha=$(git -C "$wt" rev-parse HEAD) + git -C "$origin" fetch -q "$wt" "fm/hang-first:refs/heads/fm/hang-first" + # backup sorts before origin and never answers a fetch. + mkdir -p "$fakebin" + printf '%s\n' '#!/bin/sh' 'exec sleep 30' > "$fakebin/git-remote-hang" + chmod +x "$fakebin/git-remote-hang" + git -C "$repo" remote add backup hang::backup + [ "$(git -C "$wt" remote | head -1)" = backup ] \ + || fail "fixture: the hanging remote is not listed before origin" + FM_DOD_REFRESH_FAILED= + start=$(date +%s) + PATH="$fakebin:$PATH" FM_DOD_REFRESH_REMOTES=1 FM_DOD_REFRESH_DEADLINE=$((start + 4)) \ + accept_done ship no-mistakes "$wt" "$repo" "done: PR https://gitlab.example.test/g/r/-/merge_requests/9 checks green" >/dev/null \ + || fail "a remote hanging before origin under a short refresh deadline kept origin's head from being accepted" + [ "$(git -C "$wt" rev-parse refs/remotes/origin/fm/hang-first)" = "$sha" ] \ + || fail "origin was not fetched behind a remote that hangs" + [ "$FM_DOD_REFRESH_FAILED" = backup ] \ + || fail "the hanging remote was not named as not refreshed: '$FM_DOD_REFRESH_FAILED'" + [ "$(date +%s)" -le $((start + 6)) ] || fail "the refresh outlived its deadline" + pass "a remote that hangs and sorts before origin cannot take origin's refresh; origin's head is accepted and the hanging remote is named" +} + +test_hanging_copy_remote_leaves_the_project_clone_its_refresh() { + local repo wt origin fakebin sha reason rc start + repo="$TMP_ROOT/copy-hang-repo" + wt="$TMP_ROOT/copy-hang-wt" + origin="$repo.origin.git" + fakebin="$TMP_ROOT/copy-hang-bin" + fm_git_init_commit "$repo" + fm_git_add_origin "$repo" "$origin" + # A standalone copy whose origin is the project clone and whose second remote + # never answers a fetch; only the project clone's origin holds the head. + git clone -q "$repo" "$wt" + git -C "$wt" checkout -q -b fm/copy-hang + git -C "$wt" commit -q --allow-empty -m 'fix the pipeline pushed from its gate clone' + sha=$(git -C "$wt" rev-parse HEAD) + git -C "$origin" fetch -q "$wt" "fm/copy-hang:refs/heads/fm/copy-hang" + mkdir -p "$fakebin" + printf '%s\n' '#!/bin/sh' 'exec sleep 30' > "$fakebin/git-remote-hang" + chmod +x "$fakebin/git-remote-hang" + git -C "$wt" remote add fork hang::fork + [ "$(git -C "$wt" rev-parse --path-format=absolute --git-common-dir)" != \ + "$(git -C "$repo" rev-parse --path-format=absolute --git-common-dir)" ] \ + || fail "fixture: the copy shares the project clone's git directory" + reason=$(FM_DOD_REFRESH_REMOTES=1 FM_DOD_REFRESH_DEADLINE=1 accept_done ship no-mistakes "$wt" "$repo" "done: PR https://gitlab.example.test/g/r/-/merge_requests/10 checks green") + rc=$? + [ "$rc" -eq 1 ] || fail "a head was accepted although the refresh deadline had passed (exit $rc)" + case "$reason" in + *"could not refresh remote: origin, fork, origin (project clone), so this verdict used the last-known refs"*) ;; + *) fail "the refusal did not tell the copy's remotes from the project clone's: $reason" ;; + esac + FM_DOD_REFRESH_FAILED= + start=$(date +%s) + PATH="$fakebin:$PATH" FM_DOD_REFRESH_REMOTES=1 FM_DOD_REFRESH_DEADLINE=$((start + 6)) \ + accept_done ship no-mistakes "$wt" "$repo" "done: PR https://gitlab.example.test/g/r/-/merge_requests/10 checks green" >/dev/null \ + || fail "a hanging remote in the copy kept the project clone's origin head from being accepted" + [ "$(git -C "$repo" rev-parse refs/remotes/origin/fm/copy-hang)" = "$sha" ] \ + || fail "the project clone's origin was not fetched behind the copy's hanging remote" + [ "$FM_DOD_REFRESH_FAILED" = fork ] \ + || fail "the hanging remote was not named as not refreshed: '$FM_DOD_REFRESH_FAILED'" + [ "$(date +%s)" -le $((start + 8)) ] || fail "the refresh outlived its deadline" + pass "a hanging remote in a standalone copy cannot take the project clone's refresh; its origin's head is accepted and failed remotes name their repository" +} + +test_local_only_branch_beside_gate_mirror_is_accepted() { + local repo wt sha + repo="$TMP_ROOT/mirror-local-repo" + wt="$TMP_ROOT/mirror-local-wt" + fm_git_worktree "$repo" "$wt" fm/mirror-local + git -C "$wt" commit -q --allow-empty -m 'local-only work also on the gate' + sha=$(git -C "$wt" rev-parse HEAD) + git -C "$wt" update-ref refs/remotes/no-mistakes/fm/mirror-local "$sha" + accept_done ship local-only "$wt" "$repo" "done: ready in branch fm/mirror-local" >/dev/null \ + || fail "local-only named branch in the project clone was refused beside a gate mirror ref" + pass "local-only named branch stays reachable when the gate mirror also holds it" +} + test_moved_branch_without_named_head_is_refused() { local repo wt main_sha fix_sha reason rc repo="$TMP_ROOT/moved-repo" @@ -452,6 +677,14 @@ test_scout_done_is_not_gated test_unpushed_ship_done_is_refused test_no_mistakes_prevalidation_done_is_not_gated test_remote_containing_named_head_is_accepted +test_gate_mirror_only_named_head_is_refused +test_head_pushed_from_gate_clone_is_accepted_after_refresh +test_separate_project_clone_is_refreshed +test_passed_refresh_deadline_refuses_with_last_known_refs +test_failed_refresh_refuses_with_last_known_refs +test_remote_hanging_before_origin_leaves_origin_its_refresh +test_hanging_copy_remote_leaves_the_project_clone_its_refresh +test_local_only_branch_beside_gate_mirror_is_accepted test_moved_branch_without_named_head_is_refused test_free_text_sha_is_not_the_named_head test_recorded_merged_pr_is_landed_after_prune diff --git a/tests/fm-extension-binding.test.sh b/tests/fm-extension-binding.test.sh index 9db6460188a..033ccb4a00b 100644 --- a/tests/fm-extension-binding.test.sh +++ b/tests/fm-extension-binding.test.sh @@ -18,7 +18,7 @@ fi extension_segment=${FM_EXTENSION_BINDING_SEGMENT:-all} case "$extension_segment" in - all|coordinator|early-bind|early-validation|early-handshake|early-integrity|matrix|matrix-runtime|lifecycle-flow|lifecycle-host-errors|lifecycle-order|lifecycle-lock|lifecycle-runner|lifecycle-state|lifecycle-invocation-cleanup|lifecycle-orphan-readiness|remote-envelope|remote-activation|remote-lifecycle|remote-retirement|example|coordinator-fail|coordinator-wait|coordinator-stubborn|coordinator-pass|coordinator-late-pass|coordinator-scheduler-block|coordinator-scheduler-late) ;; + all|coordinator|deadlines|early-bind|early-validation|early-handshake|early-integrity|matrix|matrix-runtime|lifecycle-flow|lifecycle-host-errors|lifecycle-order|lifecycle-lock|lifecycle-runner|lifecycle-state|lifecycle-invocation-cleanup|lifecycle-orphan-readiness|remote-envelope|remote-activation|remote-lifecycle|remote-retirement|example|coordinator-fail|coordinator-wait|coordinator-stubborn|coordinator-pass|coordinator-late-pass|coordinator-scheduler-block|coordinator-scheduler-late) ;; *) printf 'unknown extension-binding segment: %s\n' "$extension_segment" >&2; exit 64 ;; esac @@ -574,7 +574,7 @@ if [ "$extension_segment" = all ] || [ "$extension_segment" = coordinator ]; the ( trap - EXIT HUP INT trap 'terminate_section_lanes; exit 143' TERM - run_extension_section_lanes lifecycle-flow lifecycle-order remote-lifecycle example + run_extension_section_lanes lifecycle-flow lifecycle-order remote-lifecycle example deadlines ) & section_coordinator_pid=$! fi @@ -652,6 +652,154 @@ if [ "$extension_segment" = all ] || [ "$extension_segment" = coordinator ]; the exit 0 fi +# Freeze the real host and invocation group across each deadline. All cases +# have private homes and run concurrently within the existing aggregate bound. +if section_enabled deadlines; then +P_SLEEP="$PACKAGES/sleep" +make_package "$P_SLEEP" org.example.sleep ext-sleep +if ! python3 - "$ROOT" "$TMP_ROOT" "$P_SLEEP" <<'PY' +import concurrent.futures, json, os, pathlib, shutil, signal, subprocess, sys, time + +root, temporary, package = map(pathlib.Path, sys.argv[1:]) +host = root / "bin/fm-extension.mjs" + +def wait_for(predicate): + for _ in range(400): + if predicate(): return + time.sleep(.025) + raise AssertionError("deadline fixture did not reach its control boundary") + +def run_case(kind): + case = temporary / ("deadline-" + kind) + case.mkdir() + home = case / "home" + home.mkdir() + env = dict(os.environ, FM_HOME=str(home)) + marker, release = case / "marker", case / "release" + process = None + group = None + host_pid = None + try: + if kind == "startup": + owner, ready = case / "owner", case / "ready" + token = "sha256:" + "a" * 64 + entrypoint = case / "entrypoint" + entrypoint.write_text("#!/bin/sh\nexit 0\n") + entrypoint.chmod(0o755) + process = subprocess.Popen(["node", str(root / "bin/fm-extension-launch-barrier.mjs"), + token, str(owner), str(ready), str(release), str(os.getpid()), + str(entrypoint), str(case), "invoke"], start_new_session=True, + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + group = process.pid + wait_for(ready.exists) + os.killpg(group, signal.SIGSTOP) + time.sleep(8) + owner.write_text(json.dumps(dict(schema="firstmate.extension-invocation-owner.v1", + token=token, phase="group", host_pid=os.getpid(), host_identity="fixture", + group_pid=group, group_identity="barrier-token:" + token, + extension_id="org.example.sleep", binding_digest="sha256:" + "b" * 64, + request_id="sha256:" + "c" * 64, source_id=None, operation="source.poll"))) + release.write_text(json.dumps(dict(schema="firstmate.extension-invocation-release.v1", token=token))) + os.killpg(group, signal.SIGCONT) + else: + local_package = case / "package" + shutil.copytree(package, local_package) + if kind == "handshake": + (local_package / "scenario").write_text(f"handshake-block\n{marker}\n{release}\n") + executable = host + if kind == "launch": + code = case / "code" + shutil.copytree(root / "bin", code / "bin") + barrier = code / "bin/fm-extension-launch-barrier.mjs" + barrier.rename(barrier.with_name("original-barrier.mjs")) + barrier.write_text('import fs from "node:fs";\n' + f'if (!fs.existsSync({json.dumps(str(marker))})) {{\n' + f'fs.writeFileSync({json.dumps(str(marker))}, String(process.pid));\n' + 'process.kill(process.pid, "SIGSTOP"); }\n' + 'await import("./original-barrier.mjs");\n') + executable = code / "bin/fm-extension.mjs" + command = ["node", str(executable), "bind", str(local_package), "--adapter", "ext-sleep", + "--trust-same-user-code", "--timeout-ms", "1000"] + if kind in ("invoke", "stalled", "suspended-stall"): + subprocess.run(command, env=env, check=True, capture_output=True) + resolution = subprocess.run(["node", str(host), "resolve-process-event", "ext-sleep"], + env=env, check=True, capture_output=True, text=True).stdout.strip().split("\t") + expected = [] + for flag, value in zip(("--expect-extension", "--expect-version", "--expect-capability-version", + "--expect-package-digest", "--expect-binding-digest"), resolution[1:]): + expected.extend((flag, value)) + # The package waits on these paths after stdin is delivered. + command = ["node", str(host), "process-event", "ext-sleep", "source.poll", + "--source-id", "sleep-source", "--config-ref", f"active-block|{marker}|{release}", *expected] + process = subprocess.Popen(command, env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + wait_for(marker.exists) + if kind == "launch": + group = int(marker.read_text()) + else: + group = os.getpgid(int(marker.read_text())) + def invocation_owner(): + for filename in (home / "state/extension-invocations").glob("*.owner.json"): + owner = json.loads(filename.read_text()) + if kind == "launch" or owner["group_pid"] == group: + return owner + return None + wait_for(invocation_owner) + host_pid = invocation_owner()["host_pid"] + if kind != "stalled": + os.kill(host_pid, signal.SIGSTOP) + os.killpg(group, signal.SIGSTOP) + time.sleep(4 if kind in ("invoke", "suspended-stall") else 8) + if kind != "suspended-stall": release.touch() + os.kill(host_pid, signal.SIGCONT) + os.killpg(group, signal.SIGCONT) + output, error = process.communicate(timeout=6) + if kind in ("stalled", "suspended-stall"): + assert process.returncode != 0, "a genuinely stalled invocation succeeded" + evidence = json.loads(output) + assert evidence["code"] == "timeout", evidence + assert evidence["deadline"]["bound"] == "invoke", evidence + assert evidence["deadline"]["timeout_ms"] == 1000, evidence + if kind == "stalled": + assert 1000 <= evidence["deadline"]["elapsed_ms"] < 3000, evidence + assert evidence["deadline"]["grace_ms"] == 0, evidence + else: + assert 5000 <= evidence["deadline"]["elapsed_ms"] < 8000, evidence + assert evidence["deadline"]["grace_ms"] == 1000, evidence + assert evidence["deadline"]["overrun_ms"] == evidence["deadline"]["elapsed_ms"] - 1000, evidence + else: + assert process.returncode == 0, (kind, process.returncode, output, error) + if kind == "invoke": assert b"active runner completed" in output, output + print("ok - real deadline " + kind, flush=True) + finally: + if host_pid: + try: os.kill(host_pid, signal.SIGCONT) + except ProcessLookupError: pass + if group: + try: os.killpg(group, signal.SIGCONT) + except ProcessLookupError: pass + if process and process.poll() is None: + process.send_signal(signal.SIGCONT) + process.terminate() + try: process.wait(timeout=4) + except subprocess.TimeoutExpired: process.kill(); process.wait() + if group: + try: os.killpg(group, signal.SIGKILL) + except ProcessLookupError: pass + +with concurrent.futures.ThreadPoolExecutor(max_workers=6) as executor: + futures = [executor.submit(run_case, kind) for kind in ("invoke", "handshake", "launch", "startup", "stalled", "suspended-stall")] + failures = [] + for future in futures: + try: future.result() + except Exception as error: failures.append(repr(error)) + assert not failures, "\n".join(failures) +PY +then + fail "real invocation deadlines did not survive suspension with bounded stall enforcement" +fi +pass "invoke, handshake, readiness and release survive suspension while awake stalls time out" +fi + # --- permanently inert absent-registry path --------------------------------- if section_enabled early-bind; then # Exercise the executable boundary without requiring Windows: this Bash payload @@ -660,6 +808,7 @@ if section_enabled early-bind; then SHELL_ROOT="$TMP_ROOT/lifecycle shell root" mkdir -p "$SHELL_ROOT/bin" cp "$HOST" "$SHELL_ROOT/bin/fm-extension.mjs" +cp "$ROOT/bin/fm-extension-deadline.mjs" "$SHELL_ROOT/bin/" cat > "$SHELL_ROOT/bin/fm-procevent.sh" <<'SH' [[ -n "$BASH_VERSION" ]] || exit 98 if [[ "${*: -1}" = reject ]]; then @@ -2276,7 +2425,7 @@ REMOTE_SSH_COUNT="$TMP_ROOT/remote-ssh.count" mkdir -p "$H_REMOTE_CONTROL/data" "$H_REMOTE" "$REMOTE_ROOT/bin" printf 'fixture\n' > "$REMOTE_ROOT/AGENTS.md" for remote_file in \ - fm-extension.mjs fm-extension-launch-barrier.mjs fm-extension.sh fm-procevent.sh fm-procevent-lib.sh fm-procevent-extension-capture.pl fm-procevent-lavish.sh \ + fm-extension.mjs fm-extension-deadline.mjs fm-extension-launch-barrier.mjs fm-extension.sh fm-procevent.sh fm-procevent-lib.sh fm-procevent-extension-capture.pl fm-procevent-lavish.sh \ fm-pr-lib.sh fm-wake-lib.sh fm-path-lib.sh fm-remote-entrypoint.sh fm-remote-job-lib.sh \ fm-remote-job-worker.sh; do cp "$ROOT/bin/$remote_file" "$REMOTE_ROOT/bin/$remote_file" diff --git a/tests/fm-gotmp.test.sh b/tests/fm-gotmp.test.sh index 7e0ced4bec5..4509f33b88d 100755 --- a/tests/fm-gotmp.test.sh +++ b/tests/fm-gotmp.test.sh @@ -60,6 +60,7 @@ SH ln -s "$ROOT/bin/fm-tmux-lib.sh" "$fake/bin/fm-tmux-lib.sh" ln -s "$ROOT/bin/fm-hermes-process-lib.sh" "$fake/bin/fm-hermes-process-lib.sh" ln -s "$ROOT/bin/fm-session-lock-lib.sh" "$fake/bin/fm-session-lock-lib.sh" + ln -s "$ROOT/bin/fm-omp-process-lib.sh" "$fake/bin/fm-omp-process-lib.sh" ln -s "$ROOT/bin/fm-agent-process-lib.sh" "$fake/bin/fm-agent-process-lib.sh" ln -s "$ROOT/bin/fm-gemini-lib.sh" "$fake/bin/fm-gemini-lib.sh" ln -s "$ROOT/bin/fm-cursor-lib.sh" "$fake/bin/fm-cursor-lib.sh" @@ -82,6 +83,12 @@ SH ln -s "$ROOT/bin/fm-gate-refuse-lib.sh" "$fake/bin/fm-gate-refuse-lib.sh" # fm-pr-lib.sh: teardown uses its canonical task-ID validator for poll cleanup. ln -s "$ROOT/bin/fm-pr-lib.sh" "$fake/bin/fm-pr-lib.sh" + # fm-dod-lib.sh (and the fm-brief-heading-lib.sh and fm-external-review-lib.sh + # it sources from its own directory): teardown sources it for the shared + # bounded remote-ref refresh before its landed-work check. + ln -s "$ROOT/bin/fm-dod-lib.sh" "$fake/bin/fm-dod-lib.sh" + ln -s "$ROOT/bin/fm-brief-heading-lib.sh" "$fake/bin/fm-brief-heading-lib.sh" + ln -s "$ROOT/bin/fm-external-review-lib.sh" "$fake/bin/fm-external-review-lib.sh" # fm-public-followup-lib.sh (and the fm-x-lib.sh and fm-env-lib.sh it # sources): teardown sources it for the relay-activation gate on the # promised-public-reply check. None does anything in this fixture, which has @@ -169,6 +176,7 @@ SH ln -s "$ROOT/bin/fm-tmux-lib.sh" "$fake/bin/fm-tmux-lib.sh" ln -s "$ROOT/bin/fm-hermes-process-lib.sh" "$fake/bin/fm-hermes-process-lib.sh" ln -s "$ROOT/bin/fm-session-lock-lib.sh" "$fake/bin/fm-session-lock-lib.sh" + ln -s "$ROOT/bin/fm-omp-process-lib.sh" "$fake/bin/fm-omp-process-lib.sh" ln -s "$ROOT/bin/fm-agent-process-lib.sh" "$fake/bin/fm-agent-process-lib.sh" ln -s "$ROOT/bin/fm-gemini-lib.sh" "$fake/bin/fm-gemini-lib.sh" ln -s "$ROOT/bin/fm-cursor-lib.sh" "$fake/bin/fm-cursor-lib.sh" @@ -188,6 +196,12 @@ SH ln -s "$ROOT/bin/fm-gate-refuse-lib.sh" "$fake/bin/fm-gate-refuse-lib.sh" # fm-pr-lib.sh: teardown uses its canonical task-ID validator for poll cleanup. ln -s "$ROOT/bin/fm-pr-lib.sh" "$fake/bin/fm-pr-lib.sh" + # fm-dod-lib.sh (and the fm-brief-heading-lib.sh and fm-external-review-lib.sh + # it sources from its own directory): teardown sources it for the shared + # bounded remote-ref refresh before its landed-work check. + ln -s "$ROOT/bin/fm-dod-lib.sh" "$fake/bin/fm-dod-lib.sh" + ln -s "$ROOT/bin/fm-brief-heading-lib.sh" "$fake/bin/fm-brief-heading-lib.sh" + ln -s "$ROOT/bin/fm-external-review-lib.sh" "$fake/bin/fm-external-review-lib.sh" # fm-public-followup-lib.sh (and the fm-x-lib.sh and fm-env-lib.sh it # sources): teardown sources it for the relay-activation gate on the # promised-public-reply check. None does anything in this fixture, which has diff --git a/tests/fm-herdr-lab.test.sh b/tests/fm-herdr-lab.test.sh index 37f2ff05910..4c3e9f80d96 100755 --- a/tests/fm-herdr-lab.test.sh +++ b/tests/fm-herdr-lab.test.sh @@ -6,6 +6,8 @@ set -u . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" unset FM_HERDR_LAB_FLEET_SESSION +export HOME=/home/test +unset XDG_CONFIG_HOME TMP_ROOT=$(fm_test_tmproot fm-herdr-lab) FAKEBIN=$(fm_fakebin "$TMP_ROOT") @@ -57,10 +59,23 @@ case "$1 ${2:-}" in fi ;; "server --session") + if [ -n "${FM_FAKE_HERDR_SERVER_NOISE:-}" ]; then + head -c "$FM_FAKE_HERDR_SERVER_NOISE" /dev/zero | tr '\0' x >&2 + exit 1 + fi + if [ -n "${FM_FAKE_HERDR_SERVER_FAIL:-}" ]; then + printf '%s\n' "$FM_FAKE_HERDR_SERVER_FAIL" >&2 + exit 1 + fi if [ "${FM_FAKE_HERDR_SERVER_DELAY:-0}" != 0 ]; then "$FM_FAKE_HERDR_REAL_SLEEP" "$FM_FAKE_HERDR_SERVER_DELAY" fi printf '%s\n' running > "$state/$session" + if [ -n "${FM_FAKE_HERDR_SERVER_STAY:-}" ]; then + # Stay alive after reporting running, like a real server. + printf '%s\n' "$$" > "$state/$session.pid" + exec "$FM_FAKE_HERDR_REAL_SLEEP" "$FM_FAKE_HERDR_SERVER_STAY" + fi ;; "status --json") if [ "$lab_state" = running ]; then @@ -104,6 +119,8 @@ run_with_fake() { FM_FAKE_HERDR_DELETE_FAIL="${FM_FAKE_HERDR_DELETE_FAIL:-}" \ FM_FAKE_HERDR_TITLE_FAIL="${FM_FAKE_HERDR_TITLE_FAIL:-}" \ FM_FAKE_HERDR_FLEET="${FM_FAKE_HERDR_FLEET:-}" \ + FM_FAKE_HERDR_SERVER_FAIL="${FM_FAKE_HERDR_SERVER_FAIL:-}" \ + FM_FAKE_HERDR_SERVER_STAY="${FM_FAKE_HERDR_SERVER_STAY:-}" \ FM_HERDR_LAB_STATE_DIR="$TRIPWIRES" \ "$@" } @@ -122,6 +139,79 @@ test_refuses_unsafe_names() { pass "fm-herdr-lab: names fail closed and require the lab prefix" } +test_refuses_names_over_the_socket_path_limit() { + local err="$TMP_ROOT/sun-path.err" status=0 name fixed + fixed="$HOME/.config/herdr/sessions//herdr-client.sock" + name=fm-lab-$(printf 'a%.0s' $(seq 1 $((103 - ${#fixed} - 7)))) + fm_herdr_lab_check_socket_path "$name" || fail "a name exactly at the sun_path limit was refused" + fm_herdr_lab_check_socket_path "${name}a" 2>"$err" && fail "a name over the sun_path limit was accepted" + assert_contains "$(cat "$err")" "103-byte sun_path limit" "the refusal must name the limit" + HOME=/home/test/$(printf 'c%.0s' $(seq 1 90)) fm_herdr_lab_name long-label >/dev/null 2>&1 || status=$? + expect_code 1 "$status" "name must refuse a label that overflows the socket path" + # Multibyte config root: fewer than 103 characters but more than 103 bytes. + local wide chars + wide=/home/test/$(printf 'é%.0s' $(seq 1 30)) + chars=$(LC_ALL=C.UTF-8 bash -c 'p=$1/.config/herdr/sessions/fm-lab-x/herdr-client.sock; echo ${#p}' _ "$wide") + [ "$chars" -lt 103 ] || fail "multibyte fixture is not under 103 characters: $chars" + HOME=$wide fm_herdr_lab_check_socket_path fm-lab-x 2>/dev/null \ + && fail "a path under 103 characters but over 103 bytes was accepted" + # XDG_CONFIG_HOME relocates the config root: long HOME + short XDG is accepted, + # short HOME + long XDG is refused. + HOME=/home/test/$(printf 'c%.0s' $(seq 1 90)) XDG_CONFIG_HOME=/x fm_herdr_lab_check_socket_path fm-lab-x 2>/dev/null \ + || fail "a short XDG_CONFIG_HOME was refused because of a long HOME" + XDG_CONFIG_HOME=/x/$(printf 'c%.0s' $(seq 1 90)) fm_herdr_lab_check_socket_path fm-lab-x 2>/dev/null \ + && fail "a long XDG_CONFIG_HOME was accepted" + status=0 + run_with_fake fm_herdr_lab_prepare "${name}a" >/dev/null 2>&1 || status=$? + expect_code 1 "$status" "prepare must refuse an over-limit name" + status=0 + run_with_fake fm_herdr_lab_provision "${name}a" >/dev/null 2>&1 || status=$? + expect_code 1 "$status" "provision must refuse an over-limit name" + [ ! -f "$FAKE_STATE/${name}a" ] || fail "provision started a server for an over-limit name" + pass "fm-herdr-lab: names whose client socket path exceeds sun_path are refused" +} + +test_noisy_server_log_is_bounded() { + local name="fm-lab-server-noisy-$$" size + FM_FAKE_HERDR_SERVER_NOISE=300000 \ + run_with_fake fm_herdr_lab_provision "$name" >/dev/null 2>&1 || true + size=$(wc -c < "$TRIPWIRES/$name.server.log" | tr -d ' ') + [ "$size" -le 4096 ] || fail "server log grew to $size bytes, over the 4096-byte cap" + [ "$size" -gt 0 ] || fail "server output was not captured" + run_with_fake fm_herdr_lab_teardown "$name" >/dev/null 2>&1 || true + pass "fm-herdr-lab: a noisy server leaves a bounded log" +} + +test_captured_provision_returns_promptly() { + local name="fm-lab-captured-$$" started elapsed pid + started=$(date +%s) + # The server outlives provision. Command substitution waits for every + # holder of the pipe to close it, so a collector that kept the caller's + # stdout open would block until the server exits. + : "$(FM_FAKE_HERDR_SERVER_STAY=20 run_with_fake fm_herdr_lab_provision "$name" 2>&1)" + elapsed=$(($(date +%s) - started)) + pid=$(cat "$FAKE_STATE/$name.pid" 2>/dev/null || true) + [ -z "$pid" ] || kill "$pid" 2>/dev/null || true + [ "$elapsed" -lt 10 ] || fail "captured provision held its output pipe open for ${elapsed}s" + run_with_fake fm_herdr_lab_teardown "$name" >/dev/null 2>&1 || true + pass "fm-herdr-lab: provision does not hold a captured output pipe open" +} + +test_failed_server_start_reports_its_error_quickly() { + local name="fm-lab-server-fails-$$" err="$TMP_ROOT/server-fail.err" status=0 started elapsed + started=$(date +%s) + FM_FAKE_HERDR_SERVER_FAIL="Error: local socket name length exceeds capacity" \ + run_with_fake fm_herdr_lab_provision "$name" >/dev/null 2>"$err" || status=$? + elapsed=$(($(date +%s) - started)) + expect_code 1 "$status" "provision must fail when the server exits" + [ "$elapsed" -lt 10 ] || fail "provision waited ${elapsed}s for a server that had already exited" + assert_contains "$(cat "$err")" "exited before reporting running" "the failure must say the server exited" + assert_contains "$(cat "$err")" "local socket name length exceeds capacity" "the failure must show the server's own error" + assert_present "$TRIPWIRES/$name.server.log" "the server output was not kept in the state directory" + run_with_fake fm_herdr_lab_teardown "$name" >/dev/null 2>&1 || true + pass "fm-herdr-lab: a server that fails at startup reports its error without the full wait" +} + test_provision_run_and_guarded_teardown() { local name='' line_count status=0 stop_line delete_line name="fm-lab-behavior-$$" @@ -633,6 +723,10 @@ test_viewer_launcher_refuses_unsafe_arguments() { } test_refuses_unsafe_names +test_refuses_names_over_the_socket_path_limit +test_failed_server_start_reports_its_error_quickly +test_noisy_server_log_is_bounded +test_captured_provision_returns_promptly test_provision_run_and_guarded_teardown test_run_scopes_session_before_double_dash test_missing_tripwire_blocks_destruction diff --git a/tests/fm-hermes-harness.test.sh b/tests/fm-hermes-harness.test.sh index aed79197dc3..91eb8992470 100755 --- a/tests/fm-hermes-harness.test.sh +++ b/tests/fm-hermes-harness.test.sh @@ -33,6 +33,7 @@ done for args in 'node /tmp/server.js /opt/hermes-agent/ui-tui/dist/entry.js' 'python3 -c tui_gateway.entry' 'node -e hermes' 'hermes-helper'; do ! fm_hermes_process_matches "$args" || fail "false identity: $args" done +! fm_hermes_process_matches '' || fail 'empty args matched under set -u' # Use a real interpreter process so kernel-name loss and argv loss are # independent regressions, not expectations encoded into a fake harness. mkdir -p "$TMP_ROOT/process/tui_gateway" diff --git a/tests/fm-home-summary-refresh.test.sh b/tests/fm-home-summary-refresh.test.sh index 38c92ccc58a..36e4ddb0d0c 100755 --- a/tests/fm-home-summary-refresh.test.sh +++ b/tests/fm-home-summary-refresh.test.sh @@ -673,6 +673,168 @@ jq -e --arg home "$COST_HOME" ' || fail "the accumulated home published a ledger missing its open decision" pass "publication completes on a home carrying accumulated status history" +# --- collection under the whole-operation deadline --------------------------- +# +# A live home of twelve scouts repeatedly hit the 60-second deadline while the +# producer was still collecting: after the bounded concurrent reads, every task +# row - including its whole-history open-decision fold - was composed one task +# at a time, so collection grew with task count times status history and a busy +# host pushed it past the bound before validation or publication. + +# seed_fleet_home <home> <task-count>: a home of in-flight ship tasks whose +# endpoints answer through the fixture tmux and whose status logs are written by +# the caller. +seed_fleet_home() { + local home=$1 count=$2 i id gen + mkdir -p "$home/state" "$home/data" "$home/config" "$home/projects/task" + printf '# Seeded Firstmate home\n' > "$home/AGENTS.md" + printf 'fleet\n' > "$home/.fm-secondmate-home" + fm_git_init_commit "$home/projects/task" + { + printf '## In flight\n' + i=1 + while [ "$i" -le "$count" ]; do + printf -- '- [ ] fleet-%s - Fleet task %s (repo: firstmate) (kind: ship) (since 2026-08-28)\n' "$i" "$i" + i=$((i + 1)) + done + printf '\n## Queued\n\n## Done\n' + } > "$home/data/backlog.md" + i=1 + while [ "$i" -le "$count" ]; do + id="fleet-$i" + fm_write_meta "$home/state/$id.meta" \ + "window=fmtest:fm-$id" \ + "worktree=$home/projects/task" \ + "project=firstmate" \ + "harness=claude" \ + "kind=ship" \ + "mode=no-mistakes" \ + "spawn_gen=fm.fleet$i" + gen=$("$ROOT/bin/fm-busy-event.sh" arm "$home/state" "$id") + "$ROOT/bin/fm-busy-event.sh" apply "$home/state" "$id" idle \ + --gen "$gen" --source claude-hook --event stop + printf 'needs-decision [key=gate-%s]: which base to rebuild from\n' "$i" \ + > "$home/state/$id.status" + i=$((i + 1)) + done +} + +# Stand in for each task row's composition cost at the one jq call that +# receives that task's current state: hang it, or hold every row until all of +# them are being composed at once. +COMPOSEBIN="$TMP_ROOT/composebin" +mkdir -p "$COMPOSEBIN" +cat > "$COMPOSEBIN/jq" <<'SH' +#!/usr/bin/env bash +prev= +for arg in "$@"; do + if [ "$prev" = --argjson ] && [ "$arg" = current_state ]; then + case "${FM_TEST_COMPOSE_MODE:-}" in + hang) sleep 30 ;; + rendezvous) + : > "$FM_TEST_COMPOSE_DIR/composing-$$" + i=0 + while [ ! -e "$FM_TEST_COMPOSE_DIR/overlap-proved" ] && [ "$i" -lt 600 ]; do + markers=("$FM_TEST_COMPOSE_DIR"/composing-*) + if [ "${#markers[@]}" -ge "$FM_TEST_COMPOSE_COUNT" ]; then + : > "$FM_TEST_COMPOSE_DIR/overlap-proved" + fi + sleep 0.05 + i=$((i + 1)) + done + printf '%s\n' "$$" >> "$FM_TEST_COMPOSE_DIR/composed" + ;; + esac + break + fi + prev=$arg +done +exec "$FM_TEST_REAL_JQ" "$@" +SH +chmod +x "$COMPOSEBIN/jq" + +FLEET_HOME="$TMP_ROOT/fleet-home" +seed_fleet_home "$FLEET_HOME" 4 +PATH="$FAKEBIN:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$FLEET_HOME" \ + FM_SNAPSHOT_NOW="$NOW_ONE" FM_SNAPSHOT_NOW_EPOCH="$EPOCH_ONE" \ + "$WRITER" || fail "the fleet home could not publish its first complete summary" +cp -p "$FLEET_HOME/state/home-summary.json" "$TMP_ROOT/fleet-last-good.json" + +# The operator's direct refresh: collection outlives the whole-operation +# deadline, so the refresh reports the deadline and the last complete summary +# stays exactly as it was. +refresh_rc=0 +PATH="$COMPOSEBIN:$FAKEBIN:$PATH" FM_TEST_REAL_JQ="$REAL_JQ" FM_TEST_COMPOSE_MODE=hang \ + FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$FLEET_HOME" FM_HOME_SUMMARY_TIMEOUT=3 \ + FM_SNAPSHOT_NOW="$NOW_TWO" FM_SNAPSHOT_NOW_EPOCH="$EPOCH_TWO" \ + "$WRITER" >/dev/null 2>"$TMP_ROOT/fleet-deadline.err" || refresh_rc=$? +[ "$refresh_rc" -eq 124 ] \ + || fail "a refresh whose collection outlived its deadline exited $refresh_rc, not 124" +grep -F 'refresh exceeded its 3-second deadline' "$TMP_ROOT/fleet-deadline.err" >/dev/null \ + || fail "the direct refresh did not report its deadline: $(cat "$TMP_ROOT/fleet-deadline.err")" +cmp -s "$TMP_ROOT/fleet-last-good.json" "$FLEET_HOME/state/home-summary.json" \ + || fail "a refresh killed while collecting changed the last complete summary" +pass "a refresh killed while collecting keeps the last complete summary" + +# Each task's row composition must run in its own concurrent observation worker. +# Every row waits here until all four are being composed at once, so a producer +# that composes rows one task at a time never proves the overlap and runs past +# the refresh's own unchanged deadline instead of publishing. +mkdir -p "$TMP_ROOT/compose" +PATH="$COMPOSEBIN:$FAKEBIN:$PATH" FM_TEST_REAL_JQ="$REAL_JQ" \ + FM_TEST_COMPOSE_MODE=rendezvous FM_TEST_COMPOSE_DIR="$TMP_ROOT/compose" \ + FM_TEST_COMPOSE_COUNT=4 FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$FLEET_HOME" \ + FM_SNAPSHOT_NOW="$NOW_THREE" FM_SNAPSHOT_NOW_EPOCH="$EPOCH_THREE" \ + "$WRITER" 2>"$TMP_ROOT/fleet-overlap.err" \ + || fail "per-task row composition did not overlap within the refresh deadline: $(cat "$TMP_ROOT/fleet-overlap.err")" +[ -e "$TMP_ROOT/compose/overlap-proved" ] \ + || fail "the four task rows were never composed at the same time" +[ "$(grep -c . "$TMP_ROOT/compose/composed")" -eq 4 ] \ + || fail "the composition stand-in did not see exactly one row per task" +jq -e --arg now "$NOW_THREE" ' + .generated == $now + and .counts.endpoints == 4 + and ([.decisions_open[].key] | sort) == ["gate-1","gate-2","gate-3","gate-4"] +' "$FLEET_HOME/state/home-summary.json" >/dev/null \ + || fail "the overlapped refresh did not publish every task row and open decision" +pass "task rows are composed concurrently and publish within the refresh deadline" + +# The incident's shape end to end: twelve tasks whose append-only histories +# carry hundreds of decision transitions each must publish inside the default +# whole-operation deadline with every still-open decision intact. +INCIDENT_HOME="$TMP_ROOT/incident-home" +seed_fleet_home "$INCIDENT_HOME" 12 +python3 - "$INCIDENT_HOME/state" 12 <<'PY' +import sys +state, count = sys.argv[1], int(sys.argv[2]) +note = "checked the rendered prototype against the reviewed evidence " +for task in range(1, count + 1): + with open(f"{state}/fleet-{task}.status", "w") as handle: + for i in range(120): + stamp = 1791174638 + i * 60 + handle.write(f"working [at={stamp}]: {note * 2}({i})\n") + handle.write(f"paused [at={stamp}]: waiting on review round {i}\n") + handle.write(f"blocked [at={stamp}] [key=wait-{i % 7}]: {note}\n") + handle.write(f"resolved [at={stamp}] [key=wait-{i % 7}]: cleared by the answer\n") + if i % 10 == 0: + handle.write(f"needs-decision [at={stamp}] [key=pick-{i}]: which variant to keep\n") + handle.write(f"captain-held [at={stamp}] [key=pick-{i}]: held for the captain\n") + if i % 40 == 0: + handle.write(f"done [at={stamp}]: interim deliverable {i}\n") + handle.write(f"needs-decision [key=gate-{task}]: which base to rebuild from\n") +PY +PATH="$FAKEBIN:$PATH" FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$INCIDENT_HOME" \ + FM_SNAPSHOT_NOW="$NOW_ONE" FM_SNAPSHOT_NOW_EPOCH="$EPOCH_ONE" \ + "$WRITER" 2>"$TMP_ROOT/incident.err" \ + || fail "the incident-shaped fleet did not publish within the default deadline: $(cat "$TMP_ROOT/incident.err")" +jq -e ' + .counts.endpoints == 12 + and ([.decisions_open[].key] | sort) + == (["gate-1","gate-2","gate-3","gate-4","gate-5","gate-6","gate-7","gate-8","gate-9","gate-10","gate-11","gate-12"] | sort) +' "$INCIDENT_HOME/state/home-summary.json" >/dev/null \ + || fail "the incident-shaped fleet published without every task's open decision" +pass "an incident-shaped fleet publishes inside the default deadline" + # One unreachable home must not extend publication without limit. A remote # secondmate's current state is read over ssh, and ssh's own dead-peer detection # deliberately never kills a slow-but-alive remote command, so nothing under the diff --git a/tests/fm-inactive-reconcile.test.sh b/tests/fm-inactive-reconcile.test.sh index 54f38c3d973..08d4f244829 100755 --- a/tests/fm-inactive-reconcile.test.sh +++ b/tests/fm-inactive-reconcile.test.sh @@ -529,6 +529,65 @@ test_secondmate_ledger_delivery_carries_report_and_failure() { pass "ledger delivery carries the report pointer, the failed verb, and each new terminal line" } +# Children whose ready lines differ only in what config/pr-register-scan opts +# into: a ready line after one naming the recorded PR, a ready line beside a +# recorded PR the log never named, a GitLab ready line after one naming the +# recorded PR, and a GitLab ready line with no PR recorded. +write_replacement_children() { + write_child "$MATE" replaced $'done: PR https://example.test/owner/repo/pull/1 checks green\nworking: reopening\ndone: PR https://example.test/owner/repo/pull/22 checks green' + write_child "$MATE" kept 'done: PR https://example.test/owner/repo/pull/23 checks green' + write_child "$MATE" gitlab $'done: PR https://example.test/owner/repo/pull/1 checks green\ndone: PR https://gitlab.example/group/sub/project/-/merge_requests/8 checks green' + write_child "$MATE" unrecorded 'done: PR https://gitlab.example/group/sub/project/-/merge_requests/9 checks green' + awk '$0 !~ /^pr=/' "$MATE/state/unrecorded.meta" > "$MATE/state/unrecorded.meta.tmp" + mv "$MATE/state/unrecorded.meta.tmp" "$MATE/state/unrecorded.meta" +} + +# The parent-channel line delivered for <child>, without its emission stamp. +assert_child_report() { # <child> <expected line after the key> <failure> + local key + key=$(reported_outcome_key "$MATE" "$1" 'done') || fail "$1 receipt key missing" + sed -E 's/ \[at=[0-9]+\]//' "$MAIN/state/mate.status" | grep -Fxq "done [key=$key]: $2" \ + || fail "$3: $(cat "$MAIN/state/mate.status")" +} + +# With config/pr-register-scan, the child's report names the PR its newest +# ready line delivered once the log shows that line replacing the recorded PR, +# including a GitLab ready line; a recorded PR the log never named is the +# supervisor's own choice and stays. +test_ledger_report_names_a_replacement_pr() { + make_world ledger-replacement; bind_secondmate local + : > "$MATE/config/pr-register-scan" + write_replacement_children + FM_FAKE_CREW_STATE='unknown' run_reconcile "$MATE" + assert_child_report replaced "child replaced done: PR https://example.test/owner/repo/pull/22 checks green pr=https://example.test/owner/repo/pull/22 mode=no-mistakes yolo=off" \ + "the report kept the replaced PR" + assert_child_report kept "child kept done: PR https://example.test/owner/repo/pull/23 checks green pr=https://example.test/owner/repo/pull/1 mode=no-mistakes yolo=off" \ + "the report overrode a PR the log never named" + assert_child_report gitlab "child gitlab done: PR https://gitlab.example/group/sub/project/-/merge_requests/8 checks green pr=https://gitlab.example/group/sub/project/-/merge_requests/8 mode=no-mistakes yolo=off" \ + "the report kept the recorded PR instead of the GitLab replacement" + assert_child_report unrecorded "child unrecorded done: PR https://gitlab.example/group/sub/project/-/merge_requests/9 checks green pr=https://gitlab.example/group/sub/project/-/merge_requests/9 mode=no-mistakes yolo=off" \ + "the report dropped an unrecorded GitLab ready PR" + pass "with the scan opt-in, a child report names a replacement or forge PR and keeps a recorded PR the log never named" +} + +# Without config/pr-register-scan, the same children report exactly what main +# reports: a recorded PR always stands, and only a GitHub-shaped ready line +# fills in a missing one. +test_ledger_report_without_scan_opt_in_keeps_recorded_pr() { + make_world ledger-no-replacement; bind_secondmate local + write_replacement_children + FM_FAKE_CREW_STATE='unknown' run_reconcile "$MATE" + assert_child_report replaced "child replaced done: PR https://example.test/owner/repo/pull/22 checks green pr=https://example.test/owner/repo/pull/1 mode=no-mistakes yolo=off" \ + "a newer ready line replaced the recorded PR without the opt-in" + assert_child_report kept "child kept done: PR https://example.test/owner/repo/pull/23 checks green pr=https://example.test/owner/repo/pull/1 mode=no-mistakes yolo=off" \ + "the report overrode the recorded PR without the opt-in" + assert_child_report gitlab "child gitlab done: PR https://gitlab.example/group/sub/project/-/merge_requests/8 checks green pr=https://example.test/owner/repo/pull/1 mode=no-mistakes yolo=off" \ + "a GitLab ready line replaced the recorded PR without the opt-in" + assert_child_report unrecorded "child unrecorded done: PR https://gitlab.example/group/sub/project/-/merge_requests/9 checks green mode=no-mistakes yolo=off" \ + "a GitLab ready line carried a PR without the opt-in" + pass "without the scan opt-in, a child report keeps the recorded PR and accepts only a GitHub-shaped ready line" +} + # A PR URL a worker only ever mentioned in prose is never claimed as the # task's delivered PR: without a recorded PR, only a terminal line in the # ready-signal shape carries one, and a scout never carries one at all. @@ -1510,6 +1569,8 @@ test_reconciliation_never_calls_forge test_reconciliation_sets_no_forge_mode_for_state_read test_terminal_outcome_never_names_a_pr_the_run_did_not_open test_terminal_outcome_takes_the_pr_its_run_opened +test_ledger_report_names_a_replacement_pr +test_ledger_report_without_scan_opt_in_keeps_recorded_pr test_status_log_sourced_outcome_keeps_its_own_pr test_status_log_outcome_keeps_gitlab_and_gerrit_reviews test_status_log_outcome_never_borrows_an_earlier_pr diff --git a/tests/fm-omp-harness.test.sh b/tests/fm-omp-harness.test.sh index 436dc7195d3..1aca5263d30 100755 --- a/tests/fm-omp-harness.test.sh +++ b/tests/fm-omp-harness.test.sh @@ -46,6 +46,7 @@ set -u HARNESS="$ROOT/bin/fm-harness.sh" TMP_ROOT=$(fm_test_tmproot fm-omp-harness) +fm_git_identity fmtest fmtest@example.invalid export NODE_NO_WARNINGS=1 # A process whose kernel-recorded identity is the bare name `omp`: a SYMLINK to @@ -117,6 +118,83 @@ test_lock_identity_and_liveness_classification() { pass "session lock and tmux liveness: omp is anchored, decoys stay out" } +# The bun global package install: `~/.bun/bin/omp` is a `#!/usr/bin/env bun` +# script, so the process is named `bun` and only the argument right after bun +# says omp. Shells named `bun` run script files laid out like that install. +make_bun_shapes() { # <dir> -> echoes <dir> + local dir=$1 script + mkdir -p "$dir/bin" "$dir/scripts/Agent Tools/bin" + ln -sf /bin/bash "$dir/bin/bun" + ln -sf /bin/bash "$dir/bin/claude" + for script in omp ompd comp prompt.sh "Agent Tools/bin/omp" "Agent Tools/bin/ompd"; do + # shellcheck disable=SC2016 # the body expands inside the script that runs it + printf '%s\n' 'printf "%s\n" "$$" > "$1"; shift; "$@"; :' > "$dir/scripts/$script" + done + printf '%s' "$dir" +} + +# Runs <claude shell> -> <bun shell running a script> -> probe, and records what +# the probe saw. The claude shell is the nearest other harness, so a shape that +# is not omp resolves there instead of walking into the harness running this suite. +run_bun_shape() { # <dir> <name> <script>: the script path is $dir/scripts/<script> + local dir=$1 name=$2 script=$3 home + home="$dir/home-$name" + mkdir -p "$home/state" + git init -q "$home" + git -C "$home" commit -q --allow-empty -m init + : > "$home/AGENTS.md" + cat > "$dir/probe-$name.sh" <<SH +printf '%s\n' "\$(env -u CLAUDECODE -u FM_OMP_HARNESS -u PI_CODING_AGENT -u CURSOR_AGENT -u CURSOR_INVOKED_AS \\ + '$HARNESS')" > '$home/harness.out' +env -u CLAUDE_PID -u CLAUDE_CODE_SESSION_ID FM_HOME='$home' '$ROOT/bin/fm-lock.sh' > '$home/lock.out' 2>&1 +printf '%s\n' "\$?" > '$home/lock.rc' +SH + # shellcheck disable=SC2016 # the quoted body expands inside the claude shell + env -u CLAUDECODE -u FM_OMP_HARNESS -u PI_CODING_AGENT -u CURSOR_AGENT -u CURSOR_INVOKED_AS \ + "$dir/bin/claude" -c 'printf "%s\n" "$$" > "$1"; shift; "$@"; :' _ "$home/claude.pid" \ + "$dir/bin/bun" "$dir/scripts/$script" "$home/shape.pid" bash "$dir/probe-$name.sh" + printf '%s' "$home" +} + +test_bun_package_omp_is_identified_and_owns_the_lock() { + local dir home shape + dir=$(make_bun_shapes "$TMP_ROOT/bun-shapes") + for shape in omp "Agent Tools/bin/omp"; do + home=$(run_bun_shape "$dir" "pos-${shape//[\/ ]/_}" "$shape") + [ "$(cat "$home/harness.out")" = omp ] || fail "bun running $shape must detect as omp, got '$(cat "$home/harness.out")'" + [ "$(cat "$home/lock.rc")" = 0 ] || fail "bun running $shape must let the session lock be acquired: $(cat "$home/lock.out")" + [ "$(sed -n 1p "$home/state/.lock")" = "$(cat "$home/shape.pid")" ] \ + || fail "the lock for bun running $shape must name the bun-run omp process, got '$(sed -n 1p "$home/state/.lock")'" + done + for shape in ompd comp prompt.sh "Agent Tools/bin/ompd"; do + home=$(run_bun_shape "$dir" "neg-${shape//[\/ ]/_}" "$shape") + [ "$(cat "$home/harness.out")" = claude ] || fail "bun running $shape must not detect as omp, got '$(cat "$home/harness.out")'" + [ "$(sed -n 1p "$home/state/.lock")" = "$(cat "$home/claude.pid")" ] \ + || fail "the lock for bun running $shape must not name the bun process" + done + pass "fm-harness and fm-lock: a bun-run omp script owns the session; ompd, comp, and other scripts do not" +} + +test_bun_package_omp_identity_unit_shapes() { + fm_omp_process_matches bun 'bun /home/u/.bun/bin/omp' || fail "bun omp launcher must match" + fm_omp_process_matches /opt/homebrew/bin/bun '/opt/homebrew/bin/bun /x/omp --cwd /y' || fail "omp then arguments must match" + ! fm_omp_process_matches bun 'bun /x/ompd' || fail "ompd must not match" + ! fm_omp_process_matches bun 'bun /x/comp' || fail "comp must not match" + ! fm_omp_process_matches bun 'bun /x/script.js omp' || fail "a later omp argument must not match" + local spaced="$TMP_ROOT/spaced omp/bin" + mkdir -p "$spaced" + : > "$spaced/omp" + : > "$spaced/ompd" + fm_omp_process_matches bun "bun $spaced/omp" || fail "a script path with spaces must match" + fm_omp_process_matches bun "bun $spaced/omp --cwd /y" || fail "a spaced path then arguments must match" + ! fm_omp_process_matches bun "bun $spaced/ompd" || fail "a spaced ompd path must not match" + : > "$TMP_ROOT/other.js" + ! fm_omp_process_matches bun "bun $TMP_ROOT/other.js $spaced/omp" || fail "an existing script followed by a spaced omp argument must not match" + ! fm_omp_process_matches bun 'bun' || fail "bare bun must not match" + ! fm_omp_process_matches node 'node /x/omp' || fail "only bun may run the omp script" + pass "fm_omp_process_matches: the bun shape is anchored on the argument right after bun" +} + # --- 2. Launch --------------------------------------------------------------- # A fake omp that answers `models --json` with a two-provider catalog and exits @@ -977,6 +1055,8 @@ EOF test_detection_anchored_name_and_marker_precedence test_lock_identity_and_liveness_classification +test_bun_package_omp_identity_unit_shapes +test_bun_package_omp_is_identified_and_owns_the_lock test_spawn_launch_line_and_worker_wiring test_spawn_model_validation_scoped_to_listed_providers test_secondmate_launch_relies_on_discovery diff --git a/tests/fm-omp-primary-live-e2e.test.sh b/tests/fm-omp-primary-live-e2e.test.sh index 333c9b27c37..bf7d103f4e9 100755 --- a/tests/fm-omp-primary-live-e2e.test.sh +++ b/tests/fm-omp-primary-live-e2e.test.sh @@ -1,5 +1,15 @@ #!/usr/bin/env bash -# Opt-in credentialed omp (Oh My Pi) primary regression in an isolated lab +# Token-free omp (Oh My Pi) process-identity probe plus an opt-in credentialed +# omp primary regression, both in isolated labs. +# +# The token-free half starts the installed omp in JSON-RPC mode with no prompt, +# so it spends no model tokens, and has a loaded extension run bin/fm-harness.sh +# and bin/fm-lock.sh from inside that process. It proves the installed omp, +# whether the compiled binary or the bun global package (process name `bun`), +# is detected as omp and can own the session lock. +# FM_OMP_LIVE_E2E=1 selects the credentialed half below instead. +# +# The credentialed half is an omp (Oh My Pi) primary regression in an isolated lab # checkout. It drives a real omp through its JSON-RPC stdio mode so no terminal # multiplexer is needed, uses the captain's existing omp login without copying # any credential, and defaults to the captain-approved openai-codex model. @@ -19,6 +29,60 @@ set -u # shellcheck source=tests/lib.sh . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +if [ "${FM_OMP_LIVE_E2E:-0}" != 1 ]; then + fm_live_gate default-on FM_OMP_IDENTITY_LIVE omp git + ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + fm_git_identity fmtest fmtest@example.invalid + OMP_BIN=$(command -v omp) + OMP_VERSION=$(omp --version 2>/dev/null | head -1) + case "$(head -c 64 "$OMP_BIN" 2>/dev/null | head -1)" in + '#!'*bun*) INSTALL=bun-package ;; + *) INSTALL=compiled-binary ;; + esac + LAB=$(fm_test_tmproot fm-omp-identity-live) + HOME_DIR="$LAB/home" + mkdir -p "$HOME_DIR/state" + git init -q "$HOME_DIR" + git -C "$HOME_DIR" commit -q --allow-empty -m init + : > "$HOME_DIR/AGENTS.md" + cat > "$LAB/probe.sh" <<SH +env -u CLAUDECODE -u FM_OMP_HARNESS -u PI_CODING_AGENT -u CURSOR_AGENT -u CURSOR_INVOKED_AS \\ + '$ROOT/bin/fm-harness.sh' > '$LAB/harness.out' 2>&1 +env -u CLAUDE_PID -u CLAUDE_CODE_SESSION_ID FM_HOME='$HOME_DIR' '$ROOT/bin/fm-lock.sh' > '$LAB/lock.out' 2>&1 +printf '%s\\n' "\$?" > '$LAB/lock.rc' +SH + cat > "$LAB/probe.ts" <<TS +import { spawnSync } from "node:child_process"; +import { writeFileSync } from "node:fs"; +export default function () { + writeFileSync("$LAB/omp.pid", String(process.pid)); + spawnSync("bash", ["$LAB/probe.sh"], { stdio: "ignore" }); +} +TS + mkfifo "$LAB/rpc.in" + (cd "$LAB" && omp --mode rpc --no-session --cwd "$LAB" -e "$LAB/probe.ts" < "$LAB/rpc.in" > "$LAB/rpc.log" 2> "$LAB/rpc.err") & + OMP_PID=$! + exec 3> "$LAB/rpc.in" + i=0 + while [ "$i" -lt 240 ] && [ ! -s "$LAB/lock.rc" ]; do + sleep 0.5 + i=$((i + 1)) + done + exec 3>&- + sleep 1 + kill -TERM "$OMP_PID" 2>/dev/null || true + wait "$OMP_PID" 2>/dev/null || true + [ -s "$LAB/lock.rc" ] || { printf 'not ok - omp %s (%s) never ran the identity probe: %s\n' "$OMP_VERSION" "$INSTALL" "$(tail -3 "$LAB/rpc.err")" >&2; exit 1; } + [ "$(cat "$LAB/harness.out")" = omp ] \ + || { printf 'not ok - omp %s (%s) was detected as %s, not omp\n' "$OMP_VERSION" "$INSTALL" "$(cat "$LAB/harness.out")" >&2; exit 1; } + [ "$(cat "$LAB/lock.rc")" = 0 ] \ + || { printf 'not ok - omp %s (%s) could not own the session lock: %s\n' "$OMP_VERSION" "$INSTALL" "$(cat "$LAB/lock.out")" >&2; exit 1; } + [ "$(sed -n 1p "$HOME_DIR/state/.lock")" = "$(cat "$LAB/omp.pid" 2>/dev/null)" ] \ + || { printf 'not ok - omp %s (%s) acquired the session lock for pid %s, not the omp process %s\n' "$OMP_VERSION" "$INSTALL" "$(sed -n 1p "$HOME_DIR/state/.lock")" "$(cat "$LAB/omp.pid" 2>/dev/null)" >&2; exit 1; } + printf 'ok - omp %s (%s): detected as omp and acquired the session lock with no model tokens\n' "$OMP_VERSION" "$INSTALL" + exit 0 +fi + fm_live_gate opt-in FM_OMP_LIVE_E2E omp node jq ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" diff --git a/tests/fm-parent-channel-scan-exclusion.test.sh b/tests/fm-parent-channel-scan-exclusion.test.sh index 7d8a580a4c6..80a7376480f 100755 --- a/tests/fm-parent-channel-scan-exclusion.test.sh +++ b/tests/fm-parent-channel-scan-exclusion.test.sh @@ -289,6 +289,51 @@ test_watcher_scan_keeps_name_shared_files_outside_remote_mates() { pass "scan_signals keeps parent-replies.status outside remote mate homes" } +# record_hygiene_run <home> <detect-only>: run the record hygiene report over a +# home's own records, report-only when <detect-only> is 1 and with the orphan +# archive move otherwise. +record_hygiene_run() { # <home> <detect-only> + FM_TEST_LIB_SOURCED=1 FM_HOME="$1" FM_STATE_OVERRIDE="$1/state" \ + FM_BOOTSTRAP_DETECT_ONLY="$2" bash -c ' + SCRIPT_DIR="$1/bin"; STATE="$2/state"; DATA="$2/data" + # shellcheck source=bin/fm-classify-lib.sh + . "$SCRIPT_DIR/fm-classify-lib.sh" + # shellcheck source=bin/fm-tasks-axi-lib.sh + . "$SCRIPT_DIR/fm-tasks-axi-lib.sh" + # shellcheck source=bin/fm-backlog-transition-lib.sh + . "$SCRIPT_DIR/fm-backlog-transition-lib.sh" + # shellcheck source=bin/fm-backend.sh + . "$SCRIPT_DIR/fm-backend.sh" + # shellcheck source=bin/fm-timeout-lib.sh + . "$SCRIPT_DIR/fm-timeout-lib.sh" + # shellcheck source=bin/fm-record-hygiene-lib.sh + . "$SCRIPT_DIR/fm-record-hygiene-lib.sh" + record_hygiene_report + ' _ "$ROOT" "$1" +} + +test_record_hygiene_skips_remote_parent_channel() { + local dir out + dir="$TMP_ROOT/record-hygiene" + seed_remote_mate "$dir/home" + touch -t 202001010000 "$dir/home/state/parent-replies.status" "$dir/home/state/real-task.status" + mkdir -p "$dir/home/data" + out=$(record_hygiene_run "$dir/home" 1) || fail "report-only record hygiene scan failed" + case "$out" in *parent-replies*) fail "remote parent channel was reported: $out" ;; esac + printf '%s\n' "$out" | grep -F 'older than 24 hours: real-task' >/dev/null \ + || fail "the genuine orphan status log must be reported: $out" + [ -f "$dir/home/state/real-task.status" ] || fail "the report-only scan must not archive" + + # The mutating pass archives the genuine orphan but never the channel. + out=$(record_hygiene_run "$dir/home" 0) || fail "mutating record hygiene scan failed" + case "$out" in *parent-replies*) fail "remote parent channel was reported: $out" ;; esac + [ -f "$dir/home/state/parent-replies.status" ] || fail "remote parent channel was archived" + [ ! -e "$dir/home/data/parent-replies" ] || fail "remote parent channel left an archive dir" + [ -f "$dir/home/data/real-task/status.log" ] && [ ! -e "$dir/home/state/real-task.status" ] \ + || fail "the genuine orphan status log must be archived by the mutating pass" + pass "record hygiene reports and archives orphans but leaves a remote mate's parent channel in place" +} + # --- unit: the away-mode daemon's heartbeat catch-all backstop -------------- # The daemon runs the watcher's twin catch-all scan while a home is away, so it @@ -406,6 +451,7 @@ test_channel_lines_never_reach_the_remote_unread_surface test_name_shared_file_folds_in_a_main_home test_name_shared_file_folds_in_a_local_mate test_watcher_scan_skips_channel_and_keeps_task_in_remote_mate +test_record_hygiene_skips_remote_parent_channel test_heartbeat_backstop_skips_channel_in_remote_mate test_watcher_scan_keeps_name_shared_files_outside_remote_mates test_daemon_heartbeat_backstop_skips_channel_in_remote_mate diff --git a/tests/fm-park.test.sh b/tests/fm-park.test.sh new file mode 100755 index 00000000000..78922acc68d --- /dev/null +++ b/tests/fm-park.test.sh @@ -0,0 +1,1238 @@ +#!/usr/bin/env bash +# Tests for bin/fm-park.sh: the parked-on-review gate, park transition, +# working-copy protection, relaunch triggers, merge-time cleanup, and the +# watcher-facing tick. +# +# The forges are faked through gh and glab stubs that serve recorded JSON, and +# the lifecycle owners (bin/fm-control.sh, bin/fm-teardown.sh, +# bin/fm-crew-state.sh) through the script's documented FM_PARK_*_BIN +# overrides, so each case drives the real decision procedure end to end and +# asserts what it did to the lane: its record, its status log, and the exact +# lifecycle calls it made. +set -u + +# shellcheck source=tests/lib.sh disable=SC1091 +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +fm_git_identity fmtest fmtest@example.invalid + +PARK="$ROOT/bin/fm-park.sh" +TMP_ROOT=$(fm_test_tmproot fm-park-tests) +GH_URL=https://github.com/example/repo/pull/7 +GL_URL=https://gitlab.example.com/team/sub/repo/-/merge_requests/7 +NOW_ISO=$(date -u +%Y-%m-%dT%H:%M:%SZ) +OLD_ISO=2026-01-01T00:00:00Z + +# A home with one ship lane: a project, a Treehouse-shaped pool slot holding +# the lane's copy at a pushed head, an idle Claude busy record, an armed merge +# poll, and stubs for every external tool. Echoes the world dir; the copy's +# HEAD is in $WORLD/head. +make_world() { + local name=$1 url=$2 world fakebin wt head + world="$TMP_ROOT/$name" + fakebin="$world/fakebin" + mkdir -p "$world/state" "$world/config" "$fakebin" "$world/root/.treehouse/repo-1/3" + git init -q -b main "$world/project" + git -C "$world/project" commit -q --allow-empty -m base + printf '{}\n' > "$world/root/.treehouse/repo-1/treehouse-state.json" + wt="$world/root/.treehouse/repo-1/3/repo" + git -C "$world/project" worktree add -q -b fm/lane "$wt" main + printf 'work\n' > "$wt/work.txt" + git -C "$wt" add work.txt + git -C "$wt" commit -q -m work + head=$(git -C "$wt" rev-parse HEAD) + printf '%s\n' "$head" > "$world/head" + fm_write_meta "$world/state/lane.meta" \ + "window=fm:fm-lane" "endpoint_task_id=lane" "worktree=$wt" \ + "project=$world/project" "kind=ship" "mode=no-mistakes" "harness=claude" \ + "branch=fm/lane" "pr=$url" + printf 'gen1\n' > "$world/state/lane.busy-gen" + printf 'v1 gen=gen1 seq=3 state=idle source=claude-hook event=Stop ts=%s\n' "$(date +%s)" \ + > "$world/state/lane.busy-state" + printf '#!/bin/sh\nexit 0\n' > "$world/state/lane.check.sh" + printf 'done [at=1]: PR %s checks green\n' "$url" > "$world/state/lane.status" + + cat > "$fakebin/gh" <<SH +#!/usr/bin/env bash +# The base branch's required checks: gh.branch.json (unprotected by default, +# gh.branch.fail fails the read), with no rulesets. +case "\$*" in + *"/rules/branches/"*) echo '[]'; exit 0 ;; + *"/branches/"*) [ ! -e "$world/gh.branch.fail" ] || exit 1; cat "$world/gh.branch.json" 2>/dev/null || echo '{"protected":false}'; exit 0 ;; +esac +# Serves gh.<n>.json for the n-th GraphQL read, repeating the last one. +n=\$(( \$(cat "$world/gh.count" 2>/dev/null || echo 0) + 1 )) +printf '%s\n' "\$n" > "$world/gh.count" +[ ! -f "$world/gh.\$n.sh" ] || sh "$world/gh.\$n.sh" +f="$world/gh.\$n.json" +[ -f "\$f" ] || f=\$(ls "$world"/gh.*.json | sort -t. -k2 -n | tail -1) +case "\$*" in + *--paginate*) + pages="$world/gh.\$n.pages.json" + if [ -f "\$pages" ]; then cat "\$pages"; else jq -s . "\$f"; fi + ;; + *) cat "\$f" ;; +esac +SH + cat > "$fakebin/glab" <<SH +#!/usr/bin/env bash +printf '%s\n' "\$*" >> "$world/glab.log" +case "\$*" in + *"/discussions"*) + cat "$world/gl.discussions.json" + case "\$*" in + *--paginate*) + [ ! -f "$world/gl.discussions.2.json" ] || cat "$world/gl.discussions.2.json" + [ ! -e "$world/gl.discussions.fail" ] || exit 1 + ;; + esac + ;; + *) [ ! -e "$world/gl.mr.fail" ] || exit 1; cat "$world/gl.mr.json" ;; +esac +SH + cat > "$fakebin/treehouse" <<SH +#!/usr/bin/env bash +printf '%s\n' "\$*" >> "$world/treehouse.log" +if [ "\${1:-} \${2:-}" = "status --json" ]; then cat "$world/treehouse.status.json" 2>/dev/null || echo '[]'; exit 0; fi +[ "\${1:-}" = lease ] || exit 1 +[ ! -e "$world/treehouse.fail" ] || exit 1 +printf '%s\n' "$wt" +SH + cat > "$fakebin/fm-control.sh" <<SH +#!/usr/bin/env bash +printf '%s\n' "\$*" >> "$world/control.log" +if [ "\${2:-}" = relaunch ] && [ "\${3:-}" = --note-file ]; then cat "\$4" >> "$world/control.log"; fi +[ ! -e "$world/control.fail" ] || { echo "error: control refused"; exit 1; } +SH + cat > "$fakebin/fm-teardown.sh" <<SH +#!/usr/bin/env bash +printf '%s\n' "\$*" >> "$world/teardown.log" +if [ -e "$world/teardown.refuse" ]; then echo "REFUSED: worktree has work not on any remote and not landed." >&2; exit 1; fi +echo "Torn down \$1" +SH + # The lane's tmux window: no server by default (the agent is dead), and a + # live claude in the foreground while $world/agent.alive exists. + cat > "$fakebin/tmux" <<SH +#!/usr/bin/env bash +[ ! -e "$world/tmux.hang" ] || sleep 30 +[ ! -e "$world/tmux.unreadable" ] || { echo "lost server" >&2; exit 1; } +[ -e "$world/agent.alive" ] || { echo "no server running on /tmp/tmux-fake/default" >&2; exit 1; } +case "\$1" in + list-windows) echo fm-lane ;; + display-message) echo /dev/ttysfake ;; + *) exit 1 ;; +esac +SH + cat > "$fakebin/ps" <<SH +#!/usr/bin/env bash +case " \$* " in *" -t ttysfake "*) echo "1 1 1 claude"; exit 0 ;; esac +exec /bin/ps "\$@" +SH + cat > "$fakebin/fm-crew-state.sh" <<SH +#!/usr/bin/env bash +cat "$world/crew-state" 2>/dev/null || echo 'state: done · source: status-log · PR ready' +SH + chmod +x "$fakebin"/* + printf '%s\n' "$world" +} + +run_park() { # <world> <args...> + local world=$1 + shift + if [ "${FM_TEST_EXEC_PARK:-0}" = 1 ]; then + exec env \ + FM_HOME="$world" FM_STATE_OVERRIDE="$world/state" FM_CONFIG_OVERRIDE="$world/config" \ + FM_PARK_CONTROL_BIN="$world/fakebin/fm-control.sh" \ + FM_PARK_TEARDOWN_BIN="$world/fakebin/fm-teardown.sh" \ + FM_PARK_CREW_STATE_BIN="$world/fakebin/fm-crew-state.sh" \ + FM_PARK_QUIET_SECS="${FM_TEST_QUIET:-900}" \ + PATH="$world/fakebin:$PATH" "$PARK" "$@" + fi + FM_HOME="$world" FM_STATE_OVERRIDE="$world/state" FM_CONFIG_OVERRIDE="$world/config" \ + FM_PARK_CONTROL_BIN="$world/fakebin/fm-control.sh" \ + FM_PARK_TEARDOWN_BIN="$world/fakebin/fm-teardown.sh" \ + FM_PARK_CREW_STATE_BIN="$world/fakebin/fm-crew-state.sh" \ + FM_PARK_QUIET_SECS="${FM_TEST_QUIET:-900}" \ + PATH="$world/fakebin:$PATH" "$PARK" "$@" +} + +# GitHub GraphQL fixture. Fields default to a clean PR whose only reviewer bot +# reviewed the current head; key=value arguments override them. The head's one +# check suite was created at pushed= (default: the committer time; none: no +# check suite). +gh_fixture() { # <out> [key=value...] + local out=$1 kv + shift + local state=OPEN draft=false head mergeable=MERGEABLE mss=BLOCKED decision=REVIEW_REQUIRED + local rollup='"SUCCESS"' completed=$OLD_ISO committed=$OLD_ISO threads='[]' total=0 + local review_oid bot_at=$OLD_ISO comments='[]' extra_reviews='' pushed='' comment_count review_count + head=$(cat "$(dirname "$out")/head") + review_oid=$head + for kv in "$@"; do + case "$kv" in + state=*) state=${kv#*=} ;; draft=*) draft=${kv#*=} ;; head=*) head=${kv#*=} ;; + mergeable=*) mergeable=${kv#*=} ;; mss=*) mss=${kv#*=} ;; decision=*) decision=${kv#*=} ;; + rollup=*) rollup=${kv#*=} ;; completed=*) completed=${kv#*=} ;; committed=*) committed=${kv#*=} ;; + threads=*) threads=${kv#*=} ;; total=*) total=${kv#*=} ;; review_oid=*) review_oid=${kv#*=} ;; + bot_at=*) bot_at=${kv#*=} ;; comments=*) comments=${kv#*=} ;; extra_reviews=*) extra_reviews=${kv#*=} ;; + pushed=*) pushed=${kv#*=} ;; + esac + done + comment_count=$(printf '%s' "$comments" | jq 'length') + review_count=$(( 1 + $(printf '%s' "[${extra_reviews#,}]" | jq 'length') )) + threads=$(printf '%s' "$threads" | jq 'map(.comments.totalCount = (.comments.nodes | length))') + local suites="[{\"createdAt\":\"${pushed:-$committed}\"}]" + [ "$pushed" != none ] || suites='[]' + local rollup_obj=null + [ "$rollup" = null ] || rollup_obj="{\"state\":$rollup,\"contexts\":{\"nodes\":[{\"__typename\":\"CheckRun\",\"name\":\"ci\",\"completedAt\":\"$completed\"}]}}" + cat > "$out" <<JSON +{"data":{"repository":{"pullRequest":{ + "state":"$state","isDraft":$draft,"headRefOid":"$head","baseRefName":"main","mergeable":"$mergeable", + "mergeStateStatus":"$mss","reviewDecision":"$decision", + "commits":{"nodes":[{"commit":{"oid":"$head","committedDate":"$committed","checkSuites":{"nodes":$suites},"statusCheckRollup":$rollup_obj}}]}, + "reviewThreads":{"totalCount":$total,"nodes":$threads}, + "reviews":{"totalCount":$review_count,"nodes":[{"author":{"login":"coderabbitai","__typename":"Bot"},"state":"COMMENTED","body":"lgtm", + "submittedAt":"$bot_at","updatedAt":"$bot_at","commit":{"oid":"$review_oid"}}$extra_reviews]}, + "comments":{"totalCount":$comment_count,"nodes":$comments,"pageInfo":{"hasNextPage":false,"endCursor":null}} +}}}} +JSON +} + +# GitLab fixtures: the merge request and its discussions, clean by default with +# a CodeRabbit group-bot note posted after the head pipeline was created. +gl_fixture() { # <world> [key=value...] + local world=$1 kv + shift + local state=opened draft=false head has_conflicts=false dms=not_approved pstatus=success + local created=$OLD_ISO finished=$OLD_ISO notes psha='' + head=$(cat "$world/head") + notes="[{\"notes\":[{\"author\":{\"username\":\"group_42_bot_abc\"},\"system\":false,\"created_at\":\"2026-01-01T00:05:00Z\",\"resolvable\":false}]}]" + for kv in "$@"; do + case "$kv" in + state=*) state=${kv#*=} ;; draft=*) draft=${kv#*=} ;; head=*) head=${kv#*=} ;; + has_conflicts=*) has_conflicts=${kv#*=} ;; dms=*) dms=${kv#*=} ;; pstatus=*) pstatus=${kv#*=} ;; + created=*) created=${kv#*=} ;; finished=*) finished=${kv#*=} ;; notes=*) notes=${kv#*=} ;; + psha=*) psha=${kv#*=} ;; + esac + done + local pipeline=null + [ "$pstatus" = none ] || pipeline="{\"status\":\"$pstatus\",\"sha\":\"${psha:-$head}\",\"created_at\":\"$created\",\"finished_at\":\"$finished\"}" + printf '{"state":"%s","draft":%s,"sha":"%s","has_conflicts":%s,"detailed_merge_status":"%s","head_pipeline":%s}\n' \ + "$state" "$draft" "$head" "$has_conflicts" "$dms" "$pipeline" > "$world/gl.mr.json" + printf '%s\n' "$notes" > "$world/gl.discussions.json" +} + +expect_unclean() { # <world> <label> <reason-substring> + local world=$1 label=$2 want=$3 out rc=0 + rm -f "$world/gh.count" + out=$(run_park "$world" gate lane 2>&1) || rc=$? + [ "$rc" = 3 ] || fail "$label: gate exit $rc, wanted 3 (unclean): $out" + case "$out" in *"$want"*) ;; *) fail "$label: gate reason missing '$want': $out" ;; esac +} + +expect_clean() { # <world> <label> + local world=$1 label=$2 out rc=0 + rm -f "$world/gh.count" + out=$(run_park "$world" gate lane 2>&1) || rc=$? + [ "$rc" = 0 ] || fail "$label: gate exit $rc, wanted clean: $out" + [ "$out" = "clean head=$(cat "$world/head")" ] || fail "$label: unexpected clean output: $out" +} + +test_github_gate_clean_and_each_unclean_condition() { + local w + w=$(make_world gh-gate "$GH_URL") + gh_fixture "$w/gh.1.json"; expect_clean "$w" "github clean" + gh_fixture "$w/gh.1.json" draft=true; expect_unclean "$w" draft "the PR is a draft" + gh_fixture "$w/gh.1.json" state=CLOSED; expect_unclean "$w" closed "the PR is closed" + gh_fixture "$w/gh.1.json" mergeable=CONFLICTING mss=DIRTY; expect_unclean "$w" conflict "merge conflicts" + gh_fixture "$w/gh.1.json" mss=BEHIND; expect_unclean "$w" behind "behind its base" + gh_fixture "$w/gh.1.json" mergeable=UNKNOWN mss=UNKNOWN; expect_unclean "$w" unknown "mergeability is not computed" + gh_fixture "$w/gh.1.json" 'rollup="PENDING"'; expect_unclean "$w" ci-pending "CI is pending" + gh_fixture "$w/gh.1.json" 'rollup="FAILURE"'; expect_unclean "$w" ci-failing "CI is failure" + gh_fixture "$w/gh.1.json" rollup=null; expect_unclean "$w" ci-none "CI is none" + gh_fixture "$w/gh.1.json" decision=CHANGES_REQUESTED; expect_unclean "$w" changes "changes are requested" + gh_fixture "$w/gh.1.json" 'threads=[{"isResolved":true},{"isResolved":false}]' total=2 + expect_unclean "$w" threads "1 review thread(s) unresolved" + gh_fixture "$w/gh.1.json" 'threads=[]' total=150; expect_unclean "$w" threads-truncated "could not all be read" + gh_fixture "$w/gh.1.json" + jq '.data.repository.pullRequest.reviews.totalCount = 101' "$w/gh.1.json" > "$w/truncated.json" && mv "$w/truncated.json" "$w/gh.1.json" + expect_unclean "$w" reviews-truncated "review comments could not all be read" + gh_fixture "$w/gh.1.json" 'threads=[{"isResolved":true,"comments":{"nodes":[{"author":{"login":"reviewer"}}]}}]' + jq '.data.repository.pullRequest.reviewThreads.nodes[0].comments.totalCount = 101' "$w/gh.1.json" > "$w/truncated.json" && mv "$w/truncated.json" "$w/gh.1.json" + expect_unclean "$w" thread-comments-truncated "review comments could not all be read" + pass "github gate: clean PR parks, and draft, closed, conflict, behind, unknown mergeability, CI pending/failing/absent, changes requested, unresolved and unread threads each refuse" +} + +test_github_gate_refuses_until_every_required_check_reports() { + local w + w=$(make_world gh-required "$GH_URL") + gh_fixture "$w/gh.1.json" + # The rollup is green, but it holds only the checks that reported: the + # base branch also requires validate, which has not. + printf '%s\n' '{"protected":true,"protection":{"required_status_checks":{"contexts":["ci","validate"]}}}' > "$w/gh.branch.json" + expect_unclean "$w" required-missing "required check(s) validate have not reported on head" + printf '%s\n' '{"protected":true,"protection":{"required_status_checks":{"contexts":["ci"]}}}' > "$w/gh.branch.json" + expect_clean "$w" required-reported + : > "$w/gh.branch.fail" + expect_unclean "$w" required-unreadable "the required checks of the base branch could not be read" + pass "github gate refuses while a required check of the base branch has not reported on the head, or the requirement cannot be read" +} + +test_github_gate_waits_for_review_bots_or_quiet_window() { + local w + w=$(make_world gh-bots "$GH_URL") + # The bot reviewed an older head and CI just finished: neither settled. + gh_fixture "$w/gh.1.json" review_oid=0000000000000000000000000000000000000001 completed="$NOW_ISO" committed="$NOW_ISO" + expect_unclean "$w" bot-stale "automated reviewers (coderabbitai) have not submitted a review of this head" + # No reviewer bot ever posted and CI just finished: only the window can settle it. + gh_fixture "$w/gh.1.json" completed="$NOW_ISO" committed="$NOW_ISO" extra_reviews='' + sed -i.bak 's/"coderabbitai","__typename":"Bot"/"alice","__typename":"User"/' "$w/gh.1.json" + expect_unclean "$w" no-bot "no automated reviewer has submitted a review of this head" + # The same PR once the quiet window has elapsed. + FM_TEST_QUIET=0 expect_clean "$w" quiet-elapsed + # A bot comment or review edit after the push does not prove its review of + # this head finished: only a review bound to the head, or the window, does. + gh_fixture "$w/gh.1.json" review_oid=0000000000000000000000000000000000000001 completed="$NOW_ISO" committed="$OLD_ISO" bot_at="$NOW_ISO" \ + comments="[{\"author\":{\"login\":\"coderabbitai\",\"__typename\":\"Bot\"},\"createdAt\":\"$NOW_ISO\",\"updatedAt\":\"$NOW_ISO\"}]" + expect_unclean "$w" bot-comment-after-push "automated reviewers (coderabbitai) have not submitted a review of this head" + # A login named in config/contribution-review-bots counts as a reviewer bot. + gh_fixture "$w/gh.1.json" completed="$NOW_ISO" committed="$NOW_ISO" \ + comments='[{"author":{"login":"greptile","__typename":"User"},"createdAt":"2026-01-01T00:00:00Z","updatedAt":"2026-01-01T00:00:00Z"}]' + printf 'greptile\n' > "$w/config/contribution-review-bots" + expect_unclean "$w" listed-bot "automated reviewers (coderabbitai, greptile)" + rm -f "$w/config/contribution-review-bots" + # The head was committed long ago but pushed just now: the window runs from + # the push, not the commit. + gh_fixture "$w/gh.1.json" review_oid=0000000000000000000000000000000000000001 committed="$OLD_ISO" \ + pushed="$NOW_ISO" completed="$OLD_ISO" + expect_unclean "$w" committed-before-push "quiet window" + # With no check suite on the head, the committer time stands in for the push. + gh_fixture "$w/gh.1.json" review_oid=0000000000000000000000000000000000000001 committed="$NOW_ISO" \ + pushed=none completed="$OLD_ISO" + expect_unclean "$w" no-check-suite "quiet window" + pass "github gate waits for every reviewer bot to submit a review of the current head, or the quiet window since it was pushed" +} + +test_gitlab_gate_clean_and_unclean_conditions() { + local w + w=$(make_world gl-gate "$GL_URL") + gl_fixture "$w"; expect_clean "$w" "gitlab clean" + grep -qF -- '--hostname gitlab.example.com projects/team%2Fsub%2Frepo/merge_requests/7' "$w/glab.log" \ + || fail "gitlab gate did not address the merge request on its own host" + gl_fixture "$w" draft=true; expect_unclean "$w" gl-draft "the PR is a draft" + gl_fixture "$w" has_conflicts=true dms=conflict; expect_unclean "$w" gl-conflict "merge conflicts" + gl_fixture "$w" dms=need_rebase; expect_unclean "$w" gl-rebase "behind its base" + gl_fixture "$w" pstatus=running; expect_unclean "$w" gl-ci-running "CI is pending" + gl_fixture "$w" pstatus=failed; expect_unclean "$w" gl-ci-failed "CI is failure" + gl_fixture "$w" pstatus=none; expect_unclean "$w" gl-ci-none "CI is none" + # A green pipeline that ran on an older commit is not CI on this head. + gl_fixture "$w" psha=0000000000000000000000000000000000000004; expect_unclean "$w" gl-ci-old-sha "CI is none" + gl_fixture "$w" dms=requested_changes; expect_unclean "$w" gl-changes "changes are requested" + gl_fixture "$w" dms=jira_association_missing; expect_unclean "$w" gl-other "merge status jira_association_missing" + gl_fixture "$w" notes='[{"notes":[{"author":{"username":"bob"},"system":false,"created_at":"2026-01-01T00:00:00Z","resolvable":true,"resolved":false},{"author":{"username":"alice"},"system":false,"created_at":"2026-01-01T00:01:00Z","resolvable":true,"resolved":false},{"author":{"username":"carol"},"system":false,"created_at":"2026-01-01T00:02:00Z","resolvable":true,"resolved":true}]}]' + expect_unclean "$w" gl-thread "2 review thread(s) unresolved" + # CodeRabbit's group bot posted before this head's pipeline was created. + gl_fixture "$w" created="$NOW_ISO" finished="$NOW_ISO" + expect_unclean "$w" gl-bot-stale "automated reviewers (group_42_bot_abc) have not submitted a review of this head" + # A bot note after the pipeline started proves no finished review: the + # quiet window still decides. + gl_fixture "$w" created="$OLD_ISO" finished="$NOW_ISO" \ + notes="[{\"notes\":[{\"author\":{\"username\":\"group_42_bot_abc\"},\"system\":false,\"created_at\":\"$NOW_ISO\",\"resolvable\":false}]}]" + expect_unclean "$w" gl-bot-note-only "quiet window" + # A full first page of system-note discussions: every page is read. + gl_fixture "$w" notes="$(jq -cn '[range(100) | {notes: [{author: {username: "root"}, system: true, created_at: "2026-01-01T00:00:00Z", resolvable: false}]}]')" + printf '%s\n' '[{"notes":[{"author":{"username":"group_42_bot_abc"},"system":false,"created_at":"2026-01-01T00:05:00Z","resolvable":false}]}]' \ + > "$w/gl.discussions.2.json" + expect_clean "$w" gl-second-page + printf '%s\n' '[{"notes":[{"author":{"username":"bob"},"system":false,"created_at":"2026-01-01T00:06:00Z","resolvable":true,"resolved":false}]}]' \ + > "$w/gl.discussions.2.json" + expect_unclean "$w" gl-second-page-thread "1 review thread(s) unresolved" + rm -f "$w/gl.discussions.2.json" + # A failed later page leaves the read incomplete, like a truncated GitHub + # read: the visible first page is discarded, the gate refuses, and a parked + # lane relaunches with that trigger named. + gl_fixture "$w" + printf '%s\\n' '[{"notes":[{"author":{"username":"bob"},"system":false,"created_at":"2026-01-01T00:06:00Z","resolvable":true,"resolved":false}]}]' > "$w/gl.discussions.2.json" + : > "$w/gl.discussions.fail" + expect_unclean "$w" gl-failed-page "review threads could not all be read" + printf 'pr=%s\nhead=%s\nparked_at=%s\nslot_leased=yes\n' "$GL_URL" "$(cat "$w/head")" "$(date +%s)" > "$w/state/lane.review-park" + expect_relaunch "$w" gl-failed-page-parked "review threads could not all be read" + # A failed merge request read means the forge is unreachable, so an outage + # retries on the next tick and never relaunches. + rm -f "$w/gl.discussions.fail" "$w/control.log" + : > "$w/gl.mr.fail" + printf 'pr=%s\nhead=%s\nparked_at=%s\nslot_leased=yes\n' "$GL_URL" "$(cat "$w/head")" "$(date +%s)" > "$w/state/lane.review-park" + local rc=0 out + out=$(run_park "$w" tick 2>&1) || rc=$? + for _ in $(seq 1 20); do [ ! -e "$w/control.log" ] || break; sleep 0.1; done + [ "$rc" = 0 ] && [ -z "$out" ] || fail "gitlab unreachable forge: tick acted: $rc $out" + [ -e "$w/state/lane.review-park" ] && [ ! -e "$w/control.log" ] || fail "gitlab unreachable forge: parked lane was relaunched" + pass "gitlab gate: clean MR parks, and draft, conflict, rebase, CI running/failed/absent, changes requested, other merge blockers, unresolved threads, a pipeline on an older commit, and a review bot that has not finished each refuse, across every page of discussions; a failed page is unreadable for gate and tick" +} + +test_park_stops_agent_records_lane_and_leases_slot() { + local w head out mode rc=0 + w=$(make_world park-ok "$GH_URL") + head=$(cat "$w/head") + gh_fixture "$w/gh.1.json" + out=$(run_park "$w" park lane 2>&1) || rc=$? + [ "$rc" = 0 ] || fail "park: exit $rc: $out" + case "$out" in "parked lane: PR $GH_URL clean at ${head:0:12}; agent stopped") ;; *) fail "park: output $out" ;; esac + [ "$(cat "$w/control.log")" = "lane exit" ] || fail "park: expected exactly one control exit, got $(cat "$w/control.log")" + grep -qx "lease 3 --lease-holder lane --root $w/root" "$w/treehouse.log" || fail "park: slot was not durably leased: $(cat "$w/treehouse.log" 2>/dev/null)" + grep -qx "head=$head" "$w/state/lane.review-park" || fail "park: record lacks the parked head" + grep -qx "slot_leased=yes" "$w/state/lane.review-park" || fail "park: record does not note the lease" + mode=$(stat -c %a "$w/state/lane.review-park" 2>/dev/null) \ + || mode=$(stat -f %Lp "$w/state/lane.review-park" 2>/dev/null) \ + || fail "park: could not read record mode" + [ "$mode" = 600 ] || fail "park: record is not private (mode=$mode)" + tail -1 "$w/state/lane.status" | grep -q "^paused \[at=[0-9]*\]: parked on review: PR $GH_URL clean at ${head:0:12}" \ + || fail "park: no declared wait in the status log: $(tail -1 "$w/state/lane.status")" + [ "$(cat "$w/gh.count")" = 2 ] || fail "park: expected a fresh gate read and a re-check before the stop, got $(cat "$w/gh.count") reads" + out=$(run_park "$w" list) + case "$out" in "lane PR $GH_URL head ${head:0:12} parked "*) ;; *) fail "park: list output $out" ;; esac + pass "park evaluates the gate twice, leases the slot, writes a private record, stops the agent once, and declares the wait" +} + +test_park_refuses_unsupported_backend_before_side_effects() { + local w out rc=0 + w=$(make_world park-unsupported-backend "$GH_URL") + printf '%s\n' 'backend=zellij' >> "$w/state/lane.meta" + out=$(run_park "$w" park lane 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail "park accepted unsupported backend" + case "$out" in *"zellij backend cannot stop and relaunch the agent"*) ;; *) fail "park did not name unsupported backend: $out" ;; esac + [ ! -e "$w/state/lane.review-park" ] || fail "park wrote a record before refusing" + [ ! -e "$w/treehouse.log" ] || [ ! -s "$w/treehouse.log" ] || fail "park requested a Treehouse lease before refusing" + [ ! -e "$w/control.log" ] || [ ! -s "$w/control.log" ] || fail "park invoked control before refusing" + pass "park refuses an unsupported backend before leasing, recording, or stopping" +} + +test_park_refuses_when_head_moves_after_the_verdict() { + local w out rc=0 + w=$(make_world park-head-moved "$GH_URL") + gh_fixture "$w/gh.1.json" + gh_fixture "$w/gh.2.json" head=0000000000000000000000000000000000000002 + out=$(run_park "$w" park lane 2>&1) || rc=$? + [ "$rc" = 3 ] || fail "head-moved: exit $rc: $out" + case "$out" in *"the head moved from"*) ;; *) fail "head-moved: refusal does not name the moved head: $out" ;; esac + [ ! -e "$w/control.log" ] || fail "head-moved: the agent was stopped: $(cat "$w/control.log")" + [ ! -e "$w/state/lane.review-park" ] || fail "head-moved: a park record was left behind" + ! grep -q "parked on review" "$w/state/lane.status" || fail "head-moved: a parked line was appended" + pass "a head change between the verdict and the stop refuses the park and leaves the agent running" +} + +test_park_serializes_final_check_with_inbox_delivery() { + local w lock pid rc=0 out + w=$(make_world park-inbox-lock "$GH_URL") + local FM_HOME="$w" + export FM_HOME + . "$ROOT/bin/fm-task-inbox-lib.sh" + gh_fixture "$w/gh.1.json" + lock=$(fm_task_inbox_delivery_lock_path "$w/state" lane) + fm_lock_acquire_wait "$lock" || fail "inbox-lock: could not hold the task metadata lock" + run_park "$w" park lane >"$w/park.out" 2>&1 & + pid=$! + for _ in $(seq 1 200); do + [ "$(cat "$w/gh.count" 2>/dev/null || echo 0)" -ge 2 ] && break + sleep 0.05 + done + [ "$(cat "$w/gh.count" 2>/dev/null || echo 0)" -ge 2 ] || fail "inbox-lock: park did not pass its final gate check" + mkdir -p "$w/state/lane.inbox" + printf 'schema=fm-task-inbox.v1\n--\nnew instruction\n' > "$w/state/lane.inbox/001.msg" + fm_lock_release "$lock" + wait "$pid" || rc=$? + out=$(cat "$w/park.out") + case "$rc:$out" in 1:*"steering inbox"*) ;; *) fail "inbox-lock: park did not refuse the newly delivered instruction: $rc $out" ;; esac + [ -e "$w/state/lane.inbox/001.msg" ] || fail "inbox-lock: instruction was lost" + [ ! -e "$w/control.log" ] || fail "inbox-lock: park stopped the agent: $(cat "$w/control.log")" + [ ! -e "$w/state/lane.review-park" ] || fail "inbox-lock: park left a record behind" + pass "park holds the inbox delivery lock across its final check and stop, refusing a newly delivered instruction without stopping the agent" +} + +test_park_does_not_hold_metadata_lock_through_control_exit() { + local w out + w=$(make_world park-control-meta-lock "$GH_URL") + gh_fixture "$w/gh.1.json" + cat > "$w/fakebin/fm-control.sh" <<SH +#!/usr/bin/env bash +. "$ROOT/bin/fm-wake-lib.sh" +lock=\$(fm_meta_lock_path "$w/state/lane.meta") +fm_lock_acquire_wait "\$lock" || exit 1 +fm_lock_release "\$lock" +printf '%s\\n' "\$*" > "$w/control.log" +SH + chmod +x "$w/fakebin/fm-control.sh" + out=$(run_park "$w" park lane 2>&1) || fail "control-meta-lock: park failed: $out" + case "$out" in "parked lane: PR "*) ;; *) fail "control-meta-lock: unexpected output: $out" ;; esac + [ -s "$w/control.log" ] || fail "control-meta-lock: control exit did not run" + pass "park leaves the task metadata lock available to fm-control exit" +} + +test_park_refuses_when_the_lane_is_steered_before_the_stop() { + local w out rc + w=$(make_world park-steered "$GH_URL") + gh_fixture "$w/gh.1.json" + printf 'mkdir -p "%s/state/lane.inbox"; : > "%s/state/lane.inbox/001.msg"\n' "$w" "$w" > "$w/gh.2.sh" + rc=0; out=$(run_park "$w" park lane 2>&1) || rc=$? + case "$rc:$out" in 1:*"steering inbox"*) ;; *) fail "steered-inbox: $rc $out" ;; esac + [ ! -e "$w/control.log" ] || fail "steered-inbox: the agent was stopped: $(cat "$w/control.log")" + [ ! -e "$w/state/lane.review-park" ] || fail "steered-inbox: a park record was left behind" + + rm -rf "$w/state/lane.inbox" "$w/gh.count" + cat > "$w/gh.2.sh" <<SH +printf 'v1 gen=gen1 seq=9 state=busy source=claude-hook event=UserPromptSubmit ts=%s\\n' "\$(date +%s)" > "$w/state/lane.busy-state" +SH + rc=0; out=$(run_park "$w" park lane 2>&1) || rc=$? + case "$rc:$out" in 1:*"the agent is not idle"*) ;; *) fail "steered-busy: $rc $out" ;; esac + [ ! -e "$w/control.log" ] || fail "steered-busy: the agent was stopped: $(cat "$w/control.log")" + [ ! -e "$w/state/lane.review-park" ] || fail "steered-busy: a park record was left behind" + ! grep -q "parked on review" "$w/state/lane.status" || fail "steered: a parked line was appended" + pass "an instruction or a new turn arriving after the preflight refuses the park before the agent is stopped" +} + +# Starts a park whose stop hangs and kills it outright in the middle of it. +kill_park_during_stop() { # <world> + local w=$1 pid + cp "$w/fakebin/fm-control.sh" "$w/fm-control.sh.orig" + printf '#!/usr/bin/env bash\nprintf "%%s\\n" "$*" >> "%s/control.log"\nsleep 5\n' "$w" > "$w/fakebin/fm-control.sh" + TMPDIR="$w" FM_TEST_EXEC_PARK=1 run_park "$w" park lane >/dev/null 2>&1 & + pid=$! + for _ in $(seq 1 100); do [ -s "$w/control.log" ] && break; sleep 0.1; done + [ -s "$w/control.log" ] || fail "interrupted: the stop never started" + kill -KILL "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + mv "$w/fm-control.sh.orig" "$w/fakebin/fm-control.sh" + rm -f "$w/control.log" "$w/gh.count" +} + +test_interrupted_park_reads_as_parking_and_the_tick_settles_it() { + local w out + w=$(make_world park-interrupted "$GH_URL") + gh_fixture "$w/gh.1.json" + kill_park_during_stop "$w" + grep -qx 'pending=1' "$w/state/lane.review-park" || fail "interrupted: no pending park record" + out=$(FM_HOME="$w" FM_STATE_OVERRIDE="$w/state" "$ROOT/bin/fm-crew-state.sh" lane 2>&1 | head -1) + case "$out" in *"parking on review"*) ;; *) fail "interrupted: crew state does not read parking: $out" ;; esac + case "$out" in *"agent stopped"*) fail "interrupted: crew state claims the agent stopped: $out" ;; esac + case "$(run_park "$w" list)" in *"parking; stop not yet confirmed"*) ;; *) fail "interrupted: list: $(run_park "$w" list)" ;; esac + # Feedback after the crash still relaunches the lane, and the tick first + # promotes the record because the agent is confirmed dead. + gh_fixture "$w/gh.1.json" 'rollup="FAILURE"' + expect_relaunch "$w" interrupted-dead "checks are failing" + + w=$(make_world park-interrupted-quiet "$GH_URL") + gh_fixture "$w/gh.1.json" + kill_park_during_stop "$w" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] && [ ! -e "$w/control.log" ] || fail "interrupted-quiet: tick acted: $out" + [ -e "$w/state/lane.review-park" ] || fail "interrupted-quiet: the record of a stopped agent was dropped" + ! grep -q '^pending=' "$w/state/lane.review-park" || fail "interrupted-quiet: the record of a stopped agent was not promoted to parked" + + w=$(make_world park-interrupted-alive "$GH_URL") + gh_fixture "$w/gh.1.json" + kill_park_during_stop "$w" + : > "$w/agent.alive" + gh_fixture "$w/gh.1.json" 'rollup="FAILURE"' + out=$(run_park "$w" tick 2>&1) + # A relaunch would run detached, so give one the time to appear. + for _ in $(seq 1 20); do [ ! -e "$w/control.log" ] || break; sleep 0.1; done + [ -z "$out" ] && [ ! -e "$w/control.log" ] || fail "interrupted-alive: tick acted: $out" + grep -qx 'pending=1' "$w/state/lane.review-park" || fail "interrupted-alive: pending record dropped before stop bound" + + # An unreadable agent is no confirmation either, so feedback still waits. + w=$(make_world park-interrupted-unreadable "$GH_URL") + gh_fixture "$w/gh.1.json" + kill_park_during_stop "$w" + gh_fixture "$w/gh.1.json" 'rollup="FAILURE"' + : > "$w/tmux.unreadable" + run_park "$w" tick >/dev/null 2>&1 + for _ in $(seq 1 20); do [ ! -e "$w/control.log" ] || break; sleep 0.1; done + [ ! -e "$w/control.log" ] || fail "interrupted-unreadable: tick relaunched: $(cat "$w/control.log")" + grep -qx 'pending=1' "$w/state/lane.review-park" || fail "interrupted-unreadable: an unanswered check settled the record" + + w=$(make_world park-interrupted-expired "$GH_URL") + gh_fixture "$w/gh.1.json" + kill_park_during_stop "$w" + : > "$w/agent.alive" + sed -i.bak "s/^parked_at=.*/parked_at=$(( $(date +%s) - 2 ))/" "$w/state/lane.review-park" + FM_PARK_ACTION_SECS=1 run_park "$w" tick >/dev/null 2>&1 + [ ! -e "$w/state/lane.review-park" ] || fail "interrupted-expired: alive worker record was not dropped after bound" + pass "a pending park stays watched during stop, promotes a later stop, and expires only after the stop bound" +} + +test_hung_pending_check_ends_within_the_tick() { + local w out start took + w=$(make_world park-pending-hang "$GH_URL") + gh_fixture "$w/gh.1.json" + kill_park_during_stop "$w" + cp "$w/state/lane.meta" "$w/state/lane2.meta" + cp "$w/state/lane.review-park" "$w/state/lane2.review-park" + : > "$w/tmux.hang" + start=$(date +%s) + out=$(FM_PARK_TICK_SECS=2 run_park "$w" tick 2>&1) + took=$(( $(date +%s) - start )) + [ "$took" -lt 10 ] || fail "pending-hang: a hung agent check held the tick for ${took}s" + [ -z "$out" ] && [ ! -e "$w/control.log" ] || fail "pending-hang: tick acted: $out" + grep -qx 'pending=1' "$w/state/lane.review-park" || fail "pending-hang: an unanswered check settled a pending record" + grep -qx 'pending=1' "$w/state/lane2.review-park" || fail "pending-hang: an unvisited pending record was settled" + [ "$(cat "$w/state/.review-park-tick-next" 2>/dev/null)" = lane2 ] \ + || fail "pending-hang: the tick did not save the lane it never reached" + rm -f "$w/tmux.hang" + run_park "$w" tick >/dev/null 2>&1 + ! grep -q '^pending=' "$w/state/lane.review-park" "$w/state/lane2.review-park" \ + || fail "pending-hang: the next tick did not settle the pending records" + pass "a hung agent check on a pending park ends within the tick, and the next tick resumes and settles it" +} + +test_failed_park_record_write_leaves_the_agent_running() { + local w out rc=0 + w=$(make_world park-record-fail "$GH_URL") + gh_fixture "$w/gh.1.json" + cat > "$w/fakebin/mv" <<SH +#!/usr/bin/env bash +dest= +for arg; do dest=\$arg; done +[ "\$dest" != "$w/state/lane.review-park" ] || exit 1 +exec /bin/mv "\$@" +SH + chmod +x "$w/fakebin/mv" + out=$(run_park "$w" park lane 2>&1) || rc=$? + case "$rc:$out" in 1:*"its park record could not be written"*) ;; *) fail "record-fail: $rc $out" ;; esac + [ ! -e "$w/control.log" ] || fail "record-fail: the agent was stopped: $(cat "$w/control.log")" + [ ! -e "$w/state/lane.review-park" ] || fail "record-fail: a park record was left behind" + ! grep -q "parked on review" "$w/state/lane.status" || fail "record-fail: a parked line was appended" + pass "a park whose record cannot be written refuses before the agent is stopped" +} + +test_park_preflight_refusals_leave_the_agent_running() { + local w out rc + w=$(make_world park-preflight "$GH_URL") + gh_fixture "$w/gh.1.json" + + printf 'v1 gen=gen1 seq=4 state=busy source=claude-hook event=UserPromptSubmit ts=%s\n' "$(date +%s)" > "$w/state/lane.busy-state" + rc=0; out=$(run_park "$w" park lane 2>&1) || rc=$? + case "$rc:$out" in 1:*"the agent is not idle"*) ;; *) fail "busy: $rc $out" ;; esac + printf 'v1 gen=gen1 seq=5 state=idle source=claude-hook event=Stop ts=%s\n' "$(date +%s)" > "$w/state/lane.busy-state" + + printf 'state: working · source: run-step · ci\n' > "$w/crew-state" + rc=0; out=$(run_park "$w" park lane 2>&1) || rc=$? + case "$rc:$out" in 1:*"has not finished at a ready report"*) ;; *) fail "working: $rc $out" ;; esac + rm -f "$w/crew-state" + + mkdir -p "$w/state/lane.inbox"; : > "$w/state/lane.inbox/001.msg" + rc=0; out=$(run_park "$w" park lane 2>&1) || rc=$? + case "$rc:$out" in 1:*"steering inbox"*) ;; *) fail "inbox: $rc $out" ;; esac + rm -f "$w/state/lane.inbox/001.msg" + + printf 'dirty\n' > "$(fm_meta_get_test "$w" worktree)/dirty.txt" + rc=0; out=$(run_park "$w" park lane 2>&1) || rc=$? + case "$rc:$out" in 1:*"uncommitted changes"*) ;; *) fail "dirty: $rc $out" ;; esac + rm -f "$(fm_meta_get_test "$w" worktree)/dirty.txt" + + mv "$w/state/lane.check.sh" "$w/state/lane.check.off" + rc=0; out=$(run_park "$w" park lane 2>&1) || rc=$? + case "$rc:$out" in 1:*"no merge poll is armed"*) ;; *) fail "no-poll: $rc $out" ;; esac + mv "$w/state/lane.check.off" "$w/state/lane.check.sh" + + gh_fixture "$w/gh.1.json" head=0000000000000000000000000000000000000003 + rc=0; out=$(run_park "$w" park lane 2>&1) || rc=$? + case "$rc:$out" in 1:*"the local copy is at"*) ;; *) fail "local-behind: $rc $out" ;; esac + gh_fixture "$w/gh.1.json" + + : > "$w/treehouse.fail" + rc=0; out=$(run_park "$w" park lane 2>&1) || rc=$? + case "$rc:$out" in 1:*"could not be leased durably"*) ;; *) fail "lease-fail: $rc $out" ;; esac + rm -f "$w/treehouse.fail" + + : > "$w/control.fail" + rc=0; out=$(run_park "$w" park lane 2>&1) || rc=$? + case "$rc:$out" in 1:*"the agent could not be stopped"*) ;; *) fail "control-fail: $rc $out" ;; esac + grep -qx 'pending=1' "$w/state/lane.review-park" || fail "control-fail: no pending record for the tick to settle" + rm -f "$w/control.fail" "$w/control.log" + : > "$w/agent.alive" + run_park "$w" tick >/dev/null 2>&1 + grep -qx 'pending=1' "$w/state/lane.review-park" || fail "control-fail: the tick dropped the record before the stop bound" + FM_PARK_ACTION_SECS=0 run_park "$w" tick >/dev/null 2>&1 + [ ! -e "$w/state/lane.review-park" ] || fail "control-fail: the record of a running agent outlived the stop bound" + [ ! -e "$w/control.log" ] || fail "control-fail: the tick acted on the lane: $(cat "$w/control.log")" + + ! grep -q "parked on review" "$w/state/lane.status" || fail "preflight: a parked line was appended" + pass "park refuses a busy or unfinished worker, a waiting instruction, a dirty or lagging copy, an unarmed merge poll, an unleasable slot, and a failed stop" +} + +test_repark_reuses_its_own_lease_and_refuses_a_foreign_one() { + local w wt out rc=0 + w=$(make_world repark-lease "$GH_URL") + gh_fixture "$w/gh.1.json" + wt=$(cd "$(fm_meta_get_test "$w" worktree)" && pwd -P) + printf '[{"name":"3","path":"%s","status":"leased","lease_holder":"lane"}]\n' "$wt" > "$w/treehouse.status.json" + out=$(run_park "$w" park lane 2>&1) || fail "own-lease: park refused: $out" + ! grep -q '^lease ' "$w/treehouse.log" || fail "own-lease: leased the slot a second time" + grep -qx "slot_leased=yes" "$w/state/lane.review-park" || fail "own-lease: record does not note the lease" + + w=$(make_world foreign-lease "$GH_URL") + gh_fixture "$w/gh.1.json" + wt=$(cd "$(fm_meta_get_test "$w" worktree)" && pwd -P) + printf '[{"name":"3","path":"%s","status":"leased","lease_holder":"other-task"}]\n' "$wt" > "$w/treehouse.status.json" + : > "$w/treehouse.fail" + out=$(run_park "$w" park lane 2>&1) || rc=$? + case "$rc:$out" in 1:*"could not be leased durably"*) ;; *) fail "foreign-lease: $rc $out" ;; esac + [ ! -e "$w/control.log" ] || fail "foreign-lease: the agent was stopped" + pass "a lane re-parked after a relaunch keeps its own durable lease, and a slot leased to another holder refuses the park" +} + +fm_meta_get_test() { # <world> <key> + sed -n "s/^$2=//p" "$1/state/lane.meta" | tail -1 +} + +# Park a world through the real park path, then reset the forge read counter. +park_world() { # <world> + local o; o=$(run_park "$1" park lane 2>&1) || fail "setup: park failed for $1: $o" + rm -f "$1/gh.count" "$1/control.log" +} + +# Waits for a detached lane action to publish its outcome for the next tick. +wait_report() { # <world> <label> + for _ in $(seq 1 200); do + ! compgen -G "$1/state/.review-park-report-*" >/dev/null || return 0 + sleep 0.1 + done + fail "$2: no detached action reported within 20s" +} + +# Waits for a detached land to settle. It publishes its report before it +# removes the pending request, so a removed request means the report exists. +wait_land_settled() { # <world> <label> + for _ in $(seq 1 200); do + [ -e "$1/state/lane.review-park-land" ] || break + sleep 0.1 + done + [ ! -e "$1/state/lane.review-park-land" ] || fail "$2: the pending cleanup request was not cleared within 20s" + compgen -G "$1/state/.review-park-report-*" >/dev/null || fail "$2: the pending request was cleared without a reported outcome" +} + +# One tick that starts a lane action, then the next tick, which reports its +# outcome; prints that report. +tick_and_report() { # <world> <label> + local out + out=$(run_park "$1" tick 2>&1) || fail "$2: tick failed: $out" + [ -z "$out" ] || fail "$2: the starting tick printed: $out" + wait_report "$1" "$2" + run_park "$1" tick 2>&1 || fail "$2: the reporting tick failed" +} + +expect_relaunch() { # <world> <label> <trigger-substring> + local world=$1 label=$2 want=$3 out + out=$(tick_and_report "$world" "$label") + case "$out" in "relaunched lane: PR "*"$want"*) ;; *) fail "$label: tick output: $out" ;; esac + head -1 "$world/control.log" | grep -q "^lane relaunch --note-file " || fail "$label: no relaunch call: $(cat "$world/control.log")" + grep -qF "$want" "$world/control.log" || fail "$label: the relaunch note does not carry the trigger" + grep -qF "same local copy on the same branch" "$world/control.log" || fail "$label: the note does not name the same copy and branch" + [ ! -e "$world/state/lane.review-park" ] || fail "$label: park record kept after relaunch" + tail -1 "$world/state/lane.status" | grep -q "^working \[at=[0-9]*\]: relaunched from review park: .*$want" \ + || fail "$label: no working line: $(tail -1 "$world/state/lane.status")" +} + +test_each_relaunch_trigger_relaunches_a_fresh_worker() { + local w later + later=$(date -u -v+1H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '+1 hour' +%Y-%m-%dT%H:%M:%SZ) + + w=$(make_world trig-head "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + rm -f "$w"/gh.*.json; gh_fixture "$w/gh.1.json" head=0000000000000000000000000000000000000009 + expect_relaunch "$w" head "the head moved to 000000000000" + + w=$(make_world trig-ci "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" 'rollup="FAILURE"' + expect_relaunch "$w" ci "checks are failing" + + w=$(make_world trig-changes "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" decision=CHANGES_REQUESTED + expect_relaunch "$w" changes "changes were requested" + + w=$(make_world trig-thread "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" 'threads=[{"isResolved":false}]' total=1 + expect_relaunch "$w" thread "1 review thread(s) unresolved" + + w=$(make_world trig-incomplete-threads "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" 'threads=[]' total=150 + expect_relaunch "$w" incomplete-threads "review threads could not all be read" + + w=$(make_world trig-incomplete-reviews "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + jq '.data.repository.pullRequest.reviews.totalCount = 101' "$w/gh.1.json" > "$w/truncated.json" && mv "$w/truncated.json" "$w/gh.1.json" + expect_relaunch "$w" incomplete-reviews "review comments could not all be read" + + w=$(make_world trig-incomplete-thread-comments "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" 'threads=[{"isResolved":true,"comments":{"nodes":[{"author":{"login":"reviewer"}}]}}]' + jq '.data.repository.pullRequest.reviewThreads.nodes[0].comments.totalCount = 101' "$w/gh.1.json" > "$w/truncated.json" && mv "$w/truncated.json" "$w/gh.1.json" + expect_relaunch "$w" incomplete-thread-comments "review comments could not all be read" + + w=$(make_world trig-conflict "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" mergeable=CONFLICTING mss=DIRTY + expect_relaunch "$w" conflict "merge conflicts" + + w=$(make_world trig-comment "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" comments="[{\"author\":{\"login\":\"carol\",\"__typename\":\"User\"},\"createdAt\":\"$later\",\"updatedAt\":\"$later\"}]" + expect_relaunch "$w" comment "new comments from carol" + + w=$(make_world trig-same-second "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" comments="[{\"author\":{\"login\":\"carol\",\"__typename\":\"User\"},\"createdAt\":\"$NOW_ISO\",\"updatedAt\":\"$NOW_ISO\"}]" + parked_at=$(jq -nr --arg t "$NOW_ISO" '$t | fromdateiso8601') + sed -i.bak "s/^parked_at=.*/parked_at=$parked_at/" "$w/state/lane.review-park" + expect_relaunch "$w" same-second "new comments from carol" + + # An inline reply in an already resolved thread leaves no unresolved thread + # and no new review submission; the reply itself is the new feedback. + w=$(make_world trig-thread-reply "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" total=1 \ + threads="[{\"isResolved\":true,\"comments\":{\"nodes\":[{\"author\":{\"login\":\"erin\",\"__typename\":\"User\"},\"createdAt\":\"$later\",\"updatedAt\":\"$later\"}]}}]" + expect_relaunch "$w" thread-reply "new comments from erin" + + w=$(make_world trig-gl-edit "$GL_URL"); gl_fixture "$w"; park_world "$w" + gl_fixture "$w" notes="[{\"notes\":[{\"author\":{\"username\":\"group_42_bot_abc\"},\"system\":false,\"created_at\":\"$OLD_ISO\",\"updated_at\":\"$later\",\"resolvable\":false}]}]" + expect_relaunch "$w" gitlab-edited-note "new comments from group_42_bot_abc" + + w=$(make_world trig-gl-note "$GL_URL"); gl_fixture "$w"; park_world "$w" + gl_fixture "$w" notes="[{\"notes\":[{\"author\":{\"username\":\"dave\"},\"system\":false,\"created_at\":\"$later\",\"resolvable\":false}]}]" + expect_relaunch "$w" gitlab-note "new comments from dave" + pass "head change, failing checks, changes requested, an unresolved thread, a conflict, and new or edited GitHub or GitLab comments, including an inline reply in a resolved thread, each relaunch a fresh worker with the trigger in its instructions" +} + +test_edited_comment_beyond_first_hundred_triggers_relaunch() { + local w later + later=$(date -u -v+1H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '+1 hour' +%Y-%m-%dT%H:%M:%SZ) + w=$(make_world trig-edit-over-100 "$GH_URL") + gh_fixture "$w/gh.1.json"; park_world "$w" + # The old one-page response excludes this older edited comment. + gh_fixture "$w/gh.1.json" comments='[{"author":{"login":"recent","__typename":"User"},"createdAt":"2026-02-01T00:00:00Z","updatedAt":"2026-02-01T00:00:00Z"}]' + gh_fixture "$w/comment-page1.json" comments="$(jq -cn --arg old "$OLD_ISO" --arg later "$later" '[{author:{login:"greptile-apps",__typename:"Bot"},createdAt:$old,updatedAt:$later}] + [range(99) | {author:{login:"older",__typename:"User"},createdAt:$old,updatedAt:$old}]')" + gh_fixture "$w/comment-page2.json" comments='[{"author":{"login":"newest","__typename":"User"},"createdAt":"2026-03-01T00:00:00Z","updatedAt":"2026-03-01T00:00:00Z"}]' + jq -s ' + .[0] as $one | .[1] as $two + | [($one | .data.repository.pullRequest.comments.totalCount=101 + | .data.repository.pullRequest.comments.pageInfo={hasNextPage:true,endCursor:"page-1"}), + ($two | .data.repository.pullRequest.comments.totalCount=101 + | .data.repository.pullRequest.comments.pageInfo={hasNextPage:false,endCursor:null})] + ' "$w/comment-page1.json" "$w/comment-page2.json" > "$w/gh.1.pages.json" + expect_relaunch "$w" edited-comment-over-100 "new comments from greptile-apps" + pass "an edited comment older than the latest 100 is detected and relaunches the parked lane" +} + +test_behind_base_does_not_relaunch() { + local w out + w=$(make_world trig-behind "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" mss=BEHIND + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] && [ ! -e "$w/control.log" ] || fail "behind: tick acted: $out" + [ -e "$w/state/lane.review-park" ] || fail "behind: park record lost" + pass "a parked branch that merely fell behind its base stays parked" +} + +# A relaunch that takes longer than the whole tick budget still completes: the +# tick only starts it, and nothing bounded by the tick can kill it. +test_slow_relaunch_is_never_cut_short_by_the_tick() { + local w out + w=$(make_world tick-slow-relaunch "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" 'rollup="FAILURE"' + printf '#!/usr/bin/env bash\nprintf started > "%s/control.started"\nwhile [ ! -e "%s/control.release" ]; do sleep 0.05; done\nprintf "%%s\\n" "$*" >> "%s/control.log"\n' "$w" "$w" "$w" > "$w/fakebin/fm-control.sh" + out=$(FM_PARK_TICK_SECS=6 run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "slow-relaunch: the starting tick printed: $out" + for _ in $(seq 1 100); do [ -e "$w/control.started" ] && break; sleep 0.1; done + [ -e "$w/control.started" ] || fail "slow-relaunch: detached action never started" + # A tick while the relaunch runs leaves the busy lane alone. + out=$(FM_PARK_TICK_SECS=6 run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "slow-relaunch: a tick during the relaunch printed: $out" + : > "$w/control.release" + wait_report "$w" slow-relaunch + out=$(FM_PARK_TICK_SECS=6 run_park "$w" tick 2>&1) + case "$out" in "relaunched lane: PR $GH_URL - checks are failing"*) ;; *) fail "slow-relaunch: report: $out" ;; esac + [ "$(grep -c ' relaunch ' "$w/control.log")" = 1 ] || fail "slow-relaunch: relaunch calls: $(cat "$w/control.log")" + [ ! -e "$w/state/lane.review-park" ] || fail "slow-relaunch: the record of a relaunched lane was kept: $(cat "$w/state/lane.review-park")" + pass "a relaunch slower than the whole tick budget completes once, is never killed, and is reported by the next tick" +} + +test_tick_resumes_where_the_last_one_ran_out_of_time() { + local w out + w=$(make_world tick-rotate "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + cp "$w/state/lane.meta" "$w/state/lane2.meta" + cp "$w/state/lane.review-park" "$w/state/lane2.review-park" + gh_fixture "$w/gh.1.json" 'rollup="FAILURE"' + # The first lane's read outlasts the whole tick; in the next tick the second + # lane visited is the slow one, so only the lane that tick starts at relaunches. + printf 'sleep 8\n' > "$w/gh.1.sh" + printf 'sleep 8\n' > "$w/gh.3.sh" + out=$(FM_PARK_TICK_SECS=6 run_park "$w" tick 2>&1) + [ -z "$out" ] && [ ! -e "$w/control.log" ] || fail "rotate: the slow tick acted: $out" + FM_PARK_TICK_SECS=6 run_park "$w" tick >/dev/null 2>&1 + wait_report "$w" rotate + [ "$(grep ' relaunch ' "$w/control.log" | cut -d' ' -f1)" = lane2 ] \ + || fail "rotate: the next tick did not start at the lane the last one never reached: $(grep ' relaunch ' "$w/control.log")" + [ -e "$w/state/lane.review-park" ] || fail "rotate: the lane read last was relaunched" + out=$(run_park "$w" tick 2>&1) + case "$out" in "relaunched lane2: "*) ;; *) fail "rotate: the next tick did not report lane2: $out" ;; esac + wait_report "$w" rotate-rest + [ ! -e "$w/state/lane.review-park" ] && [ ! -e "$w/state/lane2.review-park" ] \ + || fail "rotate: a complete tick did not visit every lane: $out" + pass "a tick that runs out of time resumes at the next unvisited lane, so a slow early lane cannot starve later ones" +} + +test_quiet_parked_lane_and_approval_do_not_relaunch() { + local w later out + later=$(date -u -v+1H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '+1 hour' +%Y-%m-%dT%H:%M:%SZ) + w=$(make_world quiet "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "quiet: tick printed $out" + gh_fixture "$w/gh.1.json" extra_reviews=",{\"author\":{\"login\":\"erin\",\"__typename\":\"User\"},\"state\":\"APPROVED\",\"body\":\"\",\"submittedAt\":\"$later\",\"updatedAt\":\"$later\",\"commit\":{\"oid\":\"$(cat "$w/head")\"}}" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "approval: tick printed $out" + [ ! -e "$w/control.log" ] || fail "quiet: a lifecycle call was made: $(cat "$w/control.log")" + [ -e "$w/state/lane.review-park" ] || fail "quiet: park record lost" + pass "an unchanged parked PR and a bare approval stay parked silently" +} + +test_failed_relaunch_keeps_record_and_stops_retrying() { + local w out + w=$(make_world relaunch-fail "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" 'rollup="FAILURE"' + : > "$w/control.fail" + out=$(tick_and_report "$w" relaunch-fail) + case "$out" in "relaunch-failed lane: PR $GH_URL (checks are failing"*) ;; *) fail "relaunch-fail: tick output $out" ;; esac + grep -q '^resume_error=' "$w/state/lane.review-park" || fail "relaunch-fail: record does not note the failure" + rm -f "$w/control.log" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] && [ ! -e "$w/control.log" ] || fail "relaunch-fail: tick retried: $out" + pass "a failed relaunch keeps the park record, reports once, and waits for a manual resume" +} + +test_closed_pr_is_reported_once() { + local w out + w=$(make_world closed "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" state=CLOSED + out=$(run_park "$w" tick 2>&1) + [ "$out" = "closed lane: PR $GH_URL was closed without merging; the lane needs a decision" ] || fail "closed: $out" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "closed: reported twice: $out" + [ ! -e "$w/control.log" ] || fail "closed: a lifecycle call was made" + + w=$(make_world closed-default-off "$GL_URL"); gl_fixture "$w" state=closed + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] && [ ! -e "$w/glab.log" ] || fail "closed-default-off: tick acted without opt-in: $out" + + w=$(make_world closed-unparked "$GL_URL"); gl_fixture "$w" state=closed + : > "$w/config/auto-land" + out=$(run_park "$w" tick 2>&1) + [ "$out" = "closed lane: PR $GL_URL was closed without merging; the lane needs a decision" ] || fail "closed-unparked: $out" + + w=$(make_world closed-busy "$GL_URL"); gl_fixture "$w" state=closed + : > "$w/config/auto-land" + printf 'v1 gen=gen1 seq=4 state=working source=claude-hook event=Start ts=%s\n' "$(date +%s)" > "$w/state/lane.busy-state" + out=$(run_park "$w" tick 2>&1) + [ "$out" = "closed lane: PR $GL_URL was closed without merging; the lane needs a decision" ] || fail "closed-busy: $out" + [ ! -e "$w/state/lane.review-park" ] && [ ! -e "$w/control.log" ] && [ ! -e "$w/teardown.log" ] \ + || fail "closed-busy: lifecycle action occurred" + pass "closed-without-merge reporting for unparked lanes requires a feature flag and does not require an idle worker" +} + +test_tick_parks_clean_idle_lanes_only_when_opted_in() { + local w out + w=$(make_world auto-park "$GL_URL"); gl_fixture "$w" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] && [ ! -e "$w/control.log" ] || fail "auto-park: tick parked without config/park-on-review: $out" + : > "$w/config/auto-land"; rm -f "$w/glab.log" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "auto-park: unexpected output while auto-land is enabled: $out" + : > "$w/config/park-on-review" + gl_fixture "$w" pstatus=running + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] && [ ! -e "$w/control.log" ] || fail "auto-park: parked an unclean lane: $out" + gl_fixture "$w" + out=$(tick_and_report "$w" auto-park) + case "$out" in "parked lane: PR $GL_URL clean at "*) ;; *) fail "auto-park: tick output $out" ;; esac + [ -e "$w/state/lane.review-park" ] || fail "auto-park: no record" + pass "tick parks nothing without opt-in, reads nothing when opted out of cleanup too, never parks an unclean lane, and parks a clean idle one under config/park-on-review" +} + +test_land_requires_opt_in_but_always_cleans_parked() { + local w out rc + w=$(make_world land-off "$GH_URL") + out=$(run_park "$w" land lane --if-eligible 2>&1) + [ -z "$out" ] && [ ! -e "$w/teardown.log" ] || fail "land: acted without config/auto-land: $out" + : > "$w/config/auto-land" + out=$(run_park "$w" land lane --if-eligible 2>&1) + [ "$out" = "cleaned up lane: PR $GH_URL merged; lane torn down and its working copy returned" ] || fail "land: $out" + [ "$(cat "$w/teardown.log")" = lane ] || fail "land: teardown was not called exactly once without --force: $(cat "$w/teardown.log")" + + w=$(make_world land-parked "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + : > "$w/teardown.refuse" + rc=0; out=$(run_park "$w" land lane --if-eligible 2>&1) || rc=$? + [ "$rc" = 1 ] || fail "land-refused: exit $rc" + [ "$out" = "cleanup-refused lane: PR $GH_URL merged; automatic cleanup refused: REFUSED: worktree has work not on any remote and not landed." ] || fail "land-refused: $out" + pass "merge-time cleanup requires opt-in for unparked lanes and always cleans parked lanes" +} + +test_land_queue_fails_when_cleanup_request_cannot_be_written() { + local w out rc=0 + w=$(make_world land-queue-write-fail "$GH_URL") + : > "$w/config/auto-land" + cat > "$w/fakebin/mv" <<SH +#!/usr/bin/env bash +dest= +for arg; do dest=\$arg; done +[ "\$dest" != "$w/state/lane.review-park-land" ] || exit 1 +exec /bin/mv "\$@" +SH + chmod +x "$w/fakebin/mv" + out=$(run_park "$w" land lane --if-eligible --queue 2>&1) || rc=$? + [ "$rc" -ne 0 ] || fail "land-queue-write-fail: accepted an unwritable cleanup request: $out" + [ ! -e "$w/state/lane.review-park-land" ] || fail "land-queue-write-fail: request unexpectedly exists" + pass "land queue fails when the cleanup request cannot be written" +} + +test_land_report_keeps_lost_pool_slot_details() { + local w out report + w=$(make_world land-slot-lost "$GH_URL") + : > "$w/config/auto-land" + printf 'pr=%s\n' "$GH_URL" > "$w/state/lane.review-park-land" + cat > "$w/fakebin/fm-teardown.sh" <<'SH' +#!/usr/bin/env bash +printf '%s\n' 'warning: treehouse return left pool slot 7 (/pool/7) dirty or unverified; the pool lost this slot. Remaining paths:' ' untracked paths:' ' lost.txt' +exit 4 +SH + chmod +x "$w/fakebin/fm-teardown.sh" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "land-slot-lost: starting tick printed: $out" + wait_land_settled "$w" land-slot-lost + report=$(grep -l 'cleaned up lane:' "$w"/state/.review-park-report-* 2>/dev/null) || fail "land-slot-lost: no saved report" + grep -q 'pool slot 7' "$report" || fail "land-slot-lost: report omitted lost slot: $(cat "$report")" + grep -q 'lost.txt' "$report" || fail "land-slot-lost: report omitted leftover path: $(cat "$report")" + out=$(run_park "$w" tick 2>&1) + case "$out" in *'pool slot 7'*'lost.txt'*) ;; *) fail "land-slot-lost: tick omitted repair details: $out" ;; esac + pass "a lost pool slot and its leftover paths survive in the detached land report" +} + +# The merge poll's land only starts a detached teardown: it returns at once +# however slow the teardown is, and the next tick reports the outcome, even in +# a home that opted out of everything the tick otherwise watches. +test_detached_land_returns_at_once_and_the_tick_reports_it() { + local w out + w=$(make_world land-detached "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + printf '#!/usr/bin/env bash\nprintf started > "%s/teardown.started"\nwhile [ ! -e "%s/teardown.release" ]; do sleep 0.05; done\nprintf "%%s\\n" "$*" >> "%s/teardown.log"\nrm -f "%s/state/lane.review-park" "%s/state/lane.meta"\n' "$w" "$w" "$w" "$w" "$w" > "$w/fakebin/fm-teardown.sh" + out=$(run_park "$w" land lane --if-eligible --detach 2>&1) + [ "$out" = "automatic cleanup started" ] || fail "land-detached: $out" + for _ in $(seq 1 100); do [ -e "$w/teardown.started" ] && break; sleep 0.1; done + [ -e "$w/teardown.started" ] || fail "land-detached: detached teardown never started" + [ ! -e "$w/teardown.log" ] || fail "land-detached: teardown completed before release" + : > "$w/teardown.release" + wait_land_settled "$w" land-detached + [ "$(cat "$w/teardown.log")" = lane ] || fail "land-detached: teardown calls: $(cat "$w/teardown.log")" + out=$(run_park "$w" tick 2>&1) + [ "$out" = "cleaned up lane: PR $GH_URL merged; lane torn down and its working copy returned" ] || fail "land-detached: report: $out" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "land-detached: reported twice: $out" + pass "a detached merge-time cleanup returns at once, runs its slow teardown to completion, and the next tick reports it once" +} + +test_pending_land_recovers_after_watcher_restart() { + local w out + w=$(make_world land-recovery "$GH_URL"); gh_fixture "$w/gh.1.json" + : > "$w/config/auto-land" + printf 'pr=%s\n' "$GH_URL" > "$w/state/lane.review-park-land" + cat > "$w/fakebin/fm-teardown.sh" <<SH +#!/usr/bin/env bash +: > "$w/teardown.started" +while [ ! -e "$w/teardown.release" ]; do sleep 0.05; done +echo "Torn down \$1" +SH + chmod +x "$w/fakebin/fm-teardown.sh" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "land-recovery: starting tick printed: $out" + for _ in $(seq 1 100); do [ -e "$w/teardown.started" ] && break; sleep 0.1; done + [ -e "$w/teardown.started" ] || fail "land-recovery: cleanup did not start" + [ -e "$w/state/lane.review-park-land" ] || fail "land-recovery: request removed before cleanup outcome" + : > "$w/teardown.release" + wait_land_settled "$w" land-recovery + out=$(run_park "$w" tick 2>&1) + [ "$out" = "cleaned up lane: PR $GH_URL merged; lane torn down and its working copy returned" ] || fail "land-recovery: outcome: $out" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "land-recovery: outcome reported twice: $out" + pass "a pending merged-lane cleanup survives watcher restart and reports its outcome once" +} + +test_pending_land_retries_after_control_lock_refusal() { + local w out + w=$(make_world land-lock-retry "$GH_URL"); gh_fixture "$w/gh.1.json" + : > "$w/config/auto-land" + printf 'pr=%s\n' "$GH_URL" > "$w/state/lane.review-park-land" + cat > "$w/fakebin/fm-teardown.sh" <<SH +#!/usr/bin/env bash +n=\$(cat "$w/teardown.count" 2>/dev/null || echo 0) +n=\$((n + 1)); printf '%s\\n' "\$n" > "$w/teardown.count" +[ "\$n" != 1 ] || { echo 'error: another lifecycle action is already running for task lane; nothing was changed'; exit 1; } +echo "Torn down \$1" +SH + chmod +x "$w/fakebin/fm-teardown.sh" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "land-lock-retry: first tick printed: $out" + for _ in $(seq 1 100); do [ "$(cat "$w/teardown.count" 2>/dev/null || echo 0)" -ge 1 ] && break; sleep 0.1; done + [ "$(cat "$w/teardown.count")" = 1 ] || fail "land-lock-retry: first teardown did not run" + # A tick skips the lane while the first attempt still holds it, and drains + # any report it published, so tick until the retry runs: every tick must stay + # silent and the pending request must survive the refusal. + for _ in $(seq 1 200); do + [ "$(cat "$w/teardown.count")" = 1 ] || break + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "land-lock-retry: the lock refusal was reported as an outcome: $out" + sleep 0.1 + done + [ "$(cat "$w/teardown.count")" = 2 ] || fail "land-lock-retry: the pending request was never retried" + wait_land_settled "$w" land-lock-retry + out=$(run_park "$w" tick 2>&1) + [ "$out" = "cleaned up lane: PR $GH_URL merged; lane torn down and its working copy returned" ] || fail "land-lock-retry: outcome: $out" + [ "$(cat "$w/teardown.count")" = 2 ] || fail "land-lock-retry: teardown calls: $(cat "$w/teardown.count")" + pass "a temporary control-lock refusal stays pending and retries to one reported cleanup outcome" +} + +# A cleanup killed partway through its teardown leaves its request behind, so +# the next tick retries it and reports the outcome once. +test_killed_cleanup_is_retried_and_reported_once() { + local w out pgid + w=$(make_world land-killed "$GH_URL"); gh_fixture "$w/gh.1.json" + : > "$w/config/auto-land" + printf 'pr=%s\n' "$GH_URL" > "$w/state/lane.review-park-land" + cat > "$w/fakebin/fm-teardown.sh" <<SH +#!/usr/bin/env bash +n=\$(cat "$w/teardown.count" 2>/dev/null || echo 0) +n=\$((n + 1)); printf '%s\\n' "\$n" > "$w/teardown.count" +if [ "\$n" = 1 ]; then + # Record the detached cleanup's own process group: the nearest ancestor + # running fm-park.sh run. + p=\$\$ + while p=\$(ps -o ppid= -p "\$p" | tr -d ' ') && [ "\${p:-1}" -gt 1 ]; do + case "\$(ps -o command= -p "\$p")" in *"fm-park.sh run lane land"*) ps -o pgid= -p "\$p" | tr -d ' ' > "$w/cleanup.pgid"; break ;; esac + done + : > "$w/teardown.started" + sleep 30 +fi +echo "Torn down \$1" +SH + chmod +x "$w/fakebin/fm-teardown.sh" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "land-killed: first tick printed: $out" + for _ in $(seq 1 100); do [ -e "$w/teardown.started" ] && break; sleep 0.1; done + pgid=$(cat "$w/cleanup.pgid" 2>/dev/null) + [ -n "$pgid" ] && [ "$pgid" != "$(ps -o pgid= -p $$ | tr -d ' ')" ] || fail "land-killed: no detached cleanup process group found: '$pgid'" + kill -KILL -- "-$pgid" + sleep 0.5 + [ -e "$w/state/lane.review-park-land" ] || fail "land-killed: the killed cleanup lost its request" + ! compgen -G "$w/state/.review-park-report-*" >/dev/null || fail "land-killed: the killed cleanup published an outcome" + for _ in $(seq 1 200); do + [ "$(cat "$w/teardown.count")" = 1 ] || break + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "land-killed: a retry tick printed: $out" + sleep 0.1 + done + [ "$(cat "$w/teardown.count")" = 2 ] || fail "land-killed: the killed cleanup was never retried" + wait_land_settled "$w" land-killed + out=$(run_park "$w" tick 2>&1) + [ "$out" = "cleaned up lane: PR $GH_URL merged; lane torn down and its working copy returned" ] || fail "land-killed: outcome: $out" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "land-killed: reported twice: $out" + pass "a cleanup killed during its teardown keeps its request, and the next tick retries it and reports once" +} + +# A cleanup request names the PR that merged. When the lane has since moved on +# to another PR, the request is dropped without tearing anything down. +test_stale_cleanup_request_for_another_pr_is_dropped() { + local w out + w=$(make_world land-stale "$GH_URL"); gh_fixture "$w/gh.1.json" + : > "$w/config/auto-land" + printf 'pr=%s\n' "$GH_URL" > "$w/state/lane.review-park-land" + sed -i.bak "s#^pr=.*#pr=${GH_URL%/*}/8#" "$w/state/lane.meta" && rm -f "$w/state/lane.meta.bak" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "land-stale: tick printed: $out" + for _ in $(seq 1 200); do + [ -e "$w/state/lane.review-park-land" ] || break + sleep 0.1 + done + [ ! -e "$w/state/lane.review-park-land" ] || fail "land-stale: the stale request was not dropped within 20s" + [ ! -e "$w/teardown.log" ] || fail "land-stale: a stale request tore down the lane: $(cat "$w/teardown.log")" + ! compgen -G "$w/state/.review-park-report-*" >/dev/null || fail "land-stale: a stale request reported: $(cat "$w"/state/.review-park-report-*)" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "land-stale: a later tick printed: $out" + pass "a cleanup request for a PR the lane no longer names is dropped without a teardown or a report" +} + +# The tick records its wake before it removes the reports behind it, so a +# failed record loses nothing: the reports stay for the next tick. +test_reports_survive_until_their_wake_is_recorded() { + local w out report + w=$(make_world report-durable "$GH_URL") + report="$w/state/.review-park-report-lane-1-1" + printf 'cleanup-refused lane: PR %s merged; automatic cleanup refused: REFUSED\n' "$GH_URL" > "$report" + mkdir "$w/state/.wake-queue" + out=$(run_park "$w" tick 2>/dev/null) + [ -z "$out" ] || fail "report-durable: a tick that could not record its wake printed: $out" + [ -e "$report" ] || fail "report-durable: the report was removed although its wake was not recorded" + rmdir "$w/state/.wake-queue" + out=$(run_park "$w" tick 2>&1) + [ "$out" = "cleanup-refused lane: PR $GH_URL merged; automatic cleanup refused: REFUSED" ] || fail "report-durable: retry: $out" + [ ! -e "$report" ] || fail "report-durable: the report survived its recorded wake" + [ "$(awk -F '\t' '$3 == "check" && $4 == "park-on-review"' "$w/state/.wake-queue" | wc -l | tr -d ' ')" = 1 ] \ + || fail "report-durable: wake queue: $(cat "$w/state/.wake-queue")" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "report-durable: reported twice: $out" + pass "an action report survives until the tick has recorded its wake, then is reported once" +} + +# A merge-time cleanup that lands while a relaunch of the same lane is under +# way waits for it instead of tearing the lane down mid-relaunch. +test_land_waits_for_an_action_under_way() { + local w out + w=$(make_world land-waits "$GH_URL"); gh_fixture "$w/gh.1.json"; park_world "$w" + gh_fixture "$w/gh.1.json" 'rollup="FAILURE"' + printf '#!/usr/bin/env bash\nsleep 3\nprintf "%%s\\n" "$*" >> "%s/order.log"\n' "$w" > "$w/fakebin/fm-control.sh" + printf '#!/usr/bin/env bash\nprintf "teardown\\n" >> "%s/order.log"\n' "$w" > "$w/fakebin/fm-teardown.sh" + out=$(run_park "$w" tick 2>&1) + [ -z "$out" ] || fail "land-waits: the starting tick printed: $out" + for _ in $(seq 1 50); do [ -d "$w/state/.review-park-lane.lock" ] && break; sleep 0.1; done + [ -d "$w/state/.review-park-lane.lock" ] || fail "land-waits: the relaunch never took the lane lock" + out=$(run_park "$w" land lane 2>&1) + case "$out" in "cleaned up lane: "*) ;; *) fail "land-waits: $out" ;; esac + [ "$(cut -d' ' -f1-2 "$w/order.log" | paste -sd, -)" = "lane relaunch,teardown" ] \ + || fail "land-waits: the teardown did not wait for the relaunch: $(cat "$w/order.log")" + pass "a merge-time cleanup waits for a relaunch of the same lane that is under way" +} + +fm_test_run \ + test_github_gate_clean_and_each_unclean_condition \ + test_github_gate_refuses_until_every_required_check_reports \ + test_github_gate_waits_for_review_bots_or_quiet_window \ + test_gitlab_gate_clean_and_unclean_conditions \ + test_park_stops_agent_records_lane_and_leases_slot \ + test_park_refuses_unsupported_backend_before_side_effects \ + test_park_refuses_when_head_moves_after_the_verdict \ + test_park_refuses_when_the_lane_is_steered_before_the_stop \ + test_park_serializes_final_check_with_inbox_delivery \ + test_park_does_not_hold_metadata_lock_through_control_exit \ + test_interrupted_park_reads_as_parking_and_the_tick_settles_it \ + test_hung_pending_check_ends_within_the_tick \ + test_failed_park_record_write_leaves_the_agent_running \ + test_park_preflight_refusals_leave_the_agent_running \ + test_repark_reuses_its_own_lease_and_refuses_a_foreign_one \ + test_each_relaunch_trigger_relaunches_a_fresh_worker \ + test_edited_comment_beyond_first_hundred_triggers_relaunch \ + test_behind_base_does_not_relaunch \ + test_slow_relaunch_is_never_cut_short_by_the_tick \ + test_tick_resumes_where_the_last_one_ran_out_of_time \ + test_quiet_parked_lane_and_approval_do_not_relaunch \ + test_failed_relaunch_keeps_record_and_stops_retrying \ + test_closed_pr_is_reported_once \ + test_tick_parks_clean_idle_lanes_only_when_opted_in \ + test_land_requires_opt_in_but_always_cleans_parked \ + test_land_queue_fails_when_cleanup_request_cannot_be_written \ + test_land_report_keeps_lost_pool_slot_details \ + test_detached_land_returns_at_once_and_the_tick_reports_it \ + test_pending_land_recovers_after_watcher_restart \ + test_pending_land_retries_after_control_lock_refusal \ + test_killed_cleanup_is_retried_and_reported_once \ + test_stale_cleanup_request_for_another_pr_is_dropped \ + test_reports_survive_until_their_wake_is_recorded \ + test_land_waits_for_an_action_under_way diff --git a/tests/fm-pr-check-security.test.sh b/tests/fm-pr-check-security.test.sh index da3a89d1a7d..424de1ce18b 100755 --- a/tests/fm-pr-check-security.test.sh +++ b/tests/fm-pr-check-security.test.sh @@ -377,6 +377,24 @@ write_task_meta() { "mode=no-mistakes" } +# Gives the case's copy the remotes origin and backup, both served by +# fakebin/git-remote-probe, and a named head that no ref outside the copy holds. +# Each fetch appends a line to FM_TEST_REMOTE_LOG, sleeps FM_TEST_REMOTE_SLEEP +# seconds (default 0), and fails. +add_probe_remotes() { # <dir> + local dir=$1 + cat > "$dir/fakebin/git-remote-probe" <<'SH' +#!/bin/sh +printf 'fetch %s\n' "$1" >> "$FM_TEST_REMOTE_LOG" +sleep "${FM_TEST_REMOTE_SLEEP:-0}" +exit 1 +SH + chmod +x "$dir/fakebin/git-remote-probe" + git -C "$dir/wt" remote add origin probe::origin + git -C "$dir/wt" remote add backup probe::backup + git -C "$dir/wt" commit -q --allow-empty -m 'named head only in the copy' +} + # Extra "field=value" arguments are written before pr=, because # fm_pr_metadata_identity_parse rejects an unrecognised line after it. write_poll_meta() { @@ -849,10 +867,48 @@ test_direct_pr_unpushed_commit_refuses_registration() { pass "fm-pr-check refuses a direct-PR registration while a later commit is only in the copy" } -# A PR a task reported in a done line is recorded by the scan whatever the rest -# of the line says, so its merge monitoring never waits on the supervisor -# running this command by hand. A URL without the "PR " prefix is prose, a scout -# never delivers a PR, and an already recorded PR is left alone. +# bin/fm-pr-merge.sh re-records through fm-pr-check.sh (FM_PR_CHECK_MERGE=1). +# A GitLab merge request records no pr_head, so that re-record's gate reads the +# copy's own refs: it refreshes them first, accepting a head origin holds +# through the no-mistakes gate clone that the copy never fetched, and still +# refuses a head that only the copy and the local gate mirror hold. +test_merge_rerecord_refreshes_before_refusing() { + local dir state origin sha url + dir=$(make_case merge-rerecord-refresh) + state="$dir/home/state" + origin="$dir/origin.git" + url=https://gitlab.example/group/project/-/merge_requests/4 + ln -sf "$REAL_JQ" "$dir/fakebin/jq" + write_task_meta "$dir" + git init -q --bare "$origin" + git -C "$dir/wt" remote add origin "file://$origin" + git -C "$dir/wt" commit -q --allow-empty -m 'fix the pipeline validates' + sha=$(git -C "$dir/wt" rev-parse HEAD) + git -C "$dir/wt" update-ref refs/remotes/no-mistakes/fm/task-a "$sha" + + FM_PR_CHECK_MERGE=1 run_check_entry "$dir" task-a "$url" > "$dir/stdout" 2> "$dir/stderr" \ + && fail "the merge-time re-record registered an unpushed head" + grep -Fq "named head $sha is unreachable outside the worker copy" "$dir/stderr" \ + || fail "the merge-time refusal did not name the unpushed head: $(cat "$dir/stderr")" + ! grep -q '^pr=' "$state/task-a.meta" || fail "the refused merge-time re-record recorded pr=" + + # The pipeline pushes from its own gate clone, so origin holds the head while + # the copy's remote-tracking refs do not. + git -C "$origin" fetch -q "$dir/wt" "HEAD:refs/heads/fm/task-a" + ! git -C "$dir/wt" rev-parse -q --verify refs/remotes/origin/fm/task-a >/dev/null \ + || fail "fixture: the copy already tracks origin's branch" + FM_PR_CHECK_MERGE=1 run_check_entry "$dir" task-a "$url" > "$dir/stdout" 2> "$dir/stderr" \ + || fail "the merge-time re-record refused a head origin holds: $(cat "$dir/stderr")" + grep -qxF "pr=$url" "$state/task-a.meta" || fail "the merge-time re-record did not record pr=" + [ "$(git -C "$dir/wt" rev-parse refs/remotes/origin/fm/task-a)" = "$sha" ] \ + || fail "the merge-time re-record did not leave origin's head in the copy's refs" + pass "the merge-time re-record refreshes before refusing: origin's unfetched head is accepted, an unpushed one refused" +} + +# A PR a task reported on a canonical ready line is recorded by the scan, so its +# merge monitoring never waits on the supervisor running this command by hand. +# A URL without the "PR " prefix is prose, a scout never delivers a PR, and a +# recorded PR the status log never named is left alone. test_scan_records_a_pr_reported_in_a_done_line() { local dir state out dir=$(make_case scan-records-reported-pr) @@ -860,8 +916,7 @@ test_scan_records_a_pr_reported_in_a_done_line() { write_task_meta "$dir" task-a printf '%s\n' \ 'working [at=1]: rebasing the PR branch' \ - 'done [at=2]: rebased and pushed; PR https://github.com/o/r/pull/7, not draft, only Full suite red' \ - 'done [at=3]: live transcript posted on https://github.com/o/r/pull/99' > "$state/task-a.status" + 'done [at=2]: PR https://github.com/o/r/pull/7 checks green' > "$state/task-a.status" write_task_meta "$dir" task-prose printf '%s\n' 'done [at=2]: opened https://github.com/o/r/pull/8' > "$state/task-prose.status" fm_write_meta "$state/task-scout.meta" 'window=firstmate:fm-task-scout' 'endpoint_task_id=task-scout' \ @@ -957,6 +1012,46 @@ test_scan_rotates_past_a_hanging_registration() { pass "the scan starts a later sweep at a different task so a hanging registration cannot starve it" } +# A scan registration's remote-ref refresh, every remote together, ends inside +# the scan's own budget, so remotes that hang cannot get the registration cut: +# the refusal is printed once and marked instead of retried silently on every +# sweep. Once the refresh's time has passed the sweep starts no other task, so +# that task opens the next sweep and gets its own bounded refresh. +test_scan_bounds_the_remote_refresh_by_its_budget() { + local dir state sha out + dir=$(make_case scan-refresh-hang) + state="$dir/home/state" + write_task_meta "$dir" task-a + write_task_meta "$dir" task-b + add_probe_remotes "$dir" + sha=$(git -C "$dir/wt" rev-parse HEAD) + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/9 checks green' > "$state/task-a.status" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/20 checks green' > "$state/task-b.status" + + out=$(FM_CHECK_TIMEOUT=16 FM_TEST_GH_HEAD=unavailable FM_TEST_REMOTE_SLEEP=60 FM_TEST_REMOTE_LOG="$dir/remote.log" \ + run_check_entry "$dir" scan 2> "$dir/stderr") || fail "the scan failed: $(cat "$dir/stderr")" + [ -s "$dir/remote.log" ] || fail "the scan's registration did not refresh the copy's remotes" + [ "$(printf '%s\n' "$out" | wc -l | tr -d ' ')" = 1 ] || fail "a sweep behind hanging remotes did not report exactly one task: $out" + case "$out" in + "task-a reported https://github.com/o/r/pull/9 in a done line, but it was not recorded: named head $sha is unreachable outside the worker copy; could not refresh remote: "*) ;; + *) fail "the refusal behind hanging remotes was not reported: $out" ;; + esac + [ "$(cat "$state/task-a.pr-register-refused")" = https://github.com/o/r/pull/9 ] \ + || fail "the refusal behind hanging remotes was not marked as reported" + [ ! -e "$state/task-b.pr-register-refused" ] || fail "a task started after the refresh time had passed was refused" + + rm -f "$dir/remote.log" + out=$(FM_CHECK_TIMEOUT=16 FM_TEST_GH_HEAD=unavailable FM_TEST_REMOTE_SLEEP=60 FM_TEST_REMOTE_LOG="$dir/remote.log" \ + run_check_entry "$dir" scan 2> "$dir/stderr") || fail "the second scan failed: $(cat "$dir/stderr")" + [ -s "$dir/remote.log" ] || fail "the task that opened the next sweep got no refresh" + case "$out" in + "task-b reported https://github.com/o/r/pull/20 in a done line, but it was not recorded: named head $sha is unreachable outside the worker copy; could not refresh remote: "*) ;; + *) fail "the next sweep did not open with the task the previous one never started: $out" ;; + esac + [ "$(printf '%s\n' "$out" | wc -l | tr -d ' ')" = 1 ] || fail "the next sweep reported more than one task: $out" + pass "a scan bounds its registrations' remote refresh by its budget and hands the next sweep to the task it never started" +} + # A task whose pr= was written but whose poll was never published (the # registration was cut between the two) is retried, so its merge is still watched. test_scan_publishes_a_missing_poll_for_a_recorded_pr() { @@ -993,7 +1088,7 @@ test_scan_arms_a_replacement_pr_despite_an_old_merge_marker() { dir=$(make_case scan-old-marker-new-pr) state="$dir/home/state" write_poll_meta "$state" task-a https://github.com/o/r/pull/9 'kind=ship' 'mode=no-mistakes' - printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/9 open, not draft' > "$state/task-a.status" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/9 checks green' > "$state/task-a.status" fm_pr_poll_merge_mark_notified "$state" task-a github github.com o/r 7 || fail "could not write the merge marker" run_check_entry "$dir" scan > /dev/null 2> "$dir/stderr" || fail "the scan failed: $(cat "$dir/stderr")" @@ -1020,6 +1115,385 @@ test_scan_does_not_overwrite_a_pr_recorded_meanwhile() { pass "the scan keeps a PR the supervisor recorded during its registration" } +# Only the newest crew-declared line counts, and only in a ready-report shape: +# prose around the URL, a line that says the PR is not ready, and any later +# working, failed, or PR-less done line each leave nothing to register. +test_scan_ignores_non_canonical_done_lines() { + local dir state id out + dir=$(make_case scan-non-canonical) + state="$dir/home/state" + write_task_meta "$dir" task-prose + printf '%s\n' 'done [at=2]: compared against PR https://github.com/o/r/pull/1 checks green' > "$state/task-prose.status" + write_task_meta "$dir" task-red + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/2 opened, CI still red' > "$state/task-red.status" + write_task_meta "$dir" task-working + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/3 checks green' 'working [at=3]: fixing review notes' > "$state/task-working.status" + write_task_meta "$dir" task-failed + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/4 checks green' 'failed [at=3]: the branch went away' > "$state/task-failed.status" + write_task_meta "$dir" task-later-done + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/5 checks green' 'done [at=3]: cleanup finished' > "$state/task-later-done.status" + write_task_meta "$dir" task-paused + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/6 checks green' 'paused [at=3]: waiting on a rebase' > "$state/task-paused.status" + write_task_meta "$dir" task-note + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/7 checks green' 'note [at=3]: the PR is not ready' > "$state/task-note.status" + + out=$(run_check_entry "$dir" scan 2> "$dir/stderr") || fail "the scan failed: $(cat "$dir/stderr")" + [ -z "$out" ] || fail "a scan with nothing to register printed: $out" + for id in task-prose task-red task-working task-failed task-later-done task-paused task-note; do + ! grep -q '^pr=' "$state/$id.meta" || fail "$id registered a PR from a non-canonical line" + [ ! -e "$state/$id.check.sh" ] || fail "$id armed a poll from a non-canonical line" + done + pass "the scan registers nothing from a non-canonical or superseded done line" +} + +# A resolved or captain-held line is supervisor bookkeeping rather than +# the crew changing state, so it never withdraws a ready line. Each forge's +# ready shape registers. +test_scan_reads_the_ready_line_past_supervisor_annotations() { + local dir state + dir=$(make_case scan-annotations) + state="$dir/home/state" + write_task_meta "$dir" task-a + printf '%s\n' \ + 'needs-decision [at=1] [key=k1]: which base?' \ + 'done [at=2]: PR https://github.com/o/r/pull/7 checks green' \ + 'resolved [at=3] [key=k1]: main' \ + 'captain-held [at=5]: review on Monday' > "$state/task-a.status" + write_task_meta "$dir" task-bare + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/8' > "$state/task-bare.status" + write_task_meta "$dir" task-gitlab + printf '%s\n' 'done [at=2]: PR https://gitlab.example/group/project/-/merge_requests/9 checks green' > "$state/task-gitlab.status" + + run_check_entry "$dir" scan > "$dir/stdout" 2> "$dir/stderr" || fail "the scan failed: $(cat "$dir/stderr")" + grep -qxF 'pr=https://github.com/o/r/pull/7' "$state/task-a.meta" \ + || fail "a supervisor annotation withdrew the ready line: $(cat "$dir/stdout")" + grep -qxF 'pr=https://github.com/o/r/pull/8' "$state/task-bare.meta" \ + || fail "a bare PR line was not registered: $(cat "$dir/stdout")" + grep -qxF 'pr=https://gitlab.example/group/project/-/merge_requests/9' "$state/task-gitlab.meta" \ + || fail "a GitLab ready line was not registered: $(cat "$dir/stdout")" + pass "the scan reads the ready line past supervisor annotations, on each ready shape" +} + +# A lane that closes one PR and opens a replacement moves its watch: the log +# named the old PR and the newest ready line names another. +test_scan_replaces_a_recorded_pr_named_by_an_older_line() { + local dir state out + dir=$(make_case scan-replacement) + state="$dir/home/state" + write_task_meta "$dir" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/7 checks green' > "$state/task-a.status" + run_check_entry "$dir" scan > /dev/null 2> "$dir/stderr" || fail "the first scan failed: $(cat "$dir/stderr")" + grep -qxF 'pr=https://github.com/o/r/pull/7' "$state/task-a.meta" || fail "the first PR was not recorded" + printf '%s\n' 'working [at=3]: PR 7 closed, reopening' 'done [at=4]: PR https://github.com/o/r/pull/8 checks green' >> "$state/task-a.status" + + out=$(run_check_entry "$dir" scan 2> "$dir/stderr") || fail "the replacement scan failed: $(cat "$dir/stderr")" + [ -z "$out" ] || fail "a replacement printed: $out" + grep -qxF 'pr=https://github.com/o/r/pull/8' "$state/task-a.meta" || fail "the replacement PR did not take over pr=" + [ "$(grep -c '^pr=' "$state/task-a.meta")" = 1 ] || fail "the replacement left two pr= lines" + fm_pr_poll_artifacts_valid "$state" task-a "$POLL" || fail "the replacement left an invalid poll" + [ "$FM_PR_DATA_URL" = 'https://github.com/o/r/pull/8' ] || fail "the poll still watches $FM_PR_DATA_URL" + pass "a replacement PR on a newer ready line takes over pr= and the merge poll" +} + +# Only the log proves a line newer than the recorded PR. A pr= the log never +# named is the supervisor's own choice, and a URL that merely starts with the +# recorded one does not name it. +test_scan_failed_replacement_restores_previous_watch() { + local dir state before + dir=$(make_case scan-failed-replacement) + state="$dir/home/state" + rm -f "$dir/home/config/pr-degraded-watch" + write_task_meta "$dir" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/7 checks green' > "$state/task-a.status" + run_check_entry "$dir" scan > /dev/null 2> "$dir/stderr" || fail "the first scan failed: $(cat "$dir/stderr")" + grep -qxF 'pr=https://github.com/o/r/pull/7' "$state/task-a.meta" || fail "the first PR was not recorded" + before=$(poll_artifact_snapshot "$state" task-a) + printf '%s\n' 'working [at=3]: PR 7 closed, reopening' 'done [at=4]: PR https://github.com/o/r/pull/8 checks green' >> "$state/task-a.status" + cat > "$dir/fakebin/mv" <<'SH' +#!/usr/bin/env bash +if [ "${!#}" = "$FM_TEST_FAIL_REG_DEST" ] && [ ! -e "$FM_TEST_FAIL_REG_ONCE" ]; then + : > "$FM_TEST_FAIL_REG_ONCE" + exit 1 +fi +exec "$FM_TEST_REAL_MV" "$@" +SH + chmod +x "$dir/fakebin/mv" + FM_TEST_FAIL_REG_DEST="$state/task-a.pr-poll-registration" \ + FM_TEST_FAIL_REG_ONCE="$dir/fail-once" FM_TEST_REAL_MV="$REAL_MV" \ + run_check_entry "$dir" scan > "$dir/scan.out" 2> "$dir/scan.err" + [ -e "$dir/fail-once" ] || fail "the publication failure was not forced" + grep -qxF 'pr=https://github.com/o/r/pull/7' "$state/task-a.meta" || fail "failed replacement lost prior pr=: $(grep '^pr=' "$state/task-a.meta")" + [ "$(poll_artifact_snapshot "$state" task-a)" = "$before" ] || fail "failed replacement changed prior poll" + pass "a failed scan replacement restores the previous watch without degraded watching" +} + +test_scan_keeps_a_recorded_pr_the_log_never_named() { + local dir state + dir=$(make_case scan-keeps-unnamed-pr) + state="$dir/home/state" + write_poll_meta "$state" task-a https://github.com/o/r/pull/9 'kind=ship' 'mode=no-mistakes' + printf '%s\n' \ + 'done [at=2]: PR https://github.com/o/r/pull/90 checks green' \ + 'done [at=3]: PR https://github.com/o/r/pull/9-checks checks green' \ + 'done [at=4]: PR https://github.com/o/r/pull/91 checks green' > "$state/task-a.status" + cp "$state/task-a.meta" "$dir/meta.before" + + run_check_entry "$dir" scan > /dev/null 2> "$dir/stderr" || fail "the scan failed: $(cat "$dir/stderr")" + cmp -s "$dir/meta.before" "$state/task-a.meta" || fail "the scan overwrote a pr= the log never named" + pass "the scan keeps a recorded PR the status log never named" +} + +# A merged PR ended the lane's delivery, so a later line does not re-open it. +test_scan_does_not_replace_a_merged_pr() { + local dir state + dir=$(make_case scan-merged-not-replaced) + state="$dir/home/state" + write_poll_meta "$state" task-a https://github.com/o/r/pull/7 'kind=ship' 'mode=no-mistakes' + printf '%s\n' \ + 'done [at=2]: PR https://github.com/o/r/pull/7 checks green' \ + 'done [at=3]: PR https://github.com/o/r/pull/8 checks green' > "$state/task-a.status" + fm_pr_poll_merge_mark_notified "$state" task-a github github.com o/r 7 || fail "could not write the merge marker" + + run_check_entry "$dir" scan > /dev/null 2> "$dir/stderr" || fail "the scan failed: $(cat "$dir/stderr")" + grep -qxF 'pr=https://github.com/o/r/pull/7' "$state/task-a.meta" || fail "the scan replaced a merged PR" + pass "the scan does not replace a PR whose merge was already reported" +} + +# A merge record holds the task's merge-notification lock while it writes the +# merge notice, and the scan holds that same lock from its recheck through the +# pr= write, so a replacement can never interleave with the record of the PR it +# replaces. Holding the wake-queue lock stalls the real merge record mid-write +# with that lock held; the scan must wait for it and then leave the merged PR. +test_scan_does_not_replace_a_pr_whose_merge_is_being_recorded() { + local dir state holder_pid merge_pid scan_pid + dir=$(make_case scan-merge-race) + state="$dir/home/state" + write_task_meta "$dir" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/7 checks green' > "$state/task-a.status" + run_check_entry "$dir" scan >/dev/null 2>"$dir/stderr" || fail "the first scan failed: $(cat "$dir/stderr")" + grep -qxF 'pr=https://github.com/o/r/pull/7' "$state/task-a.meta" || fail "the first scan did not record pull/7" + printf '%s\n' 'done [at=3]: PR https://github.com/o/r/pull/8 checks green' >> "$state/task-a.status" + ( + export STATE=$state + # shellcheck source=/dev/null + . "$ROOT/bin/fm-wake-lib.sh" + fm_lock_acquire_wait "$state/.wake-queue.lock" || exit 1 + : > "$dir/queue-held" + # Bounded, so a failed assertion cannot strand this holder. + for _ in $(seq 600); do [ ! -e "$dir/queue-release" ] || break; sleep 0.1; done + fm_lock_release "$state/.wake-queue.lock" + ) & + holder_pid=$! + until [ -e "$dir/queue-held" ]; do + kill -0 "$holder_pid" 2>/dev/null || fail "could not hold the wake-queue lock" + sleep 0.1 + done + ( + # shellcheck source=/dev/null + . "$ROOT/bin/fm-merge-outcome-lib.sh" + FM_HOME="$dir/home" FM_CONFIG_OVERRIDE="$dir/home/config" \ + fm_merge_outcome_report "$dir/home" "$state" task-a https://github.com/o/r/pull/7 self + ) >/dev/null 2>"$dir/merge.stderr" & + merge_pid=$! + until [ -d "$state/task-a.pr-poll-merge-notified.lock" ]; do + kill -0 "$merge_pid" 2>/dev/null || fail "the merge record exited before taking its locks: $(cat "$dir/merge.stderr")" + sleep 0.1 + done + FM_CHECK_TIMEOUT=60 run_check_entry "$dir" scan >/dev/null 2>"$dir/stderr" & + scan_pid=$! + sleep 2 + kill -0 "$scan_pid" 2>/dev/null || fail "the scan did not wait for the merge record holding its lock: $(cat "$dir/stderr")" + grep -qxF 'pr=https://github.com/o/r/pull/7' "$state/task-a.meta" || fail "the scan replaced pr= while the merge record held its lock" + : > "$dir/queue-release" + wait "$holder_pid" || fail "could not release the wake-queue lock" + wait "$merge_pid" || fail "the merge record failed: $(cat "$dir/merge.stderr")" + wait "$scan_pid" || fail "the scan failed: $(cat "$dir/stderr")" + fm_pr_poll_merge_already_notified "$state" task-a github github.com o/r 7 || fail "the merge notice was not recorded" + grep -qxF 'pr=https://github.com/o/r/pull/7' "$state/task-a.meta" || fail "the merged PR was replaced after its merge record" + pass "a merge record holding its lock blocks the scan from replacing that PR" +} + +test_scan_keeps_the_watch_when_replacement_status_changes_during_reads() { + local dir state before scan_pid + dir=$(make_case scan-status-race) + state="$dir/home/state" + write_task_meta "$dir" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/7 checks green' > "$state/task-a.status" + run_check_entry "$dir" scan >/dev/null 2>"$dir/stderr" || fail "the first scan failed: $(cat "$dir/stderr")" + printf '%s\n' 'done [at=3]: PR https://github.com/o/r/pull/8 checks green' >> "$state/task-a.status" + before=$(poll_artifact_snapshot "$state" task-a) + + FM_CHECK_TIMEOUT=60 FM_TEST_GH_SLOW_URL=pull/8 FM_TEST_GH_SLOW_SECONDS=3 \ + run_check_entry "$dir" scan >/dev/null 2>"$dir/stderr" & + scan_pid=$! + for _ in $(seq 100); do + grep -q 'pull/8' "$dir/gh.log" 2>/dev/null && break + kill -0 "$scan_pid" 2>/dev/null || fail "the scan exited before reading pull/8: $(cat "$dir/stderr")" + sleep 0.1 + done + grep -q 'pull/8' "$dir/gh.log" || fail "the scan never started reading pull/8" + printf '%s\n' 'working: replacement withdrawn' >> "$state/task-a.status" + wait "$scan_pid" || fail "the scan failed: $(cat "$dir/stderr")" + + grep -qxF 'pr=https://github.com/o/r/pull/7' "$state/task-a.meta" || fail "a withdrawn replacement changed pr=" + [ "$(poll_artifact_snapshot "$state" task-a)" = "$before" ] || fail "a withdrawn replacement changed the pull/7 poll" + pass "a scan keeps the recorded PR when its ready status is withdrawn during forge reads" +} + +# A refused replacement keeps the current watch and reports the refusal, and a +# PR the supervisor records during the replacement outranks it. +test_scan_replacement_keeps_the_watch_it_cannot_take_over() { + local dir state out + dir=$(make_case scan-replacement-refused) + state="$dir/home/state" + write_task_meta "$dir" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/7 checks green' > "$state/task-a.status" + run_check_entry "$dir" scan > /dev/null 2> "$dir/stderr" || fail "the first scan failed: $(cat "$dir/stderr")" + printf '%s\n' 'done [at=3]: PR https://github.com/o/r/pull/8 checks green' >> "$state/task-a.status" + out=$(FM_TEST_GH_DRAFT=true run_check_entry "$dir" scan 2> "$dir/stderr") || fail "the scan failed: $(cat "$dir/stderr")" + case "$out" in + *'pull/8 in a done line, but it was not recorded: '*draft*) ;; + *) fail "the refused replacement was not reported: $out" ;; + esac + grep -qxF 'pr=https://github.com/o/r/pull/7' "$state/task-a.meta" || fail "a refused replacement changed pr=" + fm_pr_poll_artifacts_valid "$state" task-a "$POLL" || fail "a refused replacement broke the current poll" + + FM_CHECK_TIMEOUT=60 FM_TEST_GH_SLOW_URL=pull/8 FM_TEST_GH_SLOW_SECONDS=3 \ + run_check_entry "$dir" scan > /dev/null 2> "$dir/stderr" & + sleep 1 + write_poll_meta "$state" task-a https://github.com/o/r/pull/21 'kind=ship' 'mode=no-mistakes' + wait $! || fail "the scan failed: $(cat "$dir/stderr")" + grep -qxF 'pr=https://github.com/o/r/pull/21' "$state/task-a.meta" \ + || fail "a replacement overwrote a PR recorded while it ran" + pass "a replacement keeps the current watch when refused and yields to a PR recorded meanwhile" +} + +# A registration that never finishes publishes that the merge watch could not be +# armed, once per task and PR, while later sweeps keep retrying; a refusal after +# it still reports, and success clears both records. +test_scan_reports_a_timed_out_registration_once_and_retries() { + local dir state out + dir=$(make_case scan-timeout-notice) + state="$dir/home/state" + write_task_meta "$dir" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/9 checks green' > "$state/task-a.status" + + out=$(FM_CHECK_TIMEOUT=7 FM_TEST_GH_SLOW_URL=pull/9 run_check_entry "$dir" scan 2> "$dir/stderr") \ + || fail "the scan failed: $(cat "$dir/stderr")" + case "$out" in + 'task-a reported https://github.com/o/r/pull/9 in a done line, but its PR merge watch could not be armed: registration timed out after '[0-9]*'s, and later sweeps keep retrying') ;; + *) fail "a timed-out registration did not publish its notice: $out" ;; + esac + ! grep -q '^pr=' "$state/task-a.meta" || fail "a timed-out registration recorded the PR" + + out=$(FM_CHECK_TIMEOUT=7 FM_TEST_GH_SLOW_URL=pull/9 run_check_entry "$dir" scan 2> "$dir/stderr") \ + || fail "the repeated scan failed: $(cat "$dir/stderr")" + [ -z "$out" ] || fail "the same timeout was reported twice: $out" + + out=$(FM_TEST_GH_DRAFT=true run_check_entry "$dir" scan 2> "$dir/stderr") \ + || fail "the refusal scan failed: $(cat "$dir/stderr")" + case "$out" in + *'it was not recorded: '*draft*) ;; + *) fail "an earlier timeout hid a later refusal: $out" ;; + esac + + out=$(run_check_entry "$dir" scan 2> "$dir/stderr") || fail "the retry scan failed: $(cat "$dir/stderr")" + [ -z "$out" ] || fail "a successful retry printed: $out" + fm_pr_poll_artifacts_valid "$state" task-a "$POLL" || fail "the retry did not arm the poll" + [ ! -e "$state/task-a.pr-register-timed-out" ] || fail "a recorded PR kept its timeout marker" + [ ! -e "$state/task-a.pr-register-refused" ] || fail "a recorded PR kept its refusal marker" + pass "a timed-out registration publishes one notice, retries, and clears on success" +} + +# The once-only marker means the notice was durably queued: a queue that cannot +# take the wake leaves the notice owed, so a later sweep still delivers it, once. +test_scan_keeps_a_notice_owed_until_its_wake_is_queued() { + local dir state out reason + for reason in timeout refusal; do + dir=$(make_case "scan-notice-owed-$reason") + state="$dir/home/state" + write_task_meta "$dir" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/9 checks green' > "$state/task-a.status" + mkdir "$state/.wake-queue" + if [ "$reason" = timeout ]; then + out=$(FM_CHECK_TIMEOUT=7 FM_TEST_GH_SLOW_URL=pull/9 run_check_entry "$dir" scan 2> "$dir/stderr") \ + || fail "the $reason scan failed: $(cat "$dir/stderr")" + else + out=$(FM_TEST_GH_DRAFT=true run_check_entry "$dir" scan 2> "$dir/stderr") \ + || fail "the $reason scan failed: $(cat "$dir/stderr")" + fi + [ -z "$out" ] || fail "a $reason notice that was not queued was still reported: $out" + [ ! -e "$state/task-a.pr-register-timed-out" ] && [ ! -e "$state/task-a.pr-register-refused" ] \ + || fail "a $reason notice was marked sent before its wake was queued" + + rmdir "$state/.wake-queue" + if [ "$reason" = timeout ]; then + out=$(FM_CHECK_TIMEOUT=7 FM_TEST_GH_SLOW_URL=pull/9 run_check_entry "$dir" scan 2> "$dir/stderr") \ + || fail "the $reason retry failed: $(cat "$dir/stderr")" + else + out=$(FM_TEST_GH_DRAFT=true run_check_entry "$dir" scan 2> "$dir/stderr") \ + || fail "the $reason retry failed: $(cat "$dir/stderr")" + fi + [ -n "$out" ] || fail "a later sweep did not report the owed $reason notice" + [ "$(grep -c 'check: pr-register: task-a reported' "$state/.wake-queue")" = 1 ] \ + || fail "the owed $reason notice was not queued once: $(cat "$state/.wake-queue")" + done + pass "a timeout or refusal notice stays owed until its wake is durably queued, then is queued once" +} + +# The bound can land after the registration installed its cleanup trap, where +# the worker must still report the bound rather than an ordinary failure. +test_scan_reports_a_timeout_during_poll_preparation() { + local dir state out + dir=$(make_case scan-timeout-after-trap) + state="$dir/home/state" + write_task_meta "$dir" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/9 checks green' > "$state/task-a.status" + cat > "$dir/fakebin/cp" <<'SH' +#!/usr/bin/env bash +sleep 30 +SH + chmod +x "$dir/fakebin/cp" + + out=$(FM_CHECK_TIMEOUT=7 run_check_entry "$dir" scan 2> "$dir/stderr") \ + || fail "the scan failed: $(cat "$dir/stderr")" + case "$out" in + *'its PR merge watch could not be armed: registration timed out after '*) ;; + *) fail "a timeout during poll preparation was not reported as one: $out" ;; + esac + [ ! -e "$state/task-a.check.sh" ] || fail "a timed-out preparation published a poll" + ! find "$state" -name '.fm-pr-poll-*' -print | grep . > /dev/null \ + || fail "a timed-out preparation left temporary files" + pass "a timeout that lands after the cleanup trap is still reported as a timeout" +} + +# A scan cut by the watcher's own check bound never reached its own queueing, so +# the watcher queues that cut durably, once, and nothing the scan queued twice. +test_watcher_queues_a_scan_cut_at_the_check_bound() { + local dir state rc real_sed + real_sed=$(type -P sed) + dir=$(make_case watcher-scan-cut) + state="$dir/home/state" + : > "$dir/home/config/pr-register-scan" + write_task_meta "$dir" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/9 checks green' > "$state/task-a.status" + add_stop_custom_check "$dir" + cat > "$dir/fakebin/sed" <<SH +#!/usr/bin/env bash +case "\$*" in *'s/^kind=//p'*) sleep 30 ;; esac +exec "$real_sed" "\$@" +SH + chmod +x "$dir/fakebin/sed" + set +e + FM_TEST_CHECK_TIMEOUT=2 run_watcher_bounded "$dir/home" "$dir/fakebin" > "$dir/watch.out" 2> "$dir/watch.err" + rc=$? + set -e + [ "$rc" -eq 0 ] || fail "the cut-scan watcher failed: $(cat "$dir/watch.err")" + [ "$(grep -c 'check: pr-register: timed out after 2s' "$state/.wake-queue")" = 1 ] || + fail "a scan cut at the check bound was not queued once: $(cat "$state/.wake-queue" 2>/dev/null)" + pass "the watcher durably queues a scan cut at its check bound, once" +} + # With config/pr-register-scan present, the watcher's check sweep runs the # scan, so a reported PR gets its merge poll without the supervisor, and a # refusal becomes a durable wake. With the flag absent, the sweep is unchanged. @@ -1028,7 +1502,7 @@ test_watcher_sweep_records_a_reported_pr() { dir=$(make_case watcher-scan-off) state="$dir/home/state" write_task_meta "$dir" - printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/7 open, not draft' > "$state/task-a.status" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/7 checks green' > "$state/task-a.status" add_stop_custom_check "$dir" set +e run_watcher_bounded "$dir/home" "$dir/fakebin" > "$dir/watch.out" 2> "$dir/watch.err" @@ -1046,7 +1520,7 @@ test_watcher_sweep_records_a_reported_pr() { state="$dir/home/state" : > "$dir/home/config/pr-register-scan" write_task_meta "$dir" - printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/7 open, not draft' > "$state/task-a.status" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/7 checks green' > "$state/task-a.status" add_stop_custom_check "$dir" set +e run_watcher_bounded "$dir/home" "$dir/fakebin" > "$dir/watch.out" 2> "$dir/watch.err" @@ -1070,7 +1544,21 @@ test_watcher_sweep_records_a_reported_pr() { [ "$rc" -eq 0 ] || fail "the refusal watcher failed: $(cat "$dir/watch.err")" grep -F 'check: pr-register: task-a reported https://github.com/o/r/pull/9' "$state/.wake-queue" >/dev/null \ || fail "the refusal was not queued as a durable wake: $(cat "$state/.wake-queue" 2>/dev/null)" - pass "the watcher sweep scans only with config/pr-register-scan present, then records a reported PR and queues a refusal" + + dir=$(make_case watcher-scan-timeout) + state="$dir/home/state" + : > "$dir/home/config/pr-register-scan" + write_task_meta "$dir" + printf '%s\n' 'done [at=2]: PR https://github.com/o/r/pull/9 checks green' > "$state/task-a.status" + add_stop_custom_check "$dir" + set +e + FM_CHECK_TIMEOUT=7 FM_TEST_GH_SLOW_URL=pull/9 run_watcher_bounded "$dir/home" "$dir/fakebin" > "$dir/watch.out" 2> "$dir/watch.err" + rc=$? + set -e + [ "$rc" -eq 0 ] || fail "the timeout watcher failed: $(cat "$dir/watch.err")" + [ "$(grep -cF 'check: pr-register: task-a reported https://github.com/o/r/pull/9 in a done line, but its PR merge watch could not be armed' "$state/.wake-queue")" = 1 ] \ + || fail "the timeout was not queued as exactly one durable wake: $(cat "$state/.wake-queue" 2>/dev/null)" + pass "the watcher sweep scans only with config/pr-register-scan present, then records a reported PR and queues a refusal or a timeout" } test_valid_recording_and_merge_derivation() { @@ -2592,6 +3080,7 @@ test_teardown_removes_poll_artifacts() { printf 'degraded\n' > "$dir/home/state/task-a.pr-poll-degraded" printf 'trust\n' > "$dir/home/state/task-a.check-trust" printf 'https://github.com/o/r/pull/9\n' > "$dir/home/state/task-a.pr-register-refused" + printf 'https://github.com/o/r/pull/9\n' > "$dir/home/state/task-a.pr-register-timed-out" cat > "$fakebin/tmux" <<'SH' #!/usr/bin/env bash exit 0 @@ -2608,6 +3097,7 @@ SH [ ! -e "$dir/home/state/task-a.pr-poll-degraded" ] || fail "teardown left the degraded dedupe marker" [ ! -e "$dir/home/state/task-a.check-trust" ] || fail "teardown left the custom check registration" [ ! -e "$dir/home/state/task-a.pr-register-refused" ] || fail "teardown left the PR registration refusal marker" + [ ! -e "$dir/home/state/task-a.pr-register-timed-out" ] || fail "teardown left the PR registration timeout marker" dir=$(make_case teardown-retirement-receipt) fakebin="$dir/fakebin" @@ -3811,6 +4301,111 @@ test_merged_poll_retries_a_failed_upward_report() { pass "a failed upward merge report keeps its poll armed for repair and retry" } +# A merge this home performed through bin/fm-pr-merge.sh is recorded before the +# poll sees it; the poll absorbs the duplicate notification but still starts the +# lane's merge-time cleanup, detached so even a slow teardown never holds the +# watcher iteration, and the next park tick reports its outcome. +test_self_merged_lane_is_still_cleaned_up_by_the_poll() { + local dir state url rc + url=https://github.com/o/r/pull/1 + dir=$(make_case self-merge-cleanup) + state="$dir/home/state" + : > "$dir/home/config/auto-land" + write_task_meta "$dir" task-a + run_check_entry "$dir" task-a "$url" >/dev/null 2>"$dir/seed.err" \ + || fail "self-merge-cleanup: could not arm merge poll" + run_merge_entry "$dir" task-a "$url" >"$dir/merge.out" 2>"$dir/merge.err" \ + || fail "self-merge-cleanup: merge entrypoint failed: $(cat "$dir/merge.err")" + [ -f "$state/task-a.pr-poll-merge-notified" ] \ + || fail "self-merge-cleanup: the merge entrypoint did not record the outcome first" + printf '#!/usr/bin/env bash\nprintf started > "%s/teardown.started"\nwhile [ ! -e "%s/teardown.release" ]; do sleep 0.05; done\nprintf "%%s\\n" "$*" >> "%s/teardown.log"\n' "$dir" "$dir" "$dir" > "$dir/teardown-stub" + chmod +x "$dir/teardown-stub" + retire_contributions_observer "$dir" + add_stop_custom_check "$dir" + # The merge entrypoint's queued outcome surfaces as a recovery wake first. + FM_TEST_GH_STATE=MERGED run_watcher_bounded "$dir/home" "$dir/fakebin" >"$dir/recovery.out" 2>"$dir/recovery.err" \ + || fail "self-merge-cleanup: recovery cycle failed: $(cat "$dir/recovery.err")" + grep -F 'check: rearm-resurface' "$dir/recovery.out" >/dev/null \ + || fail "self-merge-cleanup: the recorded merge did not surface as a recovery wake: $(cat "$dir/recovery.out")" + ack_watcher_cycle "$state" || fail "self-merge-cleanup: recovery acknowledgement failed" + rm -f "$state/.last-check" + set +e + FM_PARK_TEARDOWN_BIN="$dir/teardown-stub" FM_TEST_GH_STATE=MERGED \ + run_watcher_bounded "$dir/home" "$dir/fakebin" >"$dir/watch.out" 2>"$dir/watch.err" + rc=$? + set -e + [ "$rc" -eq 0 ] || fail "self-merge-cleanup: watcher failed: $(cat "$dir/watch.err")" + for _ in $(seq 1 200); do [ -e "$dir/teardown.started" ] && break; sleep 0.05; done + [ -e "$dir/teardown.started" ] \ + || fail "self-merge-cleanup: detached teardown did not start" + [ ! -e "$dir/teardown.log" ] \ + || fail "self-merge-cleanup: teardown completed before release" + : > "$dir/teardown.release" + assert_grep 'stop-cycle' "$dir/watch.out" \ + "self-merge-cleanup: the absorbed duplicate did not fall through to the next check" + assert_poll_absent "$state" task-a + for _ in $(seq 1 200); do compgen -G "$state/.review-park-report-*" >/dev/null && break; sleep 0.1; done + [ "$(cat "$dir/teardown.log" 2>/dev/null)" = task-a ] \ + || fail "self-merge-cleanup: the poll did not clean up the self-merged lane: $(cat "$dir/teardown.log" 2>/dev/null)" + FM_HOME="$dir/home" "$ROOT/bin/fm-park.sh" tick >"$dir/tick.out" 2>&1 \ + || fail "self-merge-cleanup: the reporting tick failed: $(cat "$dir/tick.out")" + assert_grep "cleaned up task-a: PR $url merged" "$dir/tick.out" \ + "self-merge-cleanup: the cleanup outcome did not reach the captain" + pass "a lane merged through bin/fm-pr-merge.sh is still cleaned up when its poll sees the merge" +} + +test_merged_poll_stays_armed_when_cleanup_queue_fails() { + local dir state url rc + url=https://github.com/o/r/pull/1 + dir=$(make_case merge-queue-failure) + state="$dir/home/state" + : > "$dir/home/config/auto-land" + write_task_meta "$dir" task-a + run_check_entry "$dir" task-a "$url" >/dev/null 2>"$dir/seed.err" \ + || fail "merge-queue-failure: could not arm merge poll" + retire_contributions_observer "$dir" + add_stop_custom_check "$dir" + run_merge_entry "$dir" task-a "$url" >"$dir/merge.out" 2>"$dir/merge.err" \ + || fail "merge-queue-failure: merge entrypoint failed: $(cat "$dir/merge.err")" + FM_TEST_GH_STATE=MERGED run_watcher_bounded "$dir/home" "$dir/fakebin" \ + >"$dir/recovery.out" 2>"$dir/recovery.err" \ + || fail "merge-queue-failure: recovery cycle failed: $(cat "$dir/recovery.err")" + grep -F 'check: rearm-resurface' "$dir/recovery.out" >/dev/null \ + || fail "merge-queue-failure: merged result did not surface for recovery" + ack_watcher_cycle "$state" || fail "merge-queue-failure: recovery acknowledgement failed" + rm -f "$state/.last-check" + printf '#!/usr/bin/env bash\nprintf "%%s\\n" "$*" >> "%s/teardown.log"\n' "$dir" > "$dir/teardown-stub" + chmod +x "$dir/teardown-stub" + cat > "$dir/fakebin/mv" <<SH +#!/usr/bin/env bash +dest=\${!#} +[ "\$dest" != "$state/task-a.review-park-land" ] || exit 1 +exec /bin/mv "\$@" +SH + chmod +x "$dir/fakebin/mv" + set +e + FM_TEST_GH_STATE=MERGED run_watcher_bounded "$dir/home" "$dir/fakebin" \ + >"$dir/watch-failed.out" 2>"$dir/watch-failed.err" + rc=$? + set -e + [ -f "$state/task-a.check.sh" ] \ + || fail "merge-queue-failure: watcher retired poll after queue failure (rc=$rc)" + [ ! -e "$dir/teardown.log" ] \ + || fail "merge-queue-failure: teardown started without a persisted request" + + ack_watcher_cycle "$state" || fail "merge-queue-failure: failed cycle acknowledgement failed" + rm -f "$dir/fakebin/mv" "$state/.last-check" + FM_PARK_TEARDOWN_BIN="$dir/teardown-stub" FM_TEST_GH_STATE=MERGED \ + run_watcher_bounded "$dir/home" "$dir/fakebin" \ + >"$dir/watch-retry.out" 2>"$dir/watch-retry.err" \ + || fail "merge-queue-failure: retry failed: $(cat "$dir/watch-retry.err")" + assert_poll_absent "$state" task-a + for _ in $(seq 1 100); do [ -e "$dir/teardown.log" ] && break; sleep 0.1; done + [ -e "$dir/teardown.log" ] \ + || fail "merge-queue-failure: persisted retry did not start teardown" + pass "a failed cleanup queue keeps the merged poll armed until cleanup retries" +} + test_self_merge_and_poll_publish_one_outcome() { local dir state replies url rc url=https://github.com/o/r/pull/1 @@ -5081,6 +5676,8 @@ test_gerrit_nm_ready_gate_requires_recovered_custody test_merged_poll_retires_once test_merged_poll_reregistration_after_notification_is_absorbed test_merged_poll_retries_a_failed_upward_report +test_self_merged_lane_is_still_cleaned_up_by_the_poll +test_merged_poll_stays_armed_when_cleanup_queue_fails test_self_merge_and_poll_publish_one_outcome test_merged_poll_row_carries_the_merge_authority test_merged_poll_row_names_no_authority_when_no_record_grants_one @@ -5101,14 +5698,29 @@ test_draft_pull_request_is_not_armed test_secondmate_record_refuses_a_pr_watch test_unpushed_named_head_refuses_registration test_direct_pr_unpushed_commit_refuses_registration +test_merge_rerecord_refreshes_before_refusing test_scan_records_a_pr_reported_in_a_done_line test_scan_reports_a_refusal_once_and_retries test_scan_retries_refused_task_behind_a_hanging_registration test_scan_rotates_past_a_hanging_registration +test_scan_bounds_the_remote_refresh_by_its_budget test_scan_publishes_a_missing_poll_for_a_recorded_pr test_scan_does_not_overwrite_a_pr_recorded_meanwhile test_scan_does_not_rearm_a_retired_merged_poll test_scan_arms_a_replacement_pr_despite_an_old_merge_marker +test_scan_ignores_non_canonical_done_lines +test_scan_reads_the_ready_line_past_supervisor_annotations +test_scan_replaces_a_recorded_pr_named_by_an_older_line +test_scan_failed_replacement_restores_previous_watch +test_scan_keeps_a_recorded_pr_the_log_never_named +test_scan_does_not_replace_a_merged_pr +test_scan_does_not_replace_a_pr_whose_merge_is_being_recorded +test_scan_keeps_the_watch_when_replacement_status_changes_during_reads +test_scan_replacement_keeps_the_watch_it_cannot_take_over +test_scan_reports_a_timed_out_registration_once_and_retries +test_scan_keeps_a_notice_owed_until_its_wake_is_queued +test_scan_reports_a_timeout_during_poll_preparation +test_watcher_queues_a_scan_cut_at_the_check_bound test_watcher_sweep_records_a_reported_pr test_valid_recording_and_merge_derivation test_rejected_metacharacter_bytes_are_inert diff --git a/tests/fm-pr-merge.test.sh b/tests/fm-pr-merge.test.sh index 6d8bd50b3fa..5ed22460ffd 100755 --- a/tests/fm-pr-merge.test.sh +++ b/tests/fm-pr-merge.test.sh @@ -233,9 +233,15 @@ case "${1:-} ${2:-}" in cat "$FM_TEST_GH_VIEW_JSON" exit 0 ;; + *" --json title,body "*) + [ -f "${FM_TEST_GH_TEXT_JSON:-}" ] || exit 1 + cat "$FM_TEST_GH_TEXT_JSON" + exit 0 + ;; esac ;; "pr merge") + [ -z "${FM_TEST_GH_MERGE_ARGS:-}" ] || printf '%s\0' "$@" > "$FM_TEST_GH_MERGE_ARGS" if [ -n "${FM_TEST_META_AT_MERGE:-}" ] && [ -f "${FM_STATE_OVERRIDE:-}/task-x1.meta" ]; then cat "$FM_STATE_OVERRIDE/task-x1.meta" > "$FM_TEST_META_AT_MERGE" fi @@ -752,6 +758,8 @@ run_pr_merge() { FM_TEST_GH_UPDATE_DONE="$case_dir/github-update-done" \ FM_TEST_GH_UPDATE_HEAD="$case_dir/github-head-after-update" \ FM_TEST_GH_HEAD="$case_dir/github-head" \ + FM_TEST_GH_TEXT_JSON="$case_dir/github-text.json" \ + FM_TEST_GH_MERGE_ARGS="$case_dir/github-merge-args" \ FM_TEST_GH_RUNS="$case_dir/github-runs.json" \ FM_TEST_GH_MERGE_RC_FILE="$case_dir/github-merge-rc" \ FM_TEST_GH_MERGE_OUTPUT="$(cat "$case_dir/github-merge-output" 2>/dev/null || true)" \ @@ -5548,3 +5556,213 @@ test_expected_head_push_after_verify_is_refused_by_the_forge test_expected_head_later_preflight_does_not_adopt_a_new_head test_expected_head_bad_inputs_refuse_before_effects test_expected_head_keeps_existing_refusals + +# The squash-summary cases. A no-mistakes PR body: the operator intent first, +# then the drafted summary, then the appendix and its attestation comment. +SQUASH_TITLE='fix: wake once when a ready PR degrades' +SQUASH_BODY=$'## Intent\n\nraw operator brief with ~/private/paths\n\n## What Changed\n\n- Wake the merge poll once when a ready PR degrades.\n- Add shared PR polling helpers.\n\n Indented detail line.\n### Notes\n- A subheading stays in the section.\n\n## Risk Assessment\n\nlow\n\n## Pipeline\n\n<!-- no-mistakes-pipeline-attestation:v1 {} -->\n' +SQUASH_SECTION=$'- Wake the merge poll once when a ready PR degrades.\n- Add shared PR polling helpers.\n\n Indented detail line.\n### Notes\n- A subheading stays in the section.' + +# Args: case_dir title body. Writes the PR title and body gh reports. +write_github_text() { + jq -n --arg t "$2" --arg b "$3" '{title: $t, body: $b}' > "$1/github-text.json" +} + +# The value gh pr merge received for one flag, or the literal ABSENT. +# Args: case_dir flag +github_merge_flag_value() { + local arg prev='' found=ABSENT + while IFS= read -r -d '' arg; do + if [ "$prev" = "$2" ]; then + found=$arg + fi + prev=$arg + done < "$1/github-merge-args" + printf '%s' "$found" +} + +test_squash_summary_uses_title_and_what_changed() { + local case_dir + case_dir=$(make_case squash-summary) + add_gh_mocks "$case_dir" 5151515151515151515151515151515151515151 + : > "$case_dir/home/config/pr-squash-summary" + write_github_text "$case_dir" "$SQUASH_TITLE" "$SQUASH_BODY" + set +e + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/61 \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + expect_code 0 "$?" "squash-summary: the merge should succeed: $(cat "$case_dir/stderr")" + set -e + grep -qzx -- '--squash' "$case_dir/github-merge-args" \ + || fail "squash-summary: the default squash method was dropped" + [ "$(github_merge_flag_value "$case_dir" --subject)" = "$SQUASH_TITLE (#61)" ] \ + || fail "squash-summary: unexpected subject '$(github_merge_flag_value "$case_dir" --subject)'" + [ "$(github_merge_flag_value "$case_dir" --body)" = "$SQUASH_SECTION" ] \ + || fail "squash-summary: unexpected body: '$(github_merge_flag_value "$case_dir" --body)'" + pass "fm-pr-merge squash-merges with the PR title and only its What Changed section when opted in" +} + +test_squash_summary_reads_crlf_bodies() { + local case_dir + case_dir=$(make_case squash-summary-crlf) + add_gh_mocks "$case_dir" 5151515151515151515151515151515151515151 + : > "$case_dir/home/config/pr-squash-summary" + write_github_text "$case_dir" "$SQUASH_TITLE" "${SQUASH_BODY//$'\n'/$'\r\n'}" + set +e + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/62 \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + expect_code 0 "$?" "squash-summary-crlf: the merge should succeed: $(cat "$case_dir/stderr")" + set -e + [ "$(github_merge_flag_value "$case_dir" --body)" = "$SQUASH_SECTION" ] \ + || fail "squash-summary-crlf: a web-edited CRLF body did not yield the bare section" + pass "fm-pr-merge reads the What Changed section from a CRLF PR body" +} + +# Every case where the summary must not be applied leaves today's merge line. +test_squash_summary_falls_back_to_repository_default() { + local case_dir label + for label in no-flag no-section empty-section unreadable merge-method rebase-method; do + case_dir=$(make_case "squash-fallback-$label") + add_gh_mocks "$case_dir" 5151515151515151515151515151515151515151 + [ "$label" = no-flag ] || : > "$case_dir/home/config/pr-squash-summary" + case "$label" in + no-section) write_github_text "$case_dir" "$SQUASH_TITLE" $'## Summary\n\nno drafted section\n' ;; + empty-section) write_github_text "$case_dir" "$SQUASH_TITLE" $'## What Changed\n\n\n## Testing\n\nok\n' ;; + unreadable) ;; + *) write_github_text "$case_dir" "$SQUASH_TITLE" "$SQUASH_BODY" ;; + esac + set +e + if [ "$label" = merge-method ]; then + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/63 -- --merge \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + elif [ "$label" = rebase-method ]; then + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/63 -- --rebase \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + else + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/63 \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + fi + expect_code 0 "$?" "squash-fallback-$label: the merge should still succeed: $(cat "$case_dir/stderr")" + set -e + if [ "$label" = merge-method ]; then + assert_logged_gh_merge "$case_dir" 63 example/repo --merge + elif [ "$label" = rebase-method ]; then + assert_logged_gh_merge "$case_dir" 63 example/repo --rebase + else + assert_logged_gh_merge "$case_dir" 63 example/repo --squash + fi + if [ "$label" = no-flag ]; then + # Without the flag the run is today's: the exact merge line above, no + # read of the PR text, and nothing new on stderr. + assert_no_grep 'title,body' "$case_dir/gh.log" \ + "squash-fallback-no-flag: the PR text was read without the opt-in" + assert_no_grep 'squash commit' "$case_dir/stderr" \ + "squash-fallback-no-flag: the opt-in reported on a run without it" + fi + if [ "$label" = unreadable ]; then + assert_grep 'keeps the repository default' "$case_dir/stderr" \ + "squash-fallback-unreadable: the unread PR text was not reported" + fi + done + pass "fm-pr-merge keeps the repository's squash text when not opted in, the section is absent or empty, the PR text is unreadable, or the method is merge or rebase" +} + +test_squash_summary_caller_commit_text_wins() { + local case_dir spelling + for spelling in subject subject-eq body body-file short-t short-b short-cluster-b short-F; do + case_dir=$(make_case "squash-caller-$spelling") + add_gh_mocks "$case_dir" 5151515151515151515151515151515151515151 + : > "$case_dir/home/config/pr-squash-summary" + write_github_text "$case_dir" "$SQUASH_TITLE" "$SQUASH_BODY" + set -- --squash + case "$spelling" in + subject) set -- "$@" --subject 'caller subject' ;; + subject-eq) set -- "$@" '--subject=caller subject' ;; + body) set -- "$@" --body 'caller body' ;; + body-file) set -- "$@" --body-file /dev/null ;; + short-t) set -- "$@" -t 'caller subject' ;; + short-b) set -- "$@" -b 'caller body' ;; + short-cluster-b) set -- "$@" -sb 'caller body' ;; + short-F) set -- "$@" -F /dev/null ;; + esac + set +e + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/64 -- "$@" \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + expect_code 0 "$?" "squash-caller-$spelling: the merge should succeed: $(cat "$case_dir/stderr")" + set -e + assert_logged_gh_merge "$case_dir" 64 example/repo "$@" + assert_no_grep 'What Changed\|Wake the merge poll' "$case_dir/gh.log" \ + "squash-caller-$spelling: the PR summary was added beside the caller's own commit text" + done + pass "fm-pr-merge never adds the PR summary when the caller passed its own subject or body" +} + +test_squash_summary_leaves_gitlab_and_forgejo_untouched() { + local case_dir merge_line + case_dir=$(make_gitlab_case squash-gitlab) + : > "$case_dir/home/config/pr-squash-summary" + run_pr_merge "$case_dir" task-x1 "$MR_URL" > "$case_dir/stdout" 2> "$case_dir/stderr" \ + || fail "squash-gitlab: the merge should succeed: $(cat "$case_dir/stderr")" + merge_line=$(glab_merge_line "$case_dir/glab.log") + [ "$merge_line" = "GITLAB_HOST=$MR_HOST mr merge 7 -R $MR_PROJECT_URL --sha $MR_HEAD --yes" ] \ + || fail "squash-gitlab: the opt-in changed the GitLab merge: '$merge_line'" + case_dir=$(make_forgejo_case squash-forgejo) + : > "$case_dir/home/config/pr-squash-summary" + run_pr_merge "$case_dir" task-x1 "$FJ_URL" > "$case_dir/stdout" 2> "$case_dir/stderr" \ + || fail "squash-forgejo: the merge should succeed: $(cat "$case_dir/stderr")" + merge_line=$(tea_merge_line "$case_dir/tea.log") + case "$merge_line" in + *"/pulls/9/merge -f Do=merge -f head_commit_id=$FJ_HEAD") : ;; + *) fail "squash-forgejo: the opt-in changed the Forgejo merge: '$merge_line'" ;; + esac + pass "fm-pr-merge leaves GitLab and Forgejo merges unchanged under the squash-summary opt-in" +} + +# Fenced code is never read as a heading: a fenced "## What Changed" in the +# intent is not the section, and a "## " line inside a ``` or ~~~ block in the +# section does not end it (CommonMark: a fence closes only on a run of the +# same character at least as long as the one that opened it). +test_squash_summary_ignores_headings_inside_code_fences() { + local case_dir label body want + for label in fenced-heading-in-section fenced-section-before-real-one \ + inline-code-in-section inline-code-before-section; do + case "$label" in + # A backtick run whose info string holds a backtick is inline code + # (CommonMark), so it opens no fence that could hide a heading. + inline-code-in-section) + body=$'## What Changed\n\n```fm --help``` now lists the flag.\n- Keep the rest.\n\n## Risk Assessment\n\nlow\n\n<!-- no-mistakes-pipeline-attestation:v1 {} -->\n' + want=$'```fm --help``` now lists the flag.\n- Keep the rest.' + ;; + inline-code-before-section) + body=$'## Intent\n\nRun ```fm --help``` first.\n\n## What Changed\n\n- Real summary.\n\n## Testing\n\nok\n' + want='- Real summary.' + ;; + fenced-heading-in-section) + body=$'## What Changed\n\n- Document the config file.\n\n````md\n## Usage\n```\nfm --help\n````\n\n~~~\n## Not a heading\n~~~\n- Keep the rest.\n\n## Risk Assessment\n\nlow\n' + want=$'- Document the config file.\n\n````md\n## Usage\n```\nfm --help\n````\n\n~~~\n## Not a heading\n~~~\n- Keep the rest.' + ;; + fenced-section-before-real-one) + body=$'## Intent\n\nExample body:\n\n```\n## What Changed\n- fake summary\n```\n\n## What Changed\n\n- Real summary.\n\n## Testing\n\nok\n' + want='- Real summary.' + ;; + esac + case_dir=$(make_case "squash-fence-$label") + add_gh_mocks "$case_dir" 5151515151515151515151515151515151515151 + : > "$case_dir/home/config/pr-squash-summary" + write_github_text "$case_dir" "$SQUASH_TITLE" "$body" + set +e + run_pr_merge "$case_dir" task-x1 https://github.com/example/repo/pull/65 \ + > "$case_dir/stdout" 2> "$case_dir/stderr" + expect_code 0 "$?" "squash-fence-$label: the merge should succeed: $(cat "$case_dir/stderr")" + set -e + [ "$(github_merge_flag_value "$case_dir" --body)" = "$want" ] \ + || fail "squash-fence-$label: unexpected body: '$(github_merge_flag_value "$case_dir" --body)'" + done + pass "fm-pr-merge ignores headings inside fenced code, and treats a backtick run holding a backtick as inline code, when reading the What Changed section" +} + +test_squash_summary_uses_title_and_what_changed +test_squash_summary_reads_crlf_bodies +test_squash_summary_falls_back_to_repository_default +test_squash_summary_caller_commit_text_wins +test_squash_summary_leaves_gitlab_and_forgejo_untouched +test_squash_summary_ignores_headings_inside_code_fences diff --git a/tests/fm-remote-job.test.sh b/tests/fm-remote-job.test.sh index 9a548b8c6c0..58c4630fa34 100755 --- a/tests/fm-remote-job.test.sh +++ b/tests/fm-remote-job.test.sh @@ -65,6 +65,7 @@ printf 'args:' printf ' <%s>' "$@" printf '\n' if [ -n "${TOP_SECRET:-}" ]; then printf 'secret=leaked\n'; else printf 'secret=absent\n'; fi +printf 'user=%s\nlogname=%s\n' "${USER-<unset>}" "${LOGNAME-<unset>}" while IFS= read -r line || [ -n "$line" ]; do printf 'stdin=%s\n' "$line"; done exit "${FM_PROBE_EXIT:-0}" SH @@ -128,6 +129,7 @@ SH [ -z "$legacy" ] || export FM_REMOTE_JOB_POLL_SECONDS="$legacy" # shellcheck disable=SC2030 # The active override is local to this cadence fixture. [ -z "$active" ] || export FM_REMOTE_JOB_ACTIVE_POLL_SECONDS="$active" + # shellcheck disable=SC2030 # Each cadence fixture owns its subshell environment. export FM_REMOTE_JOB_STATE_ROOT="$poll_dir/state" FM_ROOT_OVERRIDE="$REMOTE_ROOT" # shellcheck disable=SC2030 # Each cadence fixture owns its subshell's bounds. export FM_REMOTE_JOB_QUEUE_TIMEOUT=60 FM_REMOTE_JOB_TIMEOUT=30 @@ -318,7 +320,7 @@ rm -rf -- "$ACCOUNT_HOME/.local/share/mise" pass "operator PATH orders discovered tool installs deterministically" HOME="$ACCOUNT_HOME" PATH="$RUNTIME_BIN:/usr/bin:/bin:/usr/sbin:/sbin" FM_FAKE_PERL_LOG="$FAKE_PERL_LOG" \ - FM_ROOT_OVERRIDE="$REMOTE_ROOT" FM_REMOTE_JOB_STATE_ROOT="$STATE_ROOT" \ + USER=fm-test-user LOGNAME=fm-test-login FM_ROOT_OVERRIDE="$REMOTE_ROOT" FM_REMOTE_JOB_STATE_ROOT="$STATE_ROOT" \ FM_REMOTE_JOB_PLATFORM_OVERRIDE=Linux FM_REMOTE_JOB_TIMEOUT=5 \ "$REMOTE_ROOT/bin/fm-remote-job-worker.sh" > "$TMP_ROOT/worker.out" 2> "$TMP_ROOT/worker.err" & for _ in $(seq 1 100); do @@ -362,12 +364,37 @@ assert_contains "$OUT" 'args: <two words> <$(not executed)>' "the worker changed assert_contains "$OUT" 'stdin=first line' "the worker lost staged stdin" assert_contains "$OUT" 'stdin=second line' "the worker lost staged stdin" assert_contains "$OUT" 'secret=absent' "ambient environment crossed into the worker child" +assert_contains "$OUT" 'user=fm-test-user' "the worker did not pass USER to the child" +assert_contains "$OUT" 'logname=fm-test-login' "the worker did not pass LOGNAME to the child" case "$OUT" in *"$REMOTE_ROOT/bin:$ACCOUNT_HOME/.local/bin:"*) : ;; *) fail "worker PATH omitted its fixed root and account head" ;; esac fm_remote_job_reap "$ACCOUNT_HOME" "$JOB_ID" || fail "the completed job could not be reaped" assert_absent "$JOB_DIR" "reap retained a completed job record" assert_absent "$FAKE_PERL_LOG" "the worker invoked an unavailable Perl runtime" pass "the worker preserves bounded argv and stdin in an empty environment" +# A worker without USER or LOGNAME must not invent them: the child gets neither +# variable, rather than empty strings a macOS credential lookup would misread. +# The fixture runs as its own bash process, so the job library's variables never +# enter this shell and the surrounding top-level assertions stay unaffected. +IDENTITY_DIR="$TMP_ROOT/identity-unset" +mkdir -p "$IDENTITY_DIR" +cat > "$IDENTITY_DIR/run.sh" <<'SH' +set -u +. "$ROOT/bin/fm-remote-job-lib.sh" +fm_remote_job_stage "$ACCOUNT_HOME" "$REMOTE_ROOT" "$REMOTE_HOME" fm-probe-job.sh </dev/null >/dev/null \ + || { echo "$FM_REMOTE_JOB_ERROR" >&2; exit 1; } +HOME="$ACCOUNT_HOME" "$BASH" "$REMOTE_ROOT/bin/fm-remote-job-worker.sh" --lane "$FM_REMOTE_JOB_ID" || exit 1 +fm_remote_job_wait "$ACCOUNT_HOME" "$FM_REMOTE_JOB_ID" || { echo "$FM_REMOTE_JOB_ERROR" >&2; exit 1; } +cat "$FM_REMOTE_JOB_STDOUT" +SH +IDENTITY_OUT=$(env -u USER -u LOGNAME ROOT="$ROOT" ACCOUNT_HOME="$ACCOUNT_HOME" REMOTE_ROOT="$REMOTE_ROOT" \ + REMOTE_HOME="$REMOTE_HOME" FM_REMOTE_JOB_STATE_ROOT="$IDENTITY_DIR/state" FM_ROOT_OVERRIDE="$REMOTE_ROOT" \ + FM_REMOTE_JOB_QUEUE_TIMEOUT=60 FM_REMOTE_JOB_TIMEOUT=30 "$BASH" "$IDENTITY_DIR/run.sh") \ + || fail "the identity-unset job did not complete" +assert_contains "$IDENTITY_OUT" 'user=<unset>' "the worker invented USER for the child" +assert_contains "$IDENTITY_OUT" 'logname=<unset>' "the worker invented LOGNAME for the child" +pass "the worker passes USER and LOGNAME through only when set" + ACTIVE_SIDE_EFFECT="$TMP_ROOT/active-side-effect" FM_REMOTE_JOB_TIMEOUT=10 fm_remote_job_stage "$ACCOUNT_HOME" "$REMOTE_ROOT" "$REMOTE_HOME" \ diff --git a/tests/fm-send-inbox.test.sh b/tests/fm-send-inbox.test.sh index 2117b74c5aa..2af182c5569 100644 --- a/tests/fm-send-inbox.test.sh +++ b/tests/fm-send-inbox.test.sh @@ -409,6 +409,44 @@ SH pass "fm-send inbox: lost reply bookkeeping never invites a resend, and the delivered steer is never duplicated" } +test_delivery_lock_serializes_enqueue() { + local dir err lock sender rc=0 waiting i=0 + dir=$(setup_case delivery-lock) + err="$dir/send.err" + waiting="$dir/delivery-lock-waiting" + . "$ROOT/bin/fm-task-inbox-lib.sh" + lock=$(fm_task_inbox_delivery_lock_path "$dir/home/state" t1) + fm_task_inbox_lock_acquire "$lock" || fail "could not acquire the inbox delivery lock" + cat > "$dir/fakebin/sleep" <<'SH' +#!/usr/bin/env bash +[ "${FM_TEST_DELIVERY_LOCK_WAITING:-}" = "" ] || : > "$FM_TEST_DELIVERY_LOCK_WAITING" +exec /bin/sleep "$@" +SH + chmod +x "$dir/fakebin/sleep" + run_send "$dir" "$err" FM_TASK_INBOX_LOCK_WAIT_SECS=10 FM_TEST_DELIVERY_LOCK_WAITING="$waiting" -- t1 "arrived during parking" & + sender=$! + while [ ! -e "$waiting" ] && [ "$i" -lt 200 ]; do + /bin/sleep 0.01 + i=$((i + 1)) + done + [ -e "$waiting" ] || { + fm_lock_release "$lock" + wait "$sender" 2>/dev/null || true + fail "fm-send did not reach the held delivery-lock wait" + } + kill -0 "$sender" 2>/dev/null || fail "fm-send exited while the delivery lock was held" + [ ! -e "$dir/home/state/t1.inbox/001.msg" ] || { + fm_lock_release "$lock" + wait "$sender" 2>/dev/null || true + fail "fm-send wrote the instruction while parking held the delivery lock" + } + fm_lock_release "$lock" + wait "$sender" || rc=$? + [ "$rc" -eq 0 ] || fail "fm-send failed after the delivery lock cleared: $(cat "$err")" + [ -f "$dir/home/state/t1.inbox/001.msg" ] || fail "fm-send did not enqueue after the delivery lock cleared" + pass "fm-send waits for the inbox delivery lock before enqueueing an instruction" +} + test_meta_lock_contention_fails_bounded() { local dir err rc holder marker lock i dir=$(setup_case meta-lock) @@ -518,5 +556,6 @@ test_key_path_never_touches_inbox test_secondmate_marker_and_enqueue_delivery test_post_enqueue_bookkeeping_failure_is_not_retryable test_meta_lock_contention_fails_bounded +test_delivery_lock_serializes_enqueue test_unwritable_inbox_fails_loudly test_empty_message_refused diff --git a/tests/fm-session-lock-ancestry.test.sh b/tests/fm-session-lock-ancestry.test.sh index bea82f5d8dc..314b10785ca 100755 --- a/tests/fm-session-lock-ancestry.test.sh +++ b/tests/fm-session-lock-ancestry.test.sh @@ -140,6 +140,20 @@ SH pass "session-lock: a harness that is pid 1 of its own namespace is examined, not skipped" } +test_bun_run_omp_is_identified_on_both_platforms() { + local comm fakebin + fakebin=$(fm_fakebin "$TMP_ROOT/bun-omp") + # macOS reports the executable path in comm, procps on Linux the exec name. + for comm in bun /Users/u/.bun/bin/bun /opt/homebrew/bin/bun; do + lib_eval "$fakebin" "fm_harness_process_matches '$comm' 'bun /Users/u/.bun/bin/omp'" \ + || fail "$comm running the omp launcher must be identified" + if lib_eval "$fakebin" "fm_harness_process_matches '$comm' 'bun /Users/u/.bun/bin/ompd'"; then + fail "$comm running ompd was identified as a harness" + fi + done + pass "session-lock: a bun-run omp script is identified under both platforms' comm reporting" +} + test_ordinary_paths_are_never_harness_processes() { local dir fakebin shape dir="$TMP_ROOT/ordinary-paths" @@ -865,6 +879,7 @@ install_autoarm_scripts() { cp "$ROOT/bin/fm-wake-lib.sh" "$dir/bin/fm-wake-lib.sh" cp "$ROOT/bin/fm-path-lib.sh" "$dir/bin/fm-path-lib.sh" cp "$ROOT/bin/fm-session-lock-lib.sh" "$dir/bin/fm-session-lock-lib.sh" + cp "$ROOT/bin/fm-omp-process-lib.sh" "$dir/bin/fm-omp-process-lib.sh" cp "$ROOT/bin/fm-cursor-lib.sh" "$dir/bin/fm-cursor-lib.sh" # Launch probe sibling of fm-cursor-lib.sh; keep it next to the copy so a # fixture that ever verifies an executable does not source a missing file. @@ -1644,6 +1659,7 @@ test_reanchor_keeps_autoarm_midturn_healthy() { test_version_named_session_is_identified_on_both_platforms test_harness_at_namespace_pid1_is_examined +test_bun_run_omp_is_identified_on_both_platforms test_ordinary_paths_are_never_harness_processes test_harness_beyond_a_gap_never_owns_the_lock test_competing_version_named_session_is_seen_as_live diff --git a/tests/fm-sessionstart-hook-live-e2e.test.sh b/tests/fm-sessionstart-hook-live-e2e.test.sh index acc295666d6..56c1f1fd978 100755 --- a/tests/fm-sessionstart-hook-live-e2e.test.sh +++ b/tests/fm-sessionstart-hook-live-e2e.test.sh @@ -123,6 +123,7 @@ make_lab() { # <harness> -> echoes lab dir ln -sf "$ROOT/bin/fm-timeout-lib.sh" "$lab/bin/fm-timeout-lib.sh" ln -sf "$ROOT/bin/fm-wake-lib.sh" "$lab/bin/fm-wake-lib.sh" ln -sf "$ROOT/bin/fm-session-lock-lib.sh" "$lab/bin/fm-session-lock-lib.sh" + ln -sf "$ROOT/bin/fm-omp-process-lib.sh" "$lab/bin/fm-omp-process-lib.sh" cat > "$lab/bin/fm-bootstrap.sh" <<'SH' #!/usr/bin/env bash # Outlives the hook on purpose: the marker can only appear if the worker was diff --git a/tests/fm-spawn-dispatch-profile.test.sh b/tests/fm-spawn-dispatch-profile.test.sh index 5580540625a..04584970075 100755 --- a/tests/fm-spawn-dispatch-profile.test.sh +++ b/tests/fm-spawn-dispatch-profile.test.sh @@ -1886,6 +1886,57 @@ JS pass "Pi worker registries report unmatched and unverified exclusions without reporting exact matches" } +# Claude Code removes a --disallowedTools entry's schema from every request +# (docs/configuration.md "Worker tool exclusions"). The option is variadic, so +# the launch must keep a later flag between it and the launch prompt. +test_claude_exclude_tools_reach_ship_and_scout_launches() { + local rec id out status launch kindflag doorbell + for kindflag in --ship --scout; do + id="excl-claude-${kindflag#--}-z9k" + rec=$(make_spawn_case "excl-claude-${kindflag#--}" claude "$id") + read_case_record "$rec" + write_exclude_file "$HOME_DIR" '# hide the unused tool schemas' '' \ + ' Workflow ' 'mcp__other-srv' 'mcp__other-srv__tool.v2' + if [ "$kindflag" = --scout ]; then + out=$(run_spawn "$HOME_DIR" "$WT_DIR" "$FAKEBIN_DIR" "$LAUNCH_LOG" "$id" "$PROJ_DIR" --scout) + else + out=$(run_ship_spawn "$HOME_DIR" "$WT_DIR" "$FAKEBIN_DIR" "$LAUNCH_LOG" "$id" "$PROJ_DIR") + fi + status=$? + expect_code 0 "$status" "claude $kindflag spawn with exclusions should succeed"$'\n'"$out" + launch=$(fm_test_launch_unframe "$(cat "$LAUNCH_LOG")") + assert_contains "$launch" "--disallowedTools 'Workflow,mcp__other-srv,mcp__other-srv__tool.v2' --append-system-prompt " \ + "claude $kindflag launch must carry the quoted list directly before the next flag" + assert_not_contains "$launch" "--exclude-tools" "claude must not receive the Pi flag" + doorbell=$(claude_launch_brief_arg "$launch") + [ -n "$doorbell" ] || fail "claude $kindflag launch lost its launch prompt after the exclusion flag" + done + pass "Claude ship and scout launches carry the configured tool exclusions ahead of the prompt" +} + +test_claude_exclude_tools_leave_the_rest_of_the_launch_unchanged() { + local rec id out status launch with_list without_list + id=excl-claude-same-z9l + rec=$(make_spawn_case excl-claude-same claude "$id") + read_case_record "$rec" + out=$(run_ship_spawn "$HOME_DIR" "$WT_DIR" "$FAKEBIN_DIR" "$LAUNCH_LOG" "$id" "$PROJ_DIR") + expect_code 0 "$?" "claude spawn without the file should succeed"$'\n'"$out" + without_list=$(fm_test_launch_unframe "$(cat "$LAUNCH_LOG")") + assert_not_contains "$without_list" "--disallowedTools" "an absent file must add no Claude exclusions" + + id=excl-claude-same-z9m + rec=$(make_spawn_case excl-claude-same2 claude "$id") + read_case_record "$rec" + write_exclude_file "$HOME_DIR" 'Workflow' + out=$(run_ship_spawn "$HOME_DIR" "$WT_DIR" "$FAKEBIN_DIR" "$LAUNCH_LOG" "$id" "$PROJ_DIR") + expect_code 0 "$?" "claude spawn with the file should succeed"$'\n'"$out" + with_list=$(fm_test_launch_unframe "$(cat "$LAUNCH_LOG")") + assert_contains "$with_list" "--disallowedTools 'Workflow' --append-system-prompt " "the exclusion flag is missing" + assert_contains "$with_list" "--dangerously-skip-permissions" "the permission flag must survive the exclusion flag" + assert_contains "$with_list" "--settings " "the settings flag must survive the exclusion flag" + pass "a Claude exclusion list adds only its own flag to the launch" +} + test_pi_exclude_tools_absent_and_empty_lists() { local rec id out status launch id=excl-absent-z9b @@ -1913,13 +1964,14 @@ test_pi_exclude_tools_absent_and_empty_lists() { status=$? expect_code 0 "$status" "$harness spawn with an empty exclusion list should succeed"$'\n'"$out" assert_not_contains "$(cat "$LAUNCH_LOG")" "--exclude-tools" "$harness must receive no exclusion flag" + assert_not_contains "$(cat "$LAUNCH_LOG")" "--disallowedTools" "$harness must receive no disallowed-tools flag" done pass "an absent or empty exclusion list leaves every runtime unaffected" } test_exclude_tools_non_pi_runtime_refuses_non_empty_list() { local harness rec id out status - for harness in claude codex grok; do + for harness in codex grok; do id="excl-refuse-${harness}-z9i" rec=$(make_spawn_case "excl-refuse-${harness}" "$harness" "$id") read_case_record "$rec" @@ -2036,6 +2088,22 @@ test_pi_exclude_tools_do_not_leak_across_homes_or_to_secondmates() { launch=$(fm_test_launch_unframe "$(cat "$LAUNCH_LOG")") assert_contains "$launch" "fm-primary-turnend-guard.ts" "secondmate launch was not the Pi secondmate shape" assert_not_contains "$launch" "--exclude-tools" "a secondmate's own agent must not receive the worker exclusions" + + # The same holds for a Claude secondmate's own agent. + id_a=excl-sm-claude-z9n + rec_a=$(make_spawn_case excl-sm-claude claude "$id_a") + read_case_record "$rec_a" + write_exclude_file "$HOME_DIR" 'Workflow' + sm="$CASE_DIR/secondmate-home" + make_seeded_secondmate_home "$sm" "$id_a" + sm=$(cd "$sm" && pwd -P) + out=$(FM_TEST_CLAUDE_CONFIG_DIR="$CASE_DIR/claude-work" \ + run_spawn "$HOME_DIR" "$WT_DIR" "$FAKEBIN_DIR" "$LAUNCH_LOG" "$id_a" "$sm" --secondmate) + status=$? + expect_code 0 "$status" "Claude secondmate spawn should succeed"$'\n'"$out" + launch=$(fm_test_launch_unframe "$(cat "$LAUNCH_LOG")") + assert_contains "$launch" "claude " "secondmate launch was not the Claude shape" + assert_not_contains "$launch" "--disallowedTools" "a Claude secondmate's own agent must not receive the worker exclusions" pass "exclusions stay in the home that configured them and skip secondmate agents" } @@ -3017,6 +3085,8 @@ test_pi_seeded_secondmate_preapproves_project_trust test_pi_worker_launch_omits_seeded_home_approve test_pi_approve_probe_omits_unsupported_flag test_pi_exclude_tools_reach_ship_and_scout_launches +test_claude_exclude_tools_reach_ship_and_scout_launches +test_claude_exclude_tools_leave_the_rest_of_the_launch_unchanged test_pi_exclude_tools_worker_registry_reports test_pi_exclude_tools_absent_and_empty_lists test_exclude_tools_non_pi_runtime_refuses_non_empty_list diff --git a/tests/fm-teardown.test.sh b/tests/fm-teardown.test.sh index 142c1657d86..19587e2ec4f 100755 --- a/tests/fm-teardown.test.sh +++ b/tests/fm-teardown.test.sh @@ -42,6 +42,9 @@ # (q3) no-mistakes + squash-merged, same file, different content -> REFUSE # (q4) no-mistakes + squash-merged rebased local plus extra commit -> REFUSE # (q5) gh down + squash-merged stale local, content not in default -> REFUSE +# (q6) GitLab MR merged by squash or train, main moved on, MR head unfetched, +# config/park-on-review or config/auto-land present -> ALLOW; open MR -> REFUSE; +# neither flag -> REFUSE without a glab read, as before the opt-in # # Also covers backlog teardown-lock-race: a git index.lock left in the worktree by a # killed crew process (bin/fm-teardown.sh's teardown_treehouse_return). @@ -911,6 +914,26 @@ test_local_only_merged_to_local_main_allows() { pass "local-only worktree with work merged into local main is torn down (no regression)" } +# A merge-time cleanup request belongs to the detached cleanup that runs this +# teardown: only that cleanup removes it, after its outcome report is saved, so +# a cleanup that dies mid-teardown is still retried by the next park tick. +test_teardown_leaves_the_merge_cleanup_request_to_its_cleanup() { + local case_dir wt_head rc + case_dir=$(make_case keep-cleanup-request) + write_meta "$case_dir" local-only ship + wt_commit "$case_dir" "merged work" + wt_head=$(git -C "$case_dir/wt" rev-parse HEAD) + git -C "$case_dir/project" update-ref refs/heads/main "$wt_head" + printf 'pr=https://github.com/example/repo/pull/7\n' > "$case_dir/state/task-x1.review-park-land" + set +e + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 0 "$rc" "keep-cleanup-request: teardown should succeed when work is merged into local main" + assert_present "$case_dir/state/task-x1.review-park-land" "teardown removed the merge cleanup request its cleanup still owns" + pass "teardown leaves a merge cleanup request for the detached cleanup that owns it" +} + # Send the task branch's commits to <target> on origin from a separate clone, so # the task copy's tracking refs do not show them. Args: case_dir target land_task_commits_from_other_clone() { @@ -1470,6 +1493,83 @@ test_squash_merged_forgejo_pr_allows_when_main_moved_on() { pass "teardown proves a Forgejo task landed from the merged pull request when the content check is inconclusive" } +add_glab_mr_for_head() { + local case_dir=$1 state=$2 head=$3 + cat > "$case_dir/fakebin/glab" <<SH +#!/usr/bin/env bash +printf '%s\n' "\$*" >> "$case_dir/glab.log" +[ "\${1:-} \${2:-}" = "mr view" ] || exit 1 +printf '{"state":"%s","sha":"%s","squash_commit_sha":"%s"}\n' '$state' '$head' 0000000000000000000000000000000000000001 +SH + chmod +x "$case_dir/fakebin/glab" +} + +# A GitLab merge request squash-merged (or merged by a merge train) lands a +# commit on main that is not the MR head, and main then moved on over the same +# lines, so the content check is inconclusive. The MR's own head is published +# only as refs/merge-requests/<n>/head, which this copy never fetched. With +# config/park-on-review or config/auto-land present, teardown must read the MR +# through glab, fetch that ref, and accept the merged record; the same history +# with the MR still open refuses. With neither flag the merged MR is never read +# through glab and teardown refuses on the inconclusive content check. +test_squash_merged_gitlab_mr_allows_when_main_moved_on() { + local case_dir rc local_head pr_head tmp scenario state expected + for scenario in merged:park-on-review merged:auto-land opened:park-on-review merged:; do + state=${scenario%%:*} + case_dir=$(make_case "gitlab-mr-$state-${scenario#*:}") + write_meta "$case_dir" no-mistakes ship + wt_commit_file "$case_dir" feature.txt hello "add feature" + local_head=$(git -C "$case_dir/wt" rev-parse HEAD) + tmp="$case_dir/_pipeline" + git clone -q "$case_dir/origin.git" "$tmp" + git -C "$tmp" fetch -q "$case_dir/wt" "fm/task-x1:fm/task-x1" + git -C "$tmp" checkout -q fm/task-x1 + printf '%s\n' pipeline > "$tmp/fix.txt" + git -C "$tmp" add -- fix.txt + git -C "$tmp" -c user.email=t@t -c user.name=t commit -q -m "no-mistakes fix" + pr_head=$(git -C "$tmp" rev-parse HEAD) + git -C "$tmp" push -q origin "HEAD:refs/merge-requests/7/head" + git -C "$tmp" checkout -q main + printf '%s\n' reworded > "$tmp/feature.txt" + git -C "$tmp" add -- feature.txt + git -C "$tmp" -c user.email=t@t -c user.name=t commit -q -m "squash then main rewords" + git -C "$tmp" push -q origin main + rm -rf "$tmp" + ! git -C "$case_dir/wt" cat-file -e "$pr_head^{commit}" 2>/dev/null \ + || fail "gitlab-mr-$state: test setup bug, the MR head is already local" + printf '%s\n' 'pr=https://gitlab.example.com/team/sub/repo/-/merge_requests/7' \ + >> "$case_dir/state/task-x1.meta" + add_gh_axi_error "$case_dir" + add_glab_mr_for_head "$case_dir" "$state" "$pr_head" + [ -z "${scenario#*:}" ] || : > "$case_dir/config/${scenario#*:}" + + set +e + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + if [ -z "${scenario#*:}" ]; then + expect_code 1 "$rc" "gitlab-mr-no-flag: teardown exit"$'\n'"$(cat "$case_dir/stderr")" + [ ! -e "$case_dir/glab.log" ] || fail "gitlab-mr-no-flag: teardown read the merge request through glab without an opt-in flag" + grep -q REFUSED "$case_dir/stderr" || fail "gitlab-mr-no-flag: no REFUSED line in stderr" + assert_refusal_retained_task_state "$case_dir" gitlab-mr-no-flag "$local_head" + continue + fi + expected=1 + [ "$state" != merged ] || expected=0 + expect_code "$expected" "$rc" "gitlab-mr-$scenario: teardown exit"$'\n'"$(cat "$case_dir/stderr")" + grep -qF -- 'mr view 7 -R https://gitlab.example.com/team/sub/repo -F json' "$case_dir/glab.log" \ + || fail "gitlab-mr-$scenario: teardown did not read the merge request through glab" + if [ "$state" = merged ]; then + ! grep -q REFUSED "$case_dir/stderr" || fail "gitlab-mr-merged: teardown printed a REFUSED line" + else + grep -q REFUSED "$case_dir/stderr" || fail "gitlab-mr-opened: no REFUSED line in stderr" + assert_refusal_retained_task_state "$case_dir" gitlab-mr-opened "$local_head" + fi + done + pass "teardown proves a squash- or train-merged GitLab MR landed from the MR head only with an opt-in flag, and refuses an open one" +} + test_squash_merged_same_file_different_content_refuses() { local case_dir rc pr_head local_head case_dir=$(make_case squash-same-path-diverged) @@ -5839,6 +5939,40 @@ test_retained_sources_still_reach_the_ordinary_refusal() { test_teardown_preserves_opencode_busy_plugin_on_pool_return +# An orphan status log archived by record hygiene must survive status-archive +# retention: the prune counts 30 days from archive time, not the log's last write. +test_record_hygiene_archive_survives_status_archive_prune() { + local case_dir rc=0 + case_dir=$(make_case record-hygiene-archive-retention) + write_meta "$case_dir" local-only ship + touch "$case_dir/config/status-archive" + printf 'done: complete\n' > "$case_dir/state/task-x1.status" + printf 'failed [at=1]: orphan from a lost task\n' > "$case_dir/state/orphan-x2.status" + touch -t 200001010000 "$case_dir/state/orphan-x2.status" + FM_TEST_LIB_SOURCED=1 bash -c ' + SCRIPT_DIR="$1/bin"; STATE="$2/state"; DATA="$2/data" + # shellcheck source=bin/fm-classify-lib.sh + . "$SCRIPT_DIR/fm-classify-lib.sh" + # shellcheck source=bin/fm-tasks-axi-lib.sh + . "$SCRIPT_DIR/fm-tasks-axi-lib.sh" + # shellcheck source=bin/fm-backlog-transition-lib.sh + . "$SCRIPT_DIR/fm-backlog-transition-lib.sh" + # shellcheck source=bin/fm-backend.sh + . "$SCRIPT_DIR/fm-backend.sh" + # shellcheck source=bin/fm-timeout-lib.sh + . "$SCRIPT_DIR/fm-timeout-lib.sh" + # shellcheck source=bin/fm-record-hygiene-lib.sh + . "$SCRIPT_DIR/fm-record-hygiene-lib.sh" + record_hygiene_report + ' _ "$ROOT" "$case_dir" > "$case_dir/hygiene.out" 2>&1 || fail "record hygiene archive pass failed: $(cat "$case_dir/hygiene.out")" + assert_present "$case_dir/data/orphan-x2/status.log" "record hygiene did not archive the orphan status log" + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" || rc=$? + expect_code 0 "$rc" "status-archive teardown after record hygiene: $(cat "$case_dir/stderr")" + assert_present "$case_dir/data/task-x1/status.log" "status-archive teardown did not archive its own task" + assert_present "$case_dir/data/orphan-x2/status.log" "teardown pruned a status log record hygiene just archived" + pass "a status log archived by record hygiene survives the next status-archive teardown" +} + # Archive coverage uses the real teardown entry point over isolated homes. test_status_archive_opt_in() { local case_dir rc mode @@ -6437,6 +6571,7 @@ test_hidden_untracked_work_refuses_before_return test_content_fallback_git_capability_boundary test_work_written_at_reap_boundary_is_preserved test_status_archive_opt_in +test_record_hygiene_archive_survives_status_archive_prune test_record_export_config_absent_never_reads_the_exporter test_record_export_unopted_symlinked_config_does_not_block_teardown test_record_export_archives_before_teardown_or_preserves_sources_on_failure @@ -6463,6 +6598,7 @@ test_teardown_closes_a_gitlab_task_with_its_merge_request_url_as_a_note test_teardown_manual_backend_leaves_the_backlog_to_the_operator test_local_only_truly_unpushed_refuses test_local_only_merged_to_local_main_allows +test_teardown_leaves_the_merge_cleanup_request_to_its_cleanup test_landed_on_other_origin_branch_with_stale_refs_allows test_refresh_failure_does_not_read_as_landed test_refreshed_refs_still_refuse_unreachable_commits @@ -6513,6 +6649,7 @@ test_merged_pr_patch_matching_preserves_indentation test_merged_pr_with_later_local_commit_refuses test_squash_merged_rebased_branch_allows test_squash_merged_forgejo_pr_allows_when_main_moved_on +test_squash_merged_gitlab_mr_allows_when_main_moved_on test_squash_merged_same_file_different_content_refuses test_squash_merged_rebased_local_with_unlanded_commit_refuses test_squash_merged_stale_local_refuses_when_forge_unreachable diff --git a/tests/fm-turnend-guard.test.sh b/tests/fm-turnend-guard.test.sh index 7ada2a5ed37..b1bd1ee8f20 100755 --- a/tests/fm-turnend-guard.test.sh +++ b/tests/fm-turnend-guard.test.sh @@ -196,6 +196,7 @@ install_guard_scripts() { cp "$ROOT/bin/fm-path-lib.sh" "$dir/bin/fm-path-lib.sh" cp "$ROOT/bin/fm-hook-host-lib.sh" "$dir/bin/fm-hook-host-lib.sh" cp "$ROOT/bin/fm-session-lock-lib.sh" "$dir/bin/fm-session-lock-lib.sh" + cp "$ROOT/bin/fm-omp-process-lib.sh" "$dir/bin/fm-omp-process-lib.sh" cp "$ROOT/bin/fm-cursor-lib.sh" "$dir/bin/fm-cursor-lib.sh" # Launch probe sibling of fm-cursor-lib.sh; keep it next to the copy so a # fixture that ever verifies an executable does not source a missing file. @@ -1497,6 +1498,7 @@ install_integrated_autoarm() { cp "$ROOT/bin/fm-path-lib.sh" "$dir/bin/fm-path-lib.sh" cp "$ROOT/bin/fm-hook-host-lib.sh" "$dir/bin/fm-hook-host-lib.sh" cp "$ROOT/bin/fm-session-lock-lib.sh" "$dir/bin/fm-session-lock-lib.sh" + cp "$ROOT/bin/fm-omp-process-lib.sh" "$dir/bin/fm-omp-process-lib.sh" cp "$ROOT/bin/fm-cursor-lib.sh" "$dir/bin/fm-cursor-lib.sh" # Launch probe sibling of fm-cursor-lib.sh; keep it next to the copy so a # fixture that ever verifies an executable does not source a missing file. diff --git a/tests/fm-wake-queue.test.sh b/tests/fm-wake-queue.test.sh index f2fef05b445..4604ea3586e 100755 --- a/tests/fm-wake-queue.test.sh +++ b/tests/fm-wake-queue.test.sh @@ -1614,6 +1614,45 @@ test_branch_actor_scoped_ack_never_swallows_a_main_owned_row() { pass "a branch-actor scoped ack never swallows an unacked main-owned row, and main's later drain sees exactly what remains" } +# A host publishes the branch's grant before its engine runs. Its branch tool +# may not be allowed to inspect the host process through ps, but it must still +# drain and acknowledge only the exact rows in that published snapshot. +test_branch_actor_keeps_its_grant_when_ps_cannot_inspect_the_host() { + local dir state sequence generation + dir=$(make_case branch-ps-denied) + state="$dir/state" + mkdir -p "$dir/fakebin" + printf '#!/usr/bin/env bash\nexit 1\n' > "$dir/fakebin/ps" + chmod +x "$dir/fakebin/ps" + + append_wake "$state" check "main-only.check.sh" "check: main-only" || fail "main append failed" + append_wake "$state" signal "task-a.status" "signal: task-a" || fail "branch append failed" + FM_PROC_ROOT_OVERRIDE="$dir/no-proc" FM_STATE_OVERRIDE="$state" \ + "$GRANT" activate "$$" ps-denied || fail "branch owner activation failed" + FM_PROC_ROOT_OVERRIDE="$dir/no-proc" FM_STATE_OVERRIDE="$state" \ + "$GRANT" publish ps-denied 2 || fail "branch grant publication failed" + + PATH="$dir/fakebin:$PATH" FM_PROC_ROOT_OVERRIDE="$dir/no-proc" FM_STATE_OVERRIDE="$state" \ + FM_SUPERVISION_ACTOR=branch "$DRAIN" > "$dir/branch.out" 2> "$dir/branch.err" \ + || fail "branch lost its live grant when ps was unavailable: $(cat "$dir/branch.err")" + grep -Fq "$(printf '\tsignal\ttask-a.status\t')" "$dir/branch.out" \ + || fail "branch drain omitted its granted row" + ! grep -Fq "$(printf '\tcheck\tmain-only.check.sh\t')" "$dir/branch.out" \ + || fail "branch drain presented the main-owned row" + sequence=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through \([0-9][0-9]*\) --recovery-generation [A-Za-z0-9._-][A-Za-z0-9._-]*$/\1/p' "$dir/branch.err") + generation=$(sed -n 's/^WAKE_ACK_REQUIRED:.*--ack-through [0-9][0-9]* --recovery-generation \([A-Za-z0-9._-][A-Za-z0-9._-]*\)$/\1/p' "$dir/branch.err") + [ "$sequence" = 2 ] && [ -n "$generation" ] || fail "branch drain omitted its exact acknowledgement" + PATH="$dir/fakebin:$PATH" FM_PROC_ROOT_OVERRIDE="$dir/no-proc" FM_STATE_OVERRIDE="$state" \ + FM_SUPERVISION_ACTOR=branch "$DRAIN" --ack-through "$sequence" --recovery-generation "$generation" \ + || fail "branch acknowledgement failed when ps was unavailable" + grep -Fq "$(printf '\tcheck\tmain-only.check.sh\t')" "$state/.wake-queue" \ + || fail "branch acknowledgement swallowed the main-owned row" + ! grep -Fq "$(printf '\tsignal\ttask-a.status\t')" "$state/.wake-queue" \ + || fail "branch acknowledgement left its granted row queued" + + pass "branch drain and acknowledgement keep their published grant when ps cannot inspect the host" +} + test_main_drain_excludes_rows_already_granted_to_branch() { local dir state out err sequence generation dir=$(make_case main-excludes-branch-grant) @@ -3818,6 +3857,7 @@ test_structural_signal_enrichment_preserves_raw_rows test_enrichment_preserves_all_unread_lines_and_status_file_failures test_slow_annotation_does_not_block_append_and_deleted_file_fails_open test_branch_actor_scoped_ack_never_swallows_a_main_owned_row +test_branch_actor_keeps_its_grant_when_ps_cannot_inspect_the_host test_main_drain_excludes_rows_already_granted_to_branch test_branch_ack_commits_secondmate_stall_receipts test_main_is_never_told_to_drain_rows_only_the_branch_owns diff --git a/tests/fm-watch-triage-pause.test.sh b/tests/fm-watch-triage-pause.test.sh index 9a10daa3f6a..8eb2de2887d 100644 --- a/tests/fm-watch-triage-pause.test.sh +++ b/tests/fm-watch-triage-pause.test.sh @@ -1634,6 +1634,132 @@ test_undated_pause_overdue_bound() { pass "undated pauses stay quiet through the bound and close their overdue decision on resume" } +# Opt-in config/paused-recheck-silent: an unchanged paused declaration that was +# already surfaced once is rechecked without a wake; first sight, a changed +# declaration, a due declared time, and an open decision still wake. +silent_recheck_fixture() { # <name> <until-epoch> <status-age-secs> -> dir + local dir state window key + dir=$(make_case "$1"); state="$dir/state"; window="test:fm-until" + printf 'idle, waiting for the reset\n' > "$dir/pane.txt" + printf 'window=%s\nkind=secondmate\n' "$window" > "$state/until.meta" + printf 'paused: rate limit resets, until %s, then resuming\n' "$(iso_utc_at "$2")" > "$state/until.status" + set_mtime "$(( $(date +%s) - $3 ))" "$state/until.status" + printf '%s' "$(seen_sig "$state/until.status")" > "$state/.seen-until_status" + key=$(watch_window_key "$window") + printf '%s' "$(hash_text 'idle, waiting for the reset')" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + mkdir -p "$dir/config" + printf '%s\n' "$dir" +} + +silent_recheck_case() { # <name> <flag:on|off> <status-body> -> dir; leaves the first-sight wake acked + local name=$1 flag=$2 body=$3 dir state key + dir=$(silent_recheck_fixture "$name" "$(( $(date +%s) + 86400 ))" 500); state="$dir/state" + printf '%s\n' "$body" > "$state/until.status" + set_mtime "$(( $(date +%s) - 500 ))" "$state/until.status" + printf '%s' "$(seen_sig "$state/until.status")" > "$state/.seen-until_status" + mkdir -p "$dir/config" + [ "$flag" = on ] && : > "$dir/config/paused-recheck-silent" + until_watch "$dir" 240 + wait_for_exit "$UNTIL_PID" 100 || { reap "$UNTIL_PID"; fail "first sight of a declared pause did not wake ($name): $(cat "$dir/watch.out")"; } + ack_stopped_cycle "$state" || fail "could not acknowledge the first-sight wake ($name)" + key=$(watch_window_key test:fm-until) + set_mtime "$(( $(date +%s) - 600 ))" "$state/.paused-resurfaced-$key" + : > "$dir/watch.out" + printf '%s\n' "$dir" +} + +test_silent_recheck_flag_on_absorbs_unchanged_pause() { + local dir state key before + dir=$(silent_recheck_case silent-on on 'paused: rate limit resets, until 2999-01-01T00:00Z, then resuming'); state="$dir/state" + key=$(watch_window_key test:fm-until) + before=$(cat "$state/.paused-resurfaced-$key") + until_watch "$dir" 240 + if ! wait_poll_cycle "$state" "$UNTIL_PID" || ! wait_poll_cycle "$state" "$UNTIL_PID"; then + reap "$UNTIL_PID"; fail "an unchanged paused recheck woke the supervisor with the flag on: $(cat "$dir/watch.out")" + fi + reap "$UNTIL_PID" + [ "$(cat "$state/.paused-resurfaced-$key")" = "$before" ] || fail "the recheck changed the declaration scope" + [ "$state/.paused-resurfaced-$key" -nt "$state/until.status" ] || fail "the recheck was not recorded in the throttle marker" + grep -F 'rechecked without a wake' "$state/.watch-triage.log" >/dev/null || fail "the silent recheck was not logged" + pass "with paused-recheck-silent on, an unchanged paused declaration is rechecked without a wake" +} + +test_silent_recheck_flag_off_still_wakes() { + local dir state + dir=$(silent_recheck_case silent-off off 'paused: rate limit resets, until 2999-01-01T00:00Z, then resuming'); state="$dir/state" + until_watch "$dir" 240 + wait_for_exit "$UNTIL_PID" 100 || { reap "$UNTIL_PID"; fail "an unchanged paused recheck did not wake with the flag off"; } + grep -F 'stale: test:fm-until' "$dir/watch.out" >/dev/null || fail "the recheck wake was not printed with the flag off: $(cat "$dir/watch.out")" + pass "with paused-recheck-silent absent, the unchanged paused recheck still wakes" +} + +test_silent_recheck_flag_on_wakes_for_an_open_decision() { + local dir state body + body=$(printf 'needs-decision [at=%s] [key=open-q]: which option\npaused: rate limit resets, until 2999-01-01T00:00Z, then resuming' "$(date +%s)") + dir=$(silent_recheck_case silent-open-decision on "$body"); state="$dir/state" + until_watch "$dir" 240 + wait_for_exit "$UNTIL_PID" 100 || { reap "$UNTIL_PID"; fail "an open decision did not wake an unchanged pause with the flag on"; } + pass "with paused-recheck-silent on, an open decision still wakes" +} + +test_silent_recheck_flag_on_wakes_for_a_changed_declaration() { + local dir state + dir=$(silent_recheck_case silent-changed on 'paused: rate limit resets, until 2999-01-01T00:00Z, then resuming'); state="$dir/state" + printf 'paused: a different wait, until 2999-01-01T00:00Z\n' > "$state/until.status" + set_mtime "$(( $(date +%s) - 500 ))" "$state/until.status" + printf '%s' "$(seen_sig "$state/until.status")" > "$state/.seen-until_status" + until_watch "$dir" 240 + wait_for_exit "$UNTIL_PID" 100 || { reap "$UNTIL_PID"; fail "a changed declaration did not wake with the flag on"; } + pass "with paused-recheck-silent on, a changed declaration still wakes" +} + +test_silent_recheck_flag_on_absorbs_unchanged_pause_for_a_live_ship_agent() { + local dir state + dir=$(silent_recheck_case silent-live-ship on 'paused: rate limit resets, until 2999-01-01T00:00Z, then resuming'); state="$dir/state" + printf 'window=test:fm-until\nkind=ship\nharness=grok\nbackend=tmux\n' > "$state/until.meta" + FM_FAKE_TMUX_CURRENT_COMMAND=grok until_watch "$dir" 240 + if ! wait_poll_cycle "$state" "$UNTIL_PID" || ! wait_poll_cycle "$state" "$UNTIL_PID"; then + reap "$UNTIL_PID"; fail "an unchanged paused recheck woke the supervisor for a live ship agent with the flag on: $(cat "$dir/watch.out")" + fi + reap "$UNTIL_PID" + pass "with paused-recheck-silent on, an unchanged paused recheck for a live ship agent is silent" +} + +test_silent_recheck_flag_on_absorbs_a_ticking_pane_for_a_live_ship_agent() { + local dir state + dir=$(silent_recheck_case silent-ticking on 'paused: waiting on the vendor reply'); state="$dir/state" + printf 'window=test:fm-until\nkind=ship\nharness=grok\nbackend=tmux\n' > "$state/until.meta" + printf 'idle, waiting for the reset 12:01\n' > "$dir/pane.txt" + printf '%s' "$(hash_text 'idle, waiting for the reset 12:01')" > "$state/.hash-$(watch_window_key test:fm-until)" + FM_FAKE_TMUX_CURRENT_COMMAND=grok until_watch "$dir" 240 + if ! wait_poll_cycle "$state" "$UNTIL_PID" || ! wait_poll_cycle "$state" "$UNTIL_PID"; then + reap "$UNTIL_PID"; fail "a changed pane on an unchanged paused declaration woke the supervisor with the flag on: $(cat "$dir/watch.out")" + fi + reap "$UNTIL_PID" + pass "with paused-recheck-silent on, a ticking pane on an unchanged paused declaration is silent" +} + +test_silent_recheck_flag_on_wakes_when_the_agent_exited() { + local dir state + dir=$(silent_recheck_case silent-dead on 'paused: rate limit resets, until 2999-01-01T00:00Z, then resuming'); state="$dir/state" + printf 'window=test:fm-until\nkind=ship\nharness=grok\nbackend=tmux\n' > "$state/until.meta" + FM_FAKE_TMUX_CURRENT_COMMAND=zsh until_watch "$dir" 240 + wait_for_exit "$UNTIL_PID" 100 || { reap "$UNTIL_PID"; fail "a paused recheck for an exited agent did not wake with the flag on"; } + grep -F 'stale: test:fm-until' "$dir/watch.out" >/dev/null || fail "the exited-agent recheck wake was not printed: $(cat "$dir/watch.out")" + pass "with paused-recheck-silent on, a paused recheck for an exited agent still wakes" +} + +test_silent_recheck_flag_on_wakes_when_declared_time_is_due() { + local dir state + dir=$(silent_recheck_fixture silent-due "$(( $(date +%s) - 30 ))" 500); state="$dir/state" + mkdir -p "$dir/config"; : > "$dir/config/paused-recheck-silent" + until_watch "$dir" 240 + wait_for_exit "$UNTIL_PID" 100 || { reap "$UNTIL_PID"; fail "a due declared time did not wake with the flag on"; } + grep -F 'declared clearing time has passed' "$dir/watch.out" >/dev/null || fail "the due wake was not the declared-time recheck" + pass "with paused-recheck-silent on, a due declared time still wakes" +} + # --- a live worker paused on its own background job -------------------------- # A worker declared `paused:` on its own background job and the job then ended # without a new status line, so only the passage of time can tell the watcher @@ -1738,6 +1864,175 @@ test_required_captain_hold_tests_fail_without_tasks_axi() { pass "all four required captain-hold tests fail when tasks-axi is unavailable" } +# The superseded-wait check is opt-in through config/paused-wait-supersession; +# cases arm it unless asked for the flag-absent default. +make_superseded_pause_case() { # <name> <status-line> [armed|unarmed] + local name=$1 status_line=$2 arm=${3:-armed} dir state window key sig + dir=$(make_case "$name"); state="$dir/state"; window="test:fm-superseded" + mkdir -p "$dir/config" + [ "$arm" = unarmed ] || : > "$dir/config/paused-wait-supersession" + printf 'idle at declared wait\n' > "$dir/pane.txt" + printf 'window=%s\nkind=ship\nharness=grok\nbackend=tmux\n' "$window" > "$state/paused.meta" + printf '%s\n' "$status_line" > "$state/paused.status" + sig=$(seen_sig "$state/paused.status"); printf '%s' "$sig" > "$state/.seen-paused_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + printf '%s' "$(hash_text 'idle at declared wait')" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + printf '%s\n' "$dir" +} + +superseded_pause_round() { # <dir> <crew-state-line> <exit|absorb> [busy] + local dir=$1 crew_state=$2 mode=$3 busy=${4:-no} state="$1/state" out="$1/watch.out" pid + local command=zsh busy_max=999999 + case "$busy" in + yes|recent) command=grok; [ "$busy" = yes ] && busy_max=1; printf 'working\nCtrl+c:cancel\n' > "$dir/pane.txt" ;; + esac + PATH="$dir/fakebin:$PATH" FM_FAKE_TMUX_WINDOW=test:fm-superseded \ + FM_FAKE_TMUX_CAPTURE="$dir/pane.txt" FM_FAKE_TMUX_CURRENT_COMMAND="$command" \ + FM_FAKE_CREW_STATE="$crew_state" FM_WATCH_HANDLING_SUCCESSOR=1 \ + FM_STATE_OVERRIDE="$state" FM_CONFIG_OVERRIDE="$dir/config" \ + FM_CREW_STATE_BIN="$dir/fakebin/fm-crew-state.sh" FM_FAKE_CREW_STATE_LOG="$dir/crew-state.log" \ + FM_BUSY_TURN_MAX_SECS="$busy_max" FM_PAUSE_RESURFACE_SECS=240 \ + FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 \ + "$WATCH" >> "$out" 2>&1 & + pid=$! + if [ "$mode" = exit ]; then + wait_for_exit "$pid" 100 || { reap "$pid"; return 1; } + else + wait_poll_cycle "$state" "$pid" 100 || { reap "$pid"; return 1; } + reap "$pid" + fi +} + +test_declared_wait_surfaces_superseding_state() { + local dir state window + dir=$(make_superseded_pause_case gate-unreported 'paused: waiting for validation run R') + state="$dir/state"; window=test:fm-superseded + superseded_pause_round "$dir" \ + 'state: parked · source: run-step · ask-user: authority decision · run: R' exit \ + || fail "idle paused run parked at an unreported human gate did not surface" + grep -F "stale: $window (paused wait superseded: run R is parked at a human-owed gate with no open decision)" \ + "$dir/watch.out" >/dev/null || fail "gate wake did not identify the run and missing decision" + ack_stopped_cycle "$state" || fail "could not acknowledge the idle gate wake" + superseded_pause_round "$dir" \ + 'state: parked · source: run-step · ask-user: authority decision · run: R' absorb \ + || fail "the same declaration and run re-alarmed inside the cadence" + [ ! -s "$state/.wake-queue" ] || fail "the same declaration and run produced a duplicate wake" + + dir=$(make_superseded_pause_case gate-reported 'paused: waiting for validation run R') + state="$dir/state" + printf 'needs-decision [key=nm-R-review]: decide the ask-user finding\npaused: waiting for validation run R\n' > "$state/paused.status" + sig=$(seen_sig "$state/paused.status"); printf '%s' "$sig" > "$state/.seen-paused_status" + superseded_pause_round "$dir" \ + 'state: parked · source: run-step · ask-user: authority decision · run: R' absorb \ + || fail "an already-open decision did not preserve the declared-wait cadence" + [ ! -s "$state/.wake-queue" ] || fail "gate with an open needs-decision generated a stale wake" + + dir=$(make_superseded_pause_case gate-working 'paused: waiting for validation run R') + state="$dir/state" + superseded_pause_round "$dir" 'state: working · source: run-step · validating (running) · run: R' absorb \ + || fail "a working validation run did not preserve the declared-wait cadence" + [ ! -s "$state/.wake-queue" ] || fail "working run generated a stale wake" + + dir=$(make_superseded_pause_case gate-terminal 'paused: waiting for validation run R') + state="$dir/state" + superseded_pause_round "$dir" 'state: done · source: run-step · run: R' exit \ + || fail "terminal validation run did not supersede the declared wait" + grep -F 'paused wait superseded: current run R is terminal (done)' "$dir/watch.out" >/dev/null \ + || fail "terminal wake did not identify the superseded wait" + + dir=$(make_superseded_pause_case gate-terminal-pr 'paused: waiting for validation run R') + state="$dir/state" + superseded_pause_round "$dir" 'state: done · source: run-step · run: R1 · pr=https://example.test/1' exit \ + || fail "terminal run with PR did not supersede the declared wait" + grep -F 'current run R1 is terminal (done)' "$dir/watch.out" >/dev/null \ + || fail "PR suffix obscured the first terminal run id" + ack_stopped_cycle "$state" || fail "could not acknowledge the first terminal wake" + superseded_pause_round "$dir" 'state: done · source: run-step · run: R2 · pr=https://example.test/2' exit \ + || fail "second terminal run with PR did not supersede the declared wait" + grep -F 'current run R2 is terminal (done)' "$dir/watch.out" >/dev/null \ + || fail "different terminal run shared the first run's throttle" + + dir=$(make_superseded_pause_case gate-busy-recent 'paused: waiting for validation run R') + superseded_pause_round "$dir" \ + 'state: parked · source: run-step · ask-user: authority decision · run: R' exit recent \ + || fail "recent busy paused run parked at an unreported human gate did not surface" + grep -F 'run R is parked at a human-owed gate' "$dir/watch.out" >/dev/null \ + || fail "recent busy gate wake did not identify the run" + + dir=$(make_superseded_pause_case gate-busy-old 'paused: waiting for validation run R') + state="$dir/state" + set_mtime "$(( $(date +%s) - 500 ))" "$state/paused.meta" + superseded_pause_round "$dir" \ + 'state: parked · source: run-step · ask-user: authority decision · run: R' exit yes \ + || fail "old busy paused run parked at an unreported human gate did not surface" + grep -F 'run R is parked at a human-owed gate' "$dir/watch.out" >/dev/null \ + || fail "old busy gate wake did not identify the run" + ack_stopped_cycle "$state" || fail "could not acknowledge the busy gate wake" + superseded_pause_round "$dir" \ + 'state: parked · source: run-step · ask-user: authority decision · run: R' absorb yes \ + || fail "the repeated busy poll failed" + [ ! -s "$state/.wake-queue" ] || fail "the busy gate repeated inside the cadence" + + pass "declared waits surface unreported gates and terminal runs once, while working runs and answered gates keep the pause cadence on idle and busy paths" +} + +# Absent config/paused-wait-supersession the same superseding states keep main's +# declared-wait handling: no superseded wake, nothing queued, and no crew-state +# read beyond the one the idle declared-wait recheck already spends (none on the +# busy paths), counted through the stub's read log. +test_superseded_wait_check_is_opt_in() { + local c n crew_state busy reads got dir state + for c in "idle-gate|state: parked · source: run-step · ask-user: authority decision · run: R|no|1" \ + "idle-terminal|state: done · source: run-step · run: R|no|1" \ + "busy-recent|state: parked · source: run-step · ask-user: authority decision · run: R|recent|0" \ + "busy-old|state: parked · source: run-step · ask-user: authority decision · run: R|yes|0"; do + IFS='|' read -r n crew_state busy reads <<< "$c" + dir=$(make_superseded_pause_case "unarmed-$n" 'paused: waiting for validation run R' unarmed) + state="$dir/state" + [ "$busy" != yes ] || set_mtime "$(( $(date +%s) - 500 ))" "$state/paused.meta" + superseded_pause_round "$dir" "$crew_state" absorb "$busy" \ + || fail "$n: flag-absent watcher round failed" + ! grep -F 'paused wait superseded' "$dir/watch.out" >/dev/null \ + || fail "$n: flag-absent watcher surfaced a superseded wait" + [ ! -s "$state/.wake-queue" ] || fail "$n: flag-absent watcher queued a wake" + got=0 + [ ! -f "$dir/crew-state.log" ] || got=$(wc -l < "$dir/crew-state.log" | tr -d ' ') + [ "$got" = "$reads" ] || fail "$n: flag-absent watcher read crew state $got times, want $reads" + done + pass "without config/paused-wait-supersession superseding states keep the declared-wait cadence and spend no extra crew-state read" +} + +# The flag scopes the check to the foreground watcher: while the away-mode flag +# exists, a busy paused worker whose run is parked at an unreported human gate +# gets no superseded wake and no crew-state read, exactly as with the flag absent. +test_superseded_wait_check_skips_away_mode() { + local c busy mode arm got dir out + for c in recent:absorb yes:exit; do + busy=${c%%:*} mode=${c#*:} out='' + for arm in unarmed armed; do + dir=$(make_superseded_pause_case "away-busy-$busy-$arm" 'paused: waiting for validation run R' "$arm") + date '+%s' > "$dir/state/.afk" + [ "$busy" != yes ] || set_mtime "$(( $(date +%s) - 500 ))" "$dir/state/paused.meta" + superseded_pause_round "$dir" \ + 'state: parked · source: run-step · ask-user: authority decision · run: R' "$mode" "$busy" \ + || fail "busy-$busy $arm: away-mode watcher round failed: $(cat "$dir/watch.out")" + ! grep -F 'paused wait superseded' "$dir/watch.out" >/dev/null \ + || fail "busy-$busy $arm: away-mode watcher surfaced a superseded wait" + got=0 + [ ! -f "$dir/crew-state.log" ] || got=$(wc -l < "$dir/crew-state.log" | tr -d ' ') + [ "$got" = 0 ] || fail "busy-$busy $arm: away-mode watcher read crew state $got times, want 0" + if [ "$arm" = unarmed ]; then + out=$(cat "$dir/watch.out") + else + [ "$(cat "$dir/watch.out")" = "$out" ] \ + || fail "busy-$busy: armed away-mode output differs from flag-absent: $(cat "$dir/watch.out") vs $out" + fi + done + done + pass "with the away-mode flag present an armed busy paused worker at a human gate behaves exactly as with the flag absent: no superseded wake and no crew-state read" +} + if [ -n "${FM_TEST_ONLY:-}" ]; then "$FM_TEST_ONLY" exit 0 @@ -1747,6 +2042,9 @@ fm_test_run \ test_required_captain_hold_tests_fail_without_tasks_axi \ test_nonterminal_stale_not_working_surfaced \ test_nonterminal_stale_paused_absorbed_then_resurfaced \ + test_declared_wait_surfaces_superseding_state \ + test_superseded_wait_check_is_opt_in \ + test_superseded_wait_check_skips_away_mode \ test_exited_declared_pause_is_bounded_but_live_gate_surfaces \ test_own_work_wait_keeps_first_alert_then_long_cadence \ test_absorbed_replacement_wait_does_not_inherit_the_old_throttle \ @@ -1781,5 +2079,13 @@ fm_test_run \ test_malformed_dated_pause_is_not_undated_overdue \ test_undated_pause_age_uses_the_pause_line \ test_undated_pause_overdue_bound \ + test_silent_recheck_flag_on_absorbs_unchanged_pause \ + test_silent_recheck_flag_off_still_wakes \ + test_silent_recheck_flag_on_wakes_for_an_open_decision \ + test_silent_recheck_flag_on_wakes_for_a_changed_declaration \ + test_silent_recheck_flag_on_absorbs_unchanged_pause_for_a_live_ship_agent \ + test_silent_recheck_flag_on_absorbs_a_ticking_pane_for_a_live_ship_agent \ + test_silent_recheck_flag_on_wakes_when_the_agent_exited \ + test_silent_recheck_flag_on_wakes_when_declared_time_is_due \ test_paused_bg_job_without_until_stays_silent_after_the_job_ends \ test_paused_bg_job_until_timeout_rechecks_after_the_job_ends diff --git a/tests/fm-watcher-lock.test.sh b/tests/fm-watcher-lock.test.sh index 91b0707c9cb..a701e805ee1 100755 --- a/tests/fm-watcher-lock.test.sh +++ b/tests/fm-watcher-lock.test.sh @@ -758,8 +758,8 @@ test_lock_steal_reap_cannot_remove_successor() { # Two reapers verify the same dead steal owner. The competitor runs to # completion exactly when the first one is about to remove the link; at most # one of them may end up believing it holds the mutex. - local dir state steal fakebin out real_rm rc - dir=$(make_case lock-steal-reap-race) + local dir state steal fakebin out real_rm rc shape=${1:-link} trigger + dir=$(make_case "lock-steal-reap-race-$shape") state="$dir/state" steal="$state/.contend.lock.steal" fakebin="$dir/fakebin" @@ -767,7 +767,13 @@ test_lock_steal_reap_cannot_remove_successor() { # The host's own rm path, not a hardcoded /bin/rm: on a host that keeps only # sh in /bin (NixOS) the fixture's shim would exec a path that does not exist. real_rm=$(command -v rm) - leave_dead_link_locks "$state" "$steal" + if [ "$shape" = directory ]; then + mkdir "$steal" + trigger=./watcher-path + else + leave_dead_link_locks "$state" "$steal" + trigger=$steal + fi # The killed fixture owner may have its PID reused during a loaded serial # shard. Pin a verified-dead PID so the race exercises reaping, not liveness. printf '%s\n' "$(dead_pid)" > "$steal/pid" @@ -775,7 +781,7 @@ test_lock_steal_reap_cannot_remove_successor() { #!/usr/bin/env bash last= for arg do last=$arg; done -if [ "$last" = "$FM_TEST_RACE_PATH" ] && mkdir "$FM_TEST_RACE_ONCE" 2>/dev/null; then +if [ "$last" = "$FM_TEST_RACE_REMOVE_PATH" ] && mkdir "$FM_TEST_RACE_ONCE" 2>/dev/null; then bash -c ' . "$1" if fm_lock_try_acquire_steal_mutex "$2"; then @@ -783,7 +789,7 @@ if [ "$last" = "$FM_TEST_RACE_PATH" ] && mkdir "$FM_TEST_RACE_ONCE" 2>/dev/null; exec sleep 30 fi printf "lost\n" > "$3" - ' _ "$FM_TEST_LIB" "$last" "$FM_TEST_RACE_OUT" >/dev/null 2>&1 & + ' _ "$FM_TEST_LIB" "$FM_TEST_RACE_PATH" "$FM_TEST_RACE_OUT" >/dev/null 2>&1 & i=0 while [ "$i" -lt 100 ] && [ ! -s "$FM_TEST_RACE_OUT" ]; do sleep 0.05 @@ -795,7 +801,7 @@ SH chmod +x "$fakebin/rm" rc=0 - PATH="$fakebin:$PATH" FM_TEST_LIB="$LIB" FM_TEST_RACE_PATH="$steal" \ + PATH="$fakebin:$PATH" FM_TEST_LIB="$LIB" FM_TEST_RACE_PATH="$steal" FM_TEST_RACE_REMOVE_PATH="$trigger" \ FM_TEST_RACE_ONCE="$dir/race-once" FM_TEST_RACE_OUT="$out" \ FM_TEST_REAL_RM="$real_rm" FM_STATE_OVERRIDE="$state" bash -c ' . "$1" @@ -813,7 +819,11 @@ SH ;; *) fail "competing reaper did not report an outcome" ;; esac - pass "a competing reaper cannot remove the successor's steal mutex" + pass "a competing $shape reaper cannot remove the successor's steal mutex" +} + +test_lock_directory_reap_cannot_remove_successor() { + test_lock_steal_reap_cannot_remove_successor directory } test_lock_stale_steal_single_winner_under_concurrency() { @@ -1052,49 +1062,53 @@ test_lock_reclaims_holder_whose_identity_no_longer_recomputes() { # identity still recomputes for its live pid. A mismatch is reclaimed # without signalling the pid; a matching identity, or no recorded identity # at all, still reads as live. - local dir state lockdir live identity out case_name recorded + local dir home state lockdir live identity out case_name recorded dir=$(make_case lock-identity-mismatch) - state="$dir/state" - lockdir="$state/.watch.lock" + mkdir -p "$dir/home/state" + ln -s "$dir/home" "$dir/logical-home" # Naming the holder under the fixture root lets the suite's cleanup reap it # if an assertion below fails before the kill at the end. A python holder # carries that name as its own argument instead of through `exec -a`, which a # multicall sleep rejects by dispatching on argv[0] and exiting at once. python3 -c 'import sys, time; time.sleep(300)' "$dir/holder" >/dev/null 2>&1 & live=$! - identity=$(FM_STATE_OVERRIDE="$state" bash -c '. "$1"; fm_pid_identity "$2"' _ "$LIB" "$live") \ - || fail "could not identify the fixture holder" - for case_name in mismatched matching unrecorded; do - rm -rf "$lockdir" - mkdir "$lockdir" - printf '%s\n' "$live" > "$lockdir/pid" - case "$case_name" in - mismatched) printf '%s\n' "identity no current rendering produces" > "$lockdir/pid-identity" ;; - matching) printf '%s\n' "$identity" > "$lockdir/pid-identity" ;; - esac - out=$(FM_STATE_OVERRIDE="$state" bash -c ' - . "$1" - if fm_lock_try_acquire "$2"; then rc=0; else rc=1; fi - printf "rc=%s held=%s recovered=%s pid=%s\n" "$rc" "${FM_LOCK_HELD_PID:-}" "${FM_LOCK_RECOVERED_PID:-}" "$(cat "$2/pid" 2>/dev/null)" - ' _ "$LIB" "$lockdir") - is_live_non_zombie "$live" || fail "$case_name identity: the live pid was signalled" - recorded=$(cat "$lockdir/pid" 2>/dev/null || true) - case "$case_name" in - mismatched) - case "$out" in - *"rc=0 held= recovered=$live "*) ;; - *) fail "lock whose recorded identity no longer recomputes was not reclaimed: $out" ;; - esac - [ "$recorded" != "$live" ] || fail "reclaimed lock still records the mismatched pid" - ;; - *) - case "$out" in - *"rc=1 held=$live "*) ;; - *) fail "$case_name identity: live holder was not refused: $out" ;; - esac - [ "$recorded" = "$live" ] || fail "$case_name identity: live holder's lock pid was clobbered ('$recorded')" - ;; - esac + for home in "$dir/home" "$dir/logical-home"; do + state="$home/state" + lockdir="$state/.watch.lock" + identity=$(FM_HOME="$home" FM_STATE_OVERRIDE="$state" bash -c '. "$1"; fm_pid_identity "$2"' _ "$LIB" "$live") \ + || fail "could not identify the fixture holder" + for case_name in mismatched matching unrecorded; do + rm -rf "$lockdir" + mkdir "$lockdir" + printf '%s\n' "$live" > "$lockdir/pid" + case "$case_name" in + mismatched) printf '%s\n' "identity no current rendering produces" > "$lockdir/pid-identity" ;; + matching) printf '%s\n' "$identity" > "$lockdir/pid-identity" ;; + esac + out=$(FM_HOME="$home" FM_STATE_OVERRIDE="$state" bash -c ' + . "$1" + if fm_lock_try_acquire "$2"; then rc=0; else rc=1; fi + printf "rc=%s held=%s recovered=%s pid=%s\n" "$rc" "${FM_LOCK_HELD_PID:-}" "${FM_LOCK_RECOVERED_PID:-}" "$(cat "$2/pid" 2>/dev/null)" + ' _ "$LIB" "$lockdir") + is_live_non_zombie "$live" || fail "$home $case_name identity: the live pid was signalled" + recorded=$(cat "$lockdir/pid" 2>/dev/null || true) + case "$case_name" in + mismatched) + case "$out" in + *"rc=0 held= recovered=$live "*) ;; + *) fail "$home: lock whose recorded identity no longer recomputes was not reclaimed: $out" ;; + esac + [ "$recorded" != "$live" ] || fail "$home: reclaimed lock still records the mismatched pid" + ;; + *) + case "$out" in + *"rc=1 held=$live "*) ;; + *) fail "$home $case_name identity: live holder was not refused: $out" ;; + esac + [ "$recorded" = "$live" ] || fail "$home $case_name identity: live holder's lock pid was clobbered ('$recorded')" + ;; + esac + done done kill "$live" 2>/dev/null || true wait "$live" 2>/dev/null || true @@ -1141,6 +1155,346 @@ test_lock_keeps_live_daemon_holder_with_stale_identity() { pass "live supervise-daemon lock with a stale local-time identity and no start time is not reclaimed" } +test_lock_real_directory_residue() { + python3 - "$LIB" "$TMP_ROOT" "$(dead_pid)" <<'PY' || fail "real-directory lock recovery" +import os +from pathlib import Path +import subprocess +import sys + +lib, root, dead = sys.argv[1:] +for name in ('.lock.acquire', '.turnend-claude-blocks.lock'): + for shape in ('nested', 'top-level'): + state = Path(root) / ('real-dir-' + name + shape) + state.mkdir() + lock = state / name + for path in (lock, Path(str(lock) + '.steal')): + path.mkdir() + owner = path / (path.name + '.owner.fixture') if shape == 'nested' else path + owner.mkdir(exist_ok=True) + (owner / 'pid').write_text(dead + '\n') + (owner / 'pid-start').write_text('dead holder\n') + os.utime(owner, (1, 1)) + os.utime(path, (1, 1)) + result = subprocess.run( + ['bash', '-c', '. "$1"; fm_lock_acquire_wait "$2" || exit 10; ' + '[ -L "$2" ] || exit 11; fm_lock_release "$2"', '_', lib, str(lock)], + env=dict(os.environ, FM_STATE_OVERRIDE=str(state)), + capture_output=True, text=True, timeout=10) + assert result.returncode == 0, (name, shape, result.stderr) + assert list(state.iterdir()) == [], list(state.iterdir()) +PY + pass "dead real-directory primary and steal locks recover without leaked owners" +} + +test_lock_real_directory_live_nested_owner() { + python3 - "$LIB" "$TMP_ROOT" "$(dead_pid)" <<'PY' || fail "live nested lock owner" +import os +from pathlib import Path +import subprocess +import sys + +lib, root, dead = sys.argv[1:] +for suffix in ('', '.steal'): + state = Path(root) / ('real-dir-live' + suffix) + state.mkdir() + lock = state / '.lock.acquire' + for path in (lock, Path(str(lock) + '.steal')): + path.mkdir() + (path / 'pid').write_text(dead + '\n') + owner = path / (path.name + '.owner.fixture') + owner.mkdir() + (owner / 'pid').write_text(str(os.getpid()) + '\n' if path == Path(str(lock) + suffix) + else dead + '\n') + os.utime(owner, (1, 1)) + os.utime(path, (1, 1)) + result = subprocess.run( + ['bash', '-c', '. "$1"; if fm_lock_try_acquire "$2"; then exit 10; fi', + '_', lib, str(lock)], env=dict(os.environ, FM_STATE_OVERRIDE=str(state)), + capture_output=True, text=True, timeout=10) + assert result.returncode == 0, result.stderr + live_owner = Path(str(lock) + suffix) / (lock.name + suffix + '.owner.fixture') + assert (live_owner / 'pid').read_text().strip() == str(os.getpid()) +PY + pass "a live nested owner blocks reclamation even in an old directory" +} + +test_lock_real_directory_named_refusal() { + python3 - "$LIB" "$TMP_ROOT" "$(dead_pid)" <<'PY' || fail "named directory-lock refusal" +import os +from pathlib import Path +import subprocess +import sys + +lib, root, dead = sys.argv[1:] +for suffix in ('', '.steal'): + state = Path(root) / ('real-dir-refusal' + suffix) + state.mkdir() + lock = state / '.lock.acquire' + wedged = Path(str(lock) + suffix) + for path in (lock, Path(str(lock) + '.steal')): + path.mkdir() + (path / 'pid').write_text(dead + '\n') + os.utime(path, (1, 1)) + (wedged / 'unrecognized-record').write_text('preserve me\n') + os.utime(wedged, (1, 1)) + result = subprocess.run( + ['bash', '-c', '. "$1"; fm_lock_acquire_wait "$2" --refuse-unreapable', '_', lib, str(lock)], + env=dict(os.environ, FM_STATE_OVERRIDE=str(state)), + capture_output=True, text=True, timeout=10) + assert result.returncode != 0, result + assert 'cannot reclaim lock directory ' + str(wedged) in result.stderr, result.stderr + assert (wedged / 'unrecognized-record').read_text() == 'preserve me\n' + assert not list(state.glob('*.owner.*')), list(state.iterdir()) +PY + pass "unreapable real-directory locks refuse promptly naming the wedged path" +} + +test_lock_real_directory_permission_refusal() { + if [ "$(id -u)" -eq 0 ]; then + pass "SKIP permission refusal: root bypasses directory search permissions" + return + fi + # Triage #8275 requires a prompt named refusal. Missing search permission + # must not turn either session claim path back into an unbounded wait. + python3 - "$LIB" "$ROOT/bin/fm-lock.sh" "$TMP_ROOT" "$(dead_pid)" <<'PY' || fail "inaccessible directory-lock refusal" +import os +from pathlib import Path +import shutil +import subprocess +import sys + +lib, script, root, dead = sys.argv[1:] +fixture = Path(root) / 'permission-refusal' +fixture.mkdir() +harness = fixture / 'claude' +harness.symlink_to(shutil.which('bash')) +for mode in ('library', 'new-session', 'own-session'): + for suffix in ('', '.steal'): + home = fixture / (mode + suffix) + state = home / 'state' + state.mkdir(parents=True) + lock = state / '.lock.acquire' + wedged = Path(str(lock) + suffix) + for path in (lock, Path(str(lock) + '.steal')): + path.mkdir() + (path / 'pid').write_text(dead + '\n') + os.utime(path, (1, 1)) + wedged.chmod(0o600) + try: + probe = subprocess.run(['bash', '-c', 'cd -- "$1"', '_', str(wedged)], + capture_output=True, timeout=5) + if probe.returncode == 0: + print('SKIP permission refusal: filesystem does not enforce search permissions') + sys.exit(0) + env = dict(os.environ, FM_HOME=str(home), FM_STATE_OVERRIDE=str(state)) + for name in ('CLAUDE_CODE_SESSION_ID', 'CLAUDE_PID'): + env.pop(name, None) + if mode == 'library': + command = ['bash', '-c', '. "$1"; fm_lock_acquire_wait "$2" --refuse-unreapable', + '_', lib, str(lock)] + else: + command = [str(harness), '-c', 'if [ "$2" = own-session ]; then ' + 'printf "%s\\n" "$$" > "$FM_HOME/state/.lock"; fi; ' + 'bash "$1"; rc=$?; exit "$rc"', '_', script, mode] + result = subprocess.run(command, env=env, capture_output=True, text=True, timeout=10) + assert result.returncode == 1, (mode, suffix, result) + assert 'cannot reclaim lock directory ' + str(wedged) in result.stderr, result.stderr + assert 'lock acquired:' not in result.stdout, result.stdout + assert wedged.is_dir() and not wedged.is_symlink() + assert not list(state.glob('*.owner.*')), list(state.iterdir()) + finally: + wedged.chmod(0o700) + assert (wedged / 'pid').read_text() == dead + '\n' +PY + pass "inaccessible primary and steal directories refuse by name on both session claim paths" +} + +test_lock_inaccessible_directory_replacement_is_retriable() { + if [ "$(id -u)" -eq 0 ]; then + pass "SKIP permission replacement: root bypasses directory search permissions" + return + fi + # Pause immediately after a real cd failure. A concurrent removal or + # successor publication must not be diagnosed as a stable blocked directory. + python3 - "$LIB" "$TMP_ROOT" <<'PY' || fail "inaccessible directory replacement safety" +import os +from pathlib import Path +import subprocess +import sys +import time + +lib, root = sys.argv[1:] +for suffix in ('', '.steal'): + for replacement in ('removed', 'directory', 'symlink'): + state = Path(root) / ('permission-replacement-' + replacement + suffix) + state.mkdir() + lock = state / ('.lock.acquire' + suffix) + lock.mkdir() + os.utime(lock, (1, 1)) + lock.chmod(0o600) + probe = subprocess.run(['bash', '-c', 'cd -- "$1"', '_', str(lock)], + capture_output=True, timeout=5) + if probe.returncode == 0: + lock.chmod(0o700) + print('SKIP permission replacement: filesystem does not enforce search permissions') + sys.exit(0) + old = state / 'old-directory' + ready, resume = state / 'ready', state / 'resume' + child = subprocess.Popen( + ['bash', '-c', ''' +. "$1" +cd() { + if [ "${1:-}" = -- ] && [ "${2:-}" = "$lock_path" ]; then + builtin cd "$@" && return 0 + : > "$ready" + while [ ! -e "$resume" ]; do sleep 0.01; done + return 1 + fi + builtin cd "$@" +} +if fm_lock_reap_dead_directory "$lock_path"; then exit 10; fi +[ -z "${FM_LOCK_RECLAIM_FAILED_PATH:-}" ] || exit 11 +''', '_', lib], + env=dict(os.environ, FM_STATE_OVERRIDE=str(state), lock_path=str(lock), + ready=str(ready), resume=str(resume)), + stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) + try: + deadline = time.monotonic() + 5 + while not ready.exists() and child.poll() is None and time.monotonic() < deadline: + time.sleep(0.01) + assert ready.exists(), 'real failed cd was never reached' + lock.rename(old) + if replacement != 'removed': + owner = state / 'successor' if replacement == 'symlink' else lock + owner.mkdir() + (owner / 'pid').write_text(str(os.getpid()) + '\n') + (owner / 'record').write_text('preserve successor\n') + if replacement == 'symlink': + lock.symlink_to(owner) + resume.touch() + out, err = child.communicate(timeout=5) + assert child.returncode == 0, (replacement, out, err) + if replacement == 'removed': + assert not lock.exists() + else: + assert (lock / 'pid').read_text() == str(os.getpid()) + '\n' + assert (lock / 'record').read_text() == 'preserve successor\n' + assert lock.is_symlink() == (replacement == 'symlink') + finally: + if child.poll() is None: + child.kill() + child.wait() + (old if old.exists() else lock).chmod(0o700) +PY + pass "failed directory entry stays retriable after concurrent removal or replacement" +} + +test_lock_real_directory_default_wait() { + python3 - "$LIB" "$TMP_ROOT" "$(dead_pid)" <<'PY' || fail "ordinary directory-lock wait" +import os +from pathlib import Path +import subprocess +import sys +import time + +lib, root, dead = sys.argv[1:] +state = Path(root) / 'real-dir-default-wait' +state.mkdir() +lock = state / '.contend.lock' +lock.mkdir() +(lock / 'pid').write_text(dead + '\n') +record = lock / 'unrecognized-record' +record.write_text('preserve me\n') +os.utime(lock, (1, 1)) +child = subprocess.Popen( + ['bash', '-c', '. "$1"; fm_lock_acquire_wait "$2" || exit 10; fm_lock_release "$2"', + '_', lib, str(lock)], env=dict(os.environ, FM_STATE_OVERRIDE=str(state)), + stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) +try: + for _ in range(100): + if not (lock / 'pid').exists(): + break + time.sleep(0.05) + assert not (lock / 'pid').exists(), 'reclamation was never attempted' + assert child.poll() is None, 'ordinary wait returned without ownership' + assert record.read_text() == 'preserve me\n' + record.unlink() + os.utime(lock, (1, 1)) + out, err = child.communicate(timeout=10) + assert child.returncode == 0, (out, err) +finally: + if child.poll() is None: + child.kill() + child.wait() +PY + pass "ordinary wait keeps waiting until directory residue can be reclaimed" +} + +test_lock_failed_create_discards_owner() { + local dir state + dir=$(make_case lock-failed-create-cleanup) + state="$dir/state" + FM_STATE_OVERRIDE="$state" bash -c ' + . "$1" + fm_lock_owner_dir() { + mktemp -d "$1.owner.XXXXXX" + return 1 + } + if fm_lock_try_create "$2"; then exit 10; fi + for owner in "$2.owner."*; do + [ ! -d "$owner" ] || exit 11 + done + ' _ "$LIB" "$state/.contend.lock" || fail "failed owner publication leaked its staging directory" + pass "failed owner publication discards the staging directory" +} + +test_session_lock_real_directory_residue() { + python3 - "$ROOT/bin/fm-lock.sh" "$TMP_ROOT" "$(dead_pid)" <<'PY' || fail "session-lock directory recovery/refusal" +import os +from pathlib import Path +import shutil +import subprocess +import sys + +script, root, dead = sys.argv[1:] +harness = Path(root) / 'claude' +harness.symlink_to(shutil.which('bash')) +for mode in ('recover', 'refuse', 'own-refuse'): + home = Path(root) / ('session-real-dir-' + mode) + state = home / 'state' + state.mkdir(parents=True) + lock = state / '.lock.acquire' + for path in (lock, Path(str(lock) + '.steal')): + path.mkdir() + owner = path / (path.name + '.owner.fixture') + owner.mkdir() + (owner / 'pid').write_text(dead + '\n') + os.utime(owner, (1, 1)) + os.utime(path, (1, 1)) + if mode != 'recover': + (lock / 'unrecognized-record').write_text('preserve me\n') + os.utime(lock, (1, 1)) + env = dict(os.environ, FM_HOME=str(home), FM_STATE_OVERRIDE=str(state)) + for name in ('CLAUDE_CODE_SESSION_ID', 'CLAUDE_PID'): + env.pop(name, None) + result = subprocess.run( + [str(harness), '-c', 'if [ "$2" = own-refuse ]; then ' + 'printf "%s\\n" "$$" > "$FM_HOME/state/.lock"; fi; ' + 'bash "$1"; rc=$?; exit "$rc"', '_', script, mode], + env=env, capture_output=True, text=True, timeout=15) + if mode == 'recover': + assert result.returncode == 0, result.stderr + assert 'lock acquired:' in result.stdout, result.stdout + else: + assert result.returncode == 1, (mode, result) + assert 'cannot reclaim lock directory ' + str(lock) in result.stderr, result.stderr + assert 'lock acquired:' not in result.stdout, result.stdout + assert not list(state.glob('*.owner.*')), list(state.iterdir()) +PY + pass "session-lock acquisition recovers directory residue and honors named refusal on both claim paths" +} + test_lock_empty_pid_uses_minimum_grace() { local dir state lockdir out dir=$(make_case lock-empty-grace) @@ -2473,6 +2827,7 @@ test_lock_stale_steal_single_winner_under_concurrency test_lock_reclaims_dead_steal_owner_without_nested_markers test_lock_recovers_dead_nested_steal_chain test_lock_steal_reap_cannot_remove_successor +test_lock_directory_reap_cannot_remove_successor test_lock_reclaims_self_held_steal_mutex test_lock_resumes_own_interrupted_steal_reap test_lock_live_steal_mutex_is_not_reclaimed @@ -2481,6 +2836,14 @@ test_lock_reclaims_reused_pid_lock test_lock_respects_live_holder_matching_its_start_time test_lock_reclaims_holder_whose_identity_no_longer_recomputes test_lock_keeps_live_daemon_holder_with_stale_identity +test_lock_real_directory_residue +test_lock_real_directory_live_nested_owner +test_lock_real_directory_named_refusal +test_lock_real_directory_permission_refusal +test_lock_inaccessible_directory_replacement_is_retriable +test_lock_real_directory_default_wait +test_lock_failed_create_discards_owner +test_session_lock_real_directory_residue test_lock_empty_pid_uses_minimum_grace test_lock_late_claim_loses_after_recreate test_lock_bounded_wait_earns_one_grace_window_for_unpublished_pid diff --git a/tests/lib.sh b/tests/lib.sh index c940aaf4204..a813ea206cd 100644 --- a/tests/lib.sh +++ b/tests/lib.sh @@ -453,6 +453,9 @@ trap 'fm_test_cleanup; exit 143' TERM trap 'fm_test_cleanup; exit 129' HUP trap 'fm_test_cleanup; exit 131' QUIT +TREEHOUSE_ROOT=$(fm_test_tmproot fm-treehouse-pool) +export TREEHOUSE_ROOT + # fm_test_reap_orphans: best-effort sweep for fixture roots left behind by a # prior run that was killed hard enough to skip the traps above (e.g. a # SIGKILL timeout). Only removes directories carrying the .fm-test-fixture