Skip to content

fix(node-tests): wait for clone Git termination before resuming HTTPS - #699

Closed
wanglongan587 wants to merge 2 commits into
ora-space:mainfrom
wanglongan587:fix/323-clone-recovery-race
Closed

wanglongan587 wants to merge 2 commits into
ora-space:mainfrom
wanglongan587:fix/323-clone-recovery-race

Conversation

@wanglongan587

Copy link
Copy Markdown
Contributor

Summary

killed_clone_recovers_original_run_without_redispatch resumed the paused HTTPS remote immediately after SIGKILL. The guardian only learns about the owner's exit at its next reconciliation (50 ms cadence) and terminates the managed Git then. On a loaded runner the blocked clone can finish inside that window, so recovery reads a real exit code 0, verifies the checkout, and reports CloneReady — the classification the terminated-attempt decision requires for an exit code. The test asserted Interrupted, which made CI fail intermittently for unrelated changes: the same head 4f628566 failed once (job) and passed on rerun (run).

 kill Node
-guardian reconciliation (≤50 ms), unsynchronized with the test
-resume paused remote → Git can finish first → Exited(0) → CloneReady
+pin managed Git → resume remote only after that process exited
+→ Terminated → Interrupted

Change

  • add Fixture::live_git() to pin the fixture's managed Git from procfs and observe its termination;
  • wait for git.has_exited() before resuming the remote, so recovery observes a confirmed termination and the test deterministically exercises Interrupted;
  • reuse that helper in the single-node acceptance test, which carried its own copy.

ADR: no ADR changes. The behavior under test is specified by decisions/node/repository/20260925-terminated-clone-attempt-fails-as-interrupted.md (implemented); this only makes its test deterministic and leaves the "Git exits before termination" race handled exactly as that decision describes.

Refs ora-space/circus#323

Evidence

  • Controlled reproduction (WSL Ubuntu, real host/guardian): freezing the guardian with SIGSTOP before SIGKILL, then resuming the remote, lets the blocked Git finish; recovery deterministically returned Completed(Clone(CloneReady(...))), matching the CI failure. With the guardian running and the new barrier, the same test returns Interrupted.
  • The targeted test passed 50/50 runs after the change. It also passed 30/30 before in isolation, which shows the race is rare on an idle machine; the controlled reproduction is the decisive evidence.
  • Final code passes the full standalone suite (cargo test -p ora-node --test standalone -- --test-threads 4: 44 passed, 1 ignored). One earlier local run hit an unrelated websocket takeover flake (controller_session_over_websocket_takes_over_clone_and_keeps_single_session, close code 1011) once and passed on rerun.
  • cargo fmt -p ora-node -- --check passes.

Merge Danger

Door: none. Test-only change: no runtime, persistence, contract or filesystem-layout changes.

The killed-clone test resumed the paused remote right after SIGKILL. The guardian learns about the owner's exit only at its next reconciliation and terminates Git then. On a loaded runner the blocked clone can finish inside that window; recovery reads a real exit code and, per the terminated-attempt decision, verifies and reports CloneReady instead of Interrupted.

Pin the fixture's managed Git and resume the remote only after that process exited, so the test deterministically exercises the terminated path.
The single-node acceptance test duplicated the procfs scan that pins the fixture's managed Git. Use the fixture helper so both tests observe the same identity.
@EricWvi

EricWvi commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Superseded by 3bef098 (#681, merged 2026-10-21): killed_clone_recovers_original_run_without_redispatch no longer pauses the HTTPS server. It now gates the managed Git with https::WorkloadGate and never releases the gate in this test, so Git cannot exit 0 before the guardian terminates it and recovery always observes Interrupted. The server.paused release this PR reorders no longer exists on main, and the branch now conflicts in repository_recovery.rs. Please close this PR; the remaining live_git dedup isn't worth a rebase on its own.

@wanglongan587

Copy link
Copy Markdown
Contributor Author

Closing without merge — confirming the check above against upstream/main 42f224ac.

  • 3bef0988 (desktop#681) rewrote killed_clone_recovers_original_run_without_redispatch to gate the managed Git with https::WorkloadGate instead of pausing the HTTPS remote, and that test never writes gate-release. The blocked git therefore cannot exit 0 before the guardian terminates it, so recovery reads a confirmed termination and settles Interrupted deterministically. The server.paused release this PR reordered no longer exists on main, and the WorkloadGate doc comment in https.rs describes exactly the race this run located.
  • CI after the rewrite runs the same assertion on Linux and passes: repository::recovery::killed_clone_recovers_original_run_without_redispatch ... ok in the ubuntu crates job at the feat(node): clone from Git Server snapshots, collaborate through ora-git and deliver Revisions over Git #681 merge commit ff9de8f9 (run 39533082357) and at the current main 42f224ac (run 39574040697). All 7 pr-tests runs on main since the merge are green.
  • The diagnosis here stands: the failure was test timing, not the recovery implementation. Exited(0) plus completed cleanup settling as CloneReady is what decisions/node/repository/20260925-terminated-clone-attempt-fails-as-interrupted.md requires when Git exits before the termination.
  • The remaining live_git dedup is a pure refactor; not worth rebasing a branch 155 commits behind with conflicts in repository_recovery.rs.

Closing this PR; the run is reported and closed in ora-space/circus#323, and the parent record in ora-space/circus#311.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants