Skip to content

Service tests restart a crashed Runner only once the console ended its calls, so a late session open no longer fails the property test - #89

Merged
codefusion-automerge[bot] merged 1 commit into
mainfrom
fix/session-open-race
Nov 12, 2026
Merged

codefusion-automerge[bot] merged 1 commit into
mainfrom
fix/session-open-race

Conversation

@XeonFX

@XeonFX XeonFX commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

The service tests' rig restarts a crashed Runner only once the console has ended every call the dead Runner made. Before this, the around-removals property test could fail on a loaded runner with more session opens than causes.

Why

around_removals_every_job_that_ran_is_heard_and_no_running_job_is_stopped failed on #87's Linux (x86_64) job (run 43312586545, seed a8f1c1f8…) with "sessions opened [1, 3] times, with cause for [2, 2]". #87 does not cause it: the seed passes on main and on #87 on a Mac, and it fails on main once the console is slowed.

The rig's crash takes GitHub and the vm-hosts down. It does not take down the console, which makes the scale set calls for a host without a key. Here is what happened, traced on main with the console's session open slowed:

  1. A listener whose session GitHub dropped asks the console to open a new one.
  2. The Runner dies at a write while that call is still at the console.
  3. start_again brings GitHub up and the next Runner starts.
  4. The console then opens the dead Runner's session at GitHub (created). The next Runner's open is refused (409) and it retries every 20 to 100 ms, so opens pass causes.

The daemon is right to retry while a session of its machine is still open. The rig's own rule, "nothing the dying Runner asks reaches GitHub", was what broke.

Fix (tests only)

  • Rig::start_again takes the console down while the old Runner is killed. Before GitHub comes back, it waits until the console has answered every call it took (FakeConsole::quiet), so the dead Runner's calls end against a GitHub that is down. Afterwards the console goes back to the down state the test had set.
  • cf-fake-console gets slow_next(route, wait) (the next scale set call to a route waits before the console asks GitHub), slowed, quiet (calls counted from arrival until answered, also when the caller goes away) and is_down. tokio (time) moves from a dev-dependency to a dependency. Cargo.lock is unchanged.

Tests

  • New a_session_the_dying_runner_asked_the_console_for_is_not_opened_once_the_next_one_started: the dying Runner's session open is held at the console across a restart. Without the wait it fails with 8 opens where 1 is right; with it, it passes.
  • The CI seed is in service.proptest-regressions.
  • All 131 tests in crates/daemon/tests/service.rs and the fake console's tests pass on macOS; fmt and clippy -D warnings are clean.

Not tested before merge: the seed itself on the loaded Linux x86_64 runner. It passes on the Mac before and after the fix, since the failure needs the load.


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

…le ended the dead one's calls

The rig's crash took GitHub and the vm-hosts down but not the console, which makes the scale set calls. A session
open the dying Runner asked the console for could reach GitHub after the next Runner started, leaving a session the
next one's opens were refused against: the around-removals property test then counted more opens than causes on a
loaded Linux runner (seed a8f1c1f8).
@XeonFX
XeonFX marked this pull request as ready for review November 12, 2026 22:47
@XeonFX XeonFX added the automerge Merges itself once every check is green label Nov 12, 2026
@codefusion-automerge
codefusion-automerge Bot merged commit db6fec5 into main Nov 12, 2026
6 checks passed
@codefusion-automerge
codefusion-automerge Bot deleted the fix/session-open-race branch November 12, 2026 22:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge Merges itself once every check is green

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant