Repository navigation
Service tests restart a crashed Runner only once the console ended its calls, so a late session open no longer fails the property test - #89
Merged
Conversation
…le ended the dead one's calls The rig's crash took GitHub and the vm-hosts down but not the console, which makes the scale set calls. A session open the dying Runner asked the console for could reach GitHub after the next Runner started, leaving a session the next one's opens were refused against: the around-removals property test then counted more opens than causes on a loaded Linux runner (seed a8f1c1f8).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The service tests' rig restarts a crashed Runner only once the console has ended every call the dead Runner made. Before this, the around-removals property test could fail on a loaded runner with more session opens than causes.
Why
around_removals_every_job_that_ran_is_heard_and_no_running_job_is_stoppedfailed on #87's Linux (x86_64) job (run 43312586545, seeda8f1c1f8…) with "sessions opened [1, 3] times, with cause for [2, 2]". #87 does not cause it: the seed passes on main and on #87 on a Mac, and it fails on main once the console is slowed.The rig's crash takes GitHub and the vm-hosts down. It does not take down the console, which makes the scale set calls for a host without a key. Here is what happened, traced on main with the console's session open slowed:
start_againbrings GitHub up and the next Runner starts.created). The next Runner's open is refused (409) and it retries every 20 to 100 ms, so opens pass causes.The daemon is right to retry while a session of its machine is still open. The rig's own rule, "nothing the dying Runner asks reaches GitHub", was what broke.
Fix (tests only)
Rig::start_againtakes the console down while the old Runner is killed. Before GitHub comes back, it waits until the console has answered every call it took (FakeConsole::quiet), so the dead Runner's calls end against a GitHub that is down. Afterwards the console goes back to the down state the test had set.cf-fake-consolegetsslow_next(route, wait)(the next scale set call to a route waits before the console asks GitHub),slowed,quiet(calls counted from arrival until answered, also when the caller goes away) andis_down.tokio(time) moves from a dev-dependency to a dependency.Cargo.lockis unchanged.Tests
a_session_the_dying_runner_asked_the_console_for_is_not_opened_once_the_next_one_started: the dying Runner's session open is held at the console across a restart. Without the wait it fails with 8 opens where 1 is right; with it, it passes.service.proptest-regressions.crates/daemon/tests/service.rsand the fake console's tests pass on macOS; fmt and clippy-D warningsare clean.Not tested before merge: the seed itself on the loaded Linux x86_64 runner. It passes on the Mac before and after the fix, since the failure needs the load.
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.