A vanished suite worker is reported as a clean run - #1371
Open
xroche wants to merge 4 commits into
Open
Conversation
The Windows suite's fork-failure counter reads the console log for the messages bash prints when MSYS fork emulation gives out, so a worker that dies without bash saying anything is invisible to it: a leg that lost a test to one still ended on "no MSYS fork failure". The tally now classifies three outcomes instead of two. A failed test keeps FAIL and its place in "failing:"; a worker that left no status gets a LOST verdict saying how far it got, a lost= field in the tally line, and an error naming the leg as one to re-run; a clean run says so with the lost count included. A lost worker stays fatal but exits 3 rather than 1 when nothing else failed, so the workflow can tell a leg to repeat from a red to investigate (#1228). The classification sits in ci_read_outcome and ci_lost_reason, above the driver's source guard, so 337_ci-lost-worker.test can drive it off the Windows runner. Closes #1352 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Review found exit 3 unreachable when the lost worker was one of the expected skips: the skip-set gate fired first and exited 1, so ~9% of tests would have reported a red to investigate rather than a leg to re-run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The gate added for that case was dead. Under pipefail the skip-set diff exits 1 when the sets differ, which is the only case it runs in, so errexit ended the script one line above it and the leg still exited 1. Guard the pipeline, gate the pass floor the same way, and stage a third run with the killer on a pinned skip: that path reached no assertion before, which is why dead code landed green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The emulated leg's rootfs carries no procps, so the ancestry walk matched nothing and the test read that as a stub that killed no worker. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Windows suite's fork-failure counter reads the console log for the messages bash prints when MSYS fork emulation gives out, so a worker that dies without bash saying anything is invisible to it: run 32484281897 ended on "no MSYS fork failure in suite-console.log" about a leg that had just lost
86_local-proxytrack-cache-longfields.test. The issue's account of the mechanism holds, with one correction: the suite step was not green. The tally already counted a missing.rcas a failure, so the leg exited 1. What was indistinguishable was the class, in the counter's verdict and in thefailing:list.The tally now has three outcomes instead of two. A failed test keeps
FAILand its place infailing:; a worker that left no status gets aLOSTverdict saying how far it got (no log, a 0-byte log, or a log its test had written), alost=field in the tally line, and an::error::naming the leg as one to re-run; a clean run says so with the lost count included. The counter reads thoseLOSTlines back and annotates them, so its clean sentence is never printed bare again. A lost worker stays fatal, since a leg that tested less than it reports must not ship green, but exits 3 rather than 1 when nothing else failed: the status is all the workflow keeps of the difference between a leg to repeat and a red to investigate (#1228). The parent's wait status is deliberately not used, becausewait -nreaps workers the pool can no longer name; the log the worker opened is what survives.Classification moved into
ci_read_outcomeandci_lost_reason, above the driver's source guard, so337_ci-lost-worker.testcan drive it off the Windows runner: the four.rcstates, the counter over a clean, a fork-failure, a lost-worker and a mixed console, a mutant with theLOSTrule removed proving the old counter misread the same log as clean, and a driver run over 115 stub tests where one stub kills its own worker, the control leg exiting 0 against the killed leg's 3 withlost=1and nothing infailing:.Closes #1352