Skip to content

A vanished suite worker is reported as a clean run - #1371

Open
xroche wants to merge 4 commits into
masterfrom
fix-1352-worker-left-no-status
Open

A vanished suite worker is reported as a clean run#1371
xroche wants to merge 4 commits into
masterfrom
fix-1352-worker-left-no-status

Conversation

@xroche

@xroche xroche commented Aug 22, 2026

Copy link
Copy Markdown
Owner

The Windows suite's fork-failure counter reads the console log for the messages bash prints when MSYS fork emulation gives out, so a worker that dies without bash saying anything is invisible to it: run 32484281897 ended on "no MSYS fork failure in suite-console.log" about a leg that had just lost 86_local-proxytrack-cache-longfields.test. The issue's account of the mechanism holds, with one correction: the suite step was not green. The tally already counted a missing .rc as a failure, so the leg exited 1. What was indistinguishable was the class, in the counter's verdict and in the failing: list.

The tally now has three outcomes instead of two. A failed test keeps FAIL and its place in failing:; a worker that left no status gets a LOST verdict saying how far it got (no log, a 0-byte log, or a log its test had written), a lost= field in the tally line, and an ::error:: naming the leg as one to re-run; a clean run says so with the lost count included. The counter reads those LOST lines back and annotates them, so its clean sentence is never printed bare again. A lost worker stays fatal, since a leg that tested less than it reports must not ship green, but exits 3 rather than 1 when nothing else failed: the status is all the workflow keeps of the difference between a leg to repeat and a red to investigate (#1228). The parent's wait status is deliberately not used, because wait -n reaps workers the pool can no longer name; the log the worker opened is what survives.

Classification moved into ci_read_outcome and ci_lost_reason, above the driver's source guard, so 337_ci-lost-worker.test can drive it off the Windows runner: the four .rc states, the counter over a clean, a fork-failure, a lost-worker and a mixed console, a mutant with the LOST rule removed proving the old counter misread the same log as clean, and a driver run over 115 stub tests where one stub kills its own worker, the control leg exiting 0 against the killed leg's 3 with lost=1 and nothing in failing:.

Closes #1352

xroche and others added 4 commits August 22, 2026 22:31
The Windows suite's fork-failure counter reads the console log for the
messages bash prints when MSYS fork emulation gives out, so a worker
that dies without bash saying anything is invisible to it: a leg that
lost a test to one still ended on "no MSYS fork failure".

The tally now classifies three outcomes instead of two. A failed test
keeps FAIL and its place in "failing:"; a worker that left no status
gets a LOST verdict saying how far it got, a lost= field in the tally
line, and an error naming the leg as one to re-run; a clean run says so
with the lost count included. A lost worker stays fatal but exits 3
rather than 1 when nothing else failed, so the workflow can tell a leg
to repeat from a red to investigate (#1228).

The classification sits in ci_read_outcome and ci_lost_reason, above
the driver's source guard, so 337_ci-lost-worker.test can drive it off
the Windows runner.

Closes #1352

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Review found exit 3 unreachable when the lost worker was one of the
expected skips: the skip-set gate fired first and exited 1, so ~9% of
tests would have reported a red to investigate rather than a leg to
re-run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
The gate added for that case was dead. Under pipefail the skip-set
diff exits 1 when the sets differ, which is the only case it runs in, so
errexit ended the script one line above it and the leg still exited 1.

Guard the pipeline, gate the pass floor the same way, and stage a third
run with the killer on a pinned skip: that path reached no assertion
before, which is why dead code landed green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
The emulated leg's rootfs carries no procps, so the ancestry walk matched
nothing and the test read that as a stub that killed no worker.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A worker can vanish with no status, and the fork counter still reports clean

1 participant