Capture mount dumps and preserve logs when a functional test fails - #2062
Open
tyrielv wants to merge 1 commit into
Open
Capture mount dumps and preserve logs when a functional test fails#2062tyrielv wants to merge 1 commit into
tyrielv wants to merge 1 commit into
Conversation
tyrielv
marked this pull request as ready for review
July 13, 2026 18:27
tyrielv
enabled auto-merge
July 13, 2026 18:28
tyrielv
force-pushed
the
tyrielv/ft-failure-diagnostics
branch
from
July 14, 2026 22:43
ae9be31 to
aa4fdf0
Compare
| proc.Start(); | ||
| proc.BeginOutputReadLine(); | ||
|
|
||
| if (!proc.WaitForExit(10000)) |
There was a problem hiding this comment.
The asynchronous stdout reader may still be flushing when the PID output is parsed, which can make mount process discovery intermittently miss a live process.
Contributor
Author
There was a problem hiding this comment.
You're right -- WaitForExit(int) only guarantees the process exited, not that the async OutputDataReceived callbacks have all fired, so parsing output right after could race the last callback and drop a trailing PID. Fixed by calling the parameterless WaitForExit() right after the timed wait succeeds, which blocks until the redirected-stream reads complete (a no-op if the process already exited). Also rebased onto latest master and squashed -- see f22daab.
When a functional test fails because its GVFS.Mount is unreachable (a hang or silent exit), we currently have no post-mortem data. Per-test enlistment inlined into the console by TestResultsHelper.OutputGVFSLogs -- lossy under parallel fixtures and empty when the mount hung. There is no process dump, so a mount deadlock leaves zero diagnostic signal. On test failure only, into a CI-uploadable diagnostics directory: GVFSFunctionalTestEnlistment.CaptureFailureDiagnostics runs first in DeleteEnlistment, gated on TestStatus.Failed, before the enlistment directory is deleted. The mount-process PID discovery is extracted from KillMountProcess into a shared GetMountProcessIds helper. CI: functional-tests.yaml sets GVFS_TEST_DIAGNOSTICS_DIR and uploads it as a FailureDiagnostics artifact with if: always(). Review follow-ups: - GetMountProcessIds doubled the backslashes in the enlistment path before using it in a PowerShell -like wildcard. In -like, '\' is a literal (not an escape), so the doubled pattern never matched a real single-backslash command line: CaptureFailureDiagnostics found no live mount and wrote no minidump, and KillMountProcess silently killed nothing. Match on the enlistment's unique leaf folder id instead -- present on the GVFS.Mount command line (launched with PrimaryEnlistmentRoot), unique, and free of path separators or wildcard metacharacters, so it needs no escaping. - WaitForExit(int) only guarantees the helper process has exited -- it does not guarantee the async OutputDataReceived callbacks (raised on the thread pool as data arrives) have all run yet. Reading the output buffer immediately afterward could race the last callback and intermittently drop a trailing PID, making GetMountProcessIds miss a live mount process. Call the parameterless WaitForExit() right after the timed wait succeeds to drain any pending async output callbacks before parsing. Assisted-by: Claude Sonnet 5 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
tyrielv
force-pushed
the
tyrielv/ft-failure-diagnostics
branch
from
August 27, 2026 16:45
e3fd01e to
f22daab
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Capture mount dumps and preserve logs when a functional test fails
When a functional test fails in CI, the only diagnostic we get today is whatever
TestResultsHelper.OutputGVFSLogshappens to inline to the console. If aGVFS.Mountprocess is hung (the common flaky-failure signature — a test timesout waiting on the mount named pipe with no crash trace), there is no way to see
why after the fact: the process is killed during teardown and its state is lost.
This adds first-class failure diagnostics for the functional tests:
GVFSFunctionalTestEnlistment.CaptureFailureDiagnostics()finds the liveGVFS.Mountprocess(es) for the enlistment and writes a full-memory minidumpvia
MiniDumpWriteDump(newMiniDumphelper, P/Invoke intodbghelp). Thisis exactly what's needed to root-cause a hang.
.gvfs/logsare copied out witha fallback that reopens locked/partially-flushed files with a shared
read handle (
FileShare.ReadWrite | Delete), so we still capture logs that alive process is holding open.
functional-tests.yamlpoints diagnostics at a fixeddirectory (
GVFS_TEST_DIAGNOSTICS_DIR) and uploads it asFailureDiagnostics_<matrix>withif: always().Everything is best-effort and never throws, so it cannot turn a passing test red
or mask the real failure.
Incidental fix
GetMountProcessIds(extracted from the existingKillMountProcess) matched themount command line with a PowerShell
-likewildcard built from the enlistmentpath with its backslashes doubled. In
-like,\is a literal, not an escape,so the doubled pattern never matched a real single-backslash command line — it
found zero PIDs. This was a pre-existing latent bug:
KillMountProcesshas beensilently killing nothing. Now matches on the enlistment's unique leaf id, which
is on the mount command line and needs no escaping.
Validation
Exercised end-to-end in CI via a throwaway harness branch (a smoke test that
Assert.Fail()s with a live mount). TheFailureDiagnosticsartifact came backcontaining
GVFS.Mount_<pid>.dmp(~125 MB full-memory dump) plus the preservedclone/mount/prefetch logs.