Skip to content

Capture mount dumps and preserve logs when a functional test fails - #2062

Open
tyrielv wants to merge 1 commit into
microsoft:masterfrom
tyrielv:tyrielv/ft-failure-diagnostics
Open

Capture mount dumps and preserve logs when a functional test fails#2062
tyrielv wants to merge 1 commit into
microsoft:masterfrom
tyrielv:tyrielv/ft-failure-diagnostics

Conversation

@tyrielv

@tyrielv tyrielv commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Capture mount dumps and preserve logs when a functional test fails

When a functional test fails in CI, the only diagnostic we get today is whatever
TestResultsHelper.OutputGVFSLogs happens to inline to the console. If a
GVFS.Mount process is hung (the common flaky-failure signature — a test times
out waiting on the mount named pipe with no crash trace), there is no way to see
why after the fact: the process is killed during teardown and its state is lost.

This adds first-class failure diagnostics for the functional tests:

  • Mount minidump. On a failed test, before teardown unmounts/kills anything,
    GVFSFunctionalTestEnlistment.CaptureFailureDiagnostics() finds the live
    GVFS.Mount process(es) for the enlistment and writes a full-memory minidump
    via MiniDumpWriteDump (new MiniDump helper, P/Invoke into dbghelp). This
    is exactly what's needed to root-cause a hang.
  • Robust log preservation. The enlistment's .gvfs/logs are copied out with
    a fallback that reopens locked/partially-flushed files with a shared
    read handle (FileShare.ReadWrite | Delete), so we still capture logs that a
    live process is holding open.
  • CI artifact upload. functional-tests.yaml points diagnostics at a fixed
    directory (GVFS_TEST_DIAGNOSTICS_DIR) and uploads it as
    FailureDiagnostics_<matrix> with if: always().

Everything is best-effort and never throws, so it cannot turn a passing test red
or mask the real failure.

Incidental fix

GetMountProcessIds (extracted from the existing KillMountProcess) matched the
mount command line with a PowerShell -like wildcard built from the enlistment
path with its backslashes doubled. In -like, \ is a literal, not an escape,
so the doubled pattern never matched a real single-backslash command line — it
found zero PIDs. This was a pre-existing latent bug: KillMountProcess has been
silently killing nothing. Now matches on the enlistment's unique leaf id, which
is on the mount command line and needs no escaping.

Validation

Exercised end-to-end in CI via a throwaway harness branch (a smoke test that
Assert.Fail()s with a live mount). The FailureDiagnostics artifact came back
containing GVFS.Mount_<pid>.dmp (~125 MB full-memory dump) plus the preserved
clone/mount/prefetch logs.

@tyrielv
tyrielv marked this pull request as ready for review July 13, 2026 18:27
@tyrielv
tyrielv enabled auto-merge July 13, 2026 18:28
Comment thread GVFS/GVFS.FunctionalTests/Tools/GVFSFunctionalTestEnlistment.cs Outdated
@tyrielv
tyrielv force-pushed the tyrielv/ft-failure-diagnostics branch from ae9be31 to aa4fdf0 Compare July 14, 2026 22:43
proc.Start();
proc.BeginOutputReadLine();

if (!proc.WaitForExit(10000))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The asynchronous stdout reader may still be flushing when the PID output is parsed, which can make mount process discovery intermittently miss a live process.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right -- WaitForExit(int) only guarantees the process exited, not that the async OutputDataReceived callbacks have all fired, so parsing output right after could race the last callback and drop a trailing PID. Fixed by calling the parameterless WaitForExit() right after the timed wait succeeds, which blocks until the redirected-stream reads complete (a no-op if the process already exited). Also rebased onto latest master and squashed -- see f22daab.

When a functional test fails because its GVFS.Mount is unreachable (a hang or
silent exit), we currently have no post-mortem data. Per-test enlistment
inlined into the console by TestResultsHelper.OutputGVFSLogs -- lossy under
parallel fixtures and empty when the mount hung. There is no process dump, so a
mount deadlock leaves zero diagnostic signal.

On test failure only, into a CI-uploadable diagnostics directory:
GVFSFunctionalTestEnlistment.CaptureFailureDiagnostics runs first in
DeleteEnlistment, gated on TestStatus.Failed, before the enlistment directory is
deleted. The mount-process PID discovery is extracted from KillMountProcess into
a shared GetMountProcessIds helper.

CI: functional-tests.yaml sets GVFS_TEST_DIAGNOSTICS_DIR and uploads it as a
FailureDiagnostics artifact with if: always().

Review follow-ups:
- GetMountProcessIds doubled the backslashes in the enlistment path before
  using it in a PowerShell -like wildcard. In -like, '\' is a literal (not an
  escape), so the doubled pattern never matched a real single-backslash
  command line: CaptureFailureDiagnostics found no live mount and wrote no
  minidump, and KillMountProcess silently killed nothing. Match on the
  enlistment's unique leaf folder id instead -- present on the GVFS.Mount
  command line (launched with PrimaryEnlistmentRoot), unique, and free of
  path separators or wildcard metacharacters, so it needs no escaping.
- WaitForExit(int) only guarantees the helper process has exited -- it does
  not guarantee the async OutputDataReceived callbacks (raised on the thread
  pool as data arrives) have all run yet. Reading the output buffer
  immediately afterward could race the last callback and intermittently drop
  a trailing PID, making GetMountProcessIds miss a live mount process. Call
  the parameterless WaitForExit() right after the timed wait succeeds to
  drain any pending async output callbacks before parsing.

Assisted-by: Claude Sonnet 5
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
tyrielv force-pushed the tyrielv/ft-failure-diagnostics branch from e3fd01e to f22daab Compare August 27, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants