Skip to content

FunctionalTests: make shared control-repo cache setup resilient - #2099

Open
tyrielv wants to merge 3 commits into
microsoft:masterfrom
tyrielv:tyrielv/fix-flaky-controlrepo
Open

FunctionalTests: make shared control-repo cache setup resilient#2099
tyrielv wants to merge 3 commits into
microsoft:masterfrom
tyrielv:tyrielv/fix-flaky-controlrepo

Conversation

@tyrielv

@tyrielv tyrielv commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Problem

The GitCommands functional tests compare a GVFS repo against a plain "control" git repo. Every control repo fetches from one machine-global bare cache. That cache is set up in ControlGitRepo's static constructor, which checked Directory.Exists and then either cloned or fetched — and discarded every git exit code.

That setup is neither atomic nor verified. Fixtures run in parallel ([assembly: Parallelizable(ParallelScope.Fixtures)]) and the cache path is shared by concurrent test processes on the same machine, so a process can observe a half-built clone directory (Directory.Exists is true before the clone finishes) and fetch from an incomplete repo. A transient clone or fetch failure has the same effect. The cache is then left missing branches, the failure is swallowed, and every GitCommands test fails its setup checkout with:

error: pathspec 'FunctionalTests/20201014' did not match any file(s) known to git

This shows up as a large batch of GitCommands failures (e.g. 21 in one run), all originating in [SetUp] / CreateEnlistment, even though the product is fine.

Root cause, reproduced

A local harness that mirrors ControlGitRepo.Initialize():

  • healthy cache → 0/50 control-repo builds fail
  • cache missing the branch → 50/50 fail, with that exact error

So the failure is fully explained by an incomplete shared cache plus swallowed git errors.

Fix

GVFS.FunctionalTests/Tools/ControlGitRepo.cs:

  • Serialize cache creation and refresh across processes with a system-wide mutex.
  • Build the cache atomically: clone into a temp directory, verify the base branch is present, then move it into place, so no other process ever observes a partial cache.
  • Retry transient clone and fetch failures, and rebuild the cache when verification fails.
  • Fail loudly with a clear message when a control repo cannot fetch or check out its branch, and retry the whole control-repo build, instead of silently producing a broken repo that fails 20+ tests with a confusing cascade.

With the fix, a control repo that starts from a broken cache self-heals and 0/50 builds fail.

Scope

Test-infrastructure only; no shipped runtime behavior changes.

Verification

  • Managed + native build clean (0 errors, 0 warnings).
  • Fix harness starting from a broken cache self-heals → 0/50 fail.
  • Full functional suite not run in this environment (needs the internal test repo, admin, and a ProjFS mount).

The GitCommands functional tests compare a GVFS repo against a plain "control"
git repo. Every control repo fetches from one machine-global bare cache. The
cache was set up in a static constructor that checked Directory.Exists and then
either cloned or fetched, and it discarded every git exit code.

That setup was not atomic and not verified. Fixtures run in parallel and the
cache path is shared by concurrent test processes on the same machine, so a
process could observe a half-built clone directory (Directory.Exists is true
before the clone finishes) and fetch from an incomplete repo. A transient
clone or fetch failure had the same effect. The cache was then left missing
branches, the failure was swallowed, and every GitCommands test failed its
setup checkout with:

    error: pathspec 'FunctionalTests/20201014' did not match any file(s) known to git

A local repro confirmed the cause: against a healthy cache 0/50 control-repo
builds fail; against a cache missing the branch 50/50 fail with that exact
error.

Make the setup robust:

- Serialize cache creation and refresh across processes with a system-wide
  mutex.
- Build the cache atomically: clone into a temporary directory, verify the
  base branch is present, then move it into place so no other process sees a
  partial cache.
- Retry transient clone and fetch failures, and rebuild the cache when
  verification fails.
- Fail loudly with a clear message when a control repo cannot fetch or check
  out its branch, and retry the whole control-repo build, instead of producing
  a broken repo that fails 20+ tests with a confusing cascade.

With the fix, a control repo that starts from a broken cache self-heals and
0/50 builds fail.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
The previous commit rebuilt the shared control-repo cache with "git clone
--bare". A --bare clone copies only refs/heads/* and tags. Some tests fetch
commits by SHA that live outside refs/heads (for example RebaseTests fetches
the tip of FunctionalTests/RebaseTestsSource_20170130). Those objects were
absent from a --bare cache, so the control repo's fetch failed with:

    fatal: git upload-pack: not our ref <sha>

and, now that Fetch throws on a non-zero exit, RebaseSmallOneFileConflict
failed on functional-test slice 4 (both architectures).

Rebuild the cache with "git clone --mirror" instead. --mirror maps
refs/*:refs/*, so the cache carries the complete ref set, matching the
refresh path's "fetch origin +refs/*:refs/*". A local test confirms a --bare
clone omits a non-refs/heads ref while a --mirror clone retains it.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
The prior commits regressed functional-test slice 4: RebaseTests fetch a
specific commit by SHA (for example 5d29951...), and those fetches failed
with "fatal: git upload-pack: not our ref". Two problems caused this.

1. The shared cache is machine-global and persistent on CI runners. It can
   hold commits reachable only from branches that upstream no longer
   advertises, which some tests still fetch by SHA. My EnsureSharedCache
   could rebuild (replace) that cache when a branch check or refresh looked
   wrong, which drops those commits. Decide fresh-build vs refresh by
   Directory.Exists (matching the original code), refresh an existing cache
   best-effort, and never delete or rebuild it.

2. upload-pack refuses to serve a SHA that is not an advertised ref tip
   unless the served repo allows it. Enable uploadpack.allowAnySHA1InWant
   (plus reachable and tip) on the cache so control repos can fetch any
   commit present in the cache by SHA.

Also make ControlGitRepo.Fetch tolerant again: it retries, but on final
failure it logs instead of throwing. Whether the cache can serve a given SHA
is a property of the cache, not the test; the test's own ValidateGitCommand
(control repo vs GVFS repo) remains the correctness gate. The loud failure
for a genuinely broken cache stays on the base-branch checkout in
Initialize, which is what caused the original swallowed cascade.

Fresh caches are still built with clone --mirror (complete ref set) into a
temporary directory and moved into place under a system-wide mutex, so no
process observes a half-built cache.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
tyrielv marked this pull request as ready for review August 27, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant