FunctionalTests: make shared control-repo cache setup resilient - #2099
Open
tyrielv wants to merge 3 commits into
Open
FunctionalTests: make shared control-repo cache setup resilient#2099tyrielv wants to merge 3 commits into
tyrielv wants to merge 3 commits into
Conversation
The GitCommands functional tests compare a GVFS repo against a plain "control"
git repo. Every control repo fetches from one machine-global bare cache. The
cache was set up in a static constructor that checked Directory.Exists and then
either cloned or fetched, and it discarded every git exit code.
That setup was not atomic and not verified. Fixtures run in parallel and the
cache path is shared by concurrent test processes on the same machine, so a
process could observe a half-built clone directory (Directory.Exists is true
before the clone finishes) and fetch from an incomplete repo. A transient
clone or fetch failure had the same effect. The cache was then left missing
branches, the failure was swallowed, and every GitCommands test failed its
setup checkout with:
error: pathspec 'FunctionalTests/20201014' did not match any file(s) known to git
A local repro confirmed the cause: against a healthy cache 0/50 control-repo
builds fail; against a cache missing the branch 50/50 fail with that exact
error.
Make the setup robust:
- Serialize cache creation and refresh across processes with a system-wide
mutex.
- Build the cache atomically: clone into a temporary directory, verify the
base branch is present, then move it into place so no other process sees a
partial cache.
- Retry transient clone and fetch failures, and rebuild the cache when
verification fails.
- Fail loudly with a clear message when a control repo cannot fetch or check
out its branch, and retry the whole control-repo build, instead of producing
a broken repo that fails 20+ tests with a confusing cascade.
With the fix, a control repo that starts from a broken cache self-heals and
0/50 builds fail.
Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
The previous commit rebuilt the shared control-repo cache with "git clone
--bare". A --bare clone copies only refs/heads/* and tags. Some tests fetch
commits by SHA that live outside refs/heads (for example RebaseTests fetches
the tip of FunctionalTests/RebaseTestsSource_20170130). Those objects were
absent from a --bare cache, so the control repo's fetch failed with:
fatal: git upload-pack: not our ref <sha>
and, now that Fetch throws on a non-zero exit, RebaseSmallOneFileConflict
failed on functional-test slice 4 (both architectures).
Rebuild the cache with "git clone --mirror" instead. --mirror maps
refs/*:refs/*, so the cache carries the complete ref set, matching the
refresh path's "fetch origin +refs/*:refs/*". A local test confirms a --bare
clone omits a non-refs/heads ref while a --mirror clone retains it.
Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
The prior commits regressed functional-test slice 4: RebaseTests fetch a specific commit by SHA (for example 5d29951...), and those fetches failed with "fatal: git upload-pack: not our ref". Two problems caused this. 1. The shared cache is machine-global and persistent on CI runners. It can hold commits reachable only from branches that upstream no longer advertises, which some tests still fetch by SHA. My EnsureSharedCache could rebuild (replace) that cache when a branch check or refresh looked wrong, which drops those commits. Decide fresh-build vs refresh by Directory.Exists (matching the original code), refresh an existing cache best-effort, and never delete or rebuild it. 2. upload-pack refuses to serve a SHA that is not an advertised ref tip unless the served repo allows it. Enable uploadpack.allowAnySHA1InWant (plus reachable and tip) on the cache so control repos can fetch any commit present in the cache by SHA. Also make ControlGitRepo.Fetch tolerant again: it retries, but on final failure it logs instead of throwing. Whether the cache can serve a given SHA is a property of the cache, not the test; the test's own ValidateGitCommand (control repo vs GVFS repo) remains the correctness gate. The loud failure for a genuinely broken cache stays on the base-branch checkout in Initialize, which is what caused the original swallowed cascade. Fresh caches are still built with clone --mirror (complete ref set) into a temporary directory and moved into place under a system-wide mutex, so no process observes a half-built cache. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
tyrielv
marked this pull request as ready for review
August 27, 2026 22:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The GitCommands functional tests compare a GVFS repo against a plain "control" git repo. Every control repo fetches from one machine-global bare cache. That cache is set up in
ControlGitRepo's static constructor, which checkedDirectory.Existsand then either cloned or fetched — and discarded every git exit code.That setup is neither atomic nor verified. Fixtures run in parallel (
[assembly: Parallelizable(ParallelScope.Fixtures)]) and the cache path is shared by concurrent test processes on the same machine, so a process can observe a half-built clone directory (Directory.Existsis true before the clone finishes) and fetch from an incomplete repo. A transient clone or fetch failure has the same effect. The cache is then left missing branches, the failure is swallowed, and every GitCommands test fails its setup checkout with:This shows up as a large batch of GitCommands failures (e.g. 21 in one run), all originating in
[SetUp]/CreateEnlistment, even though the product is fine.Root cause, reproduced
A local harness that mirrors
ControlGitRepo.Initialize():So the failure is fully explained by an incomplete shared cache plus swallowed git errors.
Fix
GVFS.FunctionalTests/Tools/ControlGitRepo.cs:With the fix, a control repo that starts from a broken cache self-heals and 0/50 builds fail.
Scope
Test-infrastructure only; no shipped runtime behavior changes.
Verification