Skip to content

Migrate the test suite from xUnit v2 to xUnit v3 (2/3) - #5758

Closed
Mikael Weaver (mikaelweave) wants to merge 1 commit into
mikaelweave-xunit3-prep-collection-definitionsfrom
mikaelweave-xunit3-core-migration
Closed

Migrate the test suite from xUnit v2 to xUnit v3 (2/3)#5758
Mikael Weaver (mikaelweave) wants to merge 1 commit into
mikaelweave-xunit3-prep-collection-definitionsfrom
mikaelweave-xunit3-core-migration

Conversation

@mikaelweave

@mikaelweave Mikael Weaver (mikaelweave) commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Layer 2 of 3. Stacked on #5756 — review that first; this PR targets its branch, so the diff here is only the migration itself.

This is the irreducible core of the xUnit v2 → v3 move: the framework, the Microsoft.Health.Extensions.Xunit infrastructure rewritten against the v3 extensibility model, and the CI legs that invoke it.

Why the pipeline YAML is in this PR

main already runs on Microsoft Testing Platform via the YTest.MTP.XUnit2 shim, which accepts the VSTest-style --filter "FullyQualifiedName!~SqlServer". xUnit v3's MTP implementation does not support --filter at all — it requires --filter-query '/[(DataStore=SqlServer)&(...)]'.

So the moment the shim is swapped, the old filter form stops selecting anything on every integration and E2E leg, and it fails silently rather than erroring. The framework swap and the YAML flip are the same atomic change; splitting them produces a commit where CI is green because it ran nothing.

What's here

Framework and packages

  • xunit / xunit.runner.visualstudio v2 → xunit.v3.*; the YTest.MTP.XUnit2 shim is removed (v3 speaks MTP natively, leaving the shim with nothing to translate).
  • Version moves in Directory.Packages.props and the test .props; 29 .csproj updated.

Extensions.Xunit rewritten for v3

  • AssemblyFixtureAttribute, TestClassWithFixtureArguments and TestClassWithFixtureArgumentsTypeInfo are deleted, not ported. v3 supplies assembly fixtures directly and no longer exposes the reflection surface those types were built on.
  • Remaining framework types rebuilt against the v3 interfaces; the Tests.Common assembly guards move with them.

Pipeline

Not in this PR

src/Microsoft.Health.Extensions.Xunit.UnitTests/ and src/Microsoft.Health.Extensions.Xunit.TestAssets/ — the test harness for the rewritten framework — follow in layer 3, so this PR stays reviewable as a mechanical migration.

Verification

Check Result
Solution build 0 errors, 0 warnings
Four E2E projects compile clean (see note)
Manifest guard Assert-UnitTestProjectsDiscovered.ps1 19/19, exit 0
Microsoft.Health.Fhir.SqlServer.UnitTests 1054 passed / 0 failed (1053 on v2 pre-migration)
Microsoft.Health.Fhir.Core.UnitTests 1054 passed / 0 failed
Microsoft.Health.TaskManagement.UnitTests 17 passed / 0 failed
No VSTest --filter left in build/jobs/ confirmed absent

Note on the E2E projects: they trip the repo's pre-existing VerifyExactSdkVersion target locally (installed SDK 10.0.303 vs global.json 10.0.302). That guard is unrelated to this change and global.json is untouched, so the four projects were verified via their Compile target, which the guard's BeforeTargets="Build" hook does not gate. All four compile with 0 errors and 0 warnings.

Scope: what is here that the migration does not strictly require

This PR was audited for minimality. The framework rewrite, package swap, --filter-query conversion and the ~109 v3 API adaptations are irreducible. The following are technically separable and were kept deliberately — listed so reviewers can see the boundary rather than guess at it.

Kept because they are this migration's safety net. Trait-based and exclusion filters report success for tests they never select, so a migration that silently stops running part of the suite still produces a green build. These are the evidence that did not happen:

  • Pre-run --list-tests discovery counts with per-leg floors, --minimum-expected-tests 1, and failTaskOnMissingResultsFile: true.
  • The Tests.Common assembly guards (16 files): no shared collection definition carries traits, the custom framework is declared, and every test class and method is selected by some CI leg. The class/method checks are what caught 12 E2E cases that had become invisible to every leg.

Kept for a practical reason. The retry reconciliation (Assert-RetriedFailuresPassed.ps1 plus the publishTestResults: false / explicit PublishTestResults@2 rework) fixes a latent bug that predates this PR: with --retry-failed-tests, a test that fails and then passes still publishes the failing attempt, reddening the leg and cancelling out the retry. It is separable, but xUnit v3 schedules test classes concurrently where v2 did not, so the flake rate rises in exactly the PR that has to go green to land. Deferring it would make the riskiest change the hardest to merge.

Small unrelated items (7 lines total), not worth their own PR: the optional Microsoft.Testing.Extensions.Telemetry reference, a duplicate xUnit1051 suppression already covered by the test ruleset, and the removal of an unused System.IO.FileSystem.AccessControl package reference from four E2E projects.

Worth a closer look during review

  • src/Microsoft.Health.Fhir.SqlServer.UnitTests/Features/Search/CustomQueriesUnitTests.cs — three await Task.Delay(1100) calls. CheckQueryHash only reaches the database once a wait period has elapsed since the last query, and that timestamp is static state shared with a sibling test. The sleep makes the first call always query, whichever test ran first. Migration-adjacent (v3's scheduling exposed it) but it is a sleep-based fix and deserves scrutiny.
  • src/Microsoft.Health.Extensions.Xunit/SingleFlag.csEquals changes to Equals(other.EnumValue, EnumValue) so default(SingleFlag), which reaches Equals with a null EnumValue without passing through the constructor, no longer throws. GetHashCode already tolerated that value. This is a behaviour change to a struct rather than a v3 adaptation.

This is the irreducible core of the migration: the framework itself, the
Microsoft.Health.Extensions.Xunit infrastructure rewritten against the v3
extensibility model, and the CI legs that invoke it. These cannot land
separately - the shim swap and the pipeline filters break each other if split.

Framework and packages:

- The xunit / xunit.runner.visualstudio v2 references are replaced by
  xunit.v3.* and the YTest.MTP.XUnit2 shim is dropped. main already ran on
  Microsoft Testing Platform through that shim; v3 speaks MTP natively, so the
  shim becomes an extra translation layer with nothing left to translate.
- Directory.Packages.props and the test .props carry the version moves.

Extensions.Xunit rewritten for the v3 extensibility model:

- AssemblyFixtureAttribute, TestClassWithFixtureArguments and
  TestClassWithFixtureArgumentsTypeInfo are deleted. v3 supplies assembly
  fixtures directly and no longer exposes the reflection surface those types
  were built on, so they are replaced rather than ported.
- The remaining framework types are rebuilt against the v3 interfaces, and the
  Tests.Common assembly guards move with them.

Pipeline:

- All five build/jobs/*.yml files switch from the VSTest-style
  --filter "FullyQualifiedName!~SqlServer" to the v3 MTP
  --filter-query '/[(DataStore=SqlServer)&(...)]' form. v3's MTP does not
  implement --filter at all, so leaving the old form in place would silently
  select nothing on every integration and E2E leg.
- Assert-RetriedFailuresPassed.ps1 reads the v3 result format.
- build.yml now calls the Assert-UnitTestProjectsDiscovered.ps1 guard added in
  the preceding change, which was left unwired until the YAML landed.

Verified: the solution builds with 0 errors and 0 warnings, the four E2E
projects compile clean (their VerifyExactSdkVersion guard is a pre-existing
local SDK-pinning trip, unrelated to this change), the manifest guard reports
19/19, and 2125 tests pass with 0 failures across
Microsoft.Health.Fhir.SqlServer.UnitTests (1054), Microsoft.Health.Fhir.Core.UnitTests
(1054) and Microsoft.Health.TaskManagement.UnitTests (17).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@github-advanced-security github-advanced-security AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

@mikaelweave

Copy link
Copy Markdown
Contributor Author

Superseded by #5761.

That PR re-does this migration from scratch against main as a single change, and is a net reduction: 144 files, +1,467 / -1,967, net -500 lines. This stack grew the custom test framework; #5761 deletes the retry subsystem (676 lines), the Xunit.SkippableFact package and its 438 call sites, and the YTest.MTP.XUnit2 shim, taking src/Microsoft.Health.Extensions.Xunit/ from 1,421 to 1,045 lines.

Test selection is preserved at name level - every CI leg was enumerated against a v2 control and diffed as a name multiset (missing 0 / added 0), not compared by count.

Closing to keep review attention on #5761. The branch is not deleted, so this can be reopened if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants