Migrate the test suite from xUnit v2 to xUnit v3 (2/3) - #5758
Conversation
This is the irreducible core of the migration: the framework itself, the Microsoft.Health.Extensions.Xunit infrastructure rewritten against the v3 extensibility model, and the CI legs that invoke it. These cannot land separately - the shim swap and the pipeline filters break each other if split. Framework and packages: - The xunit / xunit.runner.visualstudio v2 references are replaced by xunit.v3.* and the YTest.MTP.XUnit2 shim is dropped. main already ran on Microsoft Testing Platform through that shim; v3 speaks MTP natively, so the shim becomes an extra translation layer with nothing left to translate. - Directory.Packages.props and the test .props carry the version moves. Extensions.Xunit rewritten for the v3 extensibility model: - AssemblyFixtureAttribute, TestClassWithFixtureArguments and TestClassWithFixtureArgumentsTypeInfo are deleted. v3 supplies assembly fixtures directly and no longer exposes the reflection surface those types were built on, so they are replaced rather than ported. - The remaining framework types are rebuilt against the v3 interfaces, and the Tests.Common assembly guards move with them. Pipeline: - All five build/jobs/*.yml files switch from the VSTest-style --filter "FullyQualifiedName!~SqlServer" to the v3 MTP --filter-query '/[(DataStore=SqlServer)&(...)]' form. v3's MTP does not implement --filter at all, so leaving the old form in place would silently select nothing on every integration and E2E leg. - Assert-RetriedFailuresPassed.ps1 reads the v3 result format. - build.yml now calls the Assert-UnitTestProjectsDiscovered.ps1 guard added in the preceding change, which was left unwired until the YAML landed. Verified: the solution builds with 0 errors and 0 warnings, the four E2E projects compile clean (their VerifyExactSdkVersion guard is a pre-existing local SDK-pinning trip, unrelated to this change), the manifest guard reports 19/19, and 2125 tests pass with 0 failures across Microsoft.Health.Fhir.SqlServer.UnitTests (1054), Microsoft.Health.Fhir.Core.UnitTests (1054) and Microsoft.Health.TaskManagement.UnitTests (17). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
|
Superseded by #5761. That PR re-does this migration from scratch against Test selection is preserved at name level - every CI leg was enumerated against a v2 control and diffed as a name multiset (missing 0 / added 0), not compared by count. Closing to keep review attention on #5761. The branch is not deleted, so this can be reopened if needed. |
Layer 2 of 3. Stacked on #5756 — review that first; this PR targets its branch, so the diff here is only the migration itself.
This is the irreducible core of the xUnit v2 → v3 move: the framework, the
Microsoft.Health.Extensions.Xunitinfrastructure rewritten against the v3 extensibility model, and the CI legs that invoke it.Why the pipeline YAML is in this PR
mainalready runs on Microsoft Testing Platform via theYTest.MTP.XUnit2shim, which accepts the VSTest-style--filter "FullyQualifiedName!~SqlServer". xUnit v3's MTP implementation does not support--filterat all — it requires--filter-query '/[(DataStore=SqlServer)&(...)]'.So the moment the shim is swapped, the old filter form stops selecting anything on every integration and E2E leg, and it fails silently rather than erroring. The framework swap and the YAML flip are the same atomic change; splitting them produces a commit where CI is green because it ran nothing.
What's here
Framework and packages
xunit/xunit.runner.visualstudiov2 →xunit.v3.*; theYTest.MTP.XUnit2shim is removed (v3 speaks MTP natively, leaving the shim with nothing to translate).Directory.Packages.propsand the test.props; 29.csprojupdated.Extensions.Xunit rewritten for v3
AssemblyFixtureAttribute,TestClassWithFixtureArgumentsandTestClassWithFixtureArgumentsTypeInfoare deleted, not ported. v3 supplies assembly fixtures directly and no longer exposes the reflection surface those types were built on.Tests.Commonassembly guards move with them.Pipeline
build/jobs/*.ymlswitch to--filter-query.Assert-RetriedFailuresPassed.ps1reads the v3 result format.build.ymlnow wires upAssert-UnitTestProjectsDiscovered.ps1, which Extract shared collection definitions and add unit test manifest guard #5756 added but deliberately left uncalled until this YAML landed.Not in this PR
src/Microsoft.Health.Extensions.Xunit.UnitTests/andsrc/Microsoft.Health.Extensions.Xunit.TestAssets/— the test harness for the rewritten framework — follow in layer 3, so this PR stays reviewable as a mechanical migration.Verification
Assert-UnitTestProjectsDiscovered.ps1Microsoft.Health.Fhir.SqlServer.UnitTestsMicrosoft.Health.Fhir.Core.UnitTestsMicrosoft.Health.TaskManagement.UnitTests--filterleft inbuild/jobs/Note on the E2E projects: they trip the repo's pre-existing
VerifyExactSdkVersiontarget locally (installed SDK 10.0.303 vsglobal.json10.0.302). That guard is unrelated to this change andglobal.jsonis untouched, so the four projects were verified via theirCompiletarget, which the guard'sBeforeTargets="Build"hook does not gate. All four compile with 0 errors and 0 warnings.Scope: what is here that the migration does not strictly require
This PR was audited for minimality. The framework rewrite, package swap,
--filter-queryconversion and the ~109 v3 API adaptations are irreducible. The following are technically separable and were kept deliberately — listed so reviewers can see the boundary rather than guess at it.Kept because they are this migration's safety net. Trait-based and exclusion filters report success for tests they never select, so a migration that silently stops running part of the suite still produces a green build. These are the evidence that did not happen:
--list-testsdiscovery counts with per-leg floors,--minimum-expected-tests 1, andfailTaskOnMissingResultsFile: true.Tests.Commonassembly guards (16 files): no shared collection definition carries traits, the custom framework is declared, and every test class and method is selected by some CI leg. The class/method checks are what caught 12 E2E cases that had become invisible to every leg.Kept for a practical reason. The retry reconciliation (
Assert-RetriedFailuresPassed.ps1plus thepublishTestResults: false/ explicitPublishTestResults@2rework) fixes a latent bug that predates this PR: with--retry-failed-tests, a test that fails and then passes still publishes the failing attempt, reddening the leg and cancelling out the retry. It is separable, but xUnit v3 schedules test classes concurrently where v2 did not, so the flake rate rises in exactly the PR that has to go green to land. Deferring it would make the riskiest change the hardest to merge.Small unrelated items (7 lines total), not worth their own PR: the optional
Microsoft.Testing.Extensions.Telemetryreference, a duplicatexUnit1051suppression already covered by the test ruleset, and the removal of an unusedSystem.IO.FileSystem.AccessControlpackage reference from four E2E projects.Worth a closer look during review
src/Microsoft.Health.Fhir.SqlServer.UnitTests/Features/Search/CustomQueriesUnitTests.cs— threeawait Task.Delay(1100)calls.CheckQueryHashonly reaches the database once a wait period has elapsed since the last query, and that timestamp is static state shared with a sibling test. The sleep makes the first call always query, whichever test ran first. Migration-adjacent (v3's scheduling exposed it) but it is a sleep-based fix and deserves scrutiny.src/Microsoft.Health.Extensions.Xunit/SingleFlag.cs—Equalschanges toEquals(other.EnumValue, EnumValue)sodefault(SingleFlag), which reachesEqualswith a nullEnumValuewithout passing through the constructor, no longer throws.GetHashCodealready tolerated that value. This is a behaviour change to a struct rather than a v3 adaptation.