diff --git a/.config/dotnet-tools.json b/.config/dotnet-tools.json new file mode 100644 index 0000000..010a8c1 --- /dev/null +++ b/.config/dotnet-tools.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "isRoot": true, + "tools": { + "dotnet-stryker": { + "version": "4.16.0", + "commands": [ + "dotnet-stryker" + ], + "rollForward": false + } + } +} \ No newline at end of file diff --git a/.gitignore b/.gitignore index 24628a3..ebf96ff 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,7 @@ obj/ TestResults/ coverage-merged/ *.received.* +StrykerOutput/ .DS_Store *.lscache .codex diff --git a/AGENTS.md b/AGENTS.md index 48d1265..e070cf8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,6 +15,7 @@ In our Directory.Build.props files in this solution, the following rules are def - The library is not published in a stable version yet, you can make breaking changes. - `` is enabled in Release builds, so your code changes must not generate warnings. - When a type or method is properly encapsulated, make it public. We don't know how callers would like to use this library. When some types are internal, this might make it hard for callers to access these in tests or when making configuration changes. Prefer public APIs over internal ones. +- Use Conventional Commits messages. Decide whether a commit title is enough or a commit body is required. ## Testing Rules diff --git a/ai-plans/0063-0-add-stryker-mutation-testing.md b/ai-plans/0063-0-add-stryker-mutation-testing.md new file mode 100644 index 0000000..749b792 --- /dev/null +++ b/ai-plans/0063-0-add-stryker-mutation-testing.md @@ -0,0 +1,137 @@ +# Stryker.NET Mutation Testing + +## Rationale + +Line coverage is held above 95%, but it only proves that code executed, not that any assertion constrains it. Coding agents extending this library cannot currently distinguish a well-tested code path from one that is merely reached. Mutation testing narrows that gap: a surviving mutant is a concrete, machine-readable location where the suite failed to detect a changed program, which makes it a candidate for a missing assertion rather than proof of one. Some survivors are equivalent or not worth killing, so the output is a triage queue, not a defect list. + +Introduce Stryker.NET as a pinned, opt-in tool with a repository configuration that is correct by default. Two settings are load-bearing — the wrong test runner and the default coverage analysis each produce confidently wrong reports on this solution — so the configuration, not the invocation, must carry them. + +## Acceptance Criteria + +- [x] `dotnet tool restore` installs `dotnet-stryker` pinned to exactly 4.16.0, and a repository-root `stryker-config.json` supplies the shared configuration so that a scoped run needs only a `-p` argument and an optional `-m` glob. +- [x] The committed configuration selects the Microsoft Testing Platform runner and disables coverage analysis; a run against `Light.PortableResults.AspNetCore.Shared` reports a non-zero mutation score, and a run against `Result.cs` reports zero `NoCoverage` mutants. +- [x] Mutation output is contained under `StrykerOutput/`, which is ignored by git; a completed or interrupted run adds no new entries to `git status`. +- [x] `tests/AGENTS.md` documents the file-scoped and project-scoped invocations, states that a surviving mutant is a signal to investigate together with the three permitted triage outcomes, records the measured cost per project, and records the mutation blind spots that must not be read as adequate coverage. +- [x] Mutation testing runs on demand from a developer machine only. No CI workflow is added and no existing workflow is modified; pull request validation time is unchanged. +- [x] The documented invocation is verified to run the intended tests: for a run mutating `AspNetCore.Shared`, Stryker's reported test count is 337 — every test project transitively referencing it — and not the 26 tests of `AspNetCore.Shared.Tests` alone. +- [x] A baseline is recorded in `tests/AGENTS.md` for the four projects that complete in minutes (`AspNetCore.Shared`, `AspNetCore.Mvc`, `AspNetCore.MinimalApis`, `Validation.OpenApi`). Each entry carries enough provenance to be reproduced and compared: commit, `dotnet-stryker` version, concurrency, the test count actually executed, elapsed time, and the killed, timeout, survived, compile-error, ignored, and no-coverage counts — not a percentage alone. The no-coverage count is recorded even though it is expected to be zero, because a non-zero value means `coverage-analysis` is no longer taking effect; it was zero in all four measured runs. +- [x] Mutation score thresholds and `break-at` remain unset; nothing in the repository fails a build or a test run because of a mutation score. + +## Technical Details + +### Verified constraints + +The findings below come from `dotnet-stryker` 4.16.0 executed against this solution. They are the reason the configuration is not left at its defaults. + +Microsoft Testing Platform support is still a preview feature in Stryker.NET, and it announces itself as one on every run. It is tracked upstream in [stryker-mutator/stryker-net#3094](https://github.com/stryker-mutator/stryker-net/issues/3094), which is the issue to watch for the maturity of everything in this section. + +**The default `vstest` runner is unusable here.** These test projects set `UseMicrosoftTestingPlatformRunner`, and VSTest cannot drive their xUnit v3 hosts ([#3117](https://github.com/stryker-mutator/stryker-net/issues/3117)). Coverage capture fails and every mutant is reported as survived — `AspNetCore.Shared` scores 0.00% under `vstest` and 100.00% under `mtp`. The failure is a logged error plus a plausible-looking report, not a crash, so `"test-runner": "mtp"` must be committed rather than passed ad hoc. + +**MTP coverage analysis fabricates `NoCoverage`.** With the default `perTest` analysis, `Result.cs` reports 6 killed and 61 `NoCoverage` for a score of 8.96%, despite `NonGenericResultTests` covering those members directly. With `"coverage-analysis": "off"` the same file reports 67 tested, 0 uncovered, and 100.00%. A false `NoCoverage` is the most damaging possible output for the intended agent workflow, because it directs effort at tests that already exist. Per-test coverage for the MTP runner is in progress as [#3516](https://github.com/stryker-mutator/stryker-net/pull/3516); revisit this setting when it ships, since it is the main lever on run time. + +The configuration is therefore: + +```json +{ + "stryker-config": { + "solution": "Light.PortableResults.slnx", + "test-runner": "mtp", + "coverage-analysis": "off", + "configuration": "Debug", + "reporters": ["json", "html", "progress"] + } +} +``` + +Everything else about the repository is compatible without special handling: `.slnx` analysis, central package management with lock files, the `netstandard2.0;net10.0` multi-targeting (Stryker selects `net10.0` unaided), the netstandard2.0 source generator, `InterceptorsNamespaces`, and the Verify-based snapshot tests, which leave no `.received.*` files behind because 4.14.2 disables DiffEngine under MTP. No strong-naming or `InternalsVisibleTo` seam exists to work around. + +`configuration` is pinned to `Debug` in the config rather than left to the invoker. `TreatWarningsAsErrors` is Release-only, and Stryker's rollback should not have to contend with warnings promoted to errors. All measurements in this plan were taken under `Debug`. + +Reports are written to `StrykerOutput//` beneath the working directory; the JSON and HTML reporters exist to produce files, so the goal is containment, not absence. Stryker already writes a `.gitignore` containing `*` into each timestamped directory, and a completed run leaves `git status` clean without any repository change. Add `StrykerOutput/` to `.gitignore` anyway: it states the intent, covers the directory itself, and protects against a run interrupted before that inner file is written. The output location is a CLI concern — `--output` has no `stryker-config.json` equivalent — so nothing here depends on redirecting it, and no wrapper script is needed. + +### Cost + +Measured throughput is approximately three seconds of wall clock per mutant at concurrency 8, dominated by fixed per-mutant overhead rather than test count: the 337-test and 2,398-test suites cost the same per mutant. Mutant inventory: + +| Project | Mutants | CompileError | +| --- | ---: | ---: | +| `Light.PortableResults` | 4,867 | 520 | +| `Light.PortableResults.Validation` | 2,215 | 83 | +| `Validation.OpenApi.SourceGeneration` | 1,272 | 204 | +| `AspNetCore.OpenApi` | 848 | 65 | +| `Validation.OpenApi` | 114 | 2 | +| `AspNetCore.Shared` | 44 | 3 | +| `AspNetCore.Mvc` | 33 | 11 | +| `AspNetCore.MinimalApis` | 32 | 11 | + +A solution-wide run is roughly seven hours on the measured hardware, which is why this stays a local, on-demand tool rather than anything automated. Mutation testing is not added to CI in any form: no new workflow, and no change to `build-and-test.yml`. Whoever drives it locally chooses the scope, and the practical scopes are one file or one project — never the whole solution in the inner loop. + +The two large projects are long-running even in isolation: `Light.PortableResults` is roughly 3.6 hours and `Validation` roughly 1.8 hours. When one of them is worth running end to end, split it by `mutate` glob along folder boundaries (`Metadata/`, `Http/`, `CloudEvents/`, `Numbers/`) so the run is interruptible and each report arrives while it is still actionable. + +The JSON report is the agent-facing artifact; the HTML report is for humans. Agents filter for `"status": "Survived"` to obtain a work queue located at specific files and lines. A survivor is a signal to investigate, not a defect to fix on sight; see triage below. + +### Scoped invocation + +Because the configuration names the solution, every run is a solution-context run. `-p` selects which source project is mutated; it does not narrow which tests execute. Stryker discovers the test projects from the solution and runs **every test project that transitively references the mutated project**. A `-tp` argument does not override this and must not be documented as if it did. + +This was confirmed by test counts rather than inferred: mutating `AspNetCore.Shared` reports 337 tests, which is the exact sum of the six test projects referencing it (26 + 46 + 23 + 79 + 112 + 51), not the 26 in `AspNetCore.Shared.Tests`. Mutating `Light.PortableResults` reports 2,398 — the whole solution, since everything references it. + +Keep this behavior rather than forcing a single test project. It matches the sociable-testing rule in `tests/AGENTS.md`: a mutant in `AspNetCore.Shared` killed by `MinimalApis.Tests` is legitimately killed, and isolating test projects would convert those cross-project kills into false survivors. It is also close to free, because per-mutant cost is dominated by fixed overhead — the 337-test and 2,398-test sets cost the same per mutant. + +The form to document for agents is therefore source project plus optional file glob. Mutating one file takes about four minutes: + +```shell +dotnet stryker -p .csproj -m '**/TheFile.cs' +``` + +Every command runs from the repository root. Stryker resolves `stryker-config.json` relative to the current working directory and does not search parent directories, and a missing config file is not an error — it silently falls back to the defaults. From a subdirectory the run therefore loses both load-bearing settings at once, reverting to the `vstest` runner and `perTest` coverage analysis, which is exactly the combination that reports 0.00% or fabricates `NoCoverage`. The failure is a plausible report rather than a diagnostic, so `tests/AGENTS.md` must state the working directory as part of the invocation rather than assume it. + +The four small projects need no `-m` at all. Measured end to end, including analysis, build, and initial test run: + +| Project | Tests run | Created | Tested | Elapsed | +| --- | ---: | ---: | ---: | ---: | +| `AspNetCore.Mvc` | 102 | 33 | 17 | 0:38 | +| `AspNetCore.MinimalApis` | 237 | 32 | 16 | 0:57 | +| `AspNetCore.Shared` | 337 | 44 | 31 | 1:57 | +| `Validation.OpenApi` | 163 | 114 | 76 | 1:59 | + +Omitting `-p` mutates every source project in the solution and is the seven-hour path. + +Do not derive these times from the three-seconds-per-mutant figure. That rate is an upper bound taken from the large projects, where the mutated assembly is referenced by the whole solution and every test project runs. Two effects make small projects cheaper: only a fraction of created mutants is ever tested — 76 of 114 for `Validation.OpenApi` — and per-mutant cost falls with the size of the associated test set. Use the rate to size the projects in the inventory table above, and these measurements for the small ones. + +Do not use or document `--since` under this configuration. It was measured on this branch and never narrowed anything: with a completely clean working tree it still reported `ai-plans/0063-add-stryker-mutation-testing.md` as a changed test file and escalated to `16 mutants will be tested because: Non-CSharp files in test project were changed`. + +Two behaviors combine badly here. Stryker treats any non-C# file in the diff — including a Markdown plan — as a changed test file and responds by testing every mutant; and `--since:HEAD` did not pin the baseline to `HEAD`, so the diff kept resolving against the default branch. Because every feature branch in this repository begins by adding a plan under `ai-plans/`, a non-C# file is essentially always in the diff. `--since` therefore degrades to a full project run while presenting itself as scoped, which is worse than not using it: the cost is unchanged and the reported scope is wrong. + +Explicit file and project scope is the only recommended form. Revisit `--since` only once MTP coverage analysis is trustworthy — a separate concern is that mapping changed *test* files back to mutants depends on per-mutant covering tests, which `coverage-analysis: off` does not produce, so a test-only edit has no reliable path to the mutants it affects. + +### Triaging survivors + +A surviving mutant means the suite did not distinguish the mutated program from the original. That has three admissible causes, and exactly three permitted responses: + +1. **Observable behavior is genuinely unconstrained.** Add or strengthen a test. This is the outcome the tool exists to produce and should be the common one. +2. **The mutant is equivalent or invalid** — semantically identical to the original, or killable only by asserting on something that is not part of the contract. Suppress it narrowly at the source with a justification, which Stryker records in the report: + + ```csharp + // Stryker disable once Statement : equivalent - the guard is a fast path, not a behavior change + ``` + + The syntax is `Stryker [disable|restore][once][all|][: reason]`, and it is scope-aware. Prefer `disable once` over `disable all`, and never reach for the global `ignore-mutations` setting to silence a single site. +3. **The mutant sits in a construct this configuration cannot meaningfully test.** Record it and move on; see the blind spots below. + +Never restructure production code solely to make a mutant killable. The root `AGENTS.md` ranks performance above extensibility, and this library's low-allocation `in`/`ref`/`Span` style is precisely the shape that produces awkward survivors. A lower mutation score is the correct outcome when the alternative is a slower or less direct implementation. + +The measured runs produced exactly one survivor across all four small projects: a statement-removal mutant at `BuiltInValidationErrorBuilderExtensions.cs:426`, which puts `Validation.OpenApi` at 98.68%. It is untriaged — it is the first item to run through the three categories above, and it is the only evidence so far that the queue will contain anything at all. Every other scored run returned 100.00%, so these categories remain largely a priori rather than derived from survivors observed in this codebase. + +### Blind spots to document + +Roughly 9.5% of mutants fail to compile, concentrated in the low-allocation `out`/`ref` style. Stryker responds with Safe Mode, which discards every mutant in the enclosing method: + +``` +CS0165: Use of unassigned local variable 'low' (Numbers/Dragon4.cs:263) +[INF] Safe Mode! Stryker will remove all mutations in GenerateDigits +``` + +`Dragon4.GenerateDigits`, `ResultJsonReader.ReadStatusValue`, `ResultJsonReader.ReadIndexValue`, `ErrorsExtensions.WriteRichErrors`, and the whole of `LightResult.cs` (11 of 11 mutants) receive no mutation coverage at all. This is a tool limitation, not a test defect. `tests/AGENTS.md` must state it explicitly so that a high mutation score in `Numbers/` is not mistaken for verified behavior, and so that no one attempts to "fix" it by restructuring production code. + +Mutation score is a diagnostic here, not a gate. Leave the coverage threshold machinery in `build-and-test.yml` alone, and do not let a mutation score fail any automated check. diff --git a/ai-plans/0063-1-plan-deviations.md b/ai-plans/0063-1-plan-deviations.md new file mode 100644 index 0000000..1c183d1 --- /dev/null +++ b/ai-plans/0063-1-plan-deviations.md @@ -0,0 +1,86 @@ +# 0063 Plan Deviations + +This document compares `ai-plans/0063-0-add-stryker-mutation-testing.md` with the results measured while implementing it (`dotnet-stryker` 4.16.0, Apple M3 Max, 16 logical cores, source tree at commit `04aee20`). + +## Summary + +All acceptance criteria were implemented and verified as specified: the pinned tool, the committed configuration, the scoped invocations, the transitive test counts, the git-ignored output containment, the recorded baseline, and the absence of any CI or threshold changes. + +The material deviations are in the plan's Technical Details narrative about survivors and in two settings added to make the measured baseline meaningful. The claim of exactly one survivor across the four small projects did not reproduce: `Validation.OpenApi` has nineteen survivors, and the original 5,000 ms additional-timeout default hid most of them as timeouts. The committed configuration pins concurrency and raises the additional timeout so the operational report exposes that queue directly. The operational documentation in `tests/AGENTS.md` reflects the measured behavior; this file records the delta from the plan text. + +## Deviations From The Original Plan + +### 1. `Validation.OpenApi` has nineteen survivors, not one + +**Original plan:** +The plan states that the measured runs produced exactly one survivor across all four small projects — a statement-removal mutant at `BuiltInValidationErrorBuilderExtensions.cs:426`, putting `Validation.OpenApi` at 98.68% — and that the triage categories therefore "remain largely a priori rather than derived from survivors observed in this codebase." + +**Measured:** +With Stryker's 5,000 ms default additional timeout, three runs of the same unmodified sources produced three different survivor sets: + +- Concurrency 8: 54 killed, 21 timeout, 1 survived — 98.68%. The survivor was an equality mutant at `BuiltInValidationErrorBuilderExtensions.cs:388`, not the statement removal at line 426. +- Concurrency 8, repeat: 51 killed, 21 timeout, 4 survived — 94.74%. +- Concurrency 4: 56 killed, 2 timeout, 18 survived — 76.32%. + +Increasing the additional timeout exposed the stable population much more directly at concurrency 8: + +- 15,000 ms: 57 killed, 1 timeout, 18 survived — 76.32%, 2:30 elapsed. +- 20,000 ms: two runs both reported 57 killed, with 0–1 timeout and 18–19 survived — 75.00–76.32%, 2:34–2:38 elapsed. The mutant that timed out in the second run survived in the first. +- 30,000 ms: two consecutive runs both reported 57 killed, 0 timeout, and 19 survived — 75.00%, 2:32–2:38 elapsed. + +The survivors cluster in `BuiltInValidationErrorBuilderExtensions.cs` lines 367–426 (conditional, equality, object-initializer, and statement mutants around the built-in validation error OpenAPI schema customization), with a few more in `BuiltInValidationErrorContracts.cs`, `BuiltInValidationErrorContractRegistrationExtensions.cs`, and `PortableValidationOpenApiRouteHandlerBuilderExtensions.cs`. + +**Why:** +With `coverage-analysis: off`, every mutant runs the full discovered test set (163 tests for this project). A killed mutant's run aborts at the first failing test, but a genuinely surviving mutant runs the entire suite and can exceed the timeout under load. Stryker 4.16.0 calculates that timeout from the measured initial run plus `additional-timeout`, which defaults to 5,000 ms and is configurable only in the config file. `Timeout` counts as killed in the mutation score, so insufficient headroom reclassifies slow survivors as timeouts and inflates the score. The plan's "one survivor, 98.68%" is one point in this timing-dependent distribution, not a stable property of the suite. + +**Impact:** +The initial triage queue is nineteen mutants, not one, and the triage categories are no longer a priori — they now have real input. The 30,000 ms committed setting reduces the timeout count from 21 to zero on the measured hardware without changing the mutant inventory. No acceptance criterion is affected: criterion 7 deliberately requires recording the timeout count alongside the others, and the baseline in `tests/AGENTS.md` carries the measured vector. Future timeouts remain triage signals because no finite headroom can make classification independent of hardware and load. Future comparisons should match full count vectors at equal concurrency and additional timeout rather than percentages. + +### 2. The recorded baseline's provenance commit predates the tooling commit + +**Original plan:** +Criterion 7 requires the baseline to carry the commit at which it was measured. + +**Measured:** +The baseline was measured at commit `04aee20` while the implementation itself (tool manifest, `stryker-config.json`, `.gitignore`) was still uncommitted. The mutated sources were identical to `04aee20`, so mutation results are unaffected, but once this work is committed the provenance commit recorded in `tests/AGENTS.md` is an ancestor of the tooling commit rather than the commit that introduced the tool. + +**Impact:** +Cosmetic. Re-measuring the baseline at the merge commit is a valid cheap follow-up (about six minutes for all four projects) if exact provenance is ever needed. + +### 3. The committed configuration pins `concurrency: 8`, which the plan did not specify + +**Original plan:** +The plan specifies the exact contents of `stryker-config.json` (`solution`, `test-runner`, `coverage-analysis`, `configuration`, `reporters`) and requires that a scoped run needs only a `-p` argument and an optional `-m` glob. Concurrency appears only as baseline provenance, supplied via `-c 8` at invocation time — so the documented invocation was not the invocation that produced the baseline. + +**Implemented:** +`stryker-config.json` additionally pins `"concurrency": 8`, and the documented invocations pass no `-c`. + +**Why:** +The result vector is concurrency-sensitive (§1: `Validation.OpenApi` reports 2 timeouts/18 survivors at concurrency 4 vs. 21/1 at 8), so a baseline is only comparable at equal concurrency — making concurrency a load-bearing setting, which the plan's own philosophy places in the configuration rather than the invocation. Relying on the default is not an alternative: 4.16.0 defaults to half the logical processors (verified via the debug options dump; the `--help` text claiming "as many parallel processes as you have CPU cores" is stale), which is 8 on the recording machine but 4 on an 8-core machine. The pinned value ensures the same configured parallelism on machines with at least 8 logical cores, but does not make timing independent of hardware or load; it also leaves `-c` free for experiments. Verified to take effect: with `"concurrency": 3` in the config and no `-c`, the debug options dump reports `"Concurrency": 3`. + +### 4. The committed configuration pins `additional-timeout: 30000`, which the plan did not specify + +**Original plan:** +The plan leaves Stryker's additional timeout unset, so 4.16.0 uses its 5,000 ms default. It treats timeout masking as a blind spot to document rather than identifying the setting that controls it. + +**Implemented:** +`stryker-config.json` additionally pins `"additional-timeout": 30000`. The option is milliseconds of headroom added to Stryker's timeout derived from the initial test run; it is config-file-only in 4.16.0 and absent from `--help`. + +**Why:** +The default systematically hid slow survivors under `coverage-analysis: off`. At concurrency 8, 15,000 ms reduced `Validation.OpenApi` from 21 timeouts to one but still timed out a mutant known to survive, and one of two 20,000 ms runs did the same. Two consecutive 30,000 ms runs reported zero timeouts and exposed all 19 survivors. The higher setting also reclassified `AspNetCore.Mvc`'s former timeout at `BaseLightActionResult.cs:82` as a survivor. It does not make the result hardware-independent, but it removes all known masking on the measured hardware without changing concurrency or requiring readers to reinterpret dozens of timeouts. + +**Impact:** +The four small projects were remeasured under the committed setting and the baseline in `tests/AGENTS.md` was replaced. Genuine hangs can now take up to 25 seconds longer per affected test assembly than under the default, while ordinary killed mutants still stop at the first failing test. The measured small-project runs remain within minutes. A future timeout is still a triage signal, not automatically a killed mutant. + +## Notes On Items Implemented As Planned + +The following measurements remained unchanged after the timeout correction: + +- Mutant inventories and test counts for all four small projects: 33/32/44/114 mutants, 11/11/3/2 compile errors, 102/237/337/163 tests (`AspNetCore.Mvc`, `AspNetCore.MinimalApis`, `AspNetCore.Shared`, `Validation.OpenApi`). +- `AspNetCore.Shared` reporting 337 tests (all six transitively referencing test projects) and `Result.cs` reporting 2,398 tests with 67 killed, 0 `NoCoverage`, 100.00%. +- `AspNetCore.MinimalApis` and `AspNetCore.Shared` scoring 100.00%. `AspNetCore.Mvc` now scores 94.12% because its former timeout is correctly reported as survived. +- `StrykerOutput/` containment across repeated runs, no CI changes, and no `thresholds`/`break-at` settings anywhere. + +## Minor Operational Notes + +- `dotnet stryker --version` is not a tool-version query — it is Stryker's `--version ` option for the analyzed project and errors with "Missing value for option 'version'". Use `dotnet tool list` to confirm the pinned tool version. diff --git a/stryker-config.json b/stryker-config.json new file mode 100644 index 0000000..c9702d3 --- /dev/null +++ b/stryker-config.json @@ -0,0 +1,11 @@ +{ + "stryker-config": { + "solution": "Light.PortableResults.slnx", + "test-runner": "mtp", + "coverage-analysis": "off", + "configuration": "Debug", + "concurrency": 8, + "additional-timeout": 30000, + "reporters": ["json", "html", "progress"] + } +} diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 3554979..aa3e418 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -19,3 +19,74 @@ Always pass `--coverage-settings`. It excludes source-generated files under `obj/`, which otherwise dominate the line counts and report the solution at roughly 81% instead of 95%. The path must be absolute, because each test app runs with its own output directory as the working directory. Each test project writes `TestResults/.cobertura.xml`. Merge them with `reportgenerator -reports:'**/*.cobertura.xml' -targetdir:./coverage-merged -reporttypes:'Cobertura;TextSummary'`. + +## Mutation testing (Stryker.NET) + +`dotnet tool restore` installs the pinned `dotnet-stryker` 4.16.0; the shared config is `stryker-config.json` at the repository root. Local, on-demand use only: no CI integration and no score gates (`thresholds`/`break-at` are unset). A surviving mutant is a signal to investigate, not proof of a defect. + +### How to run + +Always run from the repository root: Stryker reads `stryker-config.json` only from the current directory and silently falls back to defaults without it, reverting to the `vstest` runner (cannot drive the xUnit v3 hosts, reports everything survived) and `perTest` coverage analysis (fabricates `NoCoverage`). The config also pins `Debug` (`TreatWarningsAsErrors` is Release-only), `concurrency: 8` (result vectors vary with parallelism; override with `-c` for experiments), and `additional-timeout: 30000` (the 5,000 ms default masks slow survivors as timeouts under load). + +`additional-timeout` is config-file-only in 4.16.0 and is milliseconds of headroom added to Stryker's timeout derived from the initial test run, not the total timeout. Raising it makes genuine hangs take longer to classify, but the measured value exposes all known slow survivors while keeping the four small-project runs within minutes. + +Revisit `coverage-analysis: off` when Stryker's MTP per-test coverage support ([#3516](https://github.com/stryker-mutator/stryker-net/pull/3516)) ships and proves trustworthy; it is the main lever on run time because it avoids running the full discovered test set for every mutant. + +```shell +# One project — sufficient for the four small projects +dotnet stryker -p Light.PortableResults.AspNetCore.Shared.csproj + +# One-file configuration smoke check (~4 minutes) +# Expect: 2,398 tests, 67 killed, 0 NoCoverage, 100.00% +dotnet stryker -p Light.PortableResults.csproj -m '**/Result.cs' +``` + +The smoke-check vector is tied to the current `Result.cs` and its tests; update it after intentional changes alter the mutant inventory. All mutants surviving with a 0.00% score suggests fallback to the `vstest` runner, while any non-zero `NoCoverage` count suggests `coverage-analysis: off` was lost. + +`-p` selects the mutated project, not the tests: Stryker runs every test project transitively referencing it (`AspNetCore.Shared` → 337 tests, `Light.PortableResults` → all 2,398). Cross-project kills are legitimate (sociable tests) and nearly free. Never pass `-tp` (it does not narrow tests) or `--since` (any non-C# file in the diff, e.g. an `ai-plans/` document, degrades it to a full run). Reports go to `StrykerOutput//reports/` (gitignored): JSON for agents (filter `"status"` for both `"Survived"` and `"Timeout"`; investigate the timeout cause before survivor triage), HTML for humans. + +### Cost and baseline + +~3 s per mutant at concurrency 8 (upper bound from the large projects). Omitting `-p` mutates the whole solution (~7 h; `Light.PortableResults` alone ~3.6 h) — split large projects by mutate glob along folder boundaries (`Metadata/`, `Http/`, `CloudEvents/`, `Numbers/`). Mutant inventory for sizing: + +| Project | Mutants | CompileError | +| --- | ---: | ---: | +| `Light.PortableResults` | 4,867 | 520 | +| `Light.PortableResults.Validation` | 2,215 | 83 | +| `Validation.OpenApi.SourceGeneration` | 1,272 | 204 | +| `AspNetCore.OpenApi` | 848 | 65 | +| `Validation.OpenApi` | 114 | 2 | +| `AspNetCore.Shared` | 44 | 3 | +| `AspNetCore.Mvc` | 33 | 11 | +| `AspNetCore.MinimalApis` | 32 | 11 | + +Baseline for sources at commit `04aee20`, remeasured with `dotnet-stryker` 4.16.0, concurrency 8 and 30,000 ms additional timeout (both pinned in `stryker-config.json`), `Debug`, Apple M3 Max (16 logical cores): + +| Project | Tests run | Elapsed | Killed | Timeout | Survived | CompileError | Ignored | NoCoverage | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| `AspNetCore.Mvc` | 102 | 0:53 | 16 | 0 | 1 | 11 | 5 | 0 | +| `AspNetCore.MinimalApis` | 237 | 1:22 | 16 | 0 | 0 | 11 | 5 | 0 | +| `AspNetCore.Shared` | 337 | 2:29 | 31 | 0 | 0 | 3 | 10 | 0 | +| `Validation.OpenApi` | 163 | 2:38 | 57 | 0 | 19 | 2 | 36 | 0 | + +`Validation.OpenApi` was run twice consecutively with the 30,000 ms setting; both runs produced 57 killed, 0 timeout, 19 survived, and a 75.00% score, completing in 2:32 and 2:38. At 20,000 ms, one of two runs still timed out a mutant known to survive. At the 5,000 ms default, 21 mutants were reported as timeouts, masking most survivors and inflating the score to 98.68%. + +`Ignored` is not user suppression: all 56 baseline entries are `Block removal` mutants discarded deterministically by Stryker's built-in "block already covered" filter because another active mutant exists inside the block. A different reason or count should be investigated. + +`NoCoverage` must be zero — a non-zero value means `coverage-analysis: off` is no longer taking effect. Compare full count vectors at equal concurrency, not percentages. + +### Triaging survivors + +Exactly three permitted responses to a survivor: + +1. **Behavior is genuinely unconstrained** — add or strengthen a test (the common case). +2. **Equivalent or invalid mutant** — suppress narrowly at the source with a justification: `// Stryker disable once Statement : equivalent - `. Prefer `disable once` over `disable all`; never use the global `ignore-mutations` setting for a single site. +3. **Untestable construct** — record it and move on (blind spots below). + +Never restructure production code to make a mutant killable: performance outranks mutation score here, and the low-allocation `in`/`ref`/`Span` style produces awkward survivors by nature. + +### Blind spots — do not read as adequate coverage + +- ~9.5% of mutants fail to compile (mostly the `out`/`ref` style); Stryker's Safe Mode then discards every mutant in the enclosing method: `Dragon4.GenerateDigits`, `ResultJsonReader.ReadStatusValue`/`ReadIndexValue`, `ErrorsExtensions.WriteRichErrors`, and all of `LightResult.cs` receive no mutation coverage. Tool limitation, not a test defect — a high score in `Numbers/` is not verified behavior. +- `Timeout` counts as killed. The pinned 30,000 ms additional timeout reduced `Validation.OpenApi` from 21 timeouts to zero in two consecutive concurrency-8 runs, but no finite value makes classification independent of hardware and load. Investigate any future timeout as either a genuine hang or insufficient headroom; do not assume it represents a killed mutant. +- The MTP runner is a preview (stryker-mutator/stryker-net#3094); verify surprising results against a plain `dotnet test` run.