From dd9a2664bc57113af9af5c2df1c5ec5ef0d44beb Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 18:17:19 +0200 Subject: [PATCH 01/13] docs: initial plan 63 for Stryker.NET mutation testing Introduces Stryker.NET as a pinned, local-only tool so coding agents can identify missing tests beyond what line coverage proves. The plan records two settings that must be committed rather than passed ad hoc, because each produces a confidently wrong report on this solution: the default vstest runner cannot drive the xUnit v3 hosts and reports every mutant as survived, and the MTP runner's default coverage analysis fabricates NoCoverage results for code that is demonstrably tested. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01PwpEBWeQhetkTu6CdQkuqm --- ai-plans/0063-add-stryker-mutation-testing.md | 94 +++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 ai-plans/0063-add-stryker-mutation-testing.md diff --git a/ai-plans/0063-add-stryker-mutation-testing.md b/ai-plans/0063-add-stryker-mutation-testing.md new file mode 100644 index 0000000..18ce9de --- /dev/null +++ b/ai-plans/0063-add-stryker-mutation-testing.md @@ -0,0 +1,94 @@ +# Stryker.NET Mutation Testing + +## Rationale + +Line coverage is held above 95%, but it only proves that code executed, not that any assertion constrains it. Coding agents extending this library cannot currently distinguish a well-tested code path from one that is merely reached. Mutation testing closes that gap: each surviving mutant is a concrete, machine-readable statement that some behavior can be changed without a single test noticing. + +Introduce Stryker.NET as a pinned, opt-in tool with a repository configuration that is correct by default. Two settings are load-bearing — the wrong test runner and the default coverage analysis each produce confidently wrong reports on this solution — so the configuration, not the invocation, must carry them. + +## Acceptance Criteria + +- [ ] `dotnet tool restore` installs a version-pinned `dotnet-stryker`, and a repository-root `stryker-config.json` supplies the shared configuration so that a scoped run needs only project and test-project arguments. +- [ ] The committed configuration selects the Microsoft Testing Platform runner and disables coverage analysis; a run against `Light.PortableResults.AspNetCore.Shared` reports a non-zero mutation score, and a run against `Result.cs` reports zero `NoCoverage` mutants. +- [ ] `StrykerOutput/` is ignored by git, and no mutation run leaves artifacts inside the working tree. +- [ ] `tests/AGENTS.md` documents the file-scoped and project-scoped invocations, states that surviving mutants are the signal to act on, records the measured cost per project, and records the mutation blind spots that must not be read as adequate coverage. +- [ ] Mutation testing runs on demand from a developer machine only. No CI workflow is added and no existing workflow is modified; pull request validation time is unchanged. +- [ ] A baseline mutation score is recorded in `tests/AGENTS.md` for the four projects that complete in minutes (`AspNetCore.Shared`, `AspNetCore.Mvc`, `AspNetCore.MinimalApis`, `Validation.OpenApi`). +- [ ] Mutation score thresholds and `break-at` remain unset; nothing in the repository fails a build or a test run because of a mutation score. + +## Technical Details + +### Verified constraints + +The findings below come from `dotnet-stryker` 4.16.0 executed against this solution. They are the reason the configuration is not left at its defaults. + +Microsoft Testing Platform support is still a preview feature in Stryker.NET, and it announces itself as one on every run. It is tracked upstream in [stryker-mutator/stryker-net#3094](https://github.com/stryker-mutator/stryker-net/issues/3094), which is the issue to watch for the maturity of everything in this section. + +**The default `vstest` runner is unusable here.** These test projects set `UseMicrosoftTestingPlatformRunner`, and VSTest cannot drive their xUnit v3 hosts ([#3117](https://github.com/stryker-mutator/stryker-net/issues/3117)). Coverage capture fails and every mutant is reported as survived — `AspNetCore.Shared` scores 0.00% under `vstest` and 100.00% under `mtp`. The failure is a logged error plus a plausible-looking report, not a crash, so `"test-runner": "mtp"` must be committed rather than passed ad hoc. + +**MTP coverage analysis fabricates `NoCoverage`.** With the default `perTest` analysis, `Result.cs` reports 6 killed and 61 `NoCoverage` for a score of 8.96%, despite `NonGenericResultTests` covering those members directly. With `"coverage-analysis": "off"` the same file reports 67 tested, 0 uncovered, and 100.00%. A false `NoCoverage` is the most damaging possible output for the intended agent workflow, because it directs effort at tests that already exist. Per-test coverage for the MTP runner is in progress as [#3516](https://github.com/stryker-mutator/stryker-net/pull/3516); revisit this setting when it ships, since it is the main lever on run time. + +The configuration is therefore: + +```json +{ + "stryker-config": { + "solution": "Light.PortableResults.slnx", + "test-runner": "mtp", + "coverage-analysis": "off", + "reporters": ["json", "html", "progress"] + } +} +``` + +Everything else about the repository is compatible without special handling: `.slnx` analysis, central package management with lock files, the `netstandard2.0;net10.0` multi-targeting (Stryker selects `net10.0` unaided), the netstandard2.0 source generator, `InterceptorsNamespaces`, and the Verify-based snapshot tests, which leave no `.received.*` files behind because 4.14.2 disables DiffEngine under MTP. No strong-naming or `InternalsVisibleTo` seam exists to work around. + +Run mutation testing in the `Debug` configuration. `TreatWarningsAsErrors` is Release-only, and Stryker's rollback should not have to contend with warnings promoted to errors. + +### Cost + +Measured throughput is approximately three seconds of wall clock per mutant at concurrency 8, dominated by fixed per-mutant overhead rather than test count: the 337-test and 2,398-test suites cost the same per mutant. Mutant inventory: + +| Project | Mutants | CompileError | +| --- | ---: | ---: | +| `Light.PortableResults` | 4,867 | 520 | +| `Light.PortableResults.Validation` | 2,215 | 83 | +| `Validation.OpenApi.SourceGeneration` | 1,272 | 204 | +| `AspNetCore.OpenApi` | 848 | 65 | +| `Validation.OpenApi` | 114 | 2 | +| `AspNetCore.Shared` | 44 | 3 | +| `AspNetCore.Mvc` | 33 | 11 | +| `AspNetCore.MinimalApis` | 32 | 11 | + +A solution-wide run is roughly seven hours on the measured hardware, which is why this stays a local, on-demand tool rather than anything automated. Mutation testing is not added to CI in any form: no new workflow, and no change to `build-and-test.yml`. Whoever drives it locally chooses the scope, and the practical scopes are one file or one project — never the whole solution in the inner loop. + +The two large projects are long-running even in isolation: `Light.PortableResults` is roughly 3.6 hours and `Validation` roughly 1.8 hours. When one of them is worth running end to end, split it by `mutate` glob along folder boundaries (`Metadata/`, `Http/`, `CloudEvents/`, `Numbers/`) so the run is interruptible and each report arrives while it is still actionable. + +The JSON report is the agent-facing artifact; the HTML report is for humans. Agents filter for `"status": "Survived"`, and each survivor identifies a missing assertion at a specific file and line. + +### Scoped invocation + +The workflow that actually serves the goal is per-file, not per-solution. Mutating one source file against its test project takes about four minutes and is the form to document for agents: + +```shell +dotnet stryker -p .csproj \ + -tp tests/.Tests/.Tests.csproj \ + -m '**/TheFile.cs' +``` + +`-p` resolves against the test project's references, so both arguments are required. Whole small projects (`AspNetCore.Shared`, `Mvc`, `MinimalApis`, `Validation.OpenApi`) complete in one to two minutes and need no `-m`. + +`--since:` restricts mutation to files changed against a baseline and is the natural scope for "mutation-test what I just wrote" — for example `--since:main` while working on a feature branch. It reads git history directly, so it needs no special setup locally. Document it alongside the two forms above; it is the most convenient entry point for an agent that has just finished editing. + +### Blind spots to document + +Roughly 9.5% of mutants fail to compile, concentrated in the low-allocation `out`/`ref` style. Stryker responds with Safe Mode, which discards every mutant in the enclosing method: + +``` +CS0165: Use of unassigned local variable 'low' (Numbers/Dragon4.cs:263) +[INF] Safe Mode! Stryker will remove all mutations in GenerateDigits +``` + +`Dragon4.GenerateDigits`, `ResultJsonReader.ReadStatusValue`, `ResultJsonReader.ReadIndexValue`, `ErrorsExtensions.WriteRichErrors`, and the whole of `LightResult.cs` (11 of 11 mutants) receive no mutation coverage at all. This is a tool limitation, not a test defect. `tests/AGENTS.md` must state it explicitly so that a high mutation score in `Numbers/` is not mistaken for verified behavior, and so that no one attempts to "fix" it by restructuring production code. + +Mutation score is a diagnostic here, not a gate. Leave the coverage threshold machinery in `build-and-test.yml` alone, and do not let a mutation score fail any automated check. From 7b755cca05ca0b4d21f6bf3a6c4104774ce74406 Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 18:44:17 +0200 Subject: [PATCH 02/13] docs: refine plan 63 after review Corrects four claims that did not survive verification against Stryker 4.16.0: - Solution mode: -p selects the mutated source project but does not narrow test execution, so -tp is removed from the documented command. Mutating AspNetCore.Shared runs 337 tests, the exact sum of the six test projects referencing it, not the 26 in its own test project. This is kept rather than worked around, because cross-project kills are what the sociable testing rule in tests/AGENTS.md asks for. - Artifact containment: the criterion demanded both an ignored StrykerOutput/ and an empty working tree. Reports have to be written somewhere, so the contract is now containment plus a clean git status. - --since is dropped. Measured on a clean working tree, it still reported a Markdown plan as a changed test file and escalated to testing every mutant, so it would degrade to a full run on every branch while appearing scoped. - Survivors are a triage queue, not a defect list. Equivalent and invalid mutants get narrow suppression with a justification; production code is never restructured to raise the score. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01PwpEBWeQhetkTu6CdQkuqm --- ai-plans/0063-add-stryker-mutation-testing.md | 51 +++++++++++++++---- 1 file changed, 40 insertions(+), 11 deletions(-) diff --git a/ai-plans/0063-add-stryker-mutation-testing.md b/ai-plans/0063-add-stryker-mutation-testing.md index 18ce9de..da298cf 100644 --- a/ai-plans/0063-add-stryker-mutation-testing.md +++ b/ai-plans/0063-add-stryker-mutation-testing.md @@ -2,7 +2,7 @@ ## Rationale -Line coverage is held above 95%, but it only proves that code executed, not that any assertion constrains it. Coding agents extending this library cannot currently distinguish a well-tested code path from one that is merely reached. Mutation testing closes that gap: each surviving mutant is a concrete, machine-readable statement that some behavior can be changed without a single test noticing. +Line coverage is held above 95%, but it only proves that code executed, not that any assertion constrains it. Coding agents extending this library cannot currently distinguish a well-tested code path from one that is merely reached. Mutation testing narrows that gap: a surviving mutant is a concrete, machine-readable location where the suite failed to detect a changed program, which makes it a candidate for a missing assertion rather than proof of one. Some survivors are equivalent or not worth killing, so the output is a triage queue, not a defect list. Introduce Stryker.NET as a pinned, opt-in tool with a repository configuration that is correct by default. Two settings are load-bearing — the wrong test runner and the default coverage analysis each produce confidently wrong reports on this solution — so the configuration, not the invocation, must carry them. @@ -10,10 +10,11 @@ Introduce Stryker.NET as a pinned, opt-in tool with a repository configuration t - [ ] `dotnet tool restore` installs a version-pinned `dotnet-stryker`, and a repository-root `stryker-config.json` supplies the shared configuration so that a scoped run needs only project and test-project arguments. - [ ] The committed configuration selects the Microsoft Testing Platform runner and disables coverage analysis; a run against `Light.PortableResults.AspNetCore.Shared` reports a non-zero mutation score, and a run against `Result.cs` reports zero `NoCoverage` mutants. -- [ ] `StrykerOutput/` is ignored by git, and no mutation run leaves artifacts inside the working tree. -- [ ] `tests/AGENTS.md` documents the file-scoped and project-scoped invocations, states that surviving mutants are the signal to act on, records the measured cost per project, and records the mutation blind spots that must not be read as adequate coverage. +- [ ] Mutation output is contained under `StrykerOutput/`, which is ignored by git; a completed or interrupted run leaves `git status` clean. +- [ ] `tests/AGENTS.md` documents the file-scoped and project-scoped invocations, states that a surviving mutant is a signal to investigate together with the three permitted triage outcomes, records the measured cost per project, and records the mutation blind spots that must not be read as adequate coverage. - [ ] Mutation testing runs on demand from a developer machine only. No CI workflow is added and no existing workflow is modified; pull request validation time is unchanged. -- [ ] A baseline mutation score is recorded in `tests/AGENTS.md` for the four projects that complete in minutes (`AspNetCore.Shared`, `AspNetCore.Mvc`, `AspNetCore.MinimalApis`, `Validation.OpenApi`). +- [ ] The documented invocation is verified to run the intended tests: for a run mutating `AspNetCore.Shared`, Stryker's reported test count is 337 — every test project transitively referencing it — and not the 26 tests of `AspNetCore.Shared.Tests` alone. +- [ ] A baseline mutation score is recorded in `tests/AGENTS.md` for the four projects that complete in minutes (`AspNetCore.Shared`, `AspNetCore.Mvc`, `AspNetCore.MinimalApis`, `Validation.OpenApi`), together with the test count each run executed. - [ ] Mutation score thresholds and `break-at` remain unset; nothing in the repository fails a build or a test run because of a mutation score. ## Technical Details @@ -45,6 +46,8 @@ Everything else about the repository is compatible without special handling: `.s Run mutation testing in the `Debug` configuration. `TreatWarningsAsErrors` is Release-only, and Stryker's rollback should not have to contend with warnings promoted to errors. +Reports are written to `StrykerOutput//` beneath the working directory; the JSON and HTML reporters exist to produce files, so the goal is containment, not absence. Stryker already writes a `.gitignore` containing `*` into each timestamped directory, and a completed run leaves `git status` clean without any repository change. Add `StrykerOutput/` to `.gitignore` anyway: it states the intent, covers the directory itself, and protects against a run interrupted before that inner file is written. The output location is a CLI concern — `--output` has no `stryker-config.json` equivalent — so nothing here depends on redirecting it, and no wrapper script is needed. + ### Cost Measured throughput is approximately three seconds of wall clock per mutant at concurrency 8, dominated by fixed per-mutant overhead rather than test count: the 337-test and 2,398-test suites cost the same per mutant. Mutant inventory: @@ -64,21 +67,47 @@ A solution-wide run is roughly seven hours on the measured hardware, which is wh The two large projects are long-running even in isolation: `Light.PortableResults` is roughly 3.6 hours and `Validation` roughly 1.8 hours. When one of them is worth running end to end, split it by `mutate` glob along folder boundaries (`Metadata/`, `Http/`, `CloudEvents/`, `Numbers/`) so the run is interruptible and each report arrives while it is still actionable. -The JSON report is the agent-facing artifact; the HTML report is for humans. Agents filter for `"status": "Survived"`, and each survivor identifies a missing assertion at a specific file and line. +The JSON report is the agent-facing artifact; the HTML report is for humans. Agents filter for `"status": "Survived"` to obtain a work queue located at specific files and lines. A survivor is a signal to investigate, not a defect to fix on sight; see triage below. ### Scoped invocation -The workflow that actually serves the goal is per-file, not per-solution. Mutating one source file against its test project takes about four minutes and is the form to document for agents: +Because the configuration names the solution, every run is a solution-context run. `-p` selects which source project is mutated; it does not narrow which tests execute. Stryker discovers the test projects from the solution and runs **every test project that transitively references the mutated project**. A `-tp` argument does not override this and must not be documented as if it did. + +This was confirmed by test counts rather than inferred: mutating `AspNetCore.Shared` reports 337 tests, which is the exact sum of the six test projects referencing it (26 + 46 + 23 + 79 + 112 + 51), not the 26 in `AspNetCore.Shared.Tests`. Mutating `Light.PortableResults` reports 2,398 — the whole solution, since everything references it. + +Keep this behavior rather than forcing a single test project. It matches the sociable-testing rule in `tests/AGENTS.md`: a mutant in `AspNetCore.Shared` killed by `MinimalApis.Tests` is legitimately killed, and isolating test projects would convert those cross-project kills into false survivors. It is also close to free, because per-mutant cost is dominated by fixed overhead — the 337-test and 2,398-test sets cost the same per mutant. + +The form to document for agents is therefore source project plus optional file glob. Mutating one file takes about four minutes: ```shell -dotnet stryker -p .csproj \ - -tp tests/.Tests/.Tests.csproj \ - -m '**/TheFile.cs' +dotnet stryker -p .csproj -m '**/TheFile.cs' ``` -`-p` resolves against the test project's references, so both arguments are required. Whole small projects (`AspNetCore.Shared`, `Mvc`, `MinimalApis`, `Validation.OpenApi`) complete in one to two minutes and need no `-m`. +Whole small projects (`AspNetCore.Shared`, `Mvc`, `MinimalApis`, `Validation.OpenApi`) complete in one to two minutes and need no `-m`. Omitting `-p` mutates every source project in the solution and is the seven-hour path. + +Do not use or document `--since` under this configuration. It was measured on this branch and never narrowed anything: with a completely clean working tree it still reported `ai-plans/0063-add-stryker-mutation-testing.md` as a changed test file and escalated to `16 mutants will be tested because: Non-CSharp files in test project were changed`. + +Two behaviors combine badly here. Stryker treats any non-C# file in the diff — including a Markdown plan — as a changed test file and responds by testing every mutant; and `--since:HEAD` did not pin the baseline to `HEAD`, so the diff kept resolving against the default branch. Because every feature branch in this repository begins by adding a plan under `ai-plans/`, a non-C# file is essentially always in the diff. `--since` therefore degrades to a full project run while presenting itself as scoped, which is worse than not using it: the cost is unchanged and the reported scope is wrong. + +Explicit file and project scope is the only recommended form. Revisit `--since` only once MTP coverage analysis is trustworthy — a separate concern is that mapping changed *test* files back to mutants depends on per-mutant covering tests, which `coverage-analysis: off` does not produce, so a test-only edit has no reliable path to the mutants it affects. + +### Triaging survivors + +A surviving mutant means the suite did not distinguish the mutated program from the original. That has three admissible causes, and exactly three permitted responses: + +1. **Observable behavior is genuinely unconstrained.** Add or strengthen a test. This is the outcome the tool exists to produce and should be the common one. +2. **The mutant is equivalent or invalid** — semantically identical to the original, or killable only by asserting on something that is not part of the contract. Suppress it narrowly at the source with a justification, which Stryker records in the report: + + ```csharp + // Stryker disable once Statement : equivalent - the guard is a fast path, not a behavior change + ``` + + The syntax is `Stryker [disable|restore][once][all|][: reason]`, and it is scope-aware. Prefer `disable once` over `disable all`, and never reach for the global `ignore-mutations` setting to silence a single site. +3. **The mutant sits in a construct this configuration cannot meaningfully test.** Record it and move on; see the blind spots below. + +Never restructure production code solely to make a mutant killable. The root `AGENTS.md` ranks performance above extensibility, and this library's low-allocation `in`/`ref`/`Span` style is precisely the shape that produces awkward survivors. A lower mutation score is the correct outcome when the alternative is a slower or less direct implementation. -`--since:` restricts mutation to files changed against a baseline and is the natural scope for "mutation-test what I just wrote" — for example `--since:main` while working on a feature branch. It reads git history directly, so it needs no special setup locally. Document it alongside the two forms above; it is the most convenient entry point for an agent that has just finished editing. +No run performed while preparing this plan produced a single survivor — every scored run returned 100.00%. The first real baseline is therefore also the first exercise of this guidance, and the categories above are stated a priori rather than derived from survivors observed in this codebase. ### Blind spots to document From 04aee20f17631e0d368222cbb9c3f139f99ce846 Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 18:54:28 +0200 Subject: [PATCH 03/13] docs: refine plan 63 with measured baselines Replaces estimated costs with end-to-end measurements of the four small projects and tightens several points that were left implicit: - The tool manifest must pin dotnet-stryker to exactly 4.16.0, and the scoped invocation takes only -p plus an optional -m glob. The criterion still described a -tp argument removed in the previous commit. - configuration is pinned to Debug in stryker-config.json rather than left to the invoker, since Release would build with warnings as errors. - Every command runs from the repository root. Stryker resolves stryker-config.json against the working directory, does not search parent directories, and treats a missing config as non-fatal, so a run started elsewhere silently reverts to the vstest runner and perTest coverage - the pair that reports 0.00% or fabricates NoCoverage. - Baselines must carry provenance - commit, tool version, concurrency, executed test count, elapsed time, and the killed, timeout, survived, compile-error, ignored, and no-coverage counts - rather than a percentage alone. No-coverage is expected to be zero, and a non-zero value means coverage-analysis stopped taking effect; it was zero in all four runs. - The artifact criterion now requires that a run add no new entries to git status, which stays verifiable while implementation changes are present. The timing claim is reconciled against measurement rather than arithmetic. Validation.OpenApi completes in 1:59, not the six minutes that 114 mutants at three seconds each would suggest: only 76 of its 114 mutants are tested, and per-mutant cost falls with the size of the associated test set. The three-seconds-per-mutant rate is now labelled an upper bound taken from the large projects. These runs also produced the first survivor, a statement-removal mutant at BuiltInValidationErrorBuilderExtensions.cs:426 that puts Validation.OpenApi at 98.68%. It is recorded as untriaged, and it corrects the claim in the previous commit that no run had produced a survivor. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01PwpEBWeQhetkTu6CdQkuqm --- ai-plans/0063-add-stryker-mutation-testing.md | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/ai-plans/0063-add-stryker-mutation-testing.md b/ai-plans/0063-add-stryker-mutation-testing.md index da298cf..688984e 100644 --- a/ai-plans/0063-add-stryker-mutation-testing.md +++ b/ai-plans/0063-add-stryker-mutation-testing.md @@ -8,13 +8,13 @@ Introduce Stryker.NET as a pinned, opt-in tool with a repository configuration t ## Acceptance Criteria -- [ ] `dotnet tool restore` installs a version-pinned `dotnet-stryker`, and a repository-root `stryker-config.json` supplies the shared configuration so that a scoped run needs only project and test-project arguments. +- [ ] `dotnet tool restore` installs `dotnet-stryker` pinned to exactly 4.16.0, and a repository-root `stryker-config.json` supplies the shared configuration so that a scoped run needs only a `-p` argument and an optional `-m` glob. - [ ] The committed configuration selects the Microsoft Testing Platform runner and disables coverage analysis; a run against `Light.PortableResults.AspNetCore.Shared` reports a non-zero mutation score, and a run against `Result.cs` reports zero `NoCoverage` mutants. -- [ ] Mutation output is contained under `StrykerOutput/`, which is ignored by git; a completed or interrupted run leaves `git status` clean. +- [ ] Mutation output is contained under `StrykerOutput/`, which is ignored by git; a completed or interrupted run adds no new entries to `git status`. - [ ] `tests/AGENTS.md` documents the file-scoped and project-scoped invocations, states that a surviving mutant is a signal to investigate together with the three permitted triage outcomes, records the measured cost per project, and records the mutation blind spots that must not be read as adequate coverage. - [ ] Mutation testing runs on demand from a developer machine only. No CI workflow is added and no existing workflow is modified; pull request validation time is unchanged. - [ ] The documented invocation is verified to run the intended tests: for a run mutating `AspNetCore.Shared`, Stryker's reported test count is 337 — every test project transitively referencing it — and not the 26 tests of `AspNetCore.Shared.Tests` alone. -- [ ] A baseline mutation score is recorded in `tests/AGENTS.md` for the four projects that complete in minutes (`AspNetCore.Shared`, `AspNetCore.Mvc`, `AspNetCore.MinimalApis`, `Validation.OpenApi`), together with the test count each run executed. +- [ ] A baseline is recorded in `tests/AGENTS.md` for the four projects that complete in minutes (`AspNetCore.Shared`, `AspNetCore.Mvc`, `AspNetCore.MinimalApis`, `Validation.OpenApi`). Each entry carries enough provenance to be reproduced and compared: commit, `dotnet-stryker` version, concurrency, the test count actually executed, elapsed time, and the killed, timeout, survived, compile-error, ignored, and no-coverage counts — not a percentage alone. The no-coverage count is recorded even though it is expected to be zero, because a non-zero value means `coverage-analysis` is no longer taking effect; it was zero in all four measured runs. - [ ] Mutation score thresholds and `break-at` remain unset; nothing in the repository fails a build or a test run because of a mutation score. ## Technical Details @@ -37,6 +37,7 @@ The configuration is therefore: "solution": "Light.PortableResults.slnx", "test-runner": "mtp", "coverage-analysis": "off", + "configuration": "Debug", "reporters": ["json", "html", "progress"] } } @@ -44,7 +45,7 @@ The configuration is therefore: Everything else about the repository is compatible without special handling: `.slnx` analysis, central package management with lock files, the `netstandard2.0;net10.0` multi-targeting (Stryker selects `net10.0` unaided), the netstandard2.0 source generator, `InterceptorsNamespaces`, and the Verify-based snapshot tests, which leave no `.received.*` files behind because 4.14.2 disables DiffEngine under MTP. No strong-naming or `InternalsVisibleTo` seam exists to work around. -Run mutation testing in the `Debug` configuration. `TreatWarningsAsErrors` is Release-only, and Stryker's rollback should not have to contend with warnings promoted to errors. +`configuration` is pinned to `Debug` in the config rather than left to the invoker. `TreatWarningsAsErrors` is Release-only, and Stryker's rollback should not have to contend with warnings promoted to errors. All measurements in this plan were taken under `Debug`. Reports are written to `StrykerOutput//` beneath the working directory; the JSON and HTML reporters exist to produce files, so the goal is containment, not absence. Stryker already writes a `.gitignore` containing `*` into each timestamped directory, and a completed run leaves `git status` clean without any repository change. Add `StrykerOutput/` to `.gitignore` anyway: it states the intent, covers the directory itself, and protects against a run interrupted before that inner file is written. The output location is a CLI concern — `--output` has no `stryker-config.json` equivalent — so nothing here depends on redirecting it, and no wrapper script is needed. @@ -83,7 +84,20 @@ The form to document for agents is therefore source project plus optional file g dotnet stryker -p .csproj -m '**/TheFile.cs' ``` -Whole small projects (`AspNetCore.Shared`, `Mvc`, `MinimalApis`, `Validation.OpenApi`) complete in one to two minutes and need no `-m`. Omitting `-p` mutates every source project in the solution and is the seven-hour path. +Every command runs from the repository root. Stryker resolves `stryker-config.json` relative to the current working directory and does not search parent directories, and a missing config file is not an error — it silently falls back to the defaults. From a subdirectory the run therefore loses both load-bearing settings at once, reverting to the `vstest` runner and `perTest` coverage analysis, which is exactly the combination that reports 0.00% or fabricates `NoCoverage`. The failure is a plausible report rather than a diagnostic, so `tests/AGENTS.md` must state the working directory as part of the invocation rather than assume it. + +The four small projects need no `-m` at all. Measured end to end, including analysis, build, and initial test run: + +| Project | Tests run | Created | Tested | Elapsed | +| --- | ---: | ---: | ---: | ---: | +| `AspNetCore.Mvc` | 102 | 33 | 17 | 0:38 | +| `AspNetCore.MinimalApis` | 237 | 32 | 16 | 0:57 | +| `AspNetCore.Shared` | 337 | 44 | 31 | 1:57 | +| `Validation.OpenApi` | 163 | 114 | 76 | 1:59 | + +Omitting `-p` mutates every source project in the solution and is the seven-hour path. + +Do not derive these times from the three-seconds-per-mutant figure. That rate is an upper bound taken from the large projects, where the mutated assembly is referenced by the whole solution and every test project runs. Two effects make small projects cheaper: only a fraction of created mutants is ever tested — 76 of 114 for `Validation.OpenApi` — and per-mutant cost falls with the size of the associated test set. Use the rate to size the projects in the inventory table above, and these measurements for the small ones. Do not use or document `--since` under this configuration. It was measured on this branch and never narrowed anything: with a completely clean working tree it still reported `ai-plans/0063-add-stryker-mutation-testing.md` as a changed test file and escalated to `16 mutants will be tested because: Non-CSharp files in test project were changed`. @@ -107,7 +121,7 @@ A surviving mutant means the suite did not distinguish the mutated program from Never restructure production code solely to make a mutant killable. The root `AGENTS.md` ranks performance above extensibility, and this library's low-allocation `in`/`ref`/`Span` style is precisely the shape that produces awkward survivors. A lower mutation score is the correct outcome when the alternative is a slower or less direct implementation. -No run performed while preparing this plan produced a single survivor — every scored run returned 100.00%. The first real baseline is therefore also the first exercise of this guidance, and the categories above are stated a priori rather than derived from survivors observed in this codebase. +The measured runs produced exactly one survivor across all four small projects: a statement-removal mutant at `BuiltInValidationErrorBuilderExtensions.cs:426`, which puts `Validation.OpenApi` at 98.68%. It is untriaged — it is the first item to run through the three categories above, and it is the only evidence so far that the queue will contain anything at all. Every other scored run returned 100.00%, so these categories remain largely a priori rather than derived from survivors observed in this codebase. ### Blind spots to document From 618cf2f38afe758cabb2049179ee923627ad4ec9 Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 19:48:42 +0200 Subject: [PATCH 04/13] chore: add Stryker.NET as pinned local mutation testing tool Pin dotnet-stryker 4.16.0 via the local tool manifest and commit a repository-root stryker-config.json that selects the MTP test runner (vstest cannot drive the xUnit v3 hosts) and disables coverage analysis (MTP perTest analysis fabricates NoCoverage). Ignore StrykerOutput/ so runs leave git status clean. tests/AGENTS.md documents the repo-root scoped invocations, survivor triage rules, per-project costs, the measured four-project baseline, and the blind spots (Safe Mode method discards, timeout masking). Plan deviations record that Validation.OpenApi actually yields ~18-20 survivors masked as timeouts rather than the single survivor the plan anticipated. --- .config/dotnet-tools.json | 13 ++++ .gitignore | 1 + ...=> 0063-0-add-stryker-mutation-testing.md} | 16 ++--- ai-plans/0063-1-plan-deviations.md | 56 +++++++++++++++ stryker-config.json | 9 +++ tests/AGENTS.md | 72 +++++++++++++++++++ 6 files changed, 159 insertions(+), 8 deletions(-) create mode 100644 .config/dotnet-tools.json rename ai-plans/{0063-add-stryker-mutation-testing.md => 0063-0-add-stryker-mutation-testing.md} (96%) create mode 100644 ai-plans/0063-1-plan-deviations.md create mode 100644 stryker-config.json diff --git a/.config/dotnet-tools.json b/.config/dotnet-tools.json new file mode 100644 index 0000000..010a8c1 --- /dev/null +++ b/.config/dotnet-tools.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "isRoot": true, + "tools": { + "dotnet-stryker": { + "version": "4.16.0", + "commands": [ + "dotnet-stryker" + ], + "rollForward": false + } + } +} \ No newline at end of file diff --git a/.gitignore b/.gitignore index 24628a3..ebf96ff 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,7 @@ obj/ TestResults/ coverage-merged/ *.received.* +StrykerOutput/ .DS_Store *.lscache .codex diff --git a/ai-plans/0063-add-stryker-mutation-testing.md b/ai-plans/0063-0-add-stryker-mutation-testing.md similarity index 96% rename from ai-plans/0063-add-stryker-mutation-testing.md rename to ai-plans/0063-0-add-stryker-mutation-testing.md index 688984e..749b792 100644 --- a/ai-plans/0063-add-stryker-mutation-testing.md +++ b/ai-plans/0063-0-add-stryker-mutation-testing.md @@ -8,14 +8,14 @@ Introduce Stryker.NET as a pinned, opt-in tool with a repository configuration t ## Acceptance Criteria -- [ ] `dotnet tool restore` installs `dotnet-stryker` pinned to exactly 4.16.0, and a repository-root `stryker-config.json` supplies the shared configuration so that a scoped run needs only a `-p` argument and an optional `-m` glob. -- [ ] The committed configuration selects the Microsoft Testing Platform runner and disables coverage analysis; a run against `Light.PortableResults.AspNetCore.Shared` reports a non-zero mutation score, and a run against `Result.cs` reports zero `NoCoverage` mutants. -- [ ] Mutation output is contained under `StrykerOutput/`, which is ignored by git; a completed or interrupted run adds no new entries to `git status`. -- [ ] `tests/AGENTS.md` documents the file-scoped and project-scoped invocations, states that a surviving mutant is a signal to investigate together with the three permitted triage outcomes, records the measured cost per project, and records the mutation blind spots that must not be read as adequate coverage. -- [ ] Mutation testing runs on demand from a developer machine only. No CI workflow is added and no existing workflow is modified; pull request validation time is unchanged. -- [ ] The documented invocation is verified to run the intended tests: for a run mutating `AspNetCore.Shared`, Stryker's reported test count is 337 — every test project transitively referencing it — and not the 26 tests of `AspNetCore.Shared.Tests` alone. -- [ ] A baseline is recorded in `tests/AGENTS.md` for the four projects that complete in minutes (`AspNetCore.Shared`, `AspNetCore.Mvc`, `AspNetCore.MinimalApis`, `Validation.OpenApi`). Each entry carries enough provenance to be reproduced and compared: commit, `dotnet-stryker` version, concurrency, the test count actually executed, elapsed time, and the killed, timeout, survived, compile-error, ignored, and no-coverage counts — not a percentage alone. The no-coverage count is recorded even though it is expected to be zero, because a non-zero value means `coverage-analysis` is no longer taking effect; it was zero in all four measured runs. -- [ ] Mutation score thresholds and `break-at` remain unset; nothing in the repository fails a build or a test run because of a mutation score. +- [x] `dotnet tool restore` installs `dotnet-stryker` pinned to exactly 4.16.0, and a repository-root `stryker-config.json` supplies the shared configuration so that a scoped run needs only a `-p` argument and an optional `-m` glob. +- [x] The committed configuration selects the Microsoft Testing Platform runner and disables coverage analysis; a run against `Light.PortableResults.AspNetCore.Shared` reports a non-zero mutation score, and a run against `Result.cs` reports zero `NoCoverage` mutants. +- [x] Mutation output is contained under `StrykerOutput/`, which is ignored by git; a completed or interrupted run adds no new entries to `git status`. +- [x] `tests/AGENTS.md` documents the file-scoped and project-scoped invocations, states that a surviving mutant is a signal to investigate together with the three permitted triage outcomes, records the measured cost per project, and records the mutation blind spots that must not be read as adequate coverage. +- [x] Mutation testing runs on demand from a developer machine only. No CI workflow is added and no existing workflow is modified; pull request validation time is unchanged. +- [x] The documented invocation is verified to run the intended tests: for a run mutating `AspNetCore.Shared`, Stryker's reported test count is 337 — every test project transitively referencing it — and not the 26 tests of `AspNetCore.Shared.Tests` alone. +- [x] A baseline is recorded in `tests/AGENTS.md` for the four projects that complete in minutes (`AspNetCore.Shared`, `AspNetCore.Mvc`, `AspNetCore.MinimalApis`, `Validation.OpenApi`). Each entry carries enough provenance to be reproduced and compared: commit, `dotnet-stryker` version, concurrency, the test count actually executed, elapsed time, and the killed, timeout, survived, compile-error, ignored, and no-coverage counts — not a percentage alone. The no-coverage count is recorded even though it is expected to be zero, because a non-zero value means `coverage-analysis` is no longer taking effect; it was zero in all four measured runs. +- [x] Mutation score thresholds and `break-at` remain unset; nothing in the repository fails a build or a test run because of a mutation score. ## Technical Details diff --git a/ai-plans/0063-1-plan-deviations.md b/ai-plans/0063-1-plan-deviations.md new file mode 100644 index 0000000..5c5b764 --- /dev/null +++ b/ai-plans/0063-1-plan-deviations.md @@ -0,0 +1,56 @@ +# 0063 Plan Deviations + +This document compares `ai-plans/0063-add-stryker-mutation-testing.md` with the results measured while implementing it (`dotnet-stryker` 4.16.0, Apple M3 Max, 16 logical cores, source tree at commit `04aee20`). + +## Summary + +All acceptance criteria were implemented and verified as specified: the pinned tool, the committed configuration, the scoped invocations, the transitive test counts, the git-ignored output containment, the recorded baseline, and the absence of any CI or threshold changes. + +The material deviation is in the plan's Technical Details narrative about survivors: the claim of exactly one survivor across the four small projects did not reproduce. The survivor population in `Validation.OpenApi` is roughly twenty mutants, and most of it is hidden by timeout classification to a degree that varies with machine load and concurrency. The operational documentation in `tests/AGENTS.md` already reflects the measured behavior; this file records the delta from the plan text. + +## Deviations From The Original Plan + +### 1. `Validation.OpenApi` has ~18–20 survivors masked as timeouts, not one survivor + +**Original plan:** +The plan states that the measured runs produced exactly one survivor across all four small projects — a statement-removal mutant at `BuiltInValidationErrorBuilderExtensions.cs:426`, putting `Validation.OpenApi` at 98.68% — and that the triage categories therefore "remain largely a priori rather than derived from survivors observed in this codebase." + +**Measured:** +Three runs of the same unmodified sources produced three different survivor sets: + +- Concurrency 8: 54 killed, 21 timeout, 1 survived — 98.68%. The survivor was an equality mutant at `BuiltInValidationErrorBuilderExtensions.cs:388`, not the statement removal at line 426. +- Concurrency 8, repeat: 51 killed, 21 timeout, 4 survived — 94.74%. +- Concurrency 4: 56 killed, 2 timeout, 18 survived — 76.32%. + +The survivors cluster in `BuiltInValidationErrorBuilderExtensions.cs` lines 367–426 (conditional, equality, object-initializer, and statement mutants around the built-in validation error OpenAPI schema customization), with a few more in `BuiltInValidationErrorContracts.cs`, `BuiltInValidationErrorContractRegistrationExtensions.cs`, and `PortableValidationOpenApiRouteHandlerBuilderExtensions.cs`. + +**Why:** +With `coverage-analysis: off`, every mutant runs the full discovered test set (163 tests for this project). A killed mutant's run aborts at the first failing test, but a genuinely surviving mutant runs the entire suite — and can exceed the per-mutant timeout. `Timeout` counts as killed in the mutation score, so under higher concurrency (more host contention) slow survivors are reclassified as timeouts and the score is inflated. The plan's "one survivor, 98.68%" is one point in this timing-dependent distribution, not a stable property of the test suite. + +**Impact:** +The initial triage queue is ~18–20 mutants in that file, not one, and the triage categories are no longer a priori — they now have real input. No acceptance criterion is affected: criterion 7 deliberately requires recording the timeout count alongside the others, and the baseline in `tests/AGENTS.md` carries the measured vector. The blind-spots section there directs agents to treat `Timeout` as a triage signal equal to `Survived` for this project. Future comparisons should match full count vectors at equal concurrency rather than percentages. + +### 2. The recorded baseline's provenance commit predates the tooling commit + +**Original plan:** +Criterion 7 requires the baseline to carry the commit at which it was measured. + +**Measured:** +The baseline was measured at commit `04aee20` while the implementation itself (tool manifest, `stryker-config.json`, `.gitignore`) was still uncommitted. The mutated sources were identical to `04aee20`, so mutation results are unaffected, but once this work is committed the provenance commit recorded in `tests/AGENTS.md` is an ancestor of the tooling commit rather than the commit that introduced the tool. + +**Impact:** +Cosmetic. Re-measuring the baseline at the merge commit is a valid cheap follow-up (about six minutes for all four projects) if exact provenance is ever needed. + +## Notes On Items Implemented As Planned + +The following measurements from the plan reproduced exactly or within seconds: + +- Mutant inventories and test counts for all four small projects: 33/32/44/114 mutants, 11/11/3/2 compile errors, 102/237/337/163 tests (`AspNetCore.Mvc`, `AspNetCore.MinimalApis`, `AspNetCore.Shared`, `Validation.OpenApi`). +- `AspNetCore.Shared` reporting 337 tests (all six transitively referencing test projects) and `Result.cs` reporting 2,398 tests with 67 killed, 0 `NoCoverage`, 100.00%. +- Elapsed times: 0:40 / 0:58 / 1:57 / 2:06 measured vs. 0:38 / 0:57 / 1:57 / 1:59 in the plan. +- `AspNetCore.Mvc`, `AspNetCore.MinimalApis`, and `AspNetCore.Shared` each scoring 100.00%. +- `StrykerOutput/` containment (eight runs, no `git status` entries), no CI changes, and no `thresholds`/`break-at` settings anywhere. + +## Minor Operational Notes + +- `dotnet stryker --version` is not a tool-version query — it is Stryker's `--version ` option for the analyzed project and errors with "Missing value for option 'version'". Use `dotnet tool list` to confirm the pinned tool version. diff --git a/stryker-config.json b/stryker-config.json new file mode 100644 index 0000000..71e0e58 --- /dev/null +++ b/stryker-config.json @@ -0,0 +1,9 @@ +{ + "stryker-config": { + "solution": "Light.PortableResults.slnx", + "test-runner": "mtp", + "coverage-analysis": "off", + "configuration": "Debug", + "reporters": ["json", "html", "progress"] + } +} diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 3554979..7ead9ae 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -19,3 +19,75 @@ Always pass `--coverage-settings`. It excludes source-generated files under `obj/`, which otherwise dominate the line counts and report the solution at roughly 81% instead of 95%. The path must be absolute, because each test app runs with its own output directory as the working directory. Each test project writes `TestResults/.cobertura.xml`. Merge them with `reportgenerator -reports:'**/*.cobertura.xml' -targetdir:./coverage-merged -reporttypes:'Cobertura;TextSummary'`. + +## Mutation testing (Stryker.NET) + +Stryker.NET is pinned as a local tool (`dotnet tool restore` installs `dotnet-stryker` 4.16.0) with the shared configuration committed in `stryker-config.json` at the repository root. It is a local, on-demand tool only: there is no CI integration and no mutation-score gate (`thresholds` and `break-at` are deliberately unset). A surviving mutant is a signal to investigate, not proof of a defect. + +### How to run + +Always run from the repository root. Stryker resolves `stryker-config.json` from the current working directory and a missing config is not an error — it silently falls back to the defaults, which lose the two load-bearing settings at once (the MTP test runner and the disabled coverage analysis) and produce confidently wrong reports: 0.00% scores under the `vstest` runner, or fabricated `NoCoverage` under `perTest` coverage analysis. + +```shell +# One project — the only form needed for the four small projects +dotnet stryker -p Light.PortableResults.AspNetCore.Shared.csproj + +# One file, via an optional mutate glob (about four minutes for Result.cs) +dotnet stryker -p Light.PortableResults.csproj -m '**/Result.cs' +``` + +The committed config names the solution, so every run is a solution-context run: `-p` selects which source project is mutated, but it does not narrow which tests execute. Stryker runs **every test project that transitively references the mutated project** — mutating `AspNetCore.Shared` executes 337 tests (all six referencing test projects, not the 26 of `AspNetCore.Shared.Tests` alone), and mutating `Light.PortableResults` executes all 2,398. Keep this: a mutant killed by another project's tests is legitimately killed (sociable tests), and the extra tests are nearly free because per-mutant cost is dominated by fixed overhead. `-tp` does not override this behavior and must not be passed as if it did. Do not use `--since` either: it treats any non-C# file in the diff (a plan under `ai-plans/` qualifies) as a changed test file and degrades to a full project run while presenting itself as scoped. + +The config pins `test-runner: mtp` because the default `vstest` runner cannot drive the xUnit v3 hosts used here (coverage capture fails and every mutant is reported survived), `coverage-analysis: off` because the MTP per-test analysis fabricates `NoCoverage` for members the tests demonstrably cover, and `configuration: Debug` so that the Release-only `TreatWarningsAsErrors` cannot interfere with Stryker's rollback. MTP support is a Stryker preview feature (stryker-mutator/stryker-net#3094); per-test coverage for MTP is pending upstream (#3516) and is the main lever on run time once it ships and proves trustworthy. + +Reports are written to `StrykerOutput//reports/` (gitignored). The JSON report is the agent-facing artifact — filter for `"status": "Survived"` to get a work queue located at specific files and lines; the HTML report is for humans. + +### Cost and scope + +Measured throughput is about three seconds per mutant at concurrency 8 (upper bound, taken from the large projects). Never mutate the whole solution in the inner loop: omitting `-p` is the seven-hour path. `Light.PortableResults` alone is ~3.6 h and `Validation` ~1.8 h; split them by mutate glob along folder boundaries (`Metadata/`, `Http/`, `CloudEvents/`, `Numbers/`) so runs stay interruptible and reports arrive while still actionable. Mutant inventory for sizing (measured at concurrency 8; the four small projects re-verified at the baseline below): + +| Project | Mutants | CompileError | +| --- | ---: | ---: | +| `Light.PortableResults` | 4,867 | 520 | +| `Light.PortableResults.Validation` | 2,215 | 83 | +| `Validation.OpenApi.SourceGeneration` | 1,272 | 204 | +| `AspNetCore.OpenApi` | 848 | 65 | +| `Validation.OpenApi` | 114 | 2 | +| `AspNetCore.Shared` | 44 | 3 | +| `AspNetCore.Mvc` | 33 | 11 | +| `AspNetCore.MinimalApis` | 32 | 11 | + +### Baseline + +Measured on an Apple M3 Max (16 logical cores) at commit `04aee20` (source tree; the tooling change itself was uncommitted at measurement time), `dotnet-stryker` 4.16.0, concurrency pinned to 8 via `-c 8`, `Debug`: + +| Project | Tests run | Elapsed | Killed | Timeout | Survived | CompileError | Ignored | NoCoverage | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| `AspNetCore.Mvc` | 102 | 0:40 | 16 | 1 | 0 | 11 | 5 | 0 | +| `AspNetCore.MinimalApis` | 237 | 0:58 | 16 | 0 | 0 | 11 | 5 | 0 | +| `AspNetCore.Shared` | 337 | 1:57 | 31 | 0 | 0 | 3 | 10 | 0 | +| `Validation.OpenApi` | 163 | 2:06 | 54 | 21 | 1 | 2 | 36 | 0 | + +The `NoCoverage` column must be zero even though nothing is expected there: a non-zero value means `coverage-analysis: off` is no longer taking effect. When comparing a future run against this table, compare counts, not the percentage alone. + +### Triaging survivors + +A surviving mutant means the suite did not distinguish the mutated program from the original. There are exactly three permitted responses: + +1. **Observable behavior is genuinely unconstrained** — add or strengthen a test. This is the outcome the tool exists to produce and should be the common one. +2. **The mutant is equivalent or invalid** — suppress it narrowly at the source with a justification, which Stryker records in the report: + + ```csharp + // Stryker disable once Statement : equivalent - the guard is a fast path, not a behavior change + ``` + + The syntax is `Stryker [disable|restore][once][all|][: reason]` and is scope-aware. Prefer `disable once` over `disable all`, and never use the global `ignore-mutations` setting to silence a single site. +3. **The mutant sits in a construct this configuration cannot meaningfully test** — record it and move on; see the blind spots below. + +Never restructure production code solely to make a mutant killable. Performance ranks above extensibility in this library, and the low-allocation `in`/`ref`/`Span` style is precisely the shape that produces awkward survivors; a lower mutation score is the correct outcome when the alternative is a slower implementation. + +### Blind spots — do not read these as adequate coverage + +- Roughly 9.5% of mutants fail to compile, concentrated in the low-allocation `out`/`ref` style. Stryker responds with Safe Mode, which discards **every** mutant in the enclosing method: `Dragon4.GenerateDigits`, `ResultJsonReader.ReadStatusValue`, `ResultJsonReader.ReadIndexValue`, `ErrorsExtensions.WriteRichErrors`, and the whole of `LightResult.cs` (11 of 11 mutants) receive no mutation coverage at all. This is a tool limitation, not a test defect — a high score in `Numbers/` is not verified behavior, and no one should "fix" it by restructuring production code. +- `Timeout` counts as killed in the mutation score, but with `coverage-analysis: off` every mutant runs the full discovered test set, so a genuinely surviving mutant whose tests all pass runs the entire suite and can exceed the per-mutant timeout — surfacing as `Timeout` instead of `Survived`, depending on machine load and concurrency. `Validation.OpenApi` shows this: its 21 timeouts at concurrency 8 mostly become 18 survivors at concurrency 4 (score drops from 98.68% to 76.32%). Treat `Timeout` as a triage signal equal to `Survived`, especially in that project. +- The MTP runner is a preview; verify surprising results against a plain `dotnet test` run before acting on them. From 895a8f17d01730350c47eda3bc521e9951becb06 Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 19:55:17 +0200 Subject: [PATCH 05/13] docs: tighten the mutation testing section in tests/AGENTS.md --- tests/AGENTS.md | 48 ++++++++++++++++++------------------------------ 1 file changed, 18 insertions(+), 30 deletions(-) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 7ead9ae..6e3d443 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -22,29 +22,25 @@ Each test project writes `TestResults/.cobertura.xml`. Merge them with `re ## Mutation testing (Stryker.NET) -Stryker.NET is pinned as a local tool (`dotnet tool restore` installs `dotnet-stryker` 4.16.0) with the shared configuration committed in `stryker-config.json` at the repository root. It is a local, on-demand tool only: there is no CI integration and no mutation-score gate (`thresholds` and `break-at` are deliberately unset). A surviving mutant is a signal to investigate, not proof of a defect. +`dotnet tool restore` installs the pinned `dotnet-stryker` 4.16.0; the shared config is `stryker-config.json` at the repository root. Local, on-demand use only: no CI integration and no score gates (`thresholds`/`break-at` are unset). A surviving mutant is a signal to investigate, not proof of a defect. ### How to run -Always run from the repository root. Stryker resolves `stryker-config.json` from the current working directory and a missing config is not an error — it silently falls back to the defaults, which lose the two load-bearing settings at once (the MTP test runner and the disabled coverage analysis) and produce confidently wrong reports: 0.00% scores under the `vstest` runner, or fabricated `NoCoverage` under `perTest` coverage analysis. +Always run from the repository root: Stryker reads `stryker-config.json` only from the current directory and silently falls back to defaults without it, reverting to the `vstest` runner (cannot drive the xUnit v3 hosts, reports everything survived) and `perTest` coverage analysis (fabricates `NoCoverage`). The config also pins `Debug` because `TreatWarningsAsErrors` is Release-only. ```shell -# One project — the only form needed for the four small projects +# One project — sufficient for the four small projects dotnet stryker -p Light.PortableResults.AspNetCore.Shared.csproj -# One file, via an optional mutate glob (about four minutes for Result.cs) +# One file (~4 minutes for Result.cs) dotnet stryker -p Light.PortableResults.csproj -m '**/Result.cs' ``` -The committed config names the solution, so every run is a solution-context run: `-p` selects which source project is mutated, but it does not narrow which tests execute. Stryker runs **every test project that transitively references the mutated project** — mutating `AspNetCore.Shared` executes 337 tests (all six referencing test projects, not the 26 of `AspNetCore.Shared.Tests` alone), and mutating `Light.PortableResults` executes all 2,398. Keep this: a mutant killed by another project's tests is legitimately killed (sociable tests), and the extra tests are nearly free because per-mutant cost is dominated by fixed overhead. `-tp` does not override this behavior and must not be passed as if it did. Do not use `--since` either: it treats any non-C# file in the diff (a plan under `ai-plans/` qualifies) as a changed test file and degrades to a full project run while presenting itself as scoped. +`-p` selects the mutated project, not the tests: Stryker runs every test project transitively referencing it (`AspNetCore.Shared` → 337 tests, `Light.PortableResults` → all 2,398). Cross-project kills are legitimate (sociable tests) and nearly free. Never pass `-tp` (it does not narrow tests) or `--since` (any non-C# file in the diff, e.g. an `ai-plans/` document, degrades it to a full run). Reports go to `StrykerOutput//reports/` (gitignored): JSON for agents (filter `"status": "Survived"`), HTML for humans. -The config pins `test-runner: mtp` because the default `vstest` runner cannot drive the xUnit v3 hosts used here (coverage capture fails and every mutant is reported survived), `coverage-analysis: off` because the MTP per-test analysis fabricates `NoCoverage` for members the tests demonstrably cover, and `configuration: Debug` so that the Release-only `TreatWarningsAsErrors` cannot interfere with Stryker's rollback. MTP support is a Stryker preview feature (stryker-mutator/stryker-net#3094); per-test coverage for MTP is pending upstream (#3516) and is the main lever on run time once it ships and proves trustworthy. +### Cost and baseline -Reports are written to `StrykerOutput//reports/` (gitignored). The JSON report is the agent-facing artifact — filter for `"status": "Survived"` to get a work queue located at specific files and lines; the HTML report is for humans. - -### Cost and scope - -Measured throughput is about three seconds per mutant at concurrency 8 (upper bound, taken from the large projects). Never mutate the whole solution in the inner loop: omitting `-p` is the seven-hour path. `Light.PortableResults` alone is ~3.6 h and `Validation` ~1.8 h; split them by mutate glob along folder boundaries (`Metadata/`, `Http/`, `CloudEvents/`, `Numbers/`) so runs stay interruptible and reports arrive while still actionable. Mutant inventory for sizing (measured at concurrency 8; the four small projects re-verified at the baseline below): +~3 s per mutant at concurrency 8 (upper bound from the large projects). Omitting `-p` mutates the whole solution (~7 h; `Light.PortableResults` alone ~3.6 h) — split large projects by mutate glob along folder boundaries (`Metadata/`, `Http/`, `CloudEvents/`, `Numbers/`). Mutant inventory for sizing: | Project | Mutants | CompileError | | --- | ---: | ---: | @@ -57,9 +53,7 @@ Measured throughput is about three seconds per mutant at concurrency 8 (upper bo | `AspNetCore.Mvc` | 33 | 11 | | `AspNetCore.MinimalApis` | 32 | 11 | -### Baseline - -Measured on an Apple M3 Max (16 logical cores) at commit `04aee20` (source tree; the tooling change itself was uncommitted at measurement time), `dotnet-stryker` 4.16.0, concurrency pinned to 8 via `-c 8`, `Debug`: +Baseline measured at commit `04aee20`, `dotnet-stryker` 4.16.0, `-c 8`, `Debug`, Apple M3 Max: | Project | Tests run | Elapsed | Killed | Timeout | Survived | CompileError | Ignored | NoCoverage | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | @@ -68,26 +62,20 @@ Measured on an Apple M3 Max (16 logical cores) at commit `04aee20` (source tree; | `AspNetCore.Shared` | 337 | 1:57 | 31 | 0 | 0 | 3 | 10 | 0 | | `Validation.OpenApi` | 163 | 2:06 | 54 | 21 | 1 | 2 | 36 | 0 | -The `NoCoverage` column must be zero even though nothing is expected there: a non-zero value means `coverage-analysis: off` is no longer taking effect. When comparing a future run against this table, compare counts, not the percentage alone. +`NoCoverage` must be zero — a non-zero value means `coverage-analysis: off` is no longer taking effect. Compare full count vectors, not percentages. ### Triaging survivors -A surviving mutant means the suite did not distinguish the mutated program from the original. There are exactly three permitted responses: - -1. **Observable behavior is genuinely unconstrained** — add or strengthen a test. This is the outcome the tool exists to produce and should be the common one. -2. **The mutant is equivalent or invalid** — suppress it narrowly at the source with a justification, which Stryker records in the report: - - ```csharp - // Stryker disable once Statement : equivalent - the guard is a fast path, not a behavior change - ``` +Exactly three permitted responses to a survivor: - The syntax is `Stryker [disable|restore][once][all|][: reason]` and is scope-aware. Prefer `disable once` over `disable all`, and never use the global `ignore-mutations` setting to silence a single site. -3. **The mutant sits in a construct this configuration cannot meaningfully test** — record it and move on; see the blind spots below. +1. **Behavior is genuinely unconstrained** — add or strengthen a test (the common case). +2. **Equivalent or invalid mutant** — suppress narrowly at the source with a justification: `// Stryker disable once Statement : equivalent - `. Prefer `disable once` over `disable all`; never use the global `ignore-mutations` setting for a single site. +3. **Untestable construct** — record it and move on (blind spots below). -Never restructure production code solely to make a mutant killable. Performance ranks above extensibility in this library, and the low-allocation `in`/`ref`/`Span` style is precisely the shape that produces awkward survivors; a lower mutation score is the correct outcome when the alternative is a slower implementation. +Never restructure production code to make a mutant killable: performance outranks mutation score here, and the low-allocation `in`/`ref`/`Span` style produces awkward survivors by nature. -### Blind spots — do not read these as adequate coverage +### Blind spots — do not read as adequate coverage -- Roughly 9.5% of mutants fail to compile, concentrated in the low-allocation `out`/`ref` style. Stryker responds with Safe Mode, which discards **every** mutant in the enclosing method: `Dragon4.GenerateDigits`, `ResultJsonReader.ReadStatusValue`, `ResultJsonReader.ReadIndexValue`, `ErrorsExtensions.WriteRichErrors`, and the whole of `LightResult.cs` (11 of 11 mutants) receive no mutation coverage at all. This is a tool limitation, not a test defect — a high score in `Numbers/` is not verified behavior, and no one should "fix" it by restructuring production code. -- `Timeout` counts as killed in the mutation score, but with `coverage-analysis: off` every mutant runs the full discovered test set, so a genuinely surviving mutant whose tests all pass runs the entire suite and can exceed the per-mutant timeout — surfacing as `Timeout` instead of `Survived`, depending on machine load and concurrency. `Validation.OpenApi` shows this: its 21 timeouts at concurrency 8 mostly become 18 survivors at concurrency 4 (score drops from 98.68% to 76.32%). Treat `Timeout` as a triage signal equal to `Survived`, especially in that project. -- The MTP runner is a preview; verify surprising results against a plain `dotnet test` run before acting on them. +- ~9.5% of mutants fail to compile (mostly the `out`/`ref` style); Stryker's Safe Mode then discards every mutant in the enclosing method: `Dragon4.GenerateDigits`, `ResultJsonReader.ReadStatusValue`/`ReadIndexValue`, `ErrorsExtensions.WriteRichErrors`, and all of `LightResult.cs` receive no mutation coverage. Tool limitation, not a test defect — a high score in `Numbers/` is not verified behavior. +- `Timeout` counts as killed, but full-suite survivors can exceed the per-mutant timeout under load: `Validation.OpenApi`'s 21 timeouts at `-c 8` become 18 survivors at `-c 4` (76.32%). Treat `Timeout` like `Survived` in that project. +- The MTP runner is a preview (stryker-mutator/stryker-net#3094); verify surprising results against a plain `dotnet test` run. From 374698dd804680120b8b54f20747402c85939939 Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 20:19:03 +0200 Subject: [PATCH 06/13] chore: pin Stryker concurrency for reproducible mutation baselines The result vector is concurrency-sensitive (Validation.OpenApi: 2 timeouts/18 survivors at -c 4 vs 21/1 at -c 8), so a baseline is only comparable at equal concurrency. Pin concurrency: 8 in stryker-config.json instead of passing -c ad hoc: the documented -p-only invocation now reproduces the baseline on any machine with at least 8 logical cores, and -c stays free for experiments. Note: the 4.16.0 --help text claims the default is the CPU core count, but the actual default is half the logical processors (verified via the debug options dump), i.e. machine-dependent. --- ai-plans/0063-1-plan-deviations.md | 11 +++++++++++ stryker-config.json | 1 + tests/AGENTS.md | 6 +++--- 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/ai-plans/0063-1-plan-deviations.md b/ai-plans/0063-1-plan-deviations.md index 5c5b764..8c01ff6 100644 --- a/ai-plans/0063-1-plan-deviations.md +++ b/ai-plans/0063-1-plan-deviations.md @@ -41,6 +41,17 @@ The baseline was measured at commit `04aee20` while the implementation itself (t **Impact:** Cosmetic. Re-measuring the baseline at the merge commit is a valid cheap follow-up (about six minutes for all four projects) if exact provenance is ever needed. +### 3. The committed configuration pins `concurrency: 8`, which the plan did not specify + +**Original plan:** +The plan specifies the exact contents of `stryker-config.json` (`solution`, `test-runner`, `coverage-analysis`, `configuration`, `reporters`) and requires that a scoped run needs only a `-p` argument and an optional `-m` glob. Concurrency appears only as baseline provenance, supplied via `-c 8` at invocation time — so the documented invocation was not the invocation that produced the baseline. + +**Implemented:** +`stryker-config.json` additionally pins `"concurrency": 8`, and the documented invocations pass no `-c`. + +**Why:** +The result vector is concurrency-sensitive (§1: `Validation.OpenApi` reports 2 timeouts/18 survivors at concurrency 4 vs. 21/1 at 8), so a baseline is only comparable at equal concurrency — making concurrency a load-bearing setting, which the plan's own philosophy places in the configuration rather than the invocation. Relying on the default is not an alternative: 4.16.0 defaults to half the logical processors (verified via the debug options dump; the `--help` text claiming "as many parallel processes as you have CPU cores" is stale), which is 8 on the recording machine but 4 on an 8-core machine. The pinned value reproduces the baseline on any machine with at least 8 logical cores and leaves `-c` free for experiments, e.g. lowering it to unmask slow survivors. Verified to take effect: with `"concurrency": 3` in the config and no `-c`, the debug options dump reports `"Concurrency": 3`. + ## Notes On Items Implemented As Planned The following measurements from the plan reproduced exactly or within seconds: diff --git a/stryker-config.json b/stryker-config.json index 71e0e58..db81bd4 100644 --- a/stryker-config.json +++ b/stryker-config.json @@ -4,6 +4,7 @@ "test-runner": "mtp", "coverage-analysis": "off", "configuration": "Debug", + "concurrency": 8, "reporters": ["json", "html", "progress"] } } diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 6e3d443..2c18b79 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -26,7 +26,7 @@ Each test project writes `TestResults/.cobertura.xml`. Merge them with `re ### How to run -Always run from the repository root: Stryker reads `stryker-config.json` only from the current directory and silently falls back to defaults without it, reverting to the `vstest` runner (cannot drive the xUnit v3 hosts, reports everything survived) and `perTest` coverage analysis (fabricates `NoCoverage`). The config also pins `Debug` because `TreatWarningsAsErrors` is Release-only. +Always run from the repository root: Stryker reads `stryker-config.json` only from the current directory and silently falls back to defaults without it, reverting to the `vstest` runner (cannot drive the xUnit v3 hosts, reports everything survived) and `perTest` coverage analysis (fabricates `NoCoverage`). The config also pins `Debug` (`TreatWarningsAsErrors` is Release-only) and `concurrency: 8` (result vectors vary with parallelism; override with `-c` for experiments). ```shell # One project — sufficient for the four small projects @@ -53,7 +53,7 @@ dotnet stryker -p Light.PortableResults.csproj -m '**/Result.cs' | `AspNetCore.Mvc` | 33 | 11 | | `AspNetCore.MinimalApis` | 32 | 11 | -Baseline measured at commit `04aee20`, `dotnet-stryker` 4.16.0, `-c 8`, `Debug`, Apple M3 Max: +Baseline measured at commit `04aee20`, `dotnet-stryker` 4.16.0, concurrency 8 (pinned in `stryker-config.json`; the 4.16.0 default is half the logical cores and therefore machine-dependent), `Debug`, Apple M3 Max: | Project | Tests run | Elapsed | Killed | Timeout | Survived | CompileError | Ignored | NoCoverage | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | @@ -62,7 +62,7 @@ Baseline measured at commit `04aee20`, `dotnet-stryker` 4.16.0, `-c 8`, `Debug`, | `AspNetCore.Shared` | 337 | 1:57 | 31 | 0 | 0 | 3 | 10 | 0 | | `Validation.OpenApi` | 163 | 2:06 | 54 | 21 | 1 | 2 | 36 | 0 | -`NoCoverage` must be zero — a non-zero value means `coverage-analysis: off` is no longer taking effect. Compare full count vectors, not percentages. +`NoCoverage` must be zero — a non-zero value means `coverage-analysis: off` is no longer taking effect. Compare full count vectors at equal concurrency, not percentages. ### Triaging survivors From 435df8921062202baf14baa33a51f325fe5b155b Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 20:25:24 +0200 Subject: [PATCH 07/13] docs: mark the Validation.OpenApi mutation baseline as load-dependent Footnote the row with the observed variance instead of presenting one vector unqualified: the Killed/Timeout/Survived split is machine-load dependent under coverage-analysis: off, while Tests, CompileError, and Ignored are deterministic. Point triage runs at -c 4 to unmask the slow full-suite survivors. --- tests/AGENTS.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 2c18b79..30702cb 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -60,7 +60,9 @@ Baseline measured at commit `04aee20`, `dotnet-stryker` 4.16.0, concurrency 8 (p | `AspNetCore.Mvc` | 102 | 0:40 | 16 | 1 | 0 | 11 | 5 | 0 | | `AspNetCore.MinimalApis` | 237 | 0:58 | 16 | 0 | 0 | 11 | 5 | 0 | | `AspNetCore.Shared` | 337 | 1:57 | 31 | 0 | 0 | 3 | 10 | 0 | -| `Validation.OpenApi` | 163 | 2:06 | 54 | 21 | 1 | 2 | 36 | 0 | +| `Validation.OpenApi` † | 163 | 2:06 | 54 | 21 | 1 | 2 | 36 | 0 | + +† This row is not exactly reproducible. With `coverage-analysis: off` every mutant runs the full 163-test suite, so the Killed/Timeout/Survived split is machine-load dependent (see the timeout-masking blind spot): two runs of identical sources at concurrency 8 gave Killed 51–54, Timeout 21, Survived 1–4. Tests run, CompileError, and Ignored are deterministic. For triage rather than comparison, run this project with `-c 4` to unmask the slow full-suite survivors (56 killed, 2 timeout, 18 survived). `NoCoverage` must be zero — a non-zero value means `coverage-analysis: off` is no longer taking effect. Compare full count vectors at equal concurrency, not percentages. From 2cc94c4acf451ae3b55e69f482414fa3adc0f533 Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 20:34:49 +0200 Subject: [PATCH 08/13] docs(tests): explain ignored Stryker mutants --- tests/AGENTS.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 30702cb..bf086a7 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -64,6 +64,8 @@ Baseline measured at commit `04aee20`, `dotnet-stryker` 4.16.0, concurrency 8 (p † This row is not exactly reproducible. With `coverage-analysis: off` every mutant runs the full 163-test suite, so the Killed/Timeout/Survived split is machine-load dependent (see the timeout-masking blind spot): two runs of identical sources at concurrency 8 gave Killed 51–54, Timeout 21, Survived 1–4. Tests run, CompileError, and Ignored are deterministic. For triage rather than comparison, run this project with `-c 4` to unmask the slow full-suite survivors (56 killed, 2 timeout, 18 survived). +`Ignored` is not user suppression: all 56 baseline entries are `Block removal` mutants discarded deterministically by Stryker's built-in "block already covered" filter because another active mutant exists inside the block. A different reason or count should be investigated. + `NoCoverage` must be zero — a non-zero value means `coverage-analysis: off` is no longer taking effect. Compare full count vectors at equal concurrency, not percentages. ### Triaging survivors From 42c3c7b552fc6781f8a5ccc7a23dac79905b8154 Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 20:40:42 +0200 Subject: [PATCH 09/13] chore: add Conventional Commits to AGENTS.md Signed-off-by: Kenny Pflug --- AGENTS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/AGENTS.md b/AGENTS.md index 48d1265..e070cf8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,6 +15,7 @@ In our Directory.Build.props files in this solution, the following rules are def - The library is not published in a stable version yet, you can make breaking changes. - `` is enabled in Release builds, so your code changes must not generate warnings. - When a type or method is properly encapsulated, make it public. We don't know how callers would like to use this library. When some types are internal, this might make it hard for callers to access these in tests or when making configuration changes. Prefer public APIs over internal ones. +- Use Conventional Commits messages. Decide whether a commit title is enough or a commit body is required. ## Testing Rules From 28ecf1e805c9a6fb24021eb4902f97183b6de347 Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 20:44:21 +0200 Subject: [PATCH 10/13] docs(tests): clarify Stryker baseline guidance --- tests/AGENTS.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index bf086a7..c7b4101 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -28,6 +28,8 @@ Each test project writes `TestResults/.cobertura.xml`. Merge them with `re Always run from the repository root: Stryker reads `stryker-config.json` only from the current directory and silently falls back to defaults without it, reverting to the `vstest` runner (cannot drive the xUnit v3 hosts, reports everything survived) and `perTest` coverage analysis (fabricates `NoCoverage`). The config also pins `Debug` (`TreatWarningsAsErrors` is Release-only) and `concurrency: 8` (result vectors vary with parallelism; override with `-c` for experiments). +Revisit `coverage-analysis: off` when Stryker's MTP per-test coverage support ([#3516](https://github.com/stryker-mutator/stryker-net/pull/3516)) ships and proves trustworthy; it is the main lever on run time because it avoids running the full discovered test set for every mutant. + ```shell # One project — sufficient for the four small projects dotnet stryker -p Light.PortableResults.AspNetCore.Shared.csproj @@ -53,7 +55,7 @@ dotnet stryker -p Light.PortableResults.csproj -m '**/Result.cs' | `AspNetCore.Mvc` | 33 | 11 | | `AspNetCore.MinimalApis` | 32 | 11 | -Baseline measured at commit `04aee20`, `dotnet-stryker` 4.16.0, concurrency 8 (pinned in `stryker-config.json`; the 4.16.0 default is half the logical cores and therefore machine-dependent), `Debug`, Apple M3 Max: +Baseline measured at commit `04aee20`, `dotnet-stryker` 4.16.0, concurrency 8 (pinned in `stryker-config.json`; the 4.16.0 default is half the logical cores and therefore machine-dependent), `Debug`, Apple M3 Max (16 logical cores): | Project | Tests run | Elapsed | Killed | Timeout | Survived | CompileError | Ignored | NoCoverage | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | From 9328aba166f0cad478fcc86656a90dede41b0f56 Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 20:47:50 +0200 Subject: [PATCH 11/13] docs(tests): add Stryker configuration smoke check --- tests/AGENTS.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index c7b4101..6ad1a78 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -34,10 +34,13 @@ Revisit `coverage-analysis: off` when Stryker's MTP per-test coverage support ([ # One project — sufficient for the four small projects dotnet stryker -p Light.PortableResults.AspNetCore.Shared.csproj -# One file (~4 minutes for Result.cs) +# One-file configuration smoke check (~4 minutes) +# Expect: 2,398 tests, 67 killed, 0 NoCoverage, 100.00% dotnet stryker -p Light.PortableResults.csproj -m '**/Result.cs' ``` +The smoke-check vector is tied to the current `Result.cs` and its tests; update it after intentional changes alter the mutant inventory. All mutants surviving with a 0.00% score suggests fallback to the `vstest` runner, while any non-zero `NoCoverage` count suggests `coverage-analysis: off` was lost. + `-p` selects the mutated project, not the tests: Stryker runs every test project transitively referencing it (`AspNetCore.Shared` → 337 tests, `Light.PortableResults` → all 2,398). Cross-project kills are legitimate (sociable tests) and nearly free. Never pass `-tp` (it does not narrow tests) or `--since` (any non-C# file in the diff, e.g. an `ai-plans/` document, degrades it to a full run). Reports go to `StrykerOutput//reports/` (gitignored): JSON for agents (filter `"status": "Survived"`), HTML for humans. ### Cost and baseline From c6b81b548a90b545bee2a9f31c53618d7bd1702b Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 21:32:19 +0200 Subject: [PATCH 12/13] fix(tests): prevent Stryker timeout masking --- ai-plans/0063-1-plan-deviations.md | 41 ++++++++++++++++++++++-------- stryker-config.json | 1 + tests/AGENTS.md | 18 +++++++------ 3 files changed, 41 insertions(+), 19 deletions(-) diff --git a/ai-plans/0063-1-plan-deviations.md b/ai-plans/0063-1-plan-deviations.md index 8c01ff6..1c183d1 100644 --- a/ai-plans/0063-1-plan-deviations.md +++ b/ai-plans/0063-1-plan-deviations.md @@ -1,34 +1,40 @@ # 0063 Plan Deviations -This document compares `ai-plans/0063-add-stryker-mutation-testing.md` with the results measured while implementing it (`dotnet-stryker` 4.16.0, Apple M3 Max, 16 logical cores, source tree at commit `04aee20`). +This document compares `ai-plans/0063-0-add-stryker-mutation-testing.md` with the results measured while implementing it (`dotnet-stryker` 4.16.0, Apple M3 Max, 16 logical cores, source tree at commit `04aee20`). ## Summary All acceptance criteria were implemented and verified as specified: the pinned tool, the committed configuration, the scoped invocations, the transitive test counts, the git-ignored output containment, the recorded baseline, and the absence of any CI or threshold changes. -The material deviation is in the plan's Technical Details narrative about survivors: the claim of exactly one survivor across the four small projects did not reproduce. The survivor population in `Validation.OpenApi` is roughly twenty mutants, and most of it is hidden by timeout classification to a degree that varies with machine load and concurrency. The operational documentation in `tests/AGENTS.md` already reflects the measured behavior; this file records the delta from the plan text. +The material deviations are in the plan's Technical Details narrative about survivors and in two settings added to make the measured baseline meaningful. The claim of exactly one survivor across the four small projects did not reproduce: `Validation.OpenApi` has nineteen survivors, and the original 5,000 ms additional-timeout default hid most of them as timeouts. The committed configuration pins concurrency and raises the additional timeout so the operational report exposes that queue directly. The operational documentation in `tests/AGENTS.md` reflects the measured behavior; this file records the delta from the plan text. ## Deviations From The Original Plan -### 1. `Validation.OpenApi` has ~18–20 survivors masked as timeouts, not one survivor +### 1. `Validation.OpenApi` has nineteen survivors, not one **Original plan:** The plan states that the measured runs produced exactly one survivor across all four small projects — a statement-removal mutant at `BuiltInValidationErrorBuilderExtensions.cs:426`, putting `Validation.OpenApi` at 98.68% — and that the triage categories therefore "remain largely a priori rather than derived from survivors observed in this codebase." **Measured:** -Three runs of the same unmodified sources produced three different survivor sets: +With Stryker's 5,000 ms default additional timeout, three runs of the same unmodified sources produced three different survivor sets: - Concurrency 8: 54 killed, 21 timeout, 1 survived — 98.68%. The survivor was an equality mutant at `BuiltInValidationErrorBuilderExtensions.cs:388`, not the statement removal at line 426. - Concurrency 8, repeat: 51 killed, 21 timeout, 4 survived — 94.74%. - Concurrency 4: 56 killed, 2 timeout, 18 survived — 76.32%. +Increasing the additional timeout exposed the stable population much more directly at concurrency 8: + +- 15,000 ms: 57 killed, 1 timeout, 18 survived — 76.32%, 2:30 elapsed. +- 20,000 ms: two runs both reported 57 killed, with 0–1 timeout and 18–19 survived — 75.00–76.32%, 2:34–2:38 elapsed. The mutant that timed out in the second run survived in the first. +- 30,000 ms: two consecutive runs both reported 57 killed, 0 timeout, and 19 survived — 75.00%, 2:32–2:38 elapsed. + The survivors cluster in `BuiltInValidationErrorBuilderExtensions.cs` lines 367–426 (conditional, equality, object-initializer, and statement mutants around the built-in validation error OpenAPI schema customization), with a few more in `BuiltInValidationErrorContracts.cs`, `BuiltInValidationErrorContractRegistrationExtensions.cs`, and `PortableValidationOpenApiRouteHandlerBuilderExtensions.cs`. **Why:** -With `coverage-analysis: off`, every mutant runs the full discovered test set (163 tests for this project). A killed mutant's run aborts at the first failing test, but a genuinely surviving mutant runs the entire suite — and can exceed the per-mutant timeout. `Timeout` counts as killed in the mutation score, so under higher concurrency (more host contention) slow survivors are reclassified as timeouts and the score is inflated. The plan's "one survivor, 98.68%" is one point in this timing-dependent distribution, not a stable property of the test suite. +With `coverage-analysis: off`, every mutant runs the full discovered test set (163 tests for this project). A killed mutant's run aborts at the first failing test, but a genuinely surviving mutant runs the entire suite and can exceed the timeout under load. Stryker 4.16.0 calculates that timeout from the measured initial run plus `additional-timeout`, which defaults to 5,000 ms and is configurable only in the config file. `Timeout` counts as killed in the mutation score, so insufficient headroom reclassifies slow survivors as timeouts and inflates the score. The plan's "one survivor, 98.68%" is one point in this timing-dependent distribution, not a stable property of the suite. **Impact:** -The initial triage queue is ~18–20 mutants in that file, not one, and the triage categories are no longer a priori — they now have real input. No acceptance criterion is affected: criterion 7 deliberately requires recording the timeout count alongside the others, and the baseline in `tests/AGENTS.md` carries the measured vector. The blind-spots section there directs agents to treat `Timeout` as a triage signal equal to `Survived` for this project. Future comparisons should match full count vectors at equal concurrency rather than percentages. +The initial triage queue is nineteen mutants, not one, and the triage categories are no longer a priori — they now have real input. The 30,000 ms committed setting reduces the timeout count from 21 to zero on the measured hardware without changing the mutant inventory. No acceptance criterion is affected: criterion 7 deliberately requires recording the timeout count alongside the others, and the baseline in `tests/AGENTS.md` carries the measured vector. Future timeouts remain triage signals because no finite headroom can make classification independent of hardware and load. Future comparisons should match full count vectors at equal concurrency and additional timeout rather than percentages. ### 2. The recorded baseline's provenance commit predates the tooling commit @@ -50,17 +56,30 @@ The plan specifies the exact contents of `stryker-config.json` (`solution`, `tes `stryker-config.json` additionally pins `"concurrency": 8`, and the documented invocations pass no `-c`. **Why:** -The result vector is concurrency-sensitive (§1: `Validation.OpenApi` reports 2 timeouts/18 survivors at concurrency 4 vs. 21/1 at 8), so a baseline is only comparable at equal concurrency — making concurrency a load-bearing setting, which the plan's own philosophy places in the configuration rather than the invocation. Relying on the default is not an alternative: 4.16.0 defaults to half the logical processors (verified via the debug options dump; the `--help` text claiming "as many parallel processes as you have CPU cores" is stale), which is 8 on the recording machine but 4 on an 8-core machine. The pinned value reproduces the baseline on any machine with at least 8 logical cores and leaves `-c` free for experiments, e.g. lowering it to unmask slow survivors. Verified to take effect: with `"concurrency": 3` in the config and no `-c`, the debug options dump reports `"Concurrency": 3`. +The result vector is concurrency-sensitive (§1: `Validation.OpenApi` reports 2 timeouts/18 survivors at concurrency 4 vs. 21/1 at 8), so a baseline is only comparable at equal concurrency — making concurrency a load-bearing setting, which the plan's own philosophy places in the configuration rather than the invocation. Relying on the default is not an alternative: 4.16.0 defaults to half the logical processors (verified via the debug options dump; the `--help` text claiming "as many parallel processes as you have CPU cores" is stale), which is 8 on the recording machine but 4 on an 8-core machine. The pinned value ensures the same configured parallelism on machines with at least 8 logical cores, but does not make timing independent of hardware or load; it also leaves `-c` free for experiments. Verified to take effect: with `"concurrency": 3` in the config and no `-c`, the debug options dump reports `"Concurrency": 3`. + +### 4. The committed configuration pins `additional-timeout: 30000`, which the plan did not specify + +**Original plan:** +The plan leaves Stryker's additional timeout unset, so 4.16.0 uses its 5,000 ms default. It treats timeout masking as a blind spot to document rather than identifying the setting that controls it. + +**Implemented:** +`stryker-config.json` additionally pins `"additional-timeout": 30000`. The option is milliseconds of headroom added to Stryker's timeout derived from the initial test run; it is config-file-only in 4.16.0 and absent from `--help`. + +**Why:** +The default systematically hid slow survivors under `coverage-analysis: off`. At concurrency 8, 15,000 ms reduced `Validation.OpenApi` from 21 timeouts to one but still timed out a mutant known to survive, and one of two 20,000 ms runs did the same. Two consecutive 30,000 ms runs reported zero timeouts and exposed all 19 survivors. The higher setting also reclassified `AspNetCore.Mvc`'s former timeout at `BaseLightActionResult.cs:82` as a survivor. It does not make the result hardware-independent, but it removes all known masking on the measured hardware without changing concurrency or requiring readers to reinterpret dozens of timeouts. + +**Impact:** +The four small projects were remeasured under the committed setting and the baseline in `tests/AGENTS.md` was replaced. Genuine hangs can now take up to 25 seconds longer per affected test assembly than under the default, while ordinary killed mutants still stop at the first failing test. The measured small-project runs remain within minutes. A future timeout is still a triage signal, not automatically a killed mutant. ## Notes On Items Implemented As Planned -The following measurements from the plan reproduced exactly or within seconds: +The following measurements remained unchanged after the timeout correction: - Mutant inventories and test counts for all four small projects: 33/32/44/114 mutants, 11/11/3/2 compile errors, 102/237/337/163 tests (`AspNetCore.Mvc`, `AspNetCore.MinimalApis`, `AspNetCore.Shared`, `Validation.OpenApi`). - `AspNetCore.Shared` reporting 337 tests (all six transitively referencing test projects) and `Result.cs` reporting 2,398 tests with 67 killed, 0 `NoCoverage`, 100.00%. -- Elapsed times: 0:40 / 0:58 / 1:57 / 2:06 measured vs. 0:38 / 0:57 / 1:57 / 1:59 in the plan. -- `AspNetCore.Mvc`, `AspNetCore.MinimalApis`, and `AspNetCore.Shared` each scoring 100.00%. -- `StrykerOutput/` containment (eight runs, no `git status` entries), no CI changes, and no `thresholds`/`break-at` settings anywhere. +- `AspNetCore.MinimalApis` and `AspNetCore.Shared` scoring 100.00%. `AspNetCore.Mvc` now scores 94.12% because its former timeout is correctly reported as survived. +- `StrykerOutput/` containment across repeated runs, no CI changes, and no `thresholds`/`break-at` settings anywhere. ## Minor Operational Notes diff --git a/stryker-config.json b/stryker-config.json index db81bd4..c9702d3 100644 --- a/stryker-config.json +++ b/stryker-config.json @@ -5,6 +5,7 @@ "coverage-analysis": "off", "configuration": "Debug", "concurrency": 8, + "additional-timeout": 30000, "reporters": ["json", "html", "progress"] } } diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 6ad1a78..af3f784 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -26,7 +26,9 @@ Each test project writes `TestResults/.cobertura.xml`. Merge them with `re ### How to run -Always run from the repository root: Stryker reads `stryker-config.json` only from the current directory and silently falls back to defaults without it, reverting to the `vstest` runner (cannot drive the xUnit v3 hosts, reports everything survived) and `perTest` coverage analysis (fabricates `NoCoverage`). The config also pins `Debug` (`TreatWarningsAsErrors` is Release-only) and `concurrency: 8` (result vectors vary with parallelism; override with `-c` for experiments). +Always run from the repository root: Stryker reads `stryker-config.json` only from the current directory and silently falls back to defaults without it, reverting to the `vstest` runner (cannot drive the xUnit v3 hosts, reports everything survived) and `perTest` coverage analysis (fabricates `NoCoverage`). The config also pins `Debug` (`TreatWarningsAsErrors` is Release-only), `concurrency: 8` (result vectors vary with parallelism; override with `-c` for experiments), and `additional-timeout: 30000` (the 5,000 ms default masks slow survivors as timeouts under load). + +`additional-timeout` is config-file-only in 4.16.0 and is milliseconds of headroom added to Stryker's timeout derived from the initial test run, not the total timeout. Raising it makes genuine hangs take longer to classify, but the measured value exposes all known slow survivors while keeping the four small-project runs within minutes. Revisit `coverage-analysis: off` when Stryker's MTP per-test coverage support ([#3516](https://github.com/stryker-mutator/stryker-net/pull/3516)) ships and proves trustworthy; it is the main lever on run time because it avoids running the full discovered test set for every mutant. @@ -58,16 +60,16 @@ The smoke-check vector is tied to the current `Result.cs` and its tests; update | `AspNetCore.Mvc` | 33 | 11 | | `AspNetCore.MinimalApis` | 32 | 11 | -Baseline measured at commit `04aee20`, `dotnet-stryker` 4.16.0, concurrency 8 (pinned in `stryker-config.json`; the 4.16.0 default is half the logical cores and therefore machine-dependent), `Debug`, Apple M3 Max (16 logical cores): +Baseline for sources at commit `04aee20`, remeasured with `dotnet-stryker` 4.16.0, concurrency 8 and 30,000 ms additional timeout (both pinned in `stryker-config.json`), `Debug`, Apple M3 Max (16 logical cores): | Project | Tests run | Elapsed | Killed | Timeout | Survived | CompileError | Ignored | NoCoverage | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| `AspNetCore.Mvc` | 102 | 0:40 | 16 | 1 | 0 | 11 | 5 | 0 | -| `AspNetCore.MinimalApis` | 237 | 0:58 | 16 | 0 | 0 | 11 | 5 | 0 | -| `AspNetCore.Shared` | 337 | 1:57 | 31 | 0 | 0 | 3 | 10 | 0 | -| `Validation.OpenApi` † | 163 | 2:06 | 54 | 21 | 1 | 2 | 36 | 0 | +| `AspNetCore.Mvc` | 102 | 0:53 | 16 | 0 | 1 | 11 | 5 | 0 | +| `AspNetCore.MinimalApis` | 237 | 1:22 | 16 | 0 | 0 | 11 | 5 | 0 | +| `AspNetCore.Shared` | 337 | 2:29 | 31 | 0 | 0 | 3 | 10 | 0 | +| `Validation.OpenApi` | 163 | 2:38 | 57 | 0 | 19 | 2 | 36 | 0 | -† This row is not exactly reproducible. With `coverage-analysis: off` every mutant runs the full 163-test suite, so the Killed/Timeout/Survived split is machine-load dependent (see the timeout-masking blind spot): two runs of identical sources at concurrency 8 gave Killed 51–54, Timeout 21, Survived 1–4. Tests run, CompileError, and Ignored are deterministic. For triage rather than comparison, run this project with `-c 4` to unmask the slow full-suite survivors (56 killed, 2 timeout, 18 survived). +`Validation.OpenApi` was run twice consecutively with the 30,000 ms setting; both runs produced 57 killed, 0 timeout, 19 survived, and a 75.00% score, completing in 2:32 and 2:38. At 20,000 ms, one of two runs still timed out a mutant known to survive. At the 5,000 ms default, 21 mutants were reported as timeouts, masking most survivors and inflating the score to 98.68%. `Ignored` is not user suppression: all 56 baseline entries are `Block removal` mutants discarded deterministically by Stryker's built-in "block already covered" filter because another active mutant exists inside the block. A different reason or count should be investigated. @@ -86,5 +88,5 @@ Never restructure production code to make a mutant killable: performance outrank ### Blind spots — do not read as adequate coverage - ~9.5% of mutants fail to compile (mostly the `out`/`ref` style); Stryker's Safe Mode then discards every mutant in the enclosing method: `Dragon4.GenerateDigits`, `ResultJsonReader.ReadStatusValue`/`ReadIndexValue`, `ErrorsExtensions.WriteRichErrors`, and all of `LightResult.cs` receive no mutation coverage. Tool limitation, not a test defect — a high score in `Numbers/` is not verified behavior. -- `Timeout` counts as killed, but full-suite survivors can exceed the per-mutant timeout under load: `Validation.OpenApi`'s 21 timeouts at `-c 8` become 18 survivors at `-c 4` (76.32%). Treat `Timeout` like `Survived` in that project. +- `Timeout` counts as killed. The pinned 30,000 ms additional timeout reduced `Validation.OpenApi` from 21 timeouts to zero in two consecutive concurrency-8 runs, but no finite value makes classification independent of hardware and load. Investigate any future timeout as either a genuine hang or insufficient headroom; do not assume it represents a killed mutant. - The MTP runner is a preview (stryker-mutator/stryker-net#3094); verify surprising results against a plain `dotnet test` run. From cf29e5979accd201a0e84e6c206a772a690f056e Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 21:36:44 +0200 Subject: [PATCH 13/13] docs(tests): include timeout mutants in Stryker triage --- tests/AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index af3f784..aa3e418 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -43,7 +43,7 @@ dotnet stryker -p Light.PortableResults.csproj -m '**/Result.cs' The smoke-check vector is tied to the current `Result.cs` and its tests; update it after intentional changes alter the mutant inventory. All mutants surviving with a 0.00% score suggests fallback to the `vstest` runner, while any non-zero `NoCoverage` count suggests `coverage-analysis: off` was lost. -`-p` selects the mutated project, not the tests: Stryker runs every test project transitively referencing it (`AspNetCore.Shared` → 337 tests, `Light.PortableResults` → all 2,398). Cross-project kills are legitimate (sociable tests) and nearly free. Never pass `-tp` (it does not narrow tests) or `--since` (any non-C# file in the diff, e.g. an `ai-plans/` document, degrades it to a full run). Reports go to `StrykerOutput//reports/` (gitignored): JSON for agents (filter `"status": "Survived"`), HTML for humans. +`-p` selects the mutated project, not the tests: Stryker runs every test project transitively referencing it (`AspNetCore.Shared` → 337 tests, `Light.PortableResults` → all 2,398). Cross-project kills are legitimate (sociable tests) and nearly free. Never pass `-tp` (it does not narrow tests) or `--since` (any non-C# file in the diff, e.g. an `ai-plans/` document, degrades it to a full run). Reports go to `StrykerOutput//reports/` (gitignored): JSON for agents (filter `"status"` for both `"Survived"` and `"Timeout"`; investigate the timeout cause before survivor triage), HTML for humans. ### Cost and baseline