Stryker.NET Support - #64
Merged
Merged
Conversation
Introduces Stryker.NET as a pinned, local-only tool so coding agents can identify missing tests beyond what line coverage proves. The plan records two settings that must be committed rather than passed ad hoc, because each produces a confidently wrong report on this solution: the default vstest runner cannot drive the xUnit v3 hosts and reports every mutant as survived, and the MTP runner's default coverage analysis fabricates NoCoverage results for code that is demonstrably tested. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PwpEBWeQhetkTu6CdQkuqm
Corrects four claims that did not survive verification against Stryker 4.16.0: - Solution mode: -p selects the mutated source project but does not narrow test execution, so -tp is removed from the documented command. Mutating AspNetCore.Shared runs 337 tests, the exact sum of the six test projects referencing it, not the 26 in its own test project. This is kept rather than worked around, because cross-project kills are what the sociable testing rule in tests/AGENTS.md asks for. - Artifact containment: the criterion demanded both an ignored StrykerOutput/ and an empty working tree. Reports have to be written somewhere, so the contract is now containment plus a clean git status. - --since is dropped. Measured on a clean working tree, it still reported a Markdown plan as a changed test file and escalated to testing every mutant, so it would degrade to a full run on every branch while appearing scoped. - Survivors are a triage queue, not a defect list. Equivalent and invalid mutants get narrow suppression with a justification; production code is never restructured to raise the score. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PwpEBWeQhetkTu6CdQkuqm
Replaces estimated costs with end-to-end measurements of the four small projects and tightens several points that were left implicit: - The tool manifest must pin dotnet-stryker to exactly 4.16.0, and the scoped invocation takes only -p plus an optional -m glob. The criterion still described a -tp argument removed in the previous commit. - configuration is pinned to Debug in stryker-config.json rather than left to the invoker, since Release would build with warnings as errors. - Every command runs from the repository root. Stryker resolves stryker-config.json against the working directory, does not search parent directories, and treats a missing config as non-fatal, so a run started elsewhere silently reverts to the vstest runner and perTest coverage - the pair that reports 0.00% or fabricates NoCoverage. - Baselines must carry provenance - commit, tool version, concurrency, executed test count, elapsed time, and the killed, timeout, survived, compile-error, ignored, and no-coverage counts - rather than a percentage alone. No-coverage is expected to be zero, and a non-zero value means coverage-analysis stopped taking effect; it was zero in all four runs. - The artifact criterion now requires that a run add no new entries to git status, which stays verifiable while implementation changes are present. The timing claim is reconciled against measurement rather than arithmetic. Validation.OpenApi completes in 1:59, not the six minutes that 114 mutants at three seconds each would suggest: only 76 of its 114 mutants are tested, and per-mutant cost falls with the size of the associated test set. The three-seconds-per-mutant rate is now labelled an upper bound taken from the large projects. These runs also produced the first survivor, a statement-removal mutant at BuiltInValidationErrorBuilderExtensions.cs:426 that puts Validation.OpenApi at 98.68%. It is recorded as untriaged, and it corrects the claim in the previous commit that no run had produced a survivor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PwpEBWeQhetkTu6CdQkuqm
Pin dotnet-stryker 4.16.0 via the local tool manifest and commit a repository-root stryker-config.json that selects the MTP test runner (vstest cannot drive the xUnit v3 hosts) and disables coverage analysis (MTP perTest analysis fabricates NoCoverage). Ignore StrykerOutput/ so runs leave git status clean. tests/AGENTS.md documents the repo-root scoped invocations, survivor triage rules, per-project costs, the measured four-project baseline, and the blind spots (Safe Mode method discards, timeout masking). Plan deviations record that Validation.OpenApi actually yields ~18-20 survivors masked as timeouts rather than the single survivor the plan anticipated.
The result vector is concurrency-sensitive (Validation.OpenApi: 2 timeouts/18 survivors at -c 4 vs 21/1 at -c 8), so a baseline is only comparable at equal concurrency. Pin concurrency: 8 in stryker-config.json instead of passing -c ad hoc: the documented -p-only invocation now reproduces the baseline on any machine with at least 8 logical cores, and -c stays free for experiments. Note: the 4.16.0 --help text claims the default is the CPU core count, but the actual default is half the logical processors (verified via the debug options dump), i.e. machine-dependent.
Footnote the row with the observed variance instead of presenting one vector unqualified: the Killed/Timeout/Survived split is machine-load dependent under coverage-analysis: off, while Tests, CompileError, and Ignored are deterministic. Point triage runs at -c 4 to unmask the slow full-suite survivors.
Signed-off-by: Kenny Pflug <kenny.pflug@live.de>
8 tasks
Minimum allowed line rate is |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #63