From 144e4f35baf2dbedb08c2d2677c1483dddffd027 Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 21:57:26 +0200 Subject: [PATCH 1/2] docs(tests): forbid mutant-driven assertions in Stryker triage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The triage rules told agents to add a test for an unconstrained survivor but said nothing about what that test may assert. The tempting move — pinning a mutant down by asserting on incidental output — raises the score once and then constrains an implementation detail forever, which is worse than leaving the mutant alive. State the test-side companion to the existing rule against restructuring production code: a test must read as a statement about the contract, must never name a mutant, and a survivor killable only by incidental assertions is an equivalent mutant to suppress rather than a test to write. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0184eh1G3XxvjRB11cNKyMRT --- tests/AGENTS.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index aa3e418..9161b80 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -85,6 +85,10 @@ Exactly three permitted responses to a survivor: Never restructure production code to make a mutant killable: performance outranks mutation score here, and the low-allocation `in`/`ref`/`Span` style produces awkward survivors by nature. +The same discipline applies to test code. A test written under response 1 must stand on its own as a statement about the contract: name it for the behavior it pins down, and assert only on what the public API promises. Nothing in a test may refer to a mutant — no mutant ID, no line number, no mention of Stryker in a name, comment, or assertion message. The source suppression from response 2 is the only place a mutant is named. + +If a survivor can only be killed by asserting on something incidental — exact message composition, member or property ordering, a call count, a value the contract does not fix — it is response 2, not response 1. Suppress it with that reasoning. Such a test raises the score once and then constrains an implementation detail forever, which is a worse position than the survivor: the next legitimate refactoring breaks it, and the failure carries no information about the contract. + ### Blind spots — do not read as adequate coverage - ~9.5% of mutants fail to compile (mostly the `out`/`ref` style); Stryker's Safe Mode then discards every mutant in the enclosing method: `Dragon4.GenerateDigits`, `ResultJsonReader.ReadStatusValue`/`ReadIndexValue`, `ErrorsExtensions.WriteRichErrors`, and all of `LightResult.cs` receive no mutation coverage. Tool limitation, not a test defect — a high score in `Numbers/` is not verified behavior. From 35809d26e816bd40f3bc8fcb2b4256614ec2d25c Mon Sep 17 00:00:00 2001 From: Kenny Pflug Date: Sat, 1 Aug 2026 22:02:18 +0200 Subject: [PATCH 2/2] docs(tests): make the Stryker Safe Mode blind spot actionable The blind-spot entry named five methods that Safe Mode leaves without mutation coverage, but stopped at the diagnosis. The names read as a fixed list when they are one run's observation, and nothing told a reader what to do when editing such a method. Label the list as observed and give a durable way to recover the current one: the Safe Mode announcements are console-only and the discarded mutants are absent from the JSON report, so the surviving trace is the CompileError sites. Require test adequacy for those methods to be argued by hand in the pull request, since neither mutation score nor line coverage carries information there. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0184eh1G3XxvjRB11cNKyMRT --- tests/AGENTS.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 9161b80..9999266 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -91,6 +91,10 @@ If a survivor can only be killed by asserting on something incidental — exact ### Blind spots — do not read as adequate coverage -- ~9.5% of mutants fail to compile (mostly the `out`/`ref` style); Stryker's Safe Mode then discards every mutant in the enclosing method: `Dragon4.GenerateDigits`, `ResultJsonReader.ReadStatusValue`/`ReadIndexValue`, `ErrorsExtensions.WriteRichErrors`, and all of `LightResult.cs` receive no mutation coverage. Tool limitation, not a test defect — a high score in `Numbers/` is not verified behavior. +- ~9.5% of mutants fail to compile (mostly the `out`/`ref` style); Stryker's Safe Mode then discards every mutant in the enclosing method, which receives no mutation coverage at all. Tool limitation, not a test defect — a high score in `Numbers/` is not verified behavior. Observed at the baseline: `Dragon4.GenerateDigits`, `ResultJsonReader.ReadStatusValue`/`ReadIndexValue`, `ErrorsExtensions.WriteRichErrors`, and all of `LightResult.cs` (11 of 11 mutants). + + Treat that list as observed, not fixed: any new `out`/`ref` code joins it silently. Stryker announces it as `[INF] Safe Mode! Stryker will remove all mutations in ` on the console only — no log file is written — and the discarded mutants are simply absent from the JSON report. The durable way to recover the current set is to filter the report for `"status": "CompileError"`; those sites are the only trace left, and their enclosing methods are the ones running blind. + + When changing a method in that set, mutation score carries no information about it and line coverage only proves execution. Adequacy has to be argued by hand: enumerate the behaviors the method promises and point at the test constraining each one. State that reasoning in the pull request, because no tool in this repository can check it. - `Timeout` counts as killed. The pinned 30,000 ms additional timeout reduced `Validation.OpenApi` from 21 timeouts to zero in two consecutive concurrency-8 runs, but no finite value makes classification independent of hardware and load. Investigate any future timeout as either a genuine hang or insufficient headroom; do not assume it represents a killed mutant. - The MTP runner is a preview (stryker-mutator/stryker-net#3094); verify surprising results against a plain `dotnet test` run.