Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion tests/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,16 @@ Exactly three permitted responses to a survivor:

Never restructure production code to make a mutant killable: performance outranks mutation score here, and the low-allocation `in`/`ref`/`Span` style produces awkward survivors by nature.

The same discipline applies to test code. A test written under response 1 must stand on its own as a statement about the contract: name it for the behavior it pins down, and assert only on what the public API promises. Nothing in a test may refer to a mutant — no mutant ID, no line number, no mention of Stryker in a name, comment, or assertion message. The source suppression from response 2 is the only place a mutant is named.

If a survivor can only be killed by asserting on something incidental — exact message composition, member or property ordering, a call count, a value the contract does not fix — it is response 2, not response 1. Suppress it with that reasoning. Such a test raises the score once and then constrains an implementation detail forever, which is a worse position than the survivor: the next legitimate refactoring breaks it, and the failure carries no information about the contract.

### Blind spots — do not read as adequate coverage

- ~9.5% of mutants fail to compile (mostly the `out`/`ref` style); Stryker's Safe Mode then discards every mutant in the enclosing method: `Dragon4.GenerateDigits`, `ResultJsonReader.ReadStatusValue`/`ReadIndexValue`, `ErrorsExtensions.WriteRichErrors`, and all of `LightResult.cs` receive no mutation coverage. Tool limitation, not a test defect — a high score in `Numbers/` is not verified behavior.
- ~9.5% of mutants fail to compile (mostly the `out`/`ref` style); Stryker's Safe Mode then discards every mutant in the enclosing method, which receives no mutation coverage at all. Tool limitation, not a test defect — a high score in `Numbers/` is not verified behavior. Observed at the baseline: `Dragon4.GenerateDigits`, `ResultJsonReader.ReadStatusValue`/`ReadIndexValue`, `ErrorsExtensions.WriteRichErrors`, and all of `LightResult.cs` (11 of 11 mutants).

Treat that list as observed, not fixed: any new `out`/`ref` code joins it silently. Stryker announces it as `[INF] Safe Mode! Stryker will remove all mutations in <method>` on the console only — no log file is written — and the discarded mutants are simply absent from the JSON report. The durable way to recover the current set is to filter the report for `"status": "CompileError"`; those sites are the only trace left, and their enclosing methods are the ones running blind.

When changing a method in that set, mutation score carries no information about it and line coverage only proves execution. Adequacy has to be argued by hand: enumerate the behaviors the method promises and point at the test constraining each one. State that reasoning in the pull request, because no tool in this repository can check it.
- `Timeout` counts as killed. The pinned 30,000 ms additional timeout reduced `Validation.OpenApi` from 21 timeouts to zero in two consecutive concurrency-8 runs, but no finite value makes classification independent of hardware and load. Investigate any future timeout as either a genuine hang or insufficient headroom; do not assume it represents a killed mutant.
- The MTP runner is a preview (stryker-mutator/stryker-net#3094); verify surprising results against a plain `dotnet test` run.
Loading