Skip to content

Check that every documented status is produced by a test - #241

Merged
Deval123 merged 2 commits into
mainfrom
test/openapi-status-coverage
Sep 25, 2026
Merged

Deval123 merged 2 commits into
mainfrom
test/openapi-status-coverage

Conversation

@Deval123

Copy link
Copy Markdown
Owner

What changed

OpenApiSpecIT checked paths and methods in both directions, but statuses in only one. A test producing an undocumented status failed. A documented status that no test produced passed silently. Now both directions are checked.

  • assertStatusDocumented, the method every per-status test reaches, also records the (method, path, status) it saw.
  • After the class, the documented set must equal what was recorded plus PROVED_ELSEWHERE. The documented set is every key under paths.<path>.<method>.responses in docs/openapi.yaml. PROVED_ELSEWHERE is a named map from each status this context cannot produce to the class that proves it. Today it holds the two 501s, proved in FeatureNotOfferedApiIT.
  • Three failures are reported separately: a documented status nobody produces, a stale PROVED_ELSEWHERE entry, and a response key that isn't a status code (such as default). No single test can produce the last kind, so the check fails on it rather than dropping it. The spec has none today.
  • WholeClassRun, in server/src/test/java/dev/nkap/server/support/, is a general JUnit extension with nothing OpenAPI-specific in it. It counts the tests that ran against the @Test methods declared, including inherited ones. It runs the check only when the whole class ran and passed, and logs one line when it skips. It refuses a class with parameterized, repeated or dynamic tests rather than miscounting them.
  • The javadoc sentence calling coverage "a convention maintained by review" is replaced by what the mechanism is and what it doesn't cover.

Seen failing

Each change below was reverted afterwards, and each run was the whole class, 44 tests:

  1. A status no test produces: I added "418" to GET /payments/{reference} in the spec. The check failed with documented in docs/openapi.yaml, but no test in OpenApiSpecIT produces it … ["GET /payments/{reference} 418"].
  2. A missing per-status test: I removed @Test from get_payment_404. The check failed with ["GET /payments/{reference} 404"].
  3. A missing exception: I removed GET /balance 501 from PROVED_ELSEWHERE. The check failed with ["GET /balance 501"] as documented but unproduced.
  4. A stale exception: I removed 501 from GET /balance in the spec and kept the PROVED_ELSEWHERE entry. The check failed with named in PROVED_ELSEWHERE, but docs/openapi.yaml no longer documents it: a stale exception … ["GET /balance 501"]. It was reported only as stale.
  5. One test alone: I ran OpenApiSpecIT#get_payment_404 by itself. It passed (Tests run: 1), and this line was logged: OpenApiSpecIT: 1 of 44 tests ran, a partial run, so per-status coverage of docs/openapi.yaml was not checked. Run the whole class to check it.

WholeClassRunTest also covers both branches directly: a whole run runs the check, a partial run skips it and logs, a failed test skips it and logs, a failing check fails the class, and an uncountable test is refused.

Measured and assumed

Measured:

  • The four tests that call assertStatusDocumented directly are exactly the ones the plan named.
  • One more test asserts a status outside the funnel: post_callback_with_reference_settles asserts 202. It's a behaviour test, and that status is recorded by post_callback_with_reference_202, so it leaves no gap.
  • The spec documents 43 numeric statuses and no other keys. On a whole run, all 41 that can be produced here are recorded, so no per-status test is missing.
  • mvn -B clean verify passes on the committed tree, with the check run in full.

Assumed: Surefire's default exclusion of nested classes keeps WholeClassRunTest's fixture classes from running as tests. The report confirms it: 6 tests run, none from the fixtures.

Not measured: The JUnit Platform test kit isn't on the classpath, so rather than add a dependency, the extension's tests call its callbacks directly with a mocked ExtensionContext. The real one-test run is item 5 above.

One addition to the plan: it said "equal counts: the assertion runs". This also skips the check, with its own logged line, when the whole class ran but a test failed. A failed test may stop before recording its status, so the check would report a gap that is only that failure's echo, and the build is already red.

Closes #230

OpenApiSpecIT checked paths and methods in both directions, but
statuses in one: a test producing an undocumented status failed, and
a documented status no test produced passed. Its javadoc said so,
calling per-status coverage a convention maintained by review.

assertStatusDocumented, which every per-status test reaches, now also
records the (method, path, status) it saw. After the class, the
documented set must equal what was recorded plus PROVED_ELSEWHERE, a
named map from each status this context cannot produce to the class
that proves it. A documented status nobody produces and a stale
PROVED_ELSEWHERE entry are reported separately. A response key that
is not a status code fails too, since no single test can produce it.

A check about a whole class is only true of a whole run, and one that
fails whenever someone runs a single test gets switched off. So
WholeClassRun counts the tests that ran against the @test methods the
class declares, runs the check only on a whole, passing run, and logs
one line when it skips. It knows nothing about OpenAPI and sits in
the test support package.

Signed-off-by: Devalère <28451130+Deval123@users.noreply.github.com>
The five failures shown in the pull request were made by hand and
reverted, so nothing kept them. The rule is three sets and nothing
else, so it is now tested directly, with every set built by hand and
none taken from OpenApiSpecIT.PROVED_ELSEWHERE: an unproduced status
fails naming it, the same status proved elsewhere passes, a stale
exception fails as that and as nothing else, a default response key
fails as not a status code, and the clean case passes. The rule moves
into StatusCoverage to be tested at all: OpenApiSpecIT starts an
embedded simulator when it is loaded, so calling it from a unit test
would boot a server.

A test aborted by an assumption records nothing, so WholeClassRun
skips the check, as for a failure. But Surefire reports an abort as a
skip, the build stays green, and nobody learns the check did not run.
The skip stays; it is now a warning, and the javadoc says an
assumption in such a class can switch the check off unseen. A
disabled test still counts as finished and not failed, so the check
runs and reports the coverage it lost; the javadoc says that too.

Signed-off-by: Devalère <28451130+Deval123@users.noreply.github.com>
@Deval123 Deval123 self-assigned this Sep 25, 2026
@Deval123
Deval123 merged commit aa115a1 into main Sep 25, 2026
7 checks passed
@Deval123
Deval123 deleted the test/openapi-status-coverage branch September 25, 2026 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nothing fails when docs/openapi.yaml documents a status no test produces

1 participant