Check that every documented status is produced by a test - #241
Merged
Merged
Conversation
OpenApiSpecIT checked paths and methods in both directions, but statuses in one: a test producing an undocumented status failed, and a documented status no test produced passed. Its javadoc said so, calling per-status coverage a convention maintained by review. assertStatusDocumented, which every per-status test reaches, now also records the (method, path, status) it saw. After the class, the documented set must equal what was recorded plus PROVED_ELSEWHERE, a named map from each status this context cannot produce to the class that proves it. A documented status nobody produces and a stale PROVED_ELSEWHERE entry are reported separately. A response key that is not a status code fails too, since no single test can produce it. A check about a whole class is only true of a whole run, and one that fails whenever someone runs a single test gets switched off. So WholeClassRun counts the tests that ran against the @test methods the class declares, runs the check only on a whole, passing run, and logs one line when it skips. It knows nothing about OpenAPI and sits in the test support package. Signed-off-by: Devalère <28451130+Deval123@users.noreply.github.com>
The five failures shown in the pull request were made by hand and reverted, so nothing kept them. The rule is three sets and nothing else, so it is now tested directly, with every set built by hand and none taken from OpenApiSpecIT.PROVED_ELSEWHERE: an unproduced status fails naming it, the same status proved elsewhere passes, a stale exception fails as that and as nothing else, a default response key fails as not a status code, and the clean case passes. The rule moves into StatusCoverage to be tested at all: OpenApiSpecIT starts an embedded simulator when it is loaded, so calling it from a unit test would boot a server. A test aborted by an assumption records nothing, so WholeClassRun skips the check, as for a failure. But Surefire reports an abort as a skip, the build stays green, and nobody learns the check did not run. The skip stays; it is now a warning, and the javadoc says an assumption in such a class can switch the check off unseen. A disabled test still counts as finished and not failed, so the check runs and reports the coverage it lost; the javadoc says that too. Signed-off-by: Devalère <28451130+Deval123@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
OpenApiSpecITchecked paths and methods in both directions, but statuses in only one. A test producing an undocumented status failed. A documented status that no test produced passed silently. Now both directions are checked.assertStatusDocumented, the method every per-status test reaches, also records the(method, path, status)it saw.PROVED_ELSEWHERE. The documented set is every key underpaths.<path>.<method>.responsesindocs/openapi.yaml.PROVED_ELSEWHEREis a named map from each status this context cannot produce to the class that proves it. Today it holds the two501s, proved inFeatureNotOfferedApiIT.PROVED_ELSEWHEREentry, and a response key that isn't a status code (such asdefault). No single test can produce the last kind, so the check fails on it rather than dropping it. The spec has none today.WholeClassRun, inserver/src/test/java/dev/nkap/server/support/, is a general JUnit extension with nothing OpenAPI-specific in it. It counts the tests that ran against the@Testmethods declared, including inherited ones. It runs the check only when the whole class ran and passed, and logs one line when it skips. It refuses a class with parameterized, repeated or dynamic tests rather than miscounting them.Seen failing
Each change below was reverted afterwards, and each run was the whole class, 44 tests:
"418"toGET /payments/{reference}in the spec. The check failed withdocumented in docs/openapi.yaml, but no test in OpenApiSpecIT produces it … ["GET /payments/{reference} 418"].@Testfromget_payment_404. The check failed with["GET /payments/{reference} 404"].GET /balance 501fromPROVED_ELSEWHERE. The check failed with["GET /balance 501"]as documented but unproduced.501fromGET /balancein the spec and kept thePROVED_ELSEWHEREentry. The check failed withnamed in PROVED_ELSEWHERE, but docs/openapi.yaml no longer documents it: a stale exception … ["GET /balance 501"]. It was reported only as stale.OpenApiSpecIT#get_payment_404by itself. It passed (Tests run: 1), and this line was logged:OpenApiSpecIT: 1 of 44 tests ran, a partial run, so per-status coverage of docs/openapi.yaml was not checked. Run the whole class to check it.WholeClassRunTestalso covers both branches directly: a whole run runs the check, a partial run skips it and logs, a failed test skips it and logs, a failing check fails the class, and an uncountable test is refused.Measured and assumed
Measured:
assertStatusDocumenteddirectly are exactly the ones the plan named.post_callback_with_reference_settlesasserts202. It's a behaviour test, and that status is recorded bypost_callback_with_reference_202, so it leaves no gap.mvn -B clean verifypasses on the committed tree, with the check run in full.Assumed: Surefire's default exclusion of nested classes keeps
WholeClassRunTest's fixture classes from running as tests. The report confirms it: 6 tests run, none from the fixtures.Not measured: The JUnit Platform test kit isn't on the classpath, so rather than add a dependency, the extension's tests call its callbacks directly with a mocked
ExtensionContext. The real one-test run is item 5 above.One addition to the plan: it said "equal counts: the assertion runs". This also skips the check, with its own logged line, when the whole class ran but a test failed. A failed test may stop before recording its status, so the check would report a gap that is only that failure's echo, and the build is already red.
Closes #230