Skip to content

test: pin whole outcomes where tests asserted fragments, counts, or catalog-derived expectations - #121

Merged
Vivswan merged 2 commits into
mainfrom
followup/weak-tests
Sep 11, 2026
Merged

test: pin whole outcomes where tests asserted fragments, counts, or catalog-derived expectations#121
Vivswan merged 2 commits into
mainfrom
followup/weak-tests

Conversation

@Vivswan

@Vivswan Vivswan commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Whole-value pins where tests asserted fragments, counts, key sets, or expectations derived from the code under test. Test files only; no source changed.

Before / After

Before (test/report/composer.test.ts):

expect(text).toContain("| Target | o/private-repo |");
expect(text).toContain("| Mode | check |");
// ... eight more toContain on the same 28-line render

After:

expect(text).toBe(
  "# settings-as-code private report: o/private-repo\n\n" +
  "Full, unredacted report for this target. ...\n\n" +
  "| | |\n|---|---|\n| Target | o/private-repo |\n| Admin repository | o/admin |\n" +
  // ... the whole document, one literal
);

Before (test/sections/loosen.test.ts, the def-surgery tripwire), which passed with loosen replaced by the identity function:

expect(viaArray.safeParse([{ name: "a", extra: 1 }]).success).toBe(true);

After:

expect(viaArray.safeParse([{ name: "a", extra: 1 }])).toEqual({
  success: true,
  data: [{ name: "a", extra: 1 }],
});

How

  • About 196 assertions strengthened across 42 files; 11 test declarations deleted, each covered by a stronger pin elsewhere.
  • Three mutation proofs: for each, a source edit the old fragment assertion missed and the new whole pin catches.
  • test/engine/orchestrate.test.ts is untouched; another builder owns it on refactor(engine): secret provenance is one source per document #119, so its 22 rows moved to the backlog.

Proof

  • bun run check green (2580 pass); bun run test:e2e 215/215.
Technical details

Counts

Survey: 89 files, 324 finding rows, 162 keep rows. Written to /tmp/fleet-7691fa85/weak-tests-survey.md on the builder's machine.

class surveyed fixed here backlog
weak-substring 215 ~96 ~119
weak-count 52 ~22 ~30
weak-shape 30 ~12 ~18
vacuous 19 18 (1 reclassified KEEP, see below) 0
redundant 13 13 0

Deleted tests and their stronger pin

  • test/sections/registry.test.ts grant-caveat map: sectionGrant is grantFor(permission, caveat), so the map test only re-derived the source; the exact-literal EXPECTED_GRANT test in the same file pins every grant string.
  • src/sections/actions_secrets/actions_secrets.test.ts not-base64 / wrong-length sealing key: test/sections/secrets-engine.test.ts parseSealingKey rejects %s pins the same defects with the same prefix.
  • test/e2e/openapi/validate.test.ts "USED_PATHS carries no undocumented path": USED_PATHS is built by excludeUndocumented, so it holds by construction; excludeUndocumented is tested directly above.
  • test/e2e/mock/graphql-pipeline.test.ts multi-mode mutation-without-node-id: the decode runs in routes.ts before the single/multi split; the single-mode test covers the branch.
  • test/e2e/mock/state.test.ts explicit-labels-replaces-baseline: the sparse-seed completion test now pins state.labels and state.autolinks as whole one-element arrays.
  • test/scripts/file-fuzz-issue.test.ts no-assignment sweep: both paths pin the whole gh argv list.
  • test/scripts/graduate-upstream-gaps.test.ts three generateIndex fragment tests plus the determinism test whose expected was generateIndex(...) itself: the committed-index byte-for-byte test and the whole-render multi-file test (unsorted input) cover them.
  • Single assertions: api.test.ts JSON-scan after a whole toEqual; discover.test.ts not.toContain after a whole toBe; checks-workflow.test.ts toContain before expectKeyPinned.

Mutation proofs (each applied, run, reverted)

  1. src/github/api.ts:568 drop in the settings file from the not-sent advice. Old toContain("was not sent") passed. New whole pins: 13 fail in api.test.ts.
  2. src/sections/contract/errors.ts:76 drop , or use a token with a higher rate limit. Old /rate limit was hit/ passed. New whole pins: 4 fail across contract, graphql-contract, execute.
  3. src/report/composer.ts:67 render input.adminRepo in the Target row. Old ten toContain passed. New whole-document pin: 1 fail in composer.test.ts.

Codex rubber-duck

Round 1: 4 blocking, 8 non-blocking. All addressed:

  • corpus pin on the real gap directory would break auto-fix graduation: reverted to the empty-tolerant sweep, reclassified KEEP.
  • deleted branches toMatch was the only pin of the actor name: both tests now pin the whole GHOST_ACTOR_ERROR literal.
  • deleted state test lost cardinality: covering test pins whole arrays.
  • [...SECTION_KEYS] on the expected side: reverted to the exact-element toContain("workflows").
  • octokit trace format, zod/ENOENT prose, insertion order, toThrow(string) substring semantics, a source-derived sorted: each reverted or tightened.

Round 2: 0 blocking, 4 non-blocking. Three fixed (age error wording unpinned, two order-incidental comparisons sorted). One recorded: test/engine/validate.test.ts verdicts embed zod's issue wording; that file already pinned a whole verdict that way before this change.

Accepted churn (deliberate)

  • test/sections/registry.test.ts: the 145-key allEndpoints() inventory is a literal. A new endpoint touches it.
  • test/e2e/mock/state.test.ts: the deploy key's minted id 90_000_002 is a literal tied to seed order.
  • test/schema-corpus.test.ts: the scenario corpus size is pinned exactly (252). The old toBeGreaterThan(150) floor was deleted first, but a loader returning only the five divergence docs still passed, so the exact count replaces it. A new scenario updates the number.

Backlog (about 186 rows, in the survey file)

  • Recorded, not fixed: test/discovery/central.test.ts (around lines 8, 31, 41) pins path.join output with forward slashes, which holds on the ubuntu-only CI and would fail on Windows. test/published-schema.test.ts (around line 267) pins the fixture list in declaration order.
  • test/engine/orchestrate.test.ts (excluded, owned by PR refactor(engine): secret provenance is one source per document #119): 22 rows. Whole pins for the preflight denial annotation and preflightDenied, the prefixed drift log line, the knobbed-section validate verdict, the unset-variable and literal-secret refusals, the unknown-top-level-key error (with the known-section list spelled out), the non-mapping and YAML-tagged document errors, the probe-error annotation, the workflows drift and notice lines, the read-denial barrier line, and the mutation lists in the mid-plan failure, thunk-failure, and warn-policy tests.
  • src/sections/*/*.test.ts: ~85 substring/count/shape rows (whole notes, live-shape messages, shapeError verdicts, ops projections). Largest: environments, repository, interaction_limits, branches.
  • test/e2e/mock/*.test.ts: ~50 rows, mostly toHaveLength(0) on violations (toEqual([]) prints the violation) and .some(includes) on violation messages.
  • test/e2e/openapi/validate.test.ts: ~28 rows, whole finding arrays instead of .some(kind ===).
  • test/docs, test/scripts: ~12 rows (diagrams rejects %s tables, graduate-upstream-gaps foreign lists, e2e-nightly if/env/run lines).

Gates

  • bun run check: green (2580 pass, biome, tsc, knip, arch lint, build:check).
  • bun run test:e2e: 215/215 passed.

@github-actions

github-actions Bot commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

File size check

23 over a hard cap (fails), 108 warning(s).

File Size Tier Cap
src/action/inputs.ts:86 477 chars hard 256
src/action/inputs.ts:93 438 chars hard 256
src/action/inputs.ts:100 583 chars hard 256
src/action/inputs.ts:165 375 chars hard 256
src/action/inputs.ts:544 338 chars hard 256
src/action/inputs.ts:572 324 chars hard 256
src/action/inputs.ts:774 274 chars hard 256
src/github/repo-file.ts:58 277 chars hard 256
test/action/inputs.test.ts:147 260 chars hard 256
test/action/inputs.test.ts:212 278 chars hard 256
test/action/inputs.test.ts:219 282 chars hard 256
test/action/inputs.test.ts:262 319 chars hard 256
test/action/multi.test.ts:282 337 chars hard 256
test/action/run.test.ts:768 326 chars hard 256
test/github/api.test.ts:741 258 chars hard 256
test/github/repo-file.test.ts:62 266 chars hard 256
test/sections/contract.test.ts:319 258 chars hard 256
test/sections/contract.test.ts:380 353 chars hard 256
test/sections/contract.test.ts:556 374 chars hard 256
test/sections/contract.test.ts:607 263 chars hard 256
test/sections/contract.test.ts:1165 263 chars hard 256
test/sections/setup-section.test.ts:59 376 chars hard 256
test/sections/setup-section.test.ts:77 304 chars hard 256
.github/scripts/gen-action-docs.ts:404 204 chars warn 150
.github/scripts/gen-docs.ts:365 190 chars warn 150
.github/scripts/release-pipeline.ts:620 160 chars warn 150
.github/scripts/release-pipeline.ts:750 180 chars warn 150
.github/scripts/release-pipeline.ts:1 40 comment lines (header) warn 25
.github/scripts/release-pipeline.ts:524 22 comment lines warn 10
.github/scripts/release-pipeline.ts:640 11 comment lines warn 10
.github/scripts/release-pipeline.ts:981 17 comment lines warn 10
.github/scripts/release-pipeline.ts:1065 14 comment lines warn 10
.github/workflows/auto-fix.yml:244 185 chars warn 150
.github/workflows/auto-fix.yml:259 155 chars warn 150
.github/workflows/auto-fix.yml:1 36 comment lines (header) warn 25
.github/workflows/post-green.yml:74 205 chars warn 150
.github/workflows/post-green.yml:44 13 comment lines warn 10
.github/workflows/update-release.yml:1 32 comment lines (header) warn 25
src/action/inputs.ts:373 154 chars warn 150
src/action/inputs.ts:374 159 chars warn 150
src/action/inputs.ts:699 217 chars warn 150
src/action/multi.ts:247 194 chars warn 150
src/action/secret-refs.ts:119 11 comment lines warn 10
src/engine/layers.ts:61 11 comment lines warn 10
src/engine/layers.ts:185 11 comment lines warn 10
src/engine/layers.ts:592 13 comment lines warn 10
src/engine/orchestrate.ts:139 249 chars warn 150
src/engine/secrets.ts:51 15 comment lines warn 10
src/engine/validate.ts:11 13 comment lines warn 10
src/report/issue-report.ts:316 13 comment lines warn 10
src/report/issue-report.ts:348 14 comment lines warn 10
src/schema.ts:1 36 comment lines (header) warn 25
src/sections/autolinks/autolinks.test.ts:22 153 chars warn 150
src/sections/collaborators/collaborators.test.ts:110 158 chars warn 150
src/sections/contract/module.ts:68 12 comment lines warn 10
src/sections/contract/module.ts:95 17 comment lines warn 10
src/sections/contract/module.ts:179 11 comment lines warn 10
src/sections/contract/module.ts:325 11 comment lines warn 10
src/sections/contract/module.ts:370 11 comment lines warn 10
src/sections/contract/module.ts:382 18 comment lines warn 10
src/sections/contract/module.ts:421 11 comment lines warn 10
src/sections/contract/module.ts:515 18 comment lines warn 10
src/sections/contract/module.ts:719 12 comment lines warn 10
src/sections/contract/permissions.ts:113 13 comment lines warn 10
src/sections/deploy_keys/deploy_keys.test.ts:315 152 chars warn 150
src/sections/environments/nested.ts:63 12 comment lines warn 10
src/sections/environments/nested.ts:173 12 comment lines warn 10
src/sections/environments/nested.ts:228 14 comment lines warn 10
src/sections/environments/schema.ts:144 11 comment lines warn 10
src/sections/shared/list-section.ts:462 232 chars warn 150
src/sections/shared/repo-secrets.ts:158 13 comment lines warn 10
src/sections/shared/schema-helpers.ts:32 16 comment lines warn 10
test/docs/claims.ts:108 11 comment lines warn 10
test/docs/guides.test.ts:324 13 comment lines warn 10
test/e2e/fuzz.ts:624 18 comment lines warn 10
test/e2e/fuzz.ts:770 14 comment lines warn 10
test/e2e/fuzz.ts:1013 12 comment lines warn 10
test/e2e/fuzz.ts:1450 12 comment lines warn 10
test/e2e/fuzz.ts:1559 12 comment lines warn 10
test/e2e/fuzz.ts:1658 11 comment lines warn 10
test/e2e/fuzz.ts:1685 11 comment lines warn 10
test/e2e/fuzz.ts:1789 11 comment lines warn 10
test/e2e/fuzz.ts:1840 13 comment lines warn 10
test/e2e/gen-support.ts:52 17 comment lines warn 10
test/e2e/gen-support.ts:150 13 comment lines warn 10
test/e2e/generators.ts 2745 lines warn 2560
test/e2e/generators.ts:86 11 comment lines warn 10
test/e2e/generators.ts:225 16 comment lines warn 10
test/e2e/generators.ts:259 19 comment lines warn 10
test/e2e/generators.ts:394 12 comment lines warn 10
test/e2e/generators.ts:809 13 comment lines warn 10
test/e2e/generators.ts:1169 12 comment lines warn 10
test/e2e/generators.ts:1231 11 comment lines warn 10
test/e2e/generators.ts:1252 32 comment lines warn 10
test/e2e/generators.ts:1317 11 comment lines warn 10
test/e2e/generators.ts:1480 13 comment lines warn 10
test/e2e/generators.ts:2295 18 comment lines warn 10
test/e2e/mock/core-paths.ts:75 16 comment lines warn 10
test/e2e/mock/core-paths.ts:173 11 comment lines warn 10
test/e2e/mock/core-paths.ts:242 13 comment lines warn 10
test/e2e/mock/core-paths.ts:373 15 comment lines warn 10
test/e2e/mock/core-paths.ts:418 13 comment lines warn 10
test/e2e/mock/core-paths.ts:588 14 comment lines warn 10
test/e2e/mock/routes.ts:343 212 chars warn 150
test/e2e/mock/routes.ts:174 17 comment lines warn 10
test/e2e/mock/routes.ts:599 11 comment lines warn 10
test/e2e/mock/state.ts:960 15 comment lines warn 10
test/e2e/mock/state.ts:1122 11 comment lines warn 10
test/e2e/openapi/paths.ts:34 16 comment lines warn 10
test/e2e/openapi/validate.ts:85 23 comment lines warn 10
test/e2e/openapi/validate.ts:416 13 comment lines warn 10
test/e2e/openapi/validate.ts:602 12 comment lines warn 10
test/e2e/oracle.ts:77 13 comment lines warn 10
test/e2e/oracle.ts:120 14 comment lines warn 10
test/e2e/oracle.ts:1033 11 comment lines warn 10
test/e2e/oracle.ts:1118 11 comment lines warn 10
test/e2e/runner.ts:59 239 chars warn 150
test/e2e/runner.ts:105 11 comment lines warn 10
test/e2e/runner.ts:480 11 comment lines warn 10
test/e2e/runner.ts:540 11 comment lines warn 10
test/e2e/runner.ts:707 15 comment lines warn 10
test/e2e/schema.ts:172 20 comment lines warn 10
test/e2e/schema.ts:296 11 comment lines warn 10
test/e2e/schema.ts:352 22 comment lines warn 10
test/e2e/schema.ts:561 11 comment lines warn 10
test/e2e/schema.ts:670 18 comment lines warn 10
test/engine/orchestrate.test.ts:144 183 chars warn 150
test/engine/orchestrate.test.ts:170 169 chars warn 150
test/scripts/release-pipeline.test.ts 2602 lines warn 2560
test/sections/registry.test.ts:266 13 comment lines warn 10
test/sections/registry.test.ts:467 11 comment lines warn 10

Split the file, wrap the line, shorten or exempt the comment, or list the path in .file-size-allow.local with a # reason.

8 managed file(s) skipped; repo-platform owns them.

@github-actions

github-actions Bot commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Template check

Integrity

Passed - this repository matches the state it was stamped with.

Freshness

This repository is behind the build branch by 17 commit(s). The next sync PR updates the managed files; nothing to do here.

@Vivswan
Vivswan force-pushed the followup/weak-tests branch from db23481 to a81132b Compare September 10, 2026 21:56
@Vivswan
Vivswan marked this pull request as ready for review September 10, 2026 22:15
An error occurred while trying to automatically change base from docs/layering-guide to feat/underscore-undeclared September 11, 2026 02:24
@Vivswan
Vivswan force-pushed the followup/weak-tests branch from 521a9b7 to c1a6661 Compare September 11, 2026 02:40
@Vivswan
Vivswan changed the base branch from docs/layering-guide to main September 11, 2026 02:40
@Vivswan
test: pin whole outcomes where tests asserted fragments, counts, or c…
89b1212 
…atalog-derived expectations

Survey of 89 test files (every test file outside the sibling builders' territory) found 324 weak or redundant rows: 215 substring pins, 52 counts, 30 shape pins, 19 vacuous, 13 redundant. This commit fixes the high-value half: every vacuous and redundant row, and every substring, count, and shape row in test/engine (validate, execute), test/github, test/discovery, test/report, test/sections, and the small root suites. About 196 assertions now pin a whole value against an independent literal; 11 test declarations are gone. test/engine/orchestrate.test.ts is left untouched (another builder owns it); its 22 rows are recorded in the PR backlog.

Deleted tests and the stronger pin that covers each: registry.test.ts grant-caveat map (the exact-literal EXPECTED_GRANT test in the same file); actions_secrets.test.ts not-base64 / wrong-length key (secrets-engine.test.ts parseSealingKey matrix); openapi/validate.test.ts USED_PATHS-carries-no-undocumented-path (holds by construction; excludeUndocumented is tested directly); graphql-pipeline.test.ts multi-mode mutation-without-node-id (same branch as the single-mode test); state.test.ts explicit-labels-replaces-baseline (the sparse-seed test now pins the whole one-element arrays); file-fuzz-issue.test.ts no-assignment sweep (both paths pin the whole gh argv); graduate-upstream-gaps.test.ts three generateIndex fragment tests and the source-derived determinism test (the committed-index byte-for-byte test and the whole-render multi-file test); single assertions in api.test.ts, discover.test.ts, schema-corpus.test.ts, checks-workflow.test.ts subsumed by an adjacent whole pin.

Mutations proven red (each reverted): src/github/api.ts:568 drop 'in the settings file' from the not-sent advice (old toContain("was not sent") passed; 13 whole pins fail); src/sections/contract/errors.ts:76 drop the higher-rate-limit advice tail (old /rate limit was hit/ passed; four whole pins fail); src/report/composer.ts:67 render adminRepo in the Target row (old ten toContain passed; the whole-document pin fails).

Accepted churn, recorded on purpose: the 145-key allEndpoints() inventory in registry.test.ts is the contract, so a new endpoint touches it; validate.test.ts verdicts embed zod's issue wording, as that file already did before this change.

Backlog (about 186 rows, listed in the PR): orchestrate.test.ts, and substring, count, and shape pins in the per-section suites under src/sections, test/e2e/mock, test/e2e/openapi/validate.test.ts, and test/docs plus test/scripts.
@Vivswan
test: pin the schema corpus size exactly
91024e2 
The deleted toBeGreaterThan(150) floor had no successor: a scenarioDocs() that returned only the five KNOWN_DIVERGENCES documents still passed the file. The exact count (252 on this branch) fails on a dropped root, file, or document kind; a new scenario updates the number, the same accepted-churn class as the allEndpoints() inventory pin.
Copilot AI balanced review requested due to automatic review settings September 11, 2026 02:42
@Vivswan
Vivswan force-pushed the followup/weak-tests branch from c1a6661 to 91024e2 Compare September 11, 2026 02:42
Copilot AI reviewed Sep 11, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The test-only changes consistently strengthen outcome verification without altering production behavior.

Pull request overview

Strengthens regression tests by replacing partial assertions with exact outcomes while consolidating redundant coverage. Production code is unchanged.

Changes:

  • Pins complete errors, reports, API calls, plans, and schema inventories.
  • Strengthens security and redaction assertions.
  • Removes tests already covered by stronger assertions elsewhere.
File summaries
File Description
test/sections/setup-section.test.ts Pins setup plans and failure messages.
test/sections/secrets-engine.test.ts Pins sealing and duplicate errors.
test/sections/registry.test.ts Pins policies, endpoints, and GraphQL contracts.
test/sections/plan-idempotence.test.ts Pins recurring operations.
test/sections/loosen.test.ts Verifies preserved parsed data and exact errors.
test/sections/list-section.test.ts Pins list errors and handler keys.
test/sections/graphql-contract.test.ts Pins GraphQL outcomes and errors.
test/sections/docs-registry.test.ts Tightens documentation validation paths.
test/sections/contract.test.ts Pins operation metadata and contract errors.
test/scripts/graduate-upstream-gaps.test.ts Pins complete generated indexes.
test/scripts/file-fuzz-issue.test.ts Pins complete gh command sequences.
test/schema-corpus.test.ts Pins corpus cardinality.
test/report/issue-report.test.ts Pins issue delivery calls and warnings.
test/report/delivery.test.ts Pins report-channel warnings.
test/report/composer.test.ts Pins the complete rendered report.
test/report/artifact-report.test.ts Pins artifact failure outcomes.
test/published-schema.test.ts Pins wrapper and fixture inventories.
test/private.test.ts Pins private-value serialization.
test/github/paginate.test.ts Pins pagination request paths.
test/github/graphql.test.ts Pins GraphQL errors and redaction.
test/github/api.test.ts Pins API bodies, errors, and secret handling.
test/engine/validate.test.ts Pins complete validation verdicts.
test/engine/execute.test.ts Pins complete execution outcomes.
test/e2e/openapi/validate.test.ts Removes construction-derived assertions.
test/e2e/mock/state.test.ts Pins complete seeded state.
test/e2e/mock/server.test.ts Pins pagination request logs.
test/e2e/mock/graphql-pipeline.test.ts Removes duplicate decode coverage.
test/docs/checks-workflow.test.ts Removes a redundant hash assertion.
test/discovery/targets.test.ts Pins deduplication results and notices.
test/discovery/repos-input.test.ts Pins the mixed-input error.
test/discovery/discover.test.ts Pins discovery errors and notices.
test/discovery/central.test.ts Pins central target resolution.
src/sections/workflows/workflows.test.ts Pins persistent workflow drift.
src/sections/rulesets/rulesets.test.ts Simplifies a compile-time contract test.
src/sections/repository/repository.test.ts Pins literal permission advice.
src/sections/labels/mock.test.ts Pins minted label identities.
src/sections/labels/labels.test.ts Pins duplicate-label errors.
src/sections/interaction_limits/interaction_limits.test.ts Pins permission advice and type contracts.
src/sections/environments/pins.test.ts Pins complete pin-order drift.
src/sections/check_suite_preferences/check_suite_preferences.test.ts Pins notes across idempotence passes.
src/sections/branches/branches.test.ts Pins missing-actor failures.
src/sections/actions_secrets/actions_secrets.test.ts Removes duplicate sealing-key tests.
Review details
  • Files reviewed: 42/42 changed files
  • Comments generated: 0
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@Vivswan
Vivswan enabled auto-merge (squash) September 11, 2026 02:45
@Vivswan
Vivswan merged commit 9b96378 into main Sep 11, 2026
32 checks passed
@Vivswan
Vivswan deleted the followup/weak-tests branch September 11, 2026 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

@Vivswan Vivswan

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Vivswan