Skip to content

chore(test): take the OpenAPI and GraphQL schemas from the Octokit packages instead of fetching them - #406

Merged
Vivswan merged 1 commit into
mainfrom
wt/octokit-schemas
Sep 22, 2026
Merged

Vivswan merged 1 commit into
mainfrom
wt/octokit-schemas

Conversation

@Vivswan

@Vivswan Vivswan commented Sep 22, 2026 •

Copy link
Copy Markdown
Owner

Before

$ bun run test:e2e
$ bun run test:artifacts && bun test/e2e/run.ts
$ bun .github/scripts/trim-openapi.ts --when-stale && bun .github/scripts/fetch-graphql-schema.ts --when-stale
/repo/test/e2e/openapi/github-openapi.trimmed.json is current (trimmed from https://raw.githubusercontent.com/github/rest-api-description/<sha>/descriptions/api.github.com/dereferenced/api.github.com.2022-11-28.deref.json); not fetching
/repo/test/e2e/graphql/schema.docs.graphql is current (fetched from https://raw.githubusercontent.com/github/docs/<sha>/src/graphql/data/fpt/schema.docs.graphql); not fetching
$ bun test/e2e/run.ts
  PASS  apply-idempotent-mixed
  ...
368/368 passed

After

$ bun run test:e2e
$ bun test/e2e/run.ts
  PASS  apply-idempotent-mixed
  ...
368/368 passed

Two gitignored, fetched, cache-restored test artifacts become two devDependencies. Nothing in the repository fetches from the network any more, and Dependabot moves both pins.

How

before: trim-openapi.ts ---fetch at UPSTREAM_REF---> github-openapi.trimmed.json (gitignored, actions/cache) --> validator, docs generator
        fetch-graphql-schema.ts ---fetch--------------> schema.docs.graphql          (gitignored, actions/cache) --> graphql lockstep tests

after:  @octokit/openapi/generated/api.github.com.deref.json --loadSpec(): cut to USED_PATHS in memory--> validator, docs generator
        @octokit/graphql-schema (schema.idl) --extended by src/upstream-gaps graphql-schema SDL--> graphql lockstep tests
  • Descriptor: the dereferenced api.github.com document is read by file path (the package index also loads every GHES and GHEC variant), parsed once per process, and cut to USED_PATHS. A used path the descriptor lacks, an undocumented path it now documents, or a $ref left in the slice fails the load by name.
  • GraphQL schema: the pinned package predates Repository.issueCreationPolicy, so a third gap kind, graphql-schema, carries the SDL the package lags. The lockstep test extends the published schema with each gap's SDL; graphql-js refuses the extension once the package ships the type or field, and the failure names the gap file to delete.
  • Gaps index: the generated index keys the gaps by file base name, so the file travels with each gap's SDL without a parallel list.
  • Nightly probe: installs both packages at latest with --no-save, checks the installed version is the latest, and runs the two lockstep tests, replacing the re-cut of the spec from upstream HEAD.
  • Deleted: trim-openapi.ts, fetch-graphql-schema.ts, lib/fetch-retry.ts, lib/fetched-artifact.ts, the fetch-test-artifacts composite, every cache step and test:artifacts prerequisite, and the cache-key lockstep test that guarded them.

Proof

  • Typecheck, knip, lint, build:check: green.
  • Targeted tests: 99 pass across the graphql-queries, endpoint-docs, openapi, and checks-workflow files, including the three load-failure cases and the gap-extension negative control; 33 pass in the graduate-upstream-gaps file, which pins the keyed index shape.
  • e2e: 75/75 for --sections labels,webhooks; 368/368 for the full corpus.
  • Full e2e, after over before: wall-time ratio 1.00 (identical to two decimals). The runner's peak resident set is about 1.6x, the parsed 73 MB descriptor; it stays well under the CI runner's memory, so the descriptor is loaded directly and no build-time trim artifact is generated.

Line accounting by kind

Kind Files Added Deleted
CI workflows and composites 6 +33 -116
repo scripts (.github/scripts) 9 +36 -433
tests and e2e harness 17 +249 -578
src (upstream-gaps, one comment) 4 +83 -20
manifest, lockfile, gitignore 3 +17 -14
docs (CONTRIBUTING) 1 +1 -1
Total 40 +419 -1162

Reviewer note

  • Stayed hand-rolled: trimDescriptor() (about 30 lines). It is a keyed pick over paths plus three named failure checks; an OpenAPI bundler or $ref resolver would add a dependency to do less, since the package already ships the dereferenced document.
  • Stayed hand-rolled: the graphql-schema gap extension uses graphql-js's own extendSchema; only the retire-the-gap error wording is ours.
  • Dropped, deliberately: the endpoint-docs test no longer checks the GraphQL category page. The introspected schema @octokit/graphql-schema ships cannot carry the @docsCategory directive the docs-site SDL had; the anchor name is still checked against the schema.

BEGIN_COMMIT_OVERRIDE
chore(test): take the OpenAPI and GraphQL schemas from the Octokit packages instead of fetching them

The e2e validator and the docs generator read GitHub's dereferenced OpenAPI descriptor from @octokit/openapi, cut in memory to USED_PATHS.
The GraphQL lockstep tests read GitHub's schema from @octokit/graphql-schema.
trim-openapi.ts, fetch-graphql-schema.ts, their fetch and staleness helpers, the fetch-test-artifacts composite, and every cache step are deleted.
No script fetches from the network any more, and Dependabot moves both pins.
A used path the descriptor lacks, an upstream gap it now documents, or a $ref left in the slice fails the load by name.
The pinned @octokit/graphql-schema predates Repository.issueCreationPolicy, so a graphql-schema gap kind carries the SDL the package lags and the lockstep test extends the schema with it.
The nightly probe installs both packages at latest and runs the lockstep tests instead of re-cutting the spec from upstream HEAD.
END_COMMIT_OVERRIDE

Copilot AI balanced review requested due to automatic review settings September 22, 2026 06:50
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

File size check

0 over a hard cap (fails), 30 warning(s).

File Size Tier Cap
.github/scripts/release-pipeline.ts:6 156 chars warn 150
.github/scripts/release-pipeline.ts:453 151 chars warn 150
.github/scripts/release-pipeline.ts:1247 153 chars warn 150
.github/scripts/release-pipeline.ts:1 34 comment lines (header) warn 25
.github/scripts/release-pipeline.ts:449 14 comment lines warn 10
.github/workflows/post-green.yml:29 153 chars warn 150
.github/workflows/update-release-pr.yml:109 14 comment lines warn 10
docs/upgrading/v2-to-v3.md 1276 lines warn 1040
src/engine/layers.ts:217 157 chars warn 150
src/flows/settings-write.ts:142 159 chars warn 150
src/flows/settings-write.ts:30 11 comment lines warn 10
src/flows/snapshot.ts:189 13 comment lines warn 10
src/github/secret-scan.ts:42 11 comment lines warn 10
src/schema.ts:186 153 chars warn 150
src/schema.ts:197 176 chars warn 150
src/sections/contract/errors.ts:14 14 comment lines warn 10
src/sections/contract/module.ts:963 185 chars warn 150
src/sections/contract/module.ts:627 12 comment lines warn 10
src/sections/contract/module.ts:897 12 comment lines warn 10
src/sections/secret_scanning_custom_patterns/compilable-form.ts:382 161 chars warn 150
src/sections/shared/roles.ts:43 13 comment lines warn 10
src/types.ts:16 156 chars warn 150
test/docs/guides.test.ts:451 155 chars warn 150
test/e2e/generators.ts 2633 lines warn 2560
test/e2e/generators.ts:1703 166 chars warn 150
test/e2e/generators.ts:1857 161 chars warn 150
test/e2e/generators.ts:1666 12 comment lines warn 10
test/engine/execute.test.ts:617 152 chars warn 150
test/flows/merge-parity.test.ts:63 164 chars warn 150
test/scripts/auto-fix-allowlist.test.ts:8 12 comment lines warn 10

Split the file, wrap the line, shorten or exempt the comment, or list the path in .file-size-allow.local with a # reason.

5 managed file(s) skipped; repo-platform owns them.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The nightly probe cannot read the GraphQL package manifest through its restricted exports.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity

Open (1)
What changed in this PR

Replaces fetched OpenAPI and GraphQL test artifacts with pinned Octokit packages and package-backed schema validation.

Changes:

  • Loads and trims the packaged OpenAPI descriptor in memory.
  • Extends the packaged GraphQL schema with tracked upstream gaps.
  • Removes artifact fetching, caching, and associated infrastructure.
File Description
test/​sections/​graphql-queries.test.ts Validates queries against the packaged schema and gaps.
test/​scripts/​generated.test.ts Removes the fetched-spec symlink.
test/​scripts/​fetched-artifact.test.ts Deletes obsolete artifact tests.
test/​scripts/​fetch-retry.test.ts Deletes obsolete fetch retry tests.
test/​scripts/​endpoint-docs.test.ts Uses the packaged GraphQL schema.
test/​scripts/​check-compat-markers.test.ts Removes the fetched-spec fixture.
test/​scripts/​changed-sections.test.ts Updates the composite-action fixture.
test/​scripts/​auto-fix-allowlist.test.ts Simplifies generator command parsing.
test/​e2e/​openapi/​validate.ts Loads and trims the packaged OpenAPI descriptor.
test/​e2e/​openapi/​validate.test.ts Tests descriptor loading and trimming.
test/​e2e/​openapi/​paths.ts Updates descriptor lifecycle documentation.
test/​e2e/​mock/​support.ts Updates OpenAPI terminology.
test/​e2e/​mock/​request-body.ts Updates OpenAPI terminology.
test/​e2e/​mock/​request-body.test.ts Updates OpenAPI terminology.
test/​docs/​workflow-loader.ts Removes composite-action loading support.
test/​docs/​checks-workflow.test.ts Removes artifact cache tests.
src/​upstream-gaps/​issue-creation-policy.ts Adds the missing GraphQL schema extension.
src/​upstream-gaps/​index.ts Exports aggregated GraphQL gap SDL.
src/​upstream-gaps/​gap.ts Adds the GraphQL schema gap kind.
src/​sections/​shared/​roles.ts Updates OpenAPI terminology.
package.json Adds Octokit schema dependencies and removes fetch prerequisites.
CONTRIBUTING.md Documents package-backed schemas.
bun.lock Locks the new dependencies.
.gitignore Removes obsolete artifact exclusions.
.github/​workflows/​update-release.yml Removes artifact setup.
.github/​workflows/​nightly.yml Probes latest Octokit schemas.
.github/​workflows/​nightly-fuzz.yml Removes OpenAPI fetching.
.github/​workflows/​checks.yml Removes artifact setup and caching.
.github/​workflows/​auto-fix.yml Removes artifact setup.
.github/​scripts/​trim-openapi.ts Deletes the OpenAPI fetcher.
.github/​scripts/​lib/​fetched-artifact.ts Deletes artifact staleness helpers.
.github/​scripts/​lib/​fetch-retry.ts Deletes fetch retry helpers.
.github/​scripts/​gen-gaps-index.ts Generates GraphQL gap exports.
.github/​scripts/​fetch-graphql-schema.ts Deletes the GraphQL schema fetcher.
.github/​scripts/​endpoint-docs.yml Updates descriptor terminology.
.github/​scripts/​endpoint-docs.ts Reads the packaged descriptor slice.
.github/​scripts/​check-compat-markers.ts Removes the fetched-spec exception.
.github/​actions/​fetch-test-artifacts/​action.yml Deletes artifact caching and fetching.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/nightly.yml
Copilot AI review requested due to automatic review settings September 22, 2026 07:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The nightly probe reads a package subpath that @octokit/graphql-schema does not export, so it fails before running its tests.

Review effort: Balanced
Findings: 1 High severity

Open (1)

Copilot AI review requested due to automatic review settings September 22, 2026 07:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Removing the legacy ignores breaks local linting in existing worktrees that retain the former large descriptor.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Copilot AI review requested due to automatic review settings September 22, 2026 07:15

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The package-backed schema migration is coherent and includes focused drift, failure-path, and generated-index coverage.

Review effort: Balanced
Findings: None

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The migration changes schema validation, generated artifacts, dependencies, and multiple CI workflows across 40 files.

Review effort: Balanced
Findings: None

@Vivswan Vivswan added the merge-when-green Owner approved: merge once every gate is green label Sep 22, 2026
Copilot AI review requested due to automatic review settings September 22, 2026 09:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The broad test-infrastructure migration affects schema validation and multiple CI paths, and the current all-green gate is failing.

Review effort: Balanced
Findings: None

…ckages instead of fetching them

The e2e validator and the docs generator read GitHub's dereferenced OpenAPI descriptor from @octokit/openapi, cut in memory to USED_PATHS.
The GraphQL lockstep tests read GitHub's schema from @octokit/graphql-schema.
trim-openapi.ts, fetch-graphql-schema.ts, their fetch and staleness helpers, the fetch-test-artifacts composite, and every cache step are deleted.
No script fetches from the network any more, and Dependabot moves both pins.
A used path the descriptor lacks, an upstream gap it now documents, or a $ref left in the slice fails the load by name.
The pinned @octokit/graphql-schema predates Repository.issueCreationPolicy, so a graphql-schema gap kind carries the SDL the package lags and the lockstep test extends the schema with it.
The nightly probe installs both packages at latest and runs the lockstep tests instead of re-cutting the spec from upstream HEAD.
Copilot AI review requested due to automatic review settings September 22, 2026 09:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The package-backed schema migration is coherent, removes obsolete infrastructure, and includes focused lockstep and failure-path coverage.

Review effort: Balanced
Findings: None

@Vivswan
Vivswan merged commit 9b5f68d into main Sep 22, 2026
30 checks passed
@Vivswan
Vivswan deleted the wt/octokit-schemas branch September 22, 2026 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-when-green Owner approved: merge once every gate is green

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants