fix(ci): run the regen guards, report them on the PR, and autofix CLI drift - #126
Merged
Merged
Conversation
`uv run pytest -q` collected 515 tests and not one of them came from `tests/build/`. pytest's default `norecursedirs` carries the bare pattern `build`, which matches by basename, so recursing from `testpaths = ["tests"]` skipped the directory whole. Every regen guard in there -- the `.fernignore` check, the README sync, the CLI<->SDK contract -- has never gated a PR, and guards that were red the day they were written merged to main on green CI. Drop `build` from `norecursedirs` and keep the `./build/` artifact tree out by path with `--ignore=build` instead. Collection goes 515 -> 937. That turns main red, because one of the newly-live guards is red and has been since it landed: `reference.md` documents `**request:** WorkflowRunStartIn` on three methods whose generated clients take flattened `script_text`/`source_language` kwargs, so the documented call raises TypeError before a request is sent. Replace the three hand-named assertions with a parse of every documented method, compared against `inspect.signature` in both directions, and record what the generator currently gets wrong in `_REFERENCE_DEFECTS` with reasons -- same contract as `_INTENTIONALLY_UNEXPOSED`, so the regen that fixes the generator deletes the excuse. Generalizing finds six mismatches, not three: `workspace_members.update_member_role`, `update_invite_role` and `workflows.runs_summary` (`from` documented, `from_` generated, so the documented call is not even syntactically valid) were wrong the same way and nothing said so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guards ran only on the dev path, which opens no PR, so the failures landed where there was nothing to fix forward from; the PR-opening path skipped them entirely. Guards ran where no PR existed, PRs existed where no guards ran, and the comment claiming ci.yml covered the PR side was false until the previous commit. Run them on both paths without failing the run, and post the verdict onto the PR as a comment -- including when it is clean, since "ran" being indistinguishable from "skipped" is what hid two weeks of red guards. Hand the one mechanical failure class to an agent: SDK->CLI drift, a generated param no CLI flag reaches. BE->SDK is automatic so the param lands in the client with nobody in the loop, while the hand-written CLI never follows. The agent adds the `Opt(...)`, regenerates the README block and the manifest snapshot, adds a test and pushes to the PR branch. It is barred from allowlisting a guard, weakening a test or touching generated files, and raises the `Cmd(gate=)` question for a human rather than deciding it. Reference-mismatch and renamed-method failures are explicitly out of its scope. Also close superseded `auto:sdk-regen` PRs. Each regen is a whole-SDK snapshot of one spec commit rather than an increment, so three open at once is three ways to merge the SDK backwards. Scoped to the label, the `fern-regen/` branch prefix, and a number below the PR just opened, so a human PR cannot be caught and a concurrent newer regen is never closed by an older run. The merge gate is ci.yml, not this workflow. Autofix is skipped with a notice until ANTHROPIC_API_KEY is configured; `allowed_bots` names the pipeline App rather than `*` because the repo is public. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ANTHROPIC_API_KEY` does not exist anywhere in this org, so autofix would have been skipped indefinitely waiting for a secret nobody had added. `CLAUDE_CODE_OAUTH_TOKEN` is already an org secret visible to this repo, so the job works on the next regen with no setup, and bills the subscription rather than the API. The tradeoff is that an OAuth token is tied to the account that ran `claude setup-token`, where an API key is not. Revoking or expiring it turns the autofix job red rather than skipped -- contained, since the guard verdict is posted by the regenerate job either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
.github/workflows/regen.ymlhas been failing on every dev run since 9/28 — 18 of the last 20 runs. The two that passed were prod runs, which skip the failing step entirely. It was never flaky: success and failure were two different code paths, not two outcomes of one check.Underneath that, a second and larger hole:
ci.ymlnever rantests/buildat all. pytest's defaultnorecursedirscarries the bare patternbuild, which matches by basename, so recursing fromtestpaths = ["tests"]skipped the directory whole.uv run pytest -qcollected 515 tests and not one came from there.So every guard on the Fern regen pipeline — the
.fernignorecheck, the README sync, the CLI↔SDK contract — has never gated a PR. Guards ran where no PR existed (dev), and PRs existed where no guards ran (prod). The comment inregen.ymljustifying the skip ("ci.yml runs the very sametests/buildon the regen PR") was simply not true.That let a red guard merge. #119 added
test_run_input_reference_matches_flattened_signaturesand was red on it, with all 22 checks green. The defect it caught is real:reference.mddocuments arequestparameter on methods whose generated clients take flattened kwargs, so following the docs raisesTypeError.reference.mdships in the PyPI wheel, so that lands insite-packagesfor every installer.Solution
Make the guards actually run (
656a655). Dropbuildfromnorecursedirs, keep the./build/artifact tree out by path with--ignore=build. Collection goes 515 → 937.tests/unit/test_collection.pyasserts the behaviour from a separate pytest process, because the whole failure was one word in a config file going unnoticed.Generalize the contract check. The three hand-named assertions become a parse of all 90 documented methods, compared against
inspect.signaturein both directions. It finds six mismatches, not three —workspace_members.update_member_role,update_invite_role, andworkflows.runs_summary(fromdocumented,from_generated, so the documented call isn't even syntactically valid) were wrong the same way and nothing said so. Known generator defects go in_REFERENCE_DEFECTSwith reasons, same contract as_INTENTIONALLY_UNEXPOSED: the regen that fixes the generator deletes the excuse.Stop blocking, start reporting (
77c1b2e). Guards run on both paths and fail neither. The verdict is posted onto the PR as a comment — including when clean, since "ran" being indistinguishable from "skipped" is what hid all of this. The hard gate stays the shell guard (clobbered_cli/, bare client); the merge gate isci.yml.Autofix the one mechanical class. SDK→CLI drift — a generated param no CLI flag reaches — is fully specified by the failing test, so
claude-code-actionadds theOpt(...), regenerates the README block and manifest snapshot, adds a test, and pushes to the PR branch. Explicitly barred from allowlisting a guard, weakening a test, or touching generated files; raises theCmd(gate=)question for a human instead of deciding it. Reference mismatches and renamed methods are out of scope — the former is unfixable in-repo anyway. Authenticates with the org-levelCLAUDE_CODE_OAUTH_TOKEN(already visible to this repo, so no secret to add; subscription billing rather than API).Close superseded regen PRs. Each regen is a whole-SDK snapshot of one spec commit, not an increment, so #123/#124/#125 open at once is three ways to merge the SDK backwards. Scoped to the label, the
fern-regen/prefix, and a number below the PR just opened.Verification
pytest -q→ 935 passed, 2 skipped (937 collected, up from 515);ruffclean;mypyclean; coverage 95.39%_REFERENCE_DEFECTSreds exactly the 6 known defects with actionable messages; simulating a generator fix reds the stale-allowlist guardvoices.listgainedprovider_voice_id, which is the drift alarm working as designed)./build/artifacts stay uncollected, andpytest tests/buildstill works as an explicit pathregen.ymlpassbash -n; dry-ran the supersede and comment logic against live PR data withgh pr closestubbed — new PR fix(ci): run the regen guards, report them on the PR, and autofix CLI drift #126 closes feat: sync SDK to OnePin API v0.41.142 #123/feat: sync SDK to OnePin API v0.41.144 #124/feat: sync SDK to OnePin API v0.41.145 #125 and leaves chore(main): release 0.18.0 #122 (release-please) alone; a PR-update scenario (feat: sync SDK to OnePin API v0.41.144 #124) closes only feat: sync SDK to OnePin API v0.41.142 #123 and never the newer feat: sync SDK to OnePin API v0.41.145 #125Follow-ups, not in this PR
fern/generators.ymlpins the generator tolatest, so this moved without a PR — worth pinningprovider_voice_idis a product decision: expose--provider-voice-id, or record it as intentionally unexposed. It can't be allowlisted on main today — the param isn't in main's SDK yet, so the entry would be stale on arrival. It belongs in feat: sync SDK to OnePin API v0.41.145 #125🤖 Generated with Claude Code