Skip to content

CI: fail-closed fast path for prose-only docs changes - #204

Open
harshsaver wants to merge 5 commits into
mainfrom
feat/188-ci-docs-fastpath
Open

harshsaver wants to merge 5 commits into
mainfrom
feat/188-ci-docs-fastpath

Conversation

@harshsaver

@harshsaver harshsaver commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Needs maintainer review before merge

Fixes #188.

A README-only contribution currently triggers the full runtime matrix, native builds, hosted DB tests, and browser installs. This PR adds a fail-closed docs fast path while preserving every required check name and branch protection.

Fail-closed design (redesigned per round-1 audit):

  • New classify job diffs the PR against its base and outputs docs_only=true only when every added/modified/renamed/deleted path is on an explicit prose allowlist (root README/CONTRIBUTING/CODE_OF_CONDUCT/SECURITY/CHANGELOG/ROADMAP, docs/releases.md, docs/release-notes-*.md). Protocol/dependency/operations docs, specs, schemas, examples, manifests, workflows, scripts, and any unknown status, empty diff, rename crossing the boundary, or git failure all yield docs_only=false, so the full checks run.
  • Docs validation runs exactly once, inside classify on Ubuntu, over all tracked Markdown on every PR -- never on Windows/macOS, so there is no PowerShell/python3 dependency outside Ubuntu. Mixed PRs get their docs checked too. The previous per-job validator steps, the no-op docs job, and the docs_files plumbing are gone.
  • If classify itself fails (self-test, classifier crash, or broken docs links), every required job fails closed via an explicit gate step (if: ${{ !cancelled() }} plus a first step that exits 1 when needs.classify.result != 'success'), so required checks go red and block by design -- not by a skipped-check side effect.
  • No required job is skipped or renamed: on docs-only PRs each required job runs checkout plus the gate on the real validation result, so a docs success is backed by real validation and broken links fail the required checks and block. Only non-required heavy jobs (npm-platforms, node-compatibility, hosted, workspace-ui) skip on the fast path.
  • Validator is local-only (relative targets must exist; HTML href/src/srcset and own-repo blob/tree/raw page URLs resolve to repo paths; anchors must match GitHub-style slugs with per-space hyphens; reference defs must resolve; pinned-commit/foreign/http/mailto and other schemes are skipped, never fetched). No secrets, no privileged execution, contents: read unchanged, so fork PRs gain no new exposure. Push-to-main and workflow_dispatch always run full checks.
  • Hand-rolled checker kept over lychee/remark deliberately: it runs on a pristine runner with only CPython and git, stays fully offline, approximates GitHub's heading slugs (code-span contents kept literal; formatted, setext, and duplicate headings covered; exotic nesting may differ), needs no pinned third-party install, and is covered by the repo's own tests.

What I verified:

  • scripts/ci-docs-fastpath.test.mjs (14 tests, run in CI by classify before classifying): allowlist acceptance, ~30 fail-closed cases, rename/delete handling, end-to-end git diff incl. unknown-SHA fallback, checker cases (broken file/anchor/undefined ref fail; remote links, code spans skipped), plus new coverage for HTML href/src/srcset, own-repo blob URLs (good/bad/missing/pinned-skipped/foreign-skipped), email autolinks and <placeholders> passing, A & B -> #a--b slug behavior, fenced code defining no anchors, and --all over tracked files (broken inbound link fails, untracked scratch ignored), plus root-anchored own-repo URL resolution and code-span/setext slug regressions (literal _meta/__init__, emphasised and code-span-led setext subjects, wrong-anchor negatives).
  • Checker run over all 93 tracked repo Markdown files: clean, with the new HTML and blob-URL checks active.
  • Local negative reproductions: broken nav anchor, missing img/srcset/blob targets, and wrong single-hyphen anchors each fail with the expected diagnostic.
  • This PR's own diff classifies docs_only=false (code-change path, full checks run); a canned README-only diff classifies docs_only=true.
  • Full local gates: fastpath suite 14/14 pass; workflow YAML parses; govulncheck repo-checkout input verified against the pinned action.

Validation:

  • CI on this PR should show classify passing and all required jobs running their full steps.
  • Live fast-path demonstration deferred to the first real docs-only PR after merge (no throwaway demo PRs opened); behavior proven by the local runs and tests above.
  • Honest gaps: no PowerShell available locally, so the Windows gate step (echo + exit 1, valid in both bash and pwsh) is proven by inspection plus CI, not by a local pwsh run; the gate's failure branch on Windows only executes if classify ever fails.
  • No migration or recovery steps; no secrets touched.

@harshsaver

Copy link
Copy Markdown
Member Author

🤖 Agentbox PR audit: changes requested

What needs to change

The classifier is careful and the full path is untouched (13/13 green on this head). The docs-only path is not ready. Four things must change.

  1. Windows. runtime runs on windows-latest, where run: defaults to PowerShell. "$DOCS_FILES" is empty there, so ci-docs-check.py exits 2 and the required runtime (windows-latest) check fails on every docs-only PR. The step is skipped in every job of run 43390504793, so it has never executed. Either stop running the validator on Windows and macOS (see the suggested shape) or set shell: bash and prove python3 resolves there with a live run.

  2. Make the validation real for what is on the allowlist.

    • README.md has no relative Markdown links. Its nav and images are HTML href, src and srcset, and its doc links are absolute https://github.com/october-dev/agentintercom/blob/main/... URLs. The checker skips all of them, so a README PR passes with nothing verified.
    • Check inbound links as well: validate all tracked Markdown, not only changed files. Otherwise a heading rename, or a deleted or renamed doc (D README.md currently validates zero files), breaks links silently.
    • Prefer an established offline checker over growing the regex parser: lychee with --offline --include-fragments and a remap for our own blob URLs, or remark-validate-links. If you keep the hand-rolled script, say why in the PR and add those link forms.
  3. Stop failing valid Markdown.

    • Remove the relative-path "autolink" branch (ci-docs-check.py:26,189-196). <name@example.com> and <workspace-id> are reported as missing files.
    • Fix the heading slug: GitHub turns each space into a hyphen, so A & B becomes a--b. The checker currently rejects the correct anchor.
    • Both points go away if you adopt a library.
  4. Make classify failure block by design. GitHub treats a job skipped by a failed needs as a passing required check; only the unexpanded runtime matrix name keeps merges blocked today. Give the required jobs if: ${{ !cancelled() }} and a first step that fails when needs.classify.result != 'success'. Correct the PR description on this point.

Suggested shape. Do the docs validation once, on Ubuntu, inside classify (or a single docs job that the required jobs need), over all tracked Markdown, on every PR. The four required jobs then begin with one gate step on that result and keep their docs_only != 'true' guards. This removes the four pasted steps, the Windows and macOS Python dependency, the docs_files plumbing and the no-op docs job. It also means mixed PRs get their docs checked.

How to test. Open a throwaway README-only PR against this branch (on: pull_request has no branch filter, so the fast path runs live on all three OSes). Link these runs in the description:

  • a green run;
  • a commit that breaks a nav anchor, an image path and a repo link, showing the required checks red;
  • a commit that breaks the classifier self-test, showing the required checks red and not skipped.

Add tests for HTML links, own-repo absolute links, email autolinks and a heading containing &. Optional cleanups are in the audit's non-blocking list.

What this PR changes

  • What it does: Adds a shortcut to the automated checks. When a pull request only edits a short list of plain-text documents (README, contributing and community docs, changelog, roadmap, release notes), the heavy build-and-test work is skipped and a quick documentation link check runs instead. Any other change still gets the full checks.
  • What changes in the product: Nothing visible to users. For contributors, a docs-only pull request is meant to finish its checks in well under a minute instead of about five.
  • Why it exists: Issue CI: add a fast validation path for narrowly scoped documentation changes #188. A README typo fix currently waits on the whole build and test run, and can be held up by failures that have nothing to do with it.
  • Why not yet:
    • The shortcut breaks on first real use. One of the six required checks runs on Windows, where the new step is written in a way that cannot work, so every docs-only pull request would show a red required check and could not be merged. That is worse than today.
    • The link check does not look at the README's links at all. The README's buttons, images and links are written in forms the check skips, so a README change gets six green ticks with nothing verified.
    • The shortcut has never run for real. This PR changes code, so its own checks took the normal full route.
    • The safety net for a crash in the new sorting step works by a side effect, not by design.
  • What to expect after merging (once fixed):
    • Docs-only pull requests go green quickly.
    • Code pull requests start about 10 seconds later, because the new sorting step runs first.
    • Every merge to main still gets the full checks.
    • Watch the first real docs-only pull request on all three operating systems.
    • Optional follow-up for Harsh: mark the new classify check as required in branch protection.
Full audit (Claude (fable, xhigh) × Codex (gpt-6-astra, xhigh), 1 turns, commit a8e44b2)

PR audit: #204 — CI: fail-closed fast path for prose-only docs changes

Head: a8e44b2

Sign-off

  • Claude (fable, xhigh): ✅ satisfied — first-pass audit written; decision is comment because the docs fast path cannot go green on Windows and its link check does not look at the README's links.
  • Codex (gpt-6-astra, xhigh): pending — has not reviewed yet (turn 2).

Decision

comment — the fast path would block every docs-only PR on a required Windows check, the validation behind it is empty for the README, and it has never been run live; fixing that takes a redesign plus a live demonstration.

What changes (plain language)

  • What it does: Adds a shortcut to the automated checks. When a pull request only edits a short list of plain-text documents (README, contributing and community docs, changelog, roadmap, release notes), the heavy build-and-test work is skipped and a quick documentation link check runs instead. Any other change still gets the full checks.
  • What changes in the product: Nothing visible to users. For contributors, a docs-only pull request is meant to finish its checks in well under a minute instead of about five.
  • Why it exists: Issue CI: add a fast validation path for narrowly scoped documentation changes #188. A README typo fix currently waits on the whole build and test run, and can be held up by failures that have nothing to do with it.
  • Why not yet:
    • The shortcut breaks on first real use. One of the six required checks runs on Windows, where the new step is written in a way that cannot work, so every docs-only pull request would show a red required check and could not be merged. That is worse than today.
    • The link check does not look at the README's links at all. The README's buttons, images and links are written in forms the check skips, so a README change gets six green ticks with nothing verified.
    • The shortcut has never run for real. This PR changes code, so its own checks took the normal full route.
    • The safety net for a crash in the new sorting step works by a side effect, not by design.
  • What to expect after merging (once fixed):
    • Docs-only pull requests go green quickly.
    • Code pull requests start about 10 seconds later, because the new sorting step runs first.
    • Every merge to main still gets the full checks.
    • Watch the first real docs-only pull request on all three operating systems.
    • Optional follow-up for Harsh: mark the new classify check as required in branch protection.

Findings

Blocking

  1. .github/workflows/ci.yml:64-68 — the fast-path step fails on the required runtime (windows-latest) check, so every docs-only PR is blocked.

    • The step has no shell: and the matrix includes windows-latest (ci.yml:60). GitHub's docs say the Windows default shell is pwsh and that env vars there are read as $env:NAME.
    • In PowerShell "$DOCS_FILES" is an undefined variable and expands to an empty string. The checker then exits 2 (scripts/ci-docs-check.py:215-222); I reproduced exit 2 for both an empty and a dropped argument.
    • python3 being on PATH on Windows runners is also unconfirmed, and this repo's CI has only ever used it on Ubuntu (ci.yml:246).
    • gh pr checks 204 --required lists runtime (windows-latest) as required.
    • The step has never executed: in run 43390504793 it is skipped in every required job, and the PR description says the live fast path is untested. Issue CI: add a fast validation path for narrowly scoped documentation changes #188 asks for a demonstrated docs-only PR path.
    • release.yml:115,186,200 already sets shell: explicitly for cross-OS steps.
  2. scripts/ci-docs-check.py:140-141,169 — the validation is empty for the README and misses the commonest docs breakage. Issue CI: add a fast validation path for narrowly scoped documentation changes #188 requires required results "backed by real documentation validation".

    • README.md has no relative Markdown links and no reference definitions. Its navigation and images are HTML (README.md:3-20, e.g. <a href="#quickstart">, <img src="assets/header/...">). Its 20 doc links are absolute https://github.com/october-dev/agentintercom/blob/main/... URLs (e.g. README.md:41,133,147).
    • The checker strips HTML tags before scanning (:169) and skips anything with a scheme (:140-141). A scratch README with a broken href, a missing img src and a link to a non-existent repo file passes with exit 0.
    • Across all nine allowlisted files only 8 links are actually checked.
    • Only outgoing links of changed files are checked. Renaming a heading, or deleting or renaming an allowlisted file, breaks inbound links unnoticed. Example: docs/desktop-remote-integration.md:182,240 link to releases.md.
    • Deletions take the fast path with zero files validated (scripts/ci-docs-classify.py:88-94, asserted at scripts/ci-docs-fastpath.test.mjs:86).
    • Ruleset: "Prefer established, well-maintained libraries … Do not reimplement common functionality without a clear reason." Offline checkers exist: lychee has --offline and --include-fragments; remark-validate-links checks local files and headings offline. The PR gives no reason for a 255-line regex parser.
  3. scripts/ci-docs-check.py:26,189-196 and :34-38 — valid Markdown fails the check, which blocks legitimate docs PRs on required checks.

    • The "autolink" branch treats <text> without a scheme as a relative file link. CommonMark only autolinks absolute URIs and email addresses.
    • Reproduced: <conduct@example.com> (a valid email autolink, typical in SECURITY.md and CODE_OF_CONDUCT.md) and prose placeholders <workspace-id> and <agent_id> are each reported as "missing link target".
    • github_slug collapses whitespace runs to one hyphen (:38). GitHub replaces each space and removes punctuation. For the heading at docs/release-notes-0.1.0-rc.5.md:1 the real anchor is #october-bus-010-rc5--npm-010-next15; the checker rejects it and accepts the single-hyphen form that does not exist on GitHub (reproduced).
  4. .github/workflows/ci.yml:56,127,181,273 — a failed classify job does not block the way the PR says it does.

    • Required jobs have needs: [classify] and no job-level if, so they are skipped when classify fails (checkout, Node download, self-test or runner failure).
    • GitHub's troubleshooting page says: "Successful check statuses are success, skipped, and neutral" and "The dependent job is skipped and may not block merging". Its stated fix is always() with needs.
    • So typescript-sdk, interoperability and vulnerabilities would count as passing with nothing run. CONTRIBUTING.md:61 says never to merge with a skipped required check.
    • Merging stays blocked today only because a skipped matrix job reports under the bare name runtime, leaving the three runtime (…) contexts waiting. That behaviour is described in community threads (Skipping a matrix job hangs the Pull Request when inner-matrix jobs are required status checks actions/runner#952, community discussion 9141), not in official docs.
    • This is not exploitable now, but the guarantee is one refactor of the runtime matrix away from failing open. Ruleset: long-term decisions, no stopgaps.

Non-blocking

  • ci.yml:43-52 with ci-docs-classify.py:77-106,197: docs_files is [] whenever docs_only=false. The new docs job therefore validates nothing on code, mixed and push runs, yet shows green. On this PR it passed without checking the PR's own CONTRIBUTING.md edit. On docs-only PRs it is a seventh copy of the same check.
  • The same validation step is pasted into four jobs (ci.yml:64-68,133-137,184-188,276-280) and runs six times on three operating systems.
  • The scripts are Python tested by a Node test that shells out to python3 (ci-docs-fastpath.test.mjs:13-21). The other 22 scripts are .mjs whose tests import the module directly. classify installs Node 24 only to test Python (ci.yml:27-31).
  • ci-docs-check.py:180-183: both branches append the same error, and the comment above them says the opposite of what the code does.
  • ci-docs-check.py:200-217 and ci-docs-classify.py:160-182 hand-roll argument parsing; argparse is in the standard library.
  • ci-docs-check.py:41-54 collects anchors from raw lines, so # comment lines inside code fences count as headings. This is lenient only.
  • Other gaps would surface as false failures later: indented code blocks, #top, ?query suffixes, URL-encoded paths and ../../issues-style links.
  • ci.yml:275 adds an explicit checkout to vulnerabilities, while golang/govulncheck-action already checks out by default (repo-checkout: true at the pinned SHA). The full path now checks out twice; set repo-checkout: false.
  • ci.yml:35 interpolates github.event.pull_request.base.sha into the script. A SHA is safe, but an env: variable is the usual hardening.
  • Checked and found sound:
    • Allowlist membership and the path-safety logic.
    • Rename and status parsing with -z.
    • The three-dot diff against the PR merge commit, where a stale base only adds files and so falls to full checks.
    • Output writing, which is single-line and JSON-escaped.
    • No new actions, permissions or secrets.
    • Required job names are unchanged.
    • No code, test, embed or release script reads an allowlisted doc (only release.yml:126-127 copies docs at release time).
    • Release tooling does not depend on CI job names.

Fix plan

n/a

Message to the developer

The classifier is careful and the full path is untouched (13/13 green on this head). The docs-only path is not ready. Four things must change.

  1. Windows. runtime runs on windows-latest, where run: defaults to PowerShell. "$DOCS_FILES" is empty there, so ci-docs-check.py exits 2 and the required runtime (windows-latest) check fails on every docs-only PR. The step is skipped in every job of run 43390504793, so it has never executed. Either stop running the validator on Windows and macOS (see the suggested shape) or set shell: bash and prove python3 resolves there with a live run.

  2. Make the validation real for what is on the allowlist.

    • README.md has no relative Markdown links. Its nav and images are HTML href, src and srcset, and its doc links are absolute https://github.com/october-dev/agentintercom/blob/main/... URLs. The checker skips all of them, so a README PR passes with nothing verified.
    • Check inbound links as well: validate all tracked Markdown, not only changed files. Otherwise a heading rename, or a deleted or renamed doc (D README.md currently validates zero files), breaks links silently.
    • Prefer an established offline checker over growing the regex parser: lychee with --offline --include-fragments and a remap for our own blob URLs, or remark-validate-links. If you keep the hand-rolled script, say why in the PR and add those link forms.
  3. Stop failing valid Markdown.

    • Remove the relative-path "autolink" branch (ci-docs-check.py:26,189-196). <name@example.com> and <workspace-id> are reported as missing files.
    • Fix the heading slug: GitHub turns each space into a hyphen, so A & B becomes a--b. The checker currently rejects the correct anchor.
    • Both points go away if you adopt a library.
  4. Make classify failure block by design. GitHub treats a job skipped by a failed needs as a passing required check; only the unexpanded runtime matrix name keeps merges blocked today. Give the required jobs if: ${{ !cancelled() }} and a first step that fails when needs.classify.result != 'success'. Correct the PR description on this point.

Suggested shape. Do the docs validation once, on Ubuntu, inside classify (or a single docs job that the required jobs need), over all tracked Markdown, on every PR. The four required jobs then begin with one gate step on that result and keep their docs_only != 'true' guards. This removes the four pasted steps, the Windows and macOS Python dependency, the docs_files plumbing and the no-op docs job. It also means mixed PRs get their docs checked.

How to test. Open a throwaway README-only PR against this branch (on: pull_request has no branch filter, so the fast path runs live on all three OSes). Link these runs in the description:

  • a green run;
  • a commit that breaks a nav anchor, an image path and a repo link, showing the required checks red;
  • a commit that breaks the classifier self-test, showing the required checks red and not skipped.

Add tests for HTML links, own-repo absolute links, email autolinks and a heading containing &. Optional cleanups are in the audit's non-blocking list.

Questions for Harsh

n/a

Verification

  • Read in full: the diff for all five files; ci.yml at head and base; release.yml; CODEOWNERS; CONTRIBUTING.md; README.md; SECURITY.md; CODE_OF_CONDUCT.md; CHANGELOG.md; the PR description. Issue CI: add a fast validation path for narrowly scoped documentation changes #188 was read through a web fetch that paraphrased the body but listed all five "Done when" items.
  • CI state:
    • gh pr checks 204 shows 13/13 passing.
    • --required shows 6 required checks: interoperability, runtime ×3, typescript-sdk, vulnerabilities.
    • gh run view 43390504793 --json jobs shows the fast-path step skipped in every required job and the full steps successful.
  • Docs and sources checked online:
    • GitHub workflow-syntax and variables pages (Windows default shell pwsh, $env:NAME).
    • GitHub "troubleshooting required status checks" (skipped counts as success; skipped dependents "may not block merging").
    • Skipping a matrix job hangs the Pull Request when inner-matrix jobs are required status checks actions/runner#952 and community discussion 9141 (a skipped matrix job reports under its bare name).
    • CommonMark and GFM autolink definitions.
    • github-slugger source and GitHub's section-link rules.
    • lychee CLI flags and the remark-validate-links README.
    • golang/govulncheck-action action.yml at the pinned SHA.
  • Repo searches: no Go, mjs, py or ts code, embed or test reads an allowlisted doc; release scripts do not reference CI job names.
  • Scratch reproductions: I ran python3 -I scripts/ci-docs-check.py against temporary files outside the checkout, five cases, with the results quoted in the findings. Nothing else was executed: no install, build, typecheck, lint or test suite.
  • Could not do:
    • Read job logs (gh run view --log returned empty).
    • Read branch protection through the API (denied; --required was used instead).
    • Confirm python3 on Windows runners (the runner image readmes do not say).
    • Run the fast path live, which is impossible until a docs-only PR exists.
  • Checks that matter before merge:
    • node --test scripts/ci-docs-fastpath.test.mjs (needs python3 and git).
    • A workflow lint such as actionlint on ci.yml.
    • A live docs-only PR run on Ubuntu, macOS and Windows, plus the two red-path runs described above.
    • The existing full CI on the PR head, to show code PRs are unchanged.

Audit log

  • Turn 1 (Claude (fable, xhigh)): Audited from scratch: read all changed files and surrounding workflows, the linked issue, CI run metadata and the required-check set; verified GitHub Actions, CommonMark and slug behaviour against docs; reproduced four validator behaviours on scratch files. Wrote the first audit with four blocking findings (Windows required check cannot pass on the fast path; validation empty for the README and blind to inbound links; false failures on valid Markdown; classify-failure blocking relies on an undocumented side effect). Decision: comment.

Automated review by Harsh's agentbox: an AI reviewer audited this PR against the project ruleset. It found several blocking problems, so this is a quick first review that a second reviewer has not checked; a full two-reviewer review follows after the next push.

…fail-closed gates

- Run the docs link check once in classify over all tracked Markdown
  (--all) instead of pasting a validator step into every required job;
  nothing docs-related executes on Windows/macOS anymore.
- Checker now follows HTML href/src/srcset and own-repo blob/tree/raw
  page URLs, validates inbound links across the repo, drops the false
  autolink branch, and slugs headings exactly like GitHub (each space
  becomes a hyphen). Fenced code no longer defines anchors.
- Required jobs get if: !cancelled() plus a gate step that fails when
  classify did not succeed, so a classifier failure blocks by design
  rather than by skipped-check side effect. Removed the no-op docs job
  and the docs_files plumbing; base SHA passed via env; govulncheck
  reuses the existing checkout.
@harshsaver

Copy link
Copy Markdown
Member Author

Fix pushed at 49bcf20 implementing the suggested redesign for all four blockers:

  • Validation now runs once in classify on Ubuntu over all tracked Markdown (--all); the per-job validator steps, the no-op docs job, and the docs_files plumbing are gone, so nothing docs-related executes on Windows/macOS.
  • The checker follows HTML href/src/srcset and own-repo blob/tree/raw page URLs (default branches only; pinned-commit and foreign links stay skipped), and checks inbound links across the repo. Hand-rolled script kept deliberately (pristine-runner, offline, exact GitHub slugs, no pinned install) -- noted in the description.
  • Autolink branch removed (<name@example.com>, <placeholders> pass); heading slugs now match GitHub (A & B -> #a--b); fenced code defines no anchors; fixed the inverted reference comment.
  • Required jobs have if: ${{ !cancelled() }} plus a gate step that fails when classify did not succeed (placed after checkout to avoid working-directory hazards; fails before any conditional work). PR description corrected on this point.

Evidence: fastpath suite 9/9 (3 new tests), checker clean over all 93 repo files with the new checks active, local negative reproductions for each new link form, YAML parses, and CI green at the new head (12/12, 6/6 required; classify log shows the --all validation passing 93 files).

Honestly unverified: no PowerShell locally, so the Windows gate step (echo + exit 1) is proven by inspection plus CI's green Windows run (where the gate correctly skipped), not by a local pwsh run; the gate's failure branch only executes if classify ever fails. No throwaway demo PRs opened; live fast-path demonstration deferred to the first real docs-only PR after merge.

@harshsaver

Copy link
Copy Markdown
Member Author

🤖 Agentbox PR audit: changes requested

What needs to change

The single Ubuntu validation job and the explicit required-job gates are sound by inspection, and the full path is green. Three things must change before merge.

  1. Resolve own-repo URLs from the repository root.

    • In rewrite_own_repo_url, return "/" + match.group(1).
    • Add nested-document tests: valid blob and raw links from docs/, and an invalid root target whose basename exists beside the linking document.
    • Update the two error expectations at ci-docs-fastpath.test.mjs:242-243.
  2. Stop rejecting valid Markdown before every PR is gated on this checker. These all fail today:

    • anchors to headings that contain emphasis or a link;
    • a fence longer than three markers holding a shorter one;
    • indented code;
    • HTML comments;
    • a code span containing a longer backtick run;
    • backslash-escaped brackets;
    • %20 or ?query on a relative link;
    • protocol-relative URLs;
    • setext headings as anchor targets;
    • duplicate-slug suffixes (Setup, Setup, Setup-1).

    How to get there:

    • A real Markdown parser handles the block and inline syntax, and Node is already installed in classify. For example, use a parser producing mdast with github-slugger in an .mjs script like the repo's other scripts, or remark-validate-links plus a small check for HTML nodes. Check the library's HTML and own-repo URL coverage first. mdast parser documentation
    • If you stay dependency-free, each case above needs a regression test, and the "exactly" slug claim in the docstring must go.
    • Either way, keep the HTML href/src/srcset and own-repo URL checks, and the negative tests for real breakage.
    • State the deliberate policy in one place: links that leave the repo, #top, and [a][b] with no definition are reported.
  3. Show the workflow running. Open ordinary PRs against the updated feature branch (on: pull_request has no branch filter) and link the runs in the description:

    • a README-only PR with docs_only=true, required jobs green on all three operating systems and heavy work skipped;
    • a broken local link or anchor turning the required jobs red;
    • a classifier self-test failure turning them red, not skipped;
    • the full code-path run on the final commit.

    Quote the classifier's reason= line from each run that reaches classification. A failed self-test stops before that line can exist; its failed step supplies the evidence instead. A failed diff also reports docs_only=false, so the current green job conclusions do not prove successful classification of the actual PR diff.

While you are in the scripts:

  • Remove the unused docs_files output and its test assertions, and update the classifier docstrings.
  • Point the gate message at the classify job.
  • Add the all-document check and its local command to CONTRIBUTING.md.

What this PR changes

  • What it does: Skips expensive build and test work when a pull request changes only a small list of prose documents. Checks documentation links once, before the other checks, on every pull request.
  • What changes in the product: Nothing visible to users. Contributors should get faster feedback on simple documentation changes.
  • Why it exists: Issue CI: add a fast validation path for narrowly scoped documentation changes #188: a README correction currently waits for the full build and test run and can be blocked by unrelated failures.
  • Why not yet:
    • A link written as a full GitHub address to our own repository is looked up in the wrong folder when the document is not at the top level. A correct link is rejected and a wrong one can pass.
    • The home-made link check also rejects valid ways of writing documentation that today's documents do not exercise. The first contribution that does would see every required check turn red before its tests even run.
    • The shortcut has still not been shown working in a real pull request, which the issue asks for.
  • What to expect after merging (once fixed):
    • A documentation error blocks the affected pull request, including one that also changes code. The same gate runs on main and on manual runs.
    • Build and test jobs wait for the new first job, which took seven seconds in the observed run. Queue time and total docs-only savings remain unmeasured.
    • Follow-up for Harsh: the README's "connection options" link (line 131) points to a section that does not exist. It is already broken on main.
Full audit (Claude (fable, xhigh) × Codex (gpt-6-astra, xhigh), 4 turns, commit 49bcf20)

PR audit: #204 — CI: fail-closed fast path for prose-only docs changes

Head: 49bcf20

Sign-off

  • Claude (fable, xhigh): ✅ satisfied — I accept Codex's decision and all three blocking findings; my turn-2 "fix" was too lenient on the missing live run and on the parser's false failures. I only tightened what the developer is asked to do.
  • Codex (gpt-6-astra, xhigh): ✅ satisfied — the three blockers and expanded acceptance cases are supported by the code; only evidence wording and non-blocking claims needed clarification. The comment decision and requested work stand.

Decision

comment — fix the link checker's wrong-folder lookup and its false failures on valid Markdown, and demonstrate the docs-only path in a real pull request before merging.

What changes (plain language)

  • What it does: Skips expensive build and test work when a pull request changes only a small list of prose documents. Checks documentation links once, before the other checks, on every pull request.
  • What changes in the product: Nothing visible to users. Contributors should get faster feedback on simple documentation changes.
  • Why it exists: Issue CI: add a fast validation path for narrowly scoped documentation changes #188: a README correction currently waits for the full build and test run and can be blocked by unrelated failures.
  • Why not yet:
    • A link written as a full GitHub address to our own repository is looked up in the wrong folder when the document is not at the top level. A correct link is rejected and a wrong one can pass.
    • The home-made link check also rejects valid ways of writing documentation that today's documents do not exercise. The first contribution that does would see every required check turn red before its tests even run.
    • The shortcut has still not been shown working in a real pull request, which the issue asks for.
  • What to expect after merging (once fixed):
    • A documentation error blocks the affected pull request, including one that also changes code. The same gate runs on main and on manual runs.
    • Build and test jobs wait for the new first job, which took seven seconds in the observed run. Queue time and total docs-only savings remain unmeasured.
    • Follow-up for Harsh: the README's "connection options" link (line 131) points to a section that does not exist. It is already broken on main.

Findings

Blocking

  1. scripts/ci-docs-check.py:68-75,199-202 — own-repo URLs resolve against the linking document's directory instead of the repository root.

    • Confirmed by both reviewers. https://github.com/october-dev/agentintercom/blob/main/docs/guide.md is rewritten to docs/guide.md; from docs/x.md the resolver then checks docs/docs/guide.md. A root README URL becomes docs/README.md, and a raw image URL becomes docs/assets/logo.svg.
    • The reverse also happens: a URL to a nonexistent root target.md passes when docs/target.md exists.
    • Claude reproduced both directions on scratch files. A patched copy with return "/" + match.group(1) passes all 93 tracked files and gives the right answer for the nested cases.
    • Tests exercise these URLs only from root documents (scripts/ci-docs-fastpath.test.mjs:214-243). The only non-root default-branch URL in the docs is inside a code span (deploy/hosted/README.md:248), so today's green run does not cover the defect.
    • Impact: the first affected documentation change fails classify, which fails all six required checks through .github/workflows/ci.yml:59-63,129-133,181-185,274-278.
  2. scripts/ci-docs-check.py:34-35,61-65,106-154,225-259 — the new required gate rejects valid Markdown.

    • Formatted headings: # _Quickstart_ followed by [start](#quickstart) is rejected, because github_slug keeps the underscores. A heading containing [Guide](guide.md) includes the destination in its slug. GitHub removes formatting first. GitHub section-link rules
    • Fences: FENCED captures exactly three markers, so a four-backtick fence holding a three-backtick example closes early and the example's [example](missing.md) is checked as a real link. CommonMark fenced blocks
    • Indented code and comments: a four-space-indented [example](missing.md) and <!-- [draft](missing.md) --> both reach INLINE_LINK. Neither renders a link. CommonMark indented blocks
    • Backtick runs (found by Codex, confirmed by Claude on a scratch file): text.find(run, end) at :123 also matches the start of a longer run. In `a ``` [example](missing.md)` the span closes early and the link text is checked. The comment that mistakes are always lenient is wrong. CommonMark code spans
    • Same kind, also reproduced by Claude: backslash-escaped brackets; %20 in a relative path; a ?query suffix on a relative link; protocol-relative //host URLs treated as local files; setext headings as anchor targets; the third slug for Setup, Setup, Setup-1. Codex independently traced these branches. The URL distinctions and unique-slug behavior are supported by URI syntax and github-slugger's implementation.
    • Calibration: scans of the 93 tracked files found no current examples triggering these failures: no HTML comments, longer fences, headings with the rejected emphasis/link formatting, double-backtick spans, filenames containing spaces, or relative links with queries. Claude also reported no setext headings. The repository does contain a heading with a simple code span (docs/portable-contract-verification-2026-09.md:35); that supported form is not one of these failures. These are latent failures for future edits, not a break in today's corpus.
    • Why it still blocks:
      • The gate applies to every pull request and to main.
      • Each audit round has found new parser bugs.
      • The stated reason for hand-rolling claims exact GitHub slug behaviour, which is false (ci-docs-check.py:11-13).
      • Ruleset: prefer established libraries where they improve reliability and cover edge cases; long-term decisions.
    • remark-validate-links documents offline file and heading validation, with cross-file headings in its CLI. Its HTML and URL-rewrite coverage still needs checking before it is chosen. remark-validate-links documentation
  3. .github/workflows/ci.yml:23-46,48-100 and scripts/ci-docs-fastpath.test.mjs:100-142 — the required docs-only workflow demonstration is still missing.

    • Issue CI: add a fast validation path for narrowly scoped documentation changes #188 explicitly requires demonstrating a docs-only PR path and a code-change path, with failing validation blocking the PR. Issue #188
    • Run 43404871253 verifies only the full workflow path: 12 successful jobs, with the new gate steps skipped. It does not show required checks succeeding when docs_only=true, or failing when validation fails.
    • The successful self-test does demonstrate the classifier's git diff and true/false outputs in a temporary repository on the runner (scripts/ci-docs-fastpath.test.mjs:100-138). It does not establish that classification of this PR's actual base/merge diff succeeded: a failed git diff also produces docs_only=false with exit 0 (scripts/ci-docs-classify.py:192-198). Step conclusions cannot distinguish that fallback from a real code-change verdict, and the reviewers could not retrieve the relevant job log. No observed workflow run demonstrates selecting the docs-only path for an actual PR.
    • The tests call the classifier and checker directly; none executes the workflow. The author's latest comment defers the live run until after merge.
    • The gates look correct by inspection. This is an unmet acceptance requirement, not a claim that the YAML is broken. Claude accepted the deferral in turn 2 and withdraws that: untested paths in this PR have hidden a bug in both earlier rounds.

Non-blocking

  • The round-1 failures have been addressed.
    • Validation runs only in Ubuntu classify (ci.yml:20-46).
    • HTML href/src/srcset and own-repo URLs are scanned.
    • --all selects all 93 tracked .md files.
    • The email/placeholder false failure and the per-space slug bug are fixed.
  • Classifier failure now blocks by design. Required jobs use !cancelled() and fail when needs.classify.result != 'success'; later steps keep the implicit success(). Checkout precedes the gate, which matters for the SDK job's working-directory default. Manual cancellation during classify is a separate case that neither reviewer has demonstrated. GitHub expressions
  • Obsolete docs_files plumbing remains (ci-docs-classify.py:25-27,73-107,145-157,188-206; tests at ci-docs-fastpath.test.mjs:19-20,36-45,127,137). No workflow consumes it: ci.yml:23-24 exports only docs_only, and line 46 uses --all. It is cleanup, to be done while the scripts are revised.
  • Validation still misses real links.
    • README.md:131 points to the absent #ways-to-connect heading through a repository-homepage URL, a form the checker skips (ci-docs-check.py:50-75,197-198). It is unchanged from main.
    • A shortcut reference [guide] with [guide]: missing.md is not validated (Claude reproduced the pass).
    • Own-repo URI autolinks and multi-line HTML tags are skipped.
    • So --all does not catch every inbound-link break.
  • Policy choices, not additional parser blockers. The checker reports links that leave the repository (../../issues), #top, and [a][b] with no definition. A documented checker policy may restrict these forms; that does not mean they are inherently invalid Markdown. The existing restrictions should be written down.
  • Contributor guidance needs precision.
    • The gate message says classification failed even when the self-test or link check failed (ci.yml:62,132,184,277); it should point to the classify job.
    • CONTRIBUTING.md:62 does not say that mixed and code PRs are checked too, and gives no local command.
  • Minor complexity.
    • Python is tested through a Node subprocess bridge, with Node installed in the same job (ci.yml:29-33).
    • Both scripts hand-parse arguments.
    • --files-json is now used only by tests.
    • The classifier's reason workflow output is unused, although its log line is useful.
  • Other boundaries checked and sound:
    • The allowlist, rename boundary checks, NUL-delimited diff parsing, non-PR full-path selection and PR_BASE_SHA handling.
    • No new action, permission or secret; required names unchanged.
    • The explicit vulnerabilities checkout is paired with repo-checkout: false.
    • No allowlisted prose is executed or embedded. Release packaging copies it (release.yml:126-127). scripts/artifact-integrity.mjs:11-21 hashes repository files, including prose, so documentation edits do change artifact identity. Fresh CI builds recompute it; an edit after a build intentionally invalidates the old artifact (scripts/npm-distribution.mjs:119-143). This is not evidence that prose changes affect executable behavior, nor a reason to reject the allowlist.

Fix plan

n/a

Message to the developer

The single Ubuntu validation job and the explicit required-job gates are sound by inspection, and the full path is green. Three things must change before merge.

  1. Resolve own-repo URLs from the repository root.

    • In rewrite_own_repo_url, return "/" + match.group(1).
    • Add nested-document tests: valid blob and raw links from docs/, and an invalid root target whose basename exists beside the linking document.
    • Update the two error expectations at ci-docs-fastpath.test.mjs:242-243.
  2. Stop rejecting valid Markdown before every PR is gated on this checker. These all fail today:

    • anchors to headings that contain emphasis or a link;
    • a fence longer than three markers holding a shorter one;
    • indented code;
    • HTML comments;
    • a code span containing a longer backtick run;
    • backslash-escaped brackets;
    • %20 or ?query on a relative link;
    • protocol-relative URLs;
    • setext headings as anchor targets;
    • duplicate-slug suffixes (Setup, Setup, Setup-1).

    How to get there:

    • A real Markdown parser handles the block and inline syntax, and Node is already installed in classify. For example, use a parser producing mdast with github-slugger in an .mjs script like the repo's other scripts, or remark-validate-links plus a small check for HTML nodes. Check the library's HTML and own-repo URL coverage first. mdast parser documentation
    • If you stay dependency-free, each case above needs a regression test, and the "exactly" slug claim in the docstring must go.
    • Either way, keep the HTML href/src/srcset and own-repo URL checks, and the negative tests for real breakage.
    • State the deliberate policy in one place: links that leave the repo, #top, and [a][b] with no definition are reported.
  3. Show the workflow running. Open ordinary PRs against the updated feature branch (on: pull_request has no branch filter) and link the runs in the description:

    • a README-only PR with docs_only=true, required jobs green on all three operating systems and heavy work skipped;
    • a broken local link or anchor turning the required jobs red;
    • a classifier self-test failure turning them red, not skipped;
    • the full code-path run on the final commit.

    Quote the classifier's reason= line from each run that reaches classification. A failed self-test stops before that line can exist; its failed step supplies the evidence instead. A failed diff also reports docs_only=false, so the current green job conclusions do not prove successful classification of the actual PR diff.

While you are in the scripts:

  • Remove the unused docs_files output and its test assertions, and update the classifier docstrings.
  • Point the gate message at the classify job.
  • Add the all-document check and its local command to CONTRIBUTING.md.

Questions for Harsh

n/a

Verification

  • Codex (turn 2), read only:
    • Read the full five-file diff, the redesign diff, both Python scripts, all nine tests, head and base CI workflows, the release workflow, CODEOWNERS, CONTRIBUTING, README, the allowlisted documents, the PR discussion and issue CI: add a fast validation path for narrowly scoped documentation changes #188.
    • Confirmed 12 successful jobs and six passing required checks on this head, with gate steps skipped.
    • Traced the nested-URL error, code-block and span handling, heading slugs, reference handling and all workflow guards by hand.
    • Consulted GitHub Actions and Markdown documentation, CommonMark, the github-slugger source, remark-validate-links and the pinned govulncheck action. Ran no project code or checks.
  • Claude (turn 2):
    • The current checker passes over this checkout (93 files).
    • Scratch runs of nested own-repo URL cases, a patched resolver, a broken homepage anchor and 17 Markdown constructs.
  • Claude (turn 3):
    • Scratch runs outside the checkout (python3 -I), confirming Codex's claims:
      • the unequal-backtick case fails;
      • the formatted heading fails;
      • a shortcut reference with a broken definition passes;
      • the slug-collision and setext cases fail.
    • Repo scans for the rejected constructs across the 93 tracked Markdown files. The broad original claim of no formatted headings is narrowed above: a supported code-formatted heading exists.
    • Read scripts/artifact-integrity.mjs and the README handling in scripts/npm-distribution.mjs; the packaged README is sdk/typescript/README.md, which is not on the allowlist.
    • Retried gh run view --log for the classify job; it returns empty, so the classifier's reason= line for the actual PR diff is unverified.
  • Codex (turn 4), read only:
    • Reconfirmed unchanged head 49bcf2058032e0854e3726eb791a176d86cd9e07, the 12 successful checks, the successful classifier self-test, and the author's unchanged deferral of the live demonstration.
    • Re-read the complete checker, classifier fallback/output path, end-to-end git test, workflow ordering, artifact identity and distribution consumers. Traced the expanded acceptance cases without executing them.
    • Checked tracked-document searches and the existing code-formatted heading. Rechecked github-slugger, the mdast parser documentation and URI syntax for encoding, queries and network-path references.
    • Accepted all three blockers and the requested work. Clarified only evidence scope, artifact-hash wording and why a failed self-test has no classifier reason= line. The supplied turn-3 audit was used as the current document because the on-disk copy still contained turn 2.
  • Not run by either reviewer: installs, builds, typechecks, lints, the node --test suite, or any workflow. No GitHub writes and no tracked-file changes. Only the requested untracked AUDIT.md was updated by Codex.
  • Required validation once the implementation is revised:
    • node --test scripts/ci-docs-fastpath.test.mjs with the added regressions;
    • the link check over every tracked document;
    • a workflow lint for ci.yml;
    • the live green and red runs described in the message;
    • full existing CI on the final commit.

Audit log

  • Round 1, turn 1 (Claude (fable, xhigh)), head a8e44b2: audited from scratch; four blocking findings (Windows required check could not pass on the fast path; validation empty for the README and blind to inbound links; false failures on valid Markdown; classify-failure blocking relied on a side effect). Decision: comment.
  • Round 2, turn 1 (Claude (fable, xhigh)), head 49bcf20: re-read every changed file and the new CI run; confirmed all four round-1 blockers resolved by the single-validation redesign. Found by scratch reproduction that own-repo URLs in non-root docs resolve against the wrong folder, and verified a one-line fix on a patched copy. Flagged the dead docs_files output. Recorded the remaining false-failure classes, the missing live run and the broken README.md:131 anchor as non-blocking. Decision: fix.
  • Round 2, turn 2 (Codex (gpt-6-astra, xhigh)), head 49bcf20: independently read the full change, PR discussion, issue and CI metadata; confirmed the URL defect, its proposed fix, obsolete output and redesigned failure gates. Promoted ordinary-Markdown false failures and the missing issue-required workflow demonstration to blocking; found incorrect unequal-backtick handling, shortcut-reference omissions and slug-collision gaps. Removed the unsubstantiated blanket claim that upper-case fragments must be accepted. Changed decision from fix to comment and replaced the small fix plan with concrete developer requirements. No tests or source changes performed; Claude must re-check the material changes.
  • Round 2, turn 3 (Claude (fable, xhigh)), head 49bcf20: verified Codex's new claims on scratch files (unequal backtick runs, formatted heading, shortcut reference, slug collision, setext) and accepted the decision and all three blocking findings, withdrawing my turn-2 acceptance of the deferred live run. Added calibration (no current document uses the rejected constructs), the point that a failed diff is indistinguishable from a code verdict in the green run, and the policy-versus-bug split. Tightened the developer message: a complete list of reproduced valid-Markdown rejections as the acceptance set, a concrete parser route, and quoting the reason= line in the live runs. Disputed only the word "ordinary" for constructs the repo does not currently use.
  • Round 2, turn 4 (Codex (gpt-6-astra, xhigh)), head 49bcf20: accepted the comment decision, all three blockers and Claude's expanded acceptance set. Rechecked the added cases and distinguished the successful temporary-repo git self-test from unverified classification of the actual PR diff. Corrected wording about formatted headings, artifact identity, required versus all checks, and absent reason output after a self-test failure. No blocker, acceptance case, required demonstration or implementation choice changed; no project code or checks executed. Satisfied; no material changes requiring another audit turn.

Automated review by Harsh's agentbox: two AI reviewers audited this PR against the project ruleset and agreed on it.

- Resolve own-repo blob/tree/raw URLs from the repo root ('/' prefix)
- Slugs from rendered heading text (emphasis/link/image/HTML stripped)
- Setext headings, github-slugger duplicate suffixes, exact-run code spans
- Longer fences, indented code blocks, and HTML comments never checked
- Backslash escapes, ?query strip, %XX decode, protocol-relative skip
- Drop unused docs_files classifier output; gate message points at classify
- CONTRIBUTING: validation runs on every PR + local command
@harshsaver

Copy link
Copy Markdown
Member Author

Round-2 audit fix pushed as 43e1bda (blockers 1-2 plus the script cleanups):

  • B1: own-repo blob/tree/raw URLs now rewrite root-anchored ('/'+path); nested-doc tests (valid blob+raw from docs/, invalid root target whose basename exists beside the linker); error expectations updated.
  • B2: slugs from rendered heading text (emphasis/link/image/HTML stripped, intra-word underscores kept); setext headings; github-slugger duplicate suffixes; exact-length code-span closing; longer fences, indented code, and HTML comments never checked; backslash escapes blanked; ?query stripped and %XX decoded; protocol-relative URLs skipped. Docstring 'exactly' claim softened and the deliberate policy (escaping links, #top-style anchors, undefined [a][b]) stated in one place.
  • Cleanups: docs_files output and its assertions/docstrings removed; gate message points at the classify job; CONTRIBUTING notes every-PR validation plus the local command.

Evidence: fastpath suite 13/13 (4 new tests), --all clean over 93 files, YAML parses, CI 12/12 green at the new head (run 43437062125). Classify log quotes: docs_only=false / reason=non-docs change runs full checks: .github/workflows/ci.yml / docs link check passed (93 file(s)).

Not done: B3 live demo PRs (docs-only green run, broken-link red run, self-test-failure red run) — no demo PRs opened from this watch round; still needs someone to open ordinary PRs against this branch and link the runs.

@harshsaver

Copy link
Copy Markdown
Member Author

🤖 Agentbox PR audit: changes requested

What needs to change

The root-relative URL correction and the script cleanups are accepted. The current full CI run is green. Finish these two items before merge:

  1. Complete the heading correction in scripts/ci-docs-check.py. Preserve code-span contents while removing surrounding Markdown formatting, then apply slug rules. When recognizing setext headings, distinguish real block markers from a line beginning with emphasis or inline code. Add cases beside scripts/ci-docs-fastpath.test.mjs:334-357 for:

    • ## The `_meta` field linking to #the-_meta-field;
    • ## The `OCTOBER_BUS_*` variables linking to #the-october_bus_-variables;
    • a code span containing __init__;
    • *Quickstart* followed by an equals-sign underline, linking to #quickstart;
    • a setext heading beginning with a code span.

    Include wrong-anchor cases that must still fail. Retain the existing formatted-heading, duplicate-heading and URL regressions. Claude's code-span protection patch addresses the underscore regression; the setext detection branch needs its own correction. Finding 1 describes an approach for both that was verified on a patched copy.

  2. Provide the live workflow evidence already requested by issue CI: add a fast validation path for narrowly scoped documentation changes #188. After the fixes, use an ordinary PR against feat/188-ci-docs-fastpath; the workflow has no pull-request branch filter.

    • Show a README-only commit selecting docs_only=true, all six required checks succeeding across the three operating systems, and heavy steps skipped.
    • Then show a deliberately broken local link making classify and all six required checks fail, not skip.
    • These can be successive commits in one temporary PR.
    • Link both runs and the final full code-path run in the PR description, and include the classifier's reason= lines from runs that reach classification.
    • A separate self-test-failure run is optional, because the broken-link run exercises the same explicit failure gate.

Update the stale test count and slug claims in the PR description. If a concrete access or tooling limitation prevents the demonstration, identify it; the current reply only records that it was not done.

What this PR changes

  • What it does: Skips expensive build and test work when a pull request changes only a small list of prose documents. Checks documentation links once, before the other checks, on every pull request.
  • What changes in the product: Nothing visible to users. Contributors should get faster feedback on simple documentation changes.
  • Why it exists: Issue CI: add a fast validation path for narrowly scoped documentation changes #188: a README correction currently waits for the full build and test run and can be blocked by unrelated failures.
  • Why not yet:
    • The wrong-folder lookup and the earlier concrete examples are fixed.
    • A few valid ways of writing a heading can still turn all six required checks red. None of today's documents use them.
    • The shortcut has not been demonstrated on a real documentation-only pull request. The developer has been asked twice and has not done it; its last reply says someone still needs to. It is being asked once more, and to name the obstacle if there is one. If it cannot, this comes to you as a ten-minute task.
  • What to expect after merging, once fixed:
    • Documentation errors caught by the checker block documentation, mixed and code pull requests. The same check runs on main and manual runs.
    • Build jobs wait for the new first job, which took seven seconds in the observed run. Total documentation-only savings remain unmeasured.
    • Some link forms remain unchecked.
    • The README's existing broken "connection options" link is a separate follow-up.
Full audit (Claude (fable, xhigh) × Codex (gpt-6-astra, xhigh), 3 turns, commit 43e1bda)

PR audit: #204 — CI: fail-closed fast path for prose-only docs changes

Head: 43e1bda

Sign-off

  • Claude (fable, xhigh): ✅ satisfied — I accept Codex's comment decision and both blocking findings. My "declined three times" was overstated, and the formatted setext failures reproduce on scratch files.
  • Codex (gpt-6-astra, xhigh): ❌ not satisfied — the code-span regression is real, formatted setext headings also remain broken, and the missing demonstration is an existing developer requirement rather than an unresolved product decision.

Decision

comment — correct the remaining heading failures and supply the live workflow evidence required by issue #188; the discussion establishes no obstacle requiring Harsh to choose or waive the requirement.

What changes (plain language)

  • What it does: Skips expensive build and test work when a pull request changes only a small list of prose documents. Checks documentation links once, before the other checks, on every pull request.
  • What changes in the product: Nothing visible to users. Contributors should get faster feedback on simple documentation changes.
  • Why it exists: Issue CI: add a fast validation path for narrowly scoped documentation changes #188: a README correction currently waits for the full build and test run and can be blocked by unrelated failures.
  • Why not yet:
    • The wrong-folder lookup and the earlier concrete examples are fixed.
    • A few valid ways of writing a heading can still turn all six required checks red. None of today's documents use them.
    • The shortcut has not been demonstrated on a real documentation-only pull request. The developer has been asked twice and has not done it; its last reply says someone still needs to. It is being asked once more, and to name the obstacle if there is one. If it cannot, this comes to you as a ten-minute task.
  • What to expect after merging, once fixed:
    • Documentation errors caught by the checker block documentation, mixed and code pull requests. The same check runs on main and manual runs.
    • Build jobs wait for the new first job, which took seven seconds in the observed run. Total documentation-only savings remain unmeasured.
    • Some link forms remain unchecked.
    • The README's existing broken "connection options" link is a separate follow-up.

Findings

Blocking

  1. scripts/ci-docs-check.py:81-87,149-161 — heading handling still rejects valid links, including a regression and a combination of the previously agreed cases.

    • Code-span regression, confirmed by both reviewers. Code spans are unwrapped at line 81, then their literal underscores are processed as emphasis at line 87.

      • Heading ## The `_meta` field should give #the-_meta-field; the checker generates the-meta-field.
      • Heading ## The `OCTOBER_BUS_*` variables loses the underscore before the asterisk and rejects #the-october_bus_-variables.
      • Code contents are literal; heading formatting is removed before slugging. GitHub formatting rules, github-slugger implementation, slug character filter.
    • Formatted setext headings remain broken, found by Codex and reproduced by Claude.

      • *Quickstart* over an ==== underline defines #quickstart, but the link to it fails with "unknown anchor".
      • `config` options over a ---- underline fails the same way for #config-options.
      • Cause: at line 157, startswith("*") (and the backtick entry) clears prev as though the text were a list item or fence, so line 150 records no heading. CommonMark setext headings.
    • Formatted headings and setext headings were both in last round's acceptance set, so their combination is not a new request. Claude had listed it as non-blocking and accepts the promotion.

    • Tests cover an emphasised ATX heading and plain setext headings separately (scripts/ci-docs-fastpath.test.mjs:334-357), so green results do not establish the combined case.

    • Both defects are latent in today's documents. The only code-span heading has no underscore (docs/portable-contract-verification-2026-09.md:35), and no tracked file has a setext heading. An affected edit would still fail classify and all six required checks before the heavy checks run (.github/workflows/ci.yml:59-63,129-133,181-185,274-278).

    • A small combined change was verified on a patched copy outside the checkout:

      • stash each code span's text before the markup substitutions and restore it after the underscore strip;
      • treat -, *, + and numbered markers as list items only when followed by whitespace, and keep # and > as block markers.

      With both changes, all 93 tracked files still pass, and so do:

      • the two code-span cases and `__init__`;
      • the emphasised, bold and code-span setext cases;
      • the suite's existing heading cases.

      A wrong anchor still fails, and a list item above a rule is still not a heading.

    • Add positive and negative regressions. Protecting code spans alone does not complete this finding.

  2. .github/workflows/ci.yml:20-46,48-63 — the documentation-only workflow and its failure gate still lack the required live demonstration.

    • Issue #188 requires a demonstrated documentation-only PR path and code-change path, successful required results backed by documentation validation, and failing validation that blocks the PR.
    • Run 43437062125 on this head has 12 successful jobs and all six required checks passing. It exercises the full path, with the failure-gate steps skipped. It does not show required checks succeeding with docs_only=true or failing after classify fails.
    • The latest developer reply leaves the demonstration undone. It quotes docs_only=false, reason=non-docs change runs full checks: .github/workflows/ci.yml and a 93-file validation result from the log. Those lines answer the earlier concern about a failed diff falling back, although reviewers could not retrieve the log text themselves.
    • Decision correction, accepted by Claude. The reply reports unfinished work and says someone still needs to open the demonstration PRs. It reports no denied permission, unavailable capability or product tradeoff. The demonstration was requested in two audit rounds and not done in two replies; "declined three times" was wrong. The issue already answers whether the demonstration is required, so this stays with the developer, with an explicit request to name any obstacle.
    • The gates look correct by inspection. A green documentation-only run and a broken-link run are sufficient evidence. A separate self-test-failure run is optional: both failures produce the same needs.classify.result checked by the explicit gates, independently of the docs_only output.

Non-blocking

  • The root-URL defect and concrete prior examples have been addressed.
    • Own-repo URLs now return a leading slash (ci-docs-check.py:93-105) and resolve from the repository root (:328-329). Nested-document positive and negative tests cover this (ci-docs-fastpath.test.mjs:302-332).
    • The four new tests cover the earlier formatted-heading, fence, indented-code, comment, unequal-backtick, escaped-bracket, URL and duplicate-slug examples (:334-415). Claude ran those examples on scratch files and they pass; Codex traced the branches.
    • There are 13 tests, and the CI self-test step passed. This is evidence for those examples, not complete Markdown support.
  • The requested cleanups are done.
    • docs_files and its assertions are removed (ci-docs-classify.py:71-103,141-150).
    • Gate messages point to classify (ci.yml:62,132,184,277).
    • CONTRIBUTING.md:62 explains validation on all PRs and gives the local command.
    • The checker now calls its slugs an approximation and states its policy for escaping links, missing anchors and undefined references (ci-docs-check.py:11-21).
  • Two new validation blind spots, reproduced by Claude and traced by Codex.
    • Four-space or tab-indented list content after a blank line is discarded as code (ci-docs-check.py:246-251), so a numbered-list continuation containing a broken link is hidden. The list test at ci-docs-fastpath.test.mjs:380-383 has no blank line and does not cover this. Zero lines are hidden by this rule in the current documents.
    • An inline code span containing <!-- can hide the rest of the file: blank_comments runs before blank_spans, and an unmatched comment opener blanks to the end (ci-docs-check.py:217-218,280). No tracked Markdown contains <!--.
  • Other known false failures remain outside the requested heading correction.
    • A lone backtick in one paragraph can pair with a code span in another (ci-docs-check.py:167-206,280), exposing example link syntax as a real link. The statement at lines 172-173 that mispairing can only hide text is incorrect.
    • Escapes in a destination such as docs/my\_file.md are replaced with spaces before the destination is parsed (:373-375).
    • The checker must not be described as accepting every valid Markdown document.
  • Previously identified omissions remain. Repository-homepage anchors, shortcut references and multi-line HTML tags are not validated (ci-docs-check.py:48,60-67,319-320,357-390). README.md:131 links to the absent #ways-to-connect heading; it predates this PR. An all-file scan does not catch every inbound-link break.
  • Design and maintenance. The checker is now 485 lines and its syntax passes interact. Last round allowed a dependency-free implementation, so a library is not a new mandatory condition. A maintained parser remains an implementation option under the ruleset.
  • Minor.
    • The PR description still says nine tests and exact slugs.
    • The classifier's reason output is unused by the workflow, although its log line is useful (ci-docs-classify.py:150).
    • --files-json is used only by tests.
    • Python is still tested through a Node subprocess bridge.
  • Unchanged boundaries remain sound.
    • The allowlist, rename checks, NUL-delimited diff parsing and fallback to full checks are unchanged except for the removed output.
    • Required-job guards are unchanged except for their diagnostic text. !cancelled() lets the failure gates run, and later steps keep the implicit success() condition. GitHub status-check expressions.
    • No new action, permission or secret; required check names unchanged; vulnerabilities still pairs explicit checkout with repo-checkout: false.

Fix plan

n/a

Message to the developer

The root-relative URL correction and the script cleanups are accepted. The current full CI run is green. Finish these two items before merge:

  1. Complete the heading correction in scripts/ci-docs-check.py. Preserve code-span contents while removing surrounding Markdown formatting, then apply slug rules. When recognizing setext headings, distinguish real block markers from a line beginning with emphasis or inline code. Add cases beside scripts/ci-docs-fastpath.test.mjs:334-357 for:

    • ## The `_meta` field linking to #the-_meta-field;
    • ## The `OCTOBER_BUS_*` variables linking to #the-october_bus_-variables;
    • a code span containing __init__;
    • *Quickstart* followed by an equals-sign underline, linking to #quickstart;
    • a setext heading beginning with a code span.

    Include wrong-anchor cases that must still fail. Retain the existing formatted-heading, duplicate-heading and URL regressions. Claude's code-span protection patch addresses the underscore regression; the setext detection branch needs its own correction. Finding 1 describes an approach for both that was verified on a patched copy.

  2. Provide the live workflow evidence already requested by issue CI: add a fast validation path for narrowly scoped documentation changes #188. After the fixes, use an ordinary PR against feat/188-ci-docs-fastpath; the workflow has no pull-request branch filter.

    • Show a README-only commit selecting docs_only=true, all six required checks succeeding across the three operating systems, and heavy steps skipped.
    • Then show a deliberately broken local link making classify and all six required checks fail, not skip.
    • These can be successive commits in one temporary PR.
    • Link both runs and the final full code-path run in the PR description, and include the classifier's reason= lines from runs that reach classification.
    • A separate self-test-failure run is optional, because the broken-link run exercises the same explicit failure gate.

Update the stale test count and slug claims in the PR description. If a concrete access or tooling limitation prevents the demonstration, identify it; the current reply only records that it was not done.

Questions for Harsh

n/a

Verification

  • Codex, turn 2 — read only:
    • Verified HEAD 43e1bda7e716671a685d8d8bb180c3d7abb615ad.
    • Read the complete change since 49bcf2058032, both Python scripts, all 13 tests, the current workflow, the PR description, both developer replies and issue CI: add a fast validation path for narrowly scoped documentation changes #188.
    • Traced URL resolution, every prior regression fixture, code-span underscore handling, formatted setext detection, comment and span ordering, indented-list handling, destination escapes, classifier outputs and failure gates.
    • Rechecked CommonMark, GitHub heading rules, the github-slugger source and GitHub status expressions.
    • Executed no project code.
  • CI observed: run 43437062125 has 12 successful jobs and six passing required checks. classify took seven seconds, and its self-test, classification and validation steps succeeded. Required-job failure gates were skipped. gh run view --log returned no text, so the quoted classifier reason remains author-supplied.
  • Claude, turn 1: scratch runs outside the checkout over all 93 files; the earlier acceptance examples and real-breakage cases; the code-span regression, blind spots and remaining false failures; corpus measurements; a patched code-span fix.
  • Claude, turn 3:
    • Reproduced both formatted setext failures on scratch files, with a plain setext control passing.
    • Verified the combined patch described in finding 1 on a copy outside the checkout.
    • Re-read both developer replies and corrected the count of unanswered demonstration requests to two.
    • Checkout hygiene: scripts/__pycache__/ was not pre-existing. My turn-1 measurement script created it when it imported the checker. I removed it; git status now shows only the untracked AUDIT.md.
  • Not run by either reviewer: installs, builds, typechecks, lints, the node --test suite or any workflow. No GitHub writes and no tracked-file changes.
  • Required validation after revision:
    • node --test scripts/ci-docs-fastpath.test.mjs with the added heading regressions.
    • python3 scripts/ci-docs-check.py --all over the tracked documents.
    • A workflow lint for ci.yml.
    • Full existing CI on the final commit.
    • The live green documentation-only run and red broken-link run described above.

Audit log

  • Round 1, turn 1 (Claude (fable, xhigh)), head a8e44b2: audited from scratch; four blocking findings (Windows required check could not pass on the fast path; validation empty for the README; false failures on valid Markdown; classify-failure blocking relied on a side effect). Decision: comment.
  • Round 2, turns 1-4 (Claude and Codex), head 49bcf20: confirmed the redesign resolved round 1. Found own-repo URLs resolving against the wrong folder, a list of valid-Markdown rejections, and the missing live demonstration. Agreed decision: comment, with a bounded acceptance list.
  • Round 3, turn 1 (Claude (fable, xhigh)), head 43e1bda: re-read all changed files and the developer's reply. Verified on scratch files that the URL fix and every agreed acceptance case now pass and that real breakage is still reported. Found one regression (code-span headings with edge underscores) and verified a small fix on a patched copy. Recorded two new lenient blind spots and three rare false failures as non-blocking, with measurements showing none affects today's documents. Changed the decision to needs-input because the developer has declined the live demonstration three times and says someone else must run it.
  • Round 3, turn 2 (Codex (gpt-6-astra, xhigh)), head 43e1bda: independently reviewed the new diff, full scripts/tests, PR replies, issue and CI metadata without running project code. Confirmed the URL fix, cleanup, code-span regression and remaining limitations. Promoted formatted setext failure into the heading blocker because it combines already-requested capabilities. Changed needs-input back to comment: no permission/capability blocker or unresolved product choice is established by the discussion. Accepted two live runs as sufficient and removed the proposed waiver question and timing promises. Material changes require Claude's re-check.
  • Round 3, turn 3 (Claude (fable, xhigh)), head 43e1bda: reproduced Codex's formatted setext failures on scratch files and verified a combined small patch on a copy. Accepted the comment decision and both blocking findings. Withdrew "declined three times": the demonstration was requested twice and not done twice, with no obstacle stated. Added reproduction and patch evidence to finding 1, a note for Harsh that the demonstration may come back to him, and a correction that the stray scripts/__pycache__/ was mine and is now removed. No change to the decision, the blockers or what the developer is asked to do.

Automated review by Harsh's agentbox: two AI reviewers audited this PR against the project ruleset and agreed on it.

Code spans are stashed before markup substitution so underscores inside
them survive the emphasis strip (`_meta` keeps its slug); bullet markers
only count as list items with following whitespace so *Quickstart* and
code-span-led lines stay setext candidates. Adds the five positive and
two wrong-anchor regressions (14 tests). --all still passes (93 files).
@harshsaver

Copy link
Copy Markdown
Member Author

Round-3 audit item 1 fixed at 3c2772f (all 12 CI checks green, incl. classify with the 14-test suite):

  • Code spans are stashed before markup substitution in github_slug, so underscores inside them survive the emphasis strip: _meta -> #the-meta-field, OCTOBER_BUS_* -> #the-october_bus-variables, __init__ kept literal.
  • Setext subjects: -, *, + only count as list items with following whitespace, so Quickstart over ==== gives #quickstart and a code-span-led line gives #config-options (fences are pre-blanked, so backtick/tilde no longer clear the candidate).
  • New test 'keeps code-span text literal...' with the five positive cases plus wrong-anchor negatives (#the-meta-field, #quickstart-1 still fail); verified it fails on the old code. --all still passes (93 files). PR description updated (14 tests, slugs 'approximate').

Item 2 (live docs-only + broken-link demo runs via a temp PR against this branch) is not done: opening throwaway/demo PRs needs Harsh's explicit authorization under standing orders, so that is the concrete blocker. If authorized, happy to run it.

@harshsaver harshsaver added the agentbox: needs your input Claude and Codex need a decision from Harsh label Nov 13, 2026
@harshsaver

Copy link
Copy Markdown
Member Author

🤖 Agentbox PR audit: needs Harsh's input

Questions

The developer requests your go-ahead to demonstrate the shortcut on a real pull request. How do you want that handled?

Issue #188 requires the demonstration, and the blocking code findings are resolved.

  • Option A — authorise the developer. It opens one temporary pull request against feat/188-ci-docs-fastpath and runs two successive commits:

    1. a README-only edit, which should make all six required checks green across the three operating systems, with heavy work skipped;
    2. a deliberately broken local link, which should make classify and all six required checks red.

    It links both runs on this PR and closes the temporary PR.

  • Option B — do it yourself. Perform the same two steps and provide the run links.

  • Option C — waive the pre-merge demonstration. Merge once the final checks pass and use the first real documentation-only PR as the live test. Keep issue CI: add a fast validation path for narrowly scoped documentation changes #188 open until that demonstration is complete.

Recommendation: A. It satisfies the issue before merge and tests the two workflow branches still unobserved. If you choose C, failures in the unverified shortcut could block a documentation PR or cause it to run the full checks. The classifier sends code changes to full checks by design; its current full-path run is green.

If the two runs behave as described, no further code changes are currently requested, and the PR can merge once final checks pass. If either behaves differently, return it to the developer with the run link.

The authorization request comes from the developer's latest reply, which cites its standing orders. The auditors have not inspected those underlying orders; this question resolves the constraint the developer explicitly reported.

What this PR changes

  • What it does: Skips expensive build and test work when a pull request changes only a small list of prose documents. Checks documentation links once, before the other checks, on every pull request.
  • What changes in the product: Nothing visible to users. Contributors should get faster feedback on simple documentation changes, and documentation links are checked on code and mixed pull requests too.
  • Why it exists: Issue CI: add a fast validation path for narrowly scoped documentation changes #188: a README correction currently waits for the full build and test run and can be blocked by unrelated failures.
  • Why not yet:
    • The blocking code problems from earlier rounds are fixed, and all checks are green.
    • The shortcut has not been demonstrated on a real documentation-only pull request, as the issue requires.
    • The developer now says it needs your explicit authorization to open the temporary pull request. See the question below.
  • What to expect after merging:
    • Documentation errors caught by the checker block the affected pull request, including one that also changes code. The same check runs on main and manual runs.
    • Build jobs wait for the new first job, which took 15 seconds in the latest run. Queue time and total documentation-only savings remain unmeasured.
    • The checker has known blind spots and some false failures, listed below. The current 93 documents pass; that does not mean every link is validated.
    • Follow-up for Harsh: the README's “connection options” link at line 131 points to a section that does not exist. It is already broken on main.

Answer in a comment on this PR; the next daily audit reads it.

Full audit (Claude (fable, xhigh) × Codex (gpt-6-astra, xhigh), 2 turns, commit 3c2772f)

PR audit: #204 — CI: fail-closed fast path for prose-only docs changes

Head: 3c2772f

Sign-off

  • Claude (fable, xhigh): ✅ satisfied — every code finding is fixed and verified; the only item left is the live demonstration, which the developer says needs Harsh's explicit authorization.
  • Codex (gpt-6-astra, xhigh): ✅ satisfied — the blocking heading defects are fixed, the full-path CI evidence is confirmed, and the developer now explicitly reports an authorization obstacle to the required live demonstration. I agree with needs-input.

Decision

needs-input — the blocking code findings are resolved; the required live demonstration remains undone because the developer reports needing Harsh's explicit authorization.

What changes (plain language)

  • What it does: Skips expensive build and test work when a pull request changes only a small list of prose documents. Checks documentation links once, before the other checks, on every pull request.
  • What changes in the product: Nothing visible to users. Contributors should get faster feedback on simple documentation changes, and documentation links are checked on code and mixed pull requests too.
  • Why it exists: Issue CI: add a fast validation path for narrowly scoped documentation changes #188: a README correction currently waits for the full build and test run and can be blocked by unrelated failures.
  • Why not yet:
    • The blocking code problems from earlier rounds are fixed, and all checks are green.
    • The shortcut has not been demonstrated on a real documentation-only pull request, as the issue requires.
    • The developer now says it needs your explicit authorization to open the temporary pull request. See the question below.
  • What to expect after merging:
    • Documentation errors caught by the checker block the affected pull request, including one that also changes code. The same check runs on main and manual runs.
    • Build jobs wait for the new first job, which took 15 seconds in the latest run. Queue time and total documentation-only savings remain unmeasured.
    • The checker has known blind spots and some false failures, listed below. The current 93 documents pass; that does not mean every link is validated.
    • Follow-up for Harsh: the README's “connection options” link at line 131 points to a section that does not exist. It is already broken on main.

Findings

Blocking

  1. .github/workflows/ci.yml:20-46,48-63 — the documentation-only path and its failure gate still lack a live demonstration; the developer reports that authorization is the remaining obstacle.
    • Issue #188 requires a demonstrated documentation-only PR path and code-change path, with successful required checks backed by documentation validation and failing validation blocking the PR.
    • Run 43462289141 on this head shows the full path: 12 successful jobs, all six required checks passing, and the failure-gate step skipped in every required job. It does not show required checks succeeding with docs_only=true or failing after classify fails.
    • Last round asked the developer to run the demonstration or name the obstacle. The latest reply now states: “opening throwaway/demo PRs needs Harsh's explicit authorization under standing orders”. This is an explicit reported constraint, unlike the previous replies that only left the work undone. The underlying standing orders were not supplied for independent inspection.
    • The gates remain sound by inspection. This is an unmet acceptance requirement awaiting Harsh's direction, not a finding that the workflow is broken. The requested evidence remains one green documentation-only run and one broken-link run that fails all six required checks; a separate self-test-failure run remains optional.

Non-blocking

  • Last round's heading blocker is fixed.
    • github_slug stashes code-span contents before stripping markup and restores them before applying the slug character rules (scripts/ci-docs-check.py:81-100). Literal underscores in _meta, OCTOBER_BUS_* and __init__ therefore survive.
    • The setext candidate check now distinguishes bullet markers from emphasis and no longer discards a line just because it starts with a backtick (:166-179). Emphasised, bold and code-span-led setext examples now reach slug generation. This matches the inline-content rule for CommonMark setext headings.
    • The new regression at scripts/ci-docs-fastpath.test.mjs:359-387 covers the five requested positive cases and rejects the old incorrect underscore slug and a nonexistent duplicate suffix. Codex traced these paths; Claude reports successful scratch reproductions, including a link whose label contains code and a list item above a rule remaining a non-heading.
    • The CI log confirms all 14 tests passed, with zero failures or skips. These fixes satisfy the agreed blocking cases; they do not establish complete Markdown compatibility.
  • Earlier fixes still hold. Own-repo URLs retain their root prefix (ci-docs-check.py:104-115,344-347). The existing tests retain the earlier fence, indented-code, comment, backtick-run, escape, encoded-path, query, protocol-relative URL, plain setext and duplicate-slug cases, together with negative link checks. Claude reports rechecking these on scratch files. The current CI log independently confirms validation passed over 93 files.
  • Only two files changed since the previous audit. The checker and its test file contain the heading corrections and new regression. The workflow, classifier and CONTRIBUTING.md are unchanged from 43e1bda.
  • Known blind spots remain; a broken link can pass.
    • List content indented four spaces or a tab after a blank line is treated as code (ci-docs-check.py:262-267). The list regression at ci-docs-fastpath.test.mjs:410-413 covers only a continuation without a blank line.
    • An inline code span containing an unmatched <!-- can hide the rest of the file because comments are blanked before code spans (ci-docs-check.py:233-234,296). No tracked Markdown currently contains that opener.
    • Repository-homepage anchors, shortcut references and multi-line HTML tags remain unchecked (:48,60-67,335-336,373-406). README.md:131 links to the absent #ways-to-connect heading. Thus a passing all-document check does not establish that every inbound link works.
  • Known false failures remain outside the agreed blocking cases. A lone backtick can pair with a code span in a later paragraph (ci-docs-check.py:183-222,296), and a backslash escape inside a link destination is blanked before parsing (:389-391). The “never flags valid docs” assertion at line 189 is too broad. These findings retain their previously agreed non-blocking status.
  • Design note. The checker is 501 lines of hand-written Markdown handling. The agreed dependency-free implementation option has met the requested blocking acceptance cases. This review does not require a library migration, nor endorse the checker as a complete Markdown parser. A maintained parser remains an option if broader syntax support is needed.
  • Minor. The classifier's reason step output is unread (ci-docs-classify.py:150), although its log line is useful. --files-json is used only by tests, and Python is tested through a Node bridge. The PR description now says 14 tests and approximate slugs and explicitly acknowledges the missing demonstration; its local evidence does not replace that demonstration.
  • Unchanged boundaries remain sound. The prose allowlist, rename checks, NUL-delimited diff parsing and fallback to full checks are unchanged. Required jobs use !cancelled() and explicit failure gates, while later steps retain the implicit success() condition (ci.yml:50-65,122-135,177-187,270-280). GitHub status-check expressions. No new action, permission or secret was added, and required check names remain unchanged.

Fix plan

n/a

Message to the developer

n/a

Questions for Harsh

The developer requests your go-ahead to demonstrate the shortcut on a real pull request. How do you want that handled?

Issue #188 requires the demonstration, and the blocking code findings are resolved.

  • Option A — authorise the developer. It opens one temporary pull request against feat/188-ci-docs-fastpath and runs two successive commits:

    1. a README-only edit, which should make all six required checks green across the three operating systems, with heavy work skipped;
    2. a deliberately broken local link, which should make classify and all six required checks red.

    It links both runs on this PR and closes the temporary PR.

  • Option B — do it yourself. Perform the same two steps and provide the run links.

  • Option C — waive the pre-merge demonstration. Merge once the final checks pass and use the first real documentation-only PR as the live test. Keep issue CI: add a fast validation path for narrowly scoped documentation changes #188 open until that demonstration is complete.

Recommendation: A. It satisfies the issue before merge and tests the two workflow branches still unobserved. If you choose C, failures in the unverified shortcut could block a documentation PR or cause it to run the full checks. The classifier sends code changes to full checks by design; its current full-path run is green.

If the two runs behave as described, no further code changes are currently requested, and the PR can merge once final checks pass. If either behaves differently, return it to the developer with the run link.

The authorization request comes from the developer's latest reply, which cites its standing orders. The auditors have not inspected those underlying orders; this question resolves the constraint the developer explicitly reported.

Verification

  • Codex, this turn — read only: confirmed HEAD 3c2772fa4fb1cb4e45df9cd80a54ba499cc30226; read the full two-file diff since 43e1bda, the complete current checker, classifier, workflow and 14-test file, the contributor guidance, PR description, developer discussion and issue CI: add a fast validation path for narrowly scoped documentation changes #188. Rechecked the tracked Markdown count, the README link and headings, and the absence of HTML comment openers in current Markdown.
  • Independent reasoning: traced the five new heading cases and both negative assertions, restoration of code text inside link labels, actual list-marker exclusion, the earlier URL fix, carried-over parser limitations and all required-job gates. Rechecked CommonMark, github-slugger and GitHub expression documentation. Codex ran no project code or scratch probes.
  • CI evidence: run 43462289141 on this head has 12 successful jobs and six passing required checks: interoperability, runtime on three operating systems, typescript-sdk and vulnerabilities. The heavy steps succeeded; the failure gates were skipped. classify ran for 15 seconds.
  • Logs are now readable. Codex retrieved the existing run with gh run view --log. The self-test log reports 14 tests, 14 passes, zero failures and zero skips. The actual classification step reports docs_only=false and reason=non-docs change runs full checks: .github/workflows/ci.yml; validation reports docs link check passed (93 file(s)). This independently confirms successful classification of the PR diff, not the failed-diff fallback. It supplies no documentation-only or red-path demonstration.
  • Claude's turn-1 evidence, retained as reported: scratch runs outside the checkout using python3 -I -B; all 93 files passed, seven heading cases and five earlier acceptance groups passed, and six real-breakage cases were reported. Codex did not rerun them.
  • Limitations and hygiene: plain gh pr view --comments failed on the Projects Classic API field; explicit JSON fields supplied the description and comments. No live workflow was triggered. Codex performed no installs, builds, typechecks, lints or tests and made no GitHub writes or tracked-file changes. The checkout began clean; only the requested untracked AUDIT.md was added.
  • Checks that matter before merge: node --test scripts/ci-docs-fastpath.test.mjs; python3 scripts/ci-docs-check.py --all; a workflow lint for ci.yml; full existing CI on the final commit; and the two live runs above unless Harsh explicitly waives that requirement. Local final checks were not started because the merge decision remains pending.

Audit log

  • Round 1, turn 1 (Claude (fable, xhigh)), head a8e44b2: audited from scratch; four blocking findings (Windows required check could not pass on the fast path; validation empty for the README; false failures on valid Markdown; classify-failure blocking relied on a side effect). Decision: comment.
  • Round 2, turns 1-4 (Claude and Codex), head 49bcf20: confirmed the redesign resolved round 1. Found own-repo URLs resolving against the wrong folder, a list of valid-Markdown rejections, and the missing live demonstration. Agreed decision: comment, with a bounded acceptance list.
  • Round 3, turns 1-3 (Claude and Codex), head 43e1bda: verified the URL fix and the acceptance list. Found a code-span slug regression and formatted setext failures. Agreed decision: comment, asking the developer to fix the headings and either run the live demonstration or name the obstacle.
  • Round 4, turn 1 (Claude (fable, xhigh)), head 3c2772f: read the two-file change and the developer's reply. Verified on scratch files that both heading fixes work, earlier cases still pass and real breakage is still reported; all 93 files pass and CI is green. The developer named the obstacle to the live demonstration (it needs Harsh's explicit authorization), so the decision moves to needs-input with one question.
  • Round 4, turn 2 (Codex (gpt-6-astra, xhigh)), head 3c2772f: independently traced both fixes and the new regressions, re-read the current scripts and workflow, and accepted needs-input based on the newly stated authorization constraint. Retrieved previously unavailable logs confirming 14 passing tests, successful classification of the actual PR diff and the 93-file validation result. Clarified that blocking code findings are resolved while known non-blocking limitations remain, attributed the standing-orders claim to the developer, and removed unsupported timing and absolute outcome claims. The decision, remaining blocker, demonstration requirements, options and recommendation are unchanged; no material changes requiring another audit turn.

Automated review by Harsh's agentbox: two AI reviewers audited this PR against the project ruleset and agreed on it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentbox: needs your input Claude and Codex need a decision from Harsh

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: add a fast validation path for narrowly scoped documentation changes

1 participant