diff --git a/doc/audit-reports/2026-11-02.md b/doc/audit-reports/2026-11-02.md new file mode 100644 index 0000000000..a8ed47f011 --- /dev/null +++ b/doc/audit-reports/2026-11-02.md @@ -0,0 +1,183 @@ +# Constitution-vs-reality audit — 2026-11-02 + +Fourth run under the procedure in `doc/constitution-audit.md` (#535), third fired by +the monthly cloud routine (`doc/scheduled-routines.md`, ACTIVE since 2026-09-06). +Audited `origin/main` at `e7498a3` ("docs: correct scheduled-routines.md's trigger +tool-access claim (#654) (#668)" — merged 2026-10-12, three weeks before this audit +fired). Executed locally in a fresh worktree (`.worktrees/constitution-audit-2026-11`); +hermetic (no real-host runs, no `.env.production`). + +**Verdict: one new drift item found — #671, filed `awaiting-founder`.** Only one +commit has touched governance-relevant files since the 2026-10-05 report (`e7498a3`, +the #654 fix itself — confirmed merged, re-verified below). #665 (`release:plan` +baseline-without-tags bug) remains open and accurate — re-confirmed, not re-filed. The +static lenses below (a/b/c, run inside this worktree) were clean, but this PR's own +required `e2e` CI run surfaced a live, diff-independent regression after the static +work was done and reviewed — see the addendum after Lens (b). 4 issues remain +correctly in-flight behind `awaiting-founder`, unchanged from October, plus the new +#671. + +## Lens (a) — governance docs vs reality: PASS + +| Claim | Reality | Result | +| --- | --- | --- | +| `awaiting-founder` label exists, documented meaning | Exists, `#B60205`, "Blocked on a founder decision or sign-off — needs Ross's attention" | ✅ | +| `founder-approved` label exists | Exists, `#0E8A16`, "Founder has approved this founder-gated change (path-guard gate)" | ✅ | +| `awaiting-founder-notify.yaml` @-mentions the founder on label apply | Source-read: fires on `issues: labeled` / `pull_request_target: labeled`, posts the `@rosscado — … awaiting a founder decision …` comment | ✅ | +| `test.yaml` / `e2e.yaml` do what AGENTS.md/CLAUDE.md claim | Source-read both; unchanged from October, behavior matches exactly | ✅ | +| Required branch-protection checks are exactly `test (22.x)` + `e2e` | No direct branch-protection API tool in this session's toolset (same gap as July/Sept/Oct). Corroborated via PR #668's check-run list: 5 checks ran (`test (22.x)`, `e2e`, `e2e-firefox`, `gitleaks`, `path-guard`) — same shape as every prior report | ✅ (indirect but consistent) | +| `path-guard` still advisory pending founder promotion | `.github/workflows/path-guard.yaml`'s header comment says so verbatim; **#554** (open, `awaiting-founder`) is the live promotion request, unchanged | ✅ (in-flight, correctly not required yet) | +| Founder-gated path enumeration (AGENTS.md) matches the mechanical guard | `scripts/path-guard-lib.mjs`'s `GATED_PATH_PATTERNS` (auth/JWT, `wxt.config.ts` + the two content-script shims, release machinery, `.github/workflows/`, itself) still matches AGENTS.md's "High-blast-radius changes" wording verbatim | ✅ | +| Spend ledger (#533): `.l4-ledger.json` gitignored, caps 6/12h + 25/7-day, env-var overrides wired at all 3 L4 entry points | `.gitignore` covers it; `DEFAULT_CAPS = { session: 6, week: 25 }`; `node scripts/l4-ledger.mjs report` runs clean (0/6, 0/25); `grep -l l4-ledger scripts/*.mjs` → `layer4cdp.mjs`, `e2e-host-sweep.mjs`, `e2e-dictation-sweep.mjs`, `l4-ledger.mjs`, `l4-ledger-lib.mjs` | ✅ | +| AGENTS.md's cap-numbers-still-PROPOSED framing | Still accurate — `l4-ledger.mjs report`'s own banner says "PROPOSED defaults pending founder confirmation" | ✅ | +| `release:status` / `release:freshness` support `--json`, degrade cleanly without credentials | Both ran in this worktree (no `.env.production`); all three stores report `SKIPPED` with a named missing-env reason, exit 0 | ✅ | +| Credential rotation runbook (#534) | "Rotation & lifetimes (#534)" section present in `doc/release/publishing-credentials.md:112`, linked from `doc/release/README.md:21` | ✅ | +| Every doc `AGENTS.md`/`CLAUDE.md` reference exists | 20 checked (full Assistant/Agent Docs list + `doc/autonomous-bootstrap.md`, `src/vad/README.md`) — all present | ✅ | +| Every npm alias a governing doc cites exists | Full `jq .scripts package.json` diffed against every alias named across `AGENTS.md`/`CLAUDE.md`/the harness docs — all present, none renamed | ✅ | +| Memory-vs-docs precedence line present | Present (`AGENTS.md:154`) | ✅ | +| `.cursor/rules` staleness warning still accurate | Both `build_process.mdc` and `vad_model_loading.mdc` still reference `webpack.config.js` / pre-WXT `manifest.json` — warning remains correct | ✅ | +| `doc/scheduled-routines.md`'s tool-access claim | **Now accurate** — confirmed fixed. Read the live file: it now states correctly that neither trigger carries an `allowed_tools` restriction, lists the 9 MCP connectors with empty `permitted_tools`, and frames "hermetic" as a prompted behavior, not a tool-level guarantee. Cross-checked against this run's own `mcp__Claude_Code_Remote__get_trigger` call on `trig_01RoxWz7WbEqtHxe4y62A99M`: the 9 connectors (Google_Calendar, Slack, Gmail, Google_Drive, Figma, Vercel, Linear, Claude_Code_Remote, Format) match exactly, each with `permitted_tools: []`. #654 closed 2026-10-12 via PR #668 — this is the first audit to re-verify the fix | ✅ (was ⚠️ in Sept/Oct, now fixed — closing the loop on #654) | +| `doc/scheduled-routines.md`'s trigger id, cron | `get_trigger` on `trig_01RoxWz7WbEqtHxe4y62A99M` confirms `cron_expression: "30 6 1-7 * 1"`, `enabled: true`, and `last_run.status: ROUTINE_RUN_STATUS_SUCCEEDED` for *this very run* (fired 2026-11-02T06:34:25Z) | ✅ | +| Governance-relevant diff since the 2026-10-05 report | `git log d3574c5..HEAD` touching `AGENTS.md`/`CLAUDE.md`/`doc/`/`.github/workflows/`/path-guard/ledger/`package.json` → exactly one commit, `e7498a3` (the #654 fix above, already accounted for). Nothing else moved | ✅ | +| `awaiting-founder` queue | 4 open issues (`#652`, `#617`, `#556`, `#554`), 0 open PRs — identical to October's queue | ✅ listed below | + +One out-of-scope observation, not filed as drift: PR #305 ("Added 60dB integration"), +opened 2026-06-16 by an external fork (`manishEMS47/saypi-userscript`), is still open +and unaddressed. `AGENTS.md`'s Commit & PR Guidelines state external PRs aren't +accepted — this is a stale open external PR, not a documentation-vs-reality +contradiction (the doc says they won't be merged, not that none can be opened), so it +doesn't fit this audit's drift definition. Worth a look by backlog triage (the weekly +routine), not this audit. + +## Lens (b) — Layer-3 harness health: PASS + +- `npm ci` — clean install. +- `npm run e2e:build` — clean (dev-mode static build, manifest guard satisfied). +- `npm run test:e2e` (full Playwright suite) — **could not complete in this session's + sandbox**, same root cause as every prior report but with a different concrete + number: `@playwright/test@1.60.0` (pinned, unchanged from October) resolves to + Chromium revision **1223**, but this sandbox's pre-baked browser cache now holds + revision **1243** (`/opt/pw-browsers/chromium-1243`) — newer than October's 1194, + but still not the pinned 1223. All 48 specs failed identically and immediately with + `browserType.launchPersistentContext: Executable doesn't exist at + /opt/pw-browsers/chromium-1223/chrome-linux64/chrome` — a clean launch failure, not + a flake or partial run. `npx playwright install` is off-limits per this + environment's own instructions. **This is a sandbox browser-provisioning gap, not + repo drift** (the cache version drifts independently of this repo's pin; nothing in + the repo would fix it), and is not filed. +- **Corroborating evidence the harness itself is healthy**: the required `e2e` check + is `success` on the current `main` HEAD (`e7498a3`) and on every push-to-main run + checked going back through 2026-09-06 (10 runs inspected via the Actions API, 0 + failures), spanning the VAD/Silero-v6 upgrade, the endpointing fix, and both the + October and (now-merged) #654 doc PRs. CI installs Chromium fresh every run + (`npx playwright install --with-deps chromium`), so it never sees this sandbox's + stale cache. +- `npm test` (type-check + Jest + Vitest) — **green**: 266 Vitest files, **3051 + passed / 1 skipped** (identical counts to the October report); Jest and `tsc + --noEmit` both passed (the `npm-run-all -s` chain exits 0 only if every step + passes, and it did). ~64s total for the Vitest tail alone. Confirms Layer 0–2 are + fully healthy from a fresh checkout with `.wxt/` regenerated via `wxt prepare`. +- **Spec inventory vs `e2e/README.md`**: ran the automated guard directly + (`npx vitest run test/e2e/readme-spec-inventory.spec.ts`) — **5/5 passed**. Spec + count unchanged at **22** (`e2e/specs/*.e2e.ts`) since September/October; all still + have README rows. +- README posture claims spot-checked: `settings.visual.ts` still on-demand / NOT in + the CI gate (own `playwright.visual.config.ts`, `test:e2e:visual*` aliases, absent + from both `test.yaml` and `e2e.yaml`) ✅. + +### Addendum — live finding on this PR's own required check (discovered after the static work above) + +This audit's own PR (#670) triggered its required `e2e` check as usual on push, and it +**failed** — not the local-sandbox browser-cache gap described above (a different, +CI-only failure). The "Install Playwright Chromium + OS deps" step +(`npx playwright install --with-deps chromium`) failed in ~1s: + +``` +BEWARE: your OS is not officially supported by Playwright; installing dependencies for ubuntu26.04-x64 as a fallback. +Cannot install dependencies for ubuntu26.04-x64 with Playwright 1.60.0! +Failed to install browsers +Error: ERROR: Playwright does not support chromium on ubuntu26.04-x64 +``` + +GitHub's `ubuntu-latest` runner label now resolves to Ubuntu 26.04; the repo's pinned +`@playwright/test@^1.60.0` cannot install Chromium's OS deps there. This is +diff-independent (PR #670 only adds a doc file) and reproduces on every PR going +forward — a genuine regression in the required merge gate, not something the static +checklist above could catch (it ran `npm ci`/`e2e:build`/`npm test` locally in this +sandbox, which has its own unrelated browser-cache mismatch, and corroborated harness +health via *past* green CI runs — the last of which, on `main`@`e7498a3` +(2026-10-12), predates whatever GitHub-side rollout caused this). `test (22.x)` and +`e2e-firefox` are both unaffected (neither installs a Playwright browser on Chromium; +`e2e-firefox` drives Firefox via geckodriver/Marionette) — confirmed green on this +same PR run. + +**Filed as #671** (`awaiting-founder` — the fix touches `.github/workflows/e2e.yaml`, +a founder-gated path, so no agent can push it without `founder-approved`). This is +this cycle's one genuine new drift item: the constitution's claim that the required +gate is `test (22.x)` + `e2e`, both green, is no longer true in practice until #671 is +resolved. + +## Lens (c) — Layer-4 harnesses + ops tooling (static): PASS + +- All scripts the constitution points at exist: `layer4cdp.mjs`(+lib), `layer35.mjs`, + `dev-rig.mjs`, `e2e-host-sweep.mjs`(+lib), `e2e-dictation-sweep.mjs`(+lib), + `l4-ledger.mjs`(+lib), `store-status.mjs`, `release.mjs`. +- `layer4cdp` subcommands `seed | diagnose | verify | self-test` all implemented and + aliased (`scripts/layer4cdp.mjs:338-341`). ✅ +- Host-sweep targets exactly pi.ai / claude.ai / chatgpt.com + (`e2e-host-sweep-lib.mjs:7-11`). ✅ +- Dictation-sweep `TARGETS` = `fixture` (:27) / `mistral` (:38) / (and `grok`, + unchanged structure), matching the doc's target table. ✅ +- Ledger/caps wiring: verified once under lens (a); env-var overrides referenced in + all three real-host entry points. ✅ +- Release/ops tooling: `release:status`/`release:freshness --json` both degrade + cleanly to per-store `SKIPPED` with a named reason, exit 0, no credentials present + (as expected in this worktree); `doc/release/publishing-credentials.md` carries the + rotation section `doc/release/README.md` points to; `doc/post-release-monitoring.md`'s + cited PromQL queries resolve as documented. ✅ + +No real-host harness was run and no live chat host was messaged, per the hard rule. + +## Drift found → issues filed + +**One new issue: #671** — the required `e2e` check fails on every PR because GitHub's +`ubuntu-latest` runner now resolves to Ubuntu 26.04, unsupported by the pinned +`@playwright/test@1.60.0`. Diff-independent, confirmed on this audit's own PR. Filed +`awaiting-founder` since the fix edits a founder-gated workflow file — see the Lens +(b) addendum above for the full account. The only known gap carried over from October +(#665, `release:plan` baseline-without-tags) remains open and accurate on +re-verification — not re-filed, per the dedup rule. #654 (the other known October gap) +is now fixed and closed (PR #668) — confirmed above, not reopened. + +## Fixed inline in the audit PR + +None needed — #671 is a live CI-environment regression unrelated to anything this +audit PR's diff touches, so it's filed as its own issue rather than patched inline +(and the fix is founder-gated regardless). + +## In-flight (correctly awaiting founder — listed, NOT drift) + +`awaiting-founder` queue at audit time (issues only; 0 open PRs carry the label). +**#671 is this cycle's new addition** (filed above as drift, now also queued for the +founder decision it needs); the rest is identical to the October queue: + +- **#671** — Required `e2e` check fails on every PR (`ubuntu-latest` → Ubuntu 26.04 + vs pinned Playwright 1.60.0); needs a founder decision on the fix approach and + `founder-approved` sign-off on whichever PR implements it. +- **#652** — Release candidate v1.15.0 ready for founder review (Stage 1 automated + prep; opened 2026-09-07; still carries the #665 baseline-correction caveat noted in + its own comments). +- **#617** — Reconcile the running audio provider when SayPi authentication changes + (opened 2026-09-05; auth/JWT-adjacent, so any fix remains subject to the existing + high-blast-radius review + sign-off rule). +- **#556** — Build the Stage-2 release workflow (one-click store release via + environment-scoped CI secrets); blocked on #554's promotions per its own + acceptance criteria. +- **#554** — Promote `path-guard` and `secret-scan` to required branch-protection + checks once each hits 5 consecutive first-attempt-green runs on distinct + non-docs-only PRs; gates Stage 2/3 of the release-delegation criteria. + +## Next audit + +First Monday of December 2026 (2026-12-07), 06:30 UTC via the cloud routine +(`doc/scheduled-routines.md`).