Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
183 changes: 183 additions & 0 deletions doc/audit-reports/2026-11-02.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
# Constitution-vs-reality audit — 2026-11-02

Fourth run under the procedure in `doc/constitution-audit.md` (#535), third fired by
the monthly cloud routine (`doc/scheduled-routines.md`, ACTIVE since 2026-09-06).
Audited `origin/main` at `e7498a3` ("docs: correct scheduled-routines.md's trigger
tool-access claim (#654) (#668)" — merged 2026-10-12, three weeks before this audit
fired). Executed locally in a fresh worktree (`.worktrees/constitution-audit-2026-11`);
hermetic (no real-host runs, no `.env.production`).

**Verdict: one new drift item found — #671, filed `awaiting-founder`.** Only one
commit has touched governance-relevant files since the 2026-10-05 report (`e7498a3`,
the #654 fix itself — confirmed merged, re-verified below). #665 (`release:plan`
baseline-without-tags bug) remains open and accurate — re-confirmed, not re-filed. The
static lenses below (a/b/c, run inside this worktree) were clean, but this PR's own
required `e2e` CI run surfaced a live, diff-independent regression after the static
work was done and reviewed — see the addendum after Lens (b). 4 issues remain
correctly in-flight behind `awaiting-founder`, unchanged from October, plus the new
#671.

## Lens (a) — governance docs vs reality: PASS

| Claim | Reality | Result |
| --- | --- | --- |
| `awaiting-founder` label exists, documented meaning | Exists, `#B60205`, "Blocked on a founder decision or sign-off — needs Ross's attention" | ✅ |
| `founder-approved` label exists | Exists, `#0E8A16`, "Founder has approved this founder-gated change (path-guard gate)" | ✅ |
| `awaiting-founder-notify.yaml` @-mentions the founder on label apply | Source-read: fires on `issues: labeled` / `pull_request_target: labeled`, posts the `@rosscado — … awaiting a founder decision …` comment | ✅ |
| `test.yaml` / `e2e.yaml` do what AGENTS.md/CLAUDE.md claim | Source-read both; unchanged from October, behavior matches exactly | ✅ |
| Required branch-protection checks are exactly `test (22.x)` + `e2e` | No direct branch-protection API tool in this session's toolset (same gap as July/Sept/Oct). Corroborated via PR #668's check-run list: 5 checks ran (`test (22.x)`, `e2e`, `e2e-firefox`, `gitleaks`, `path-guard`) — same shape as every prior report | ✅ (indirect but consistent) |
| `path-guard` still advisory pending founder promotion | `.github/workflows/path-guard.yaml`'s header comment says so verbatim; **#554** (open, `awaiting-founder`) is the live promotion request, unchanged | ✅ (in-flight, correctly not required yet) |
| Founder-gated path enumeration (AGENTS.md) matches the mechanical guard | `scripts/path-guard-lib.mjs`'s `GATED_PATH_PATTERNS` (auth/JWT, `wxt.config.ts` + the two content-script shims, release machinery, `.github/workflows/`, itself) still matches AGENTS.md's "High-blast-radius changes" wording verbatim | ✅ |
| Spend ledger (#533): `.l4-ledger.json` gitignored, caps 6/12h + 25/7-day, env-var overrides wired at all 3 L4 entry points | `.gitignore` covers it; `DEFAULT_CAPS = { session: 6, week: 25 }`; `node scripts/l4-ledger.mjs report` runs clean (0/6, 0/25); `grep -l l4-ledger scripts/*.mjs` → `layer4cdp.mjs`, `e2e-host-sweep.mjs`, `e2e-dictation-sweep.mjs`, `l4-ledger.mjs`, `l4-ledger-lib.mjs` | ✅ |
| AGENTS.md's cap-numbers-still-PROPOSED framing | Still accurate — `l4-ledger.mjs report`'s own banner says "PROPOSED defaults pending founder confirmation" | ✅ |
| `release:status` / `release:freshness` support `--json`, degrade cleanly without credentials | Both ran in this worktree (no `.env.production`); all three stores report `SKIPPED` with a named missing-env reason, exit 0 | ✅ |
| Credential rotation runbook (#534) | "Rotation & lifetimes (#534)" section present in `doc/release/publishing-credentials.md:112`, linked from `doc/release/README.md:21` | ✅ |
| Every doc `AGENTS.md`/`CLAUDE.md` reference exists | 20 checked (full Assistant/Agent Docs list + `doc/autonomous-bootstrap.md`, `src/vad/README.md`) — all present | ✅ |
| Every npm alias a governing doc cites exists | Full `jq .scripts package.json` diffed against every alias named across `AGENTS.md`/`CLAUDE.md`/the harness docs — all present, none renamed | ✅ |
| Memory-vs-docs precedence line present | Present (`AGENTS.md:154`) | ✅ |
| `.cursor/rules` staleness warning still accurate | Both `build_process.mdc` and `vad_model_loading.mdc` still reference `webpack.config.js` / pre-WXT `manifest.json` — warning remains correct | ✅ |
| `doc/scheduled-routines.md`'s tool-access claim | **Now accurate** — confirmed fixed. Read the live file: it now states correctly that neither trigger carries an `allowed_tools` restriction, lists the 9 MCP connectors with empty `permitted_tools`, and frames "hermetic" as a prompted behavior, not a tool-level guarantee. Cross-checked against this run's own `mcp__Claude_Code_Remote__get_trigger` call on `trig_01RoxWz7WbEqtHxe4y62A99M`: the 9 connectors (Google_Calendar, Slack, Gmail, Google_Drive, Figma, Vercel, Linear, Claude_Code_Remote, Format) match exactly, each with `permitted_tools: []`. #654 closed 2026-10-12 via PR #668 — this is the first audit to re-verify the fix | ✅ (was ⚠️ in Sept/Oct, now fixed — closing the loop on #654) |
| `doc/scheduled-routines.md`'s trigger id, cron | `get_trigger` on `trig_01RoxWz7WbEqtHxe4y62A99M` confirms `cron_expression: "30 6 1-7 * 1"`, `enabled: true`, and `last_run.status: ROUTINE_RUN_STATUS_SUCCEEDED` for *this very run* (fired 2026-11-02T06:34:25Z) | ✅ |
| Governance-relevant diff since the 2026-10-05 report | `git log d3574c5..HEAD` touching `AGENTS.md`/`CLAUDE.md`/`doc/`/`.github/workflows/`/path-guard/ledger/`package.json` → exactly one commit, `e7498a3` (the #654 fix above, already accounted for). Nothing else moved | ✅ |
| `awaiting-founder` queue | 4 open issues (`#652`, `#617`, `#556`, `#554`), 0 open PRs — identical to October's queue | ✅ listed below |

One out-of-scope observation, not filed as drift: PR #305 ("Added 60dB integration"),
opened 2026-06-16 by an external fork (`manishEMS47/saypi-userscript`), is still open
and unaddressed. `AGENTS.md`'s Commit & PR Guidelines state external PRs aren't
accepted — this is a stale open external PR, not a documentation-vs-reality
contradiction (the doc says they won't be merged, not that none can be opened), so it
doesn't fit this audit's drift definition. Worth a look by backlog triage (the weekly
routine), not this audit.

## Lens (b) — Layer-3 harness health: PASS

- `npm ci` — clean install.
- `npm run e2e:build` — clean (dev-mode static build, manifest guard satisfied).
- `npm run test:e2e` (full Playwright suite) — **could not complete in this session's
sandbox**, same root cause as every prior report but with a different concrete
number: `@playwright/test@1.60.0` (pinned, unchanged from October) resolves to
Chromium revision **1223**, but this sandbox's pre-baked browser cache now holds
revision **1243** (`/opt/pw-browsers/chromium-1243`) — newer than October's 1194,
but still not the pinned 1223. All 48 specs failed identically and immediately with
`browserType.launchPersistentContext: Executable doesn't exist at
/opt/pw-browsers/chromium-1223/chrome-linux64/chrome` — a clean launch failure, not
a flake or partial run. `npx playwright install` is off-limits per this
environment's own instructions. **This is a sandbox browser-provisioning gap, not
repo drift** (the cache version drifts independently of this repo's pin; nothing in
the repo would fix it), and is not filed.
- **Corroborating evidence the harness itself is healthy**: the required `e2e` check
is `success` on the current `main` HEAD (`e7498a3`) and on every push-to-main run
checked going back through 2026-09-06 (10 runs inspected via the Actions API, 0
failures), spanning the VAD/Silero-v6 upgrade, the endpointing fix, and both the
October and (now-merged) #654 doc PRs. CI installs Chromium fresh every run
(`npx playwright install --with-deps chromium`), so it never sees this sandbox's
stale cache.
- `npm test` (type-check + Jest + Vitest) — **green**: 266 Vitest files, **3051
passed / 1 skipped** (identical counts to the October report); Jest and `tsc
--noEmit` both passed (the `npm-run-all -s` chain exits 0 only if every step
passes, and it did). ~64s total for the Vitest tail alone. Confirms Layer 0–2 are
fully healthy from a fresh checkout with `.wxt/` regenerated via `wxt prepare`.
- **Spec inventory vs `e2e/README.md`**: ran the automated guard directly
(`npx vitest run test/e2e/readme-spec-inventory.spec.ts`) — **5/5 passed**. Spec
count unchanged at **22** (`e2e/specs/*.e2e.ts`) since September/October; all still
have README rows.
- README posture claims spot-checked: `settings.visual.ts` still on-demand / NOT in
the CI gate (own `playwright.visual.config.ts`, `test:e2e:visual*` aliases, absent
from both `test.yaml` and `e2e.yaml`) ✅.

### Addendum — live finding on this PR's own required check (discovered after the static work above)

This audit's own PR (#670) triggered its required `e2e` check as usual on push, and it
**failed** — not the local-sandbox browser-cache gap described above (a different,
CI-only failure). The "Install Playwright Chromium + OS deps" step
(`npx playwright install --with-deps chromium`) failed in ~1s:

```
BEWARE: your OS is not officially supported by Playwright; installing dependencies for ubuntu26.04-x64 as a fallback.
Cannot install dependencies for ubuntu26.04-x64 with Playwright 1.60.0!
Failed to install browsers
Error: ERROR: Playwright does not support chromium on ubuntu26.04-x64
```

GitHub's `ubuntu-latest` runner label now resolves to Ubuntu 26.04; the repo's pinned
`@playwright/test@^1.60.0` cannot install Chromium's OS deps there. This is
diff-independent (PR #670 only adds a doc file) and reproduces on every PR going
forward — a genuine regression in the required merge gate, not something the static
checklist above could catch (it ran `npm ci`/`e2e:build`/`npm test` locally in this
sandbox, which has its own unrelated browser-cache mismatch, and corroborated harness
health via *past* green CI runs — the last of which, on `main`@`e7498a3`
(2026-10-12), predates whatever GitHub-side rollout caused this). `test (22.x)` and
`e2e-firefox` are both unaffected (neither installs a Playwright browser on Chromium;
`e2e-firefox` drives Firefox via geckodriver/Marionette) — confirmed green on this
same PR run.

**Filed as #671** (`awaiting-founder` — the fix touches `.github/workflows/e2e.yaml`,
a founder-gated path, so no agent can push it without `founder-approved`). This is
this cycle's one genuine new drift item: the constitution's claim that the required
gate is `test (22.x)` + `e2e`, both green, is no longer true in practice until #671 is
resolved.

## Lens (c) — Layer-4 harnesses + ops tooling (static): PASS

- All scripts the constitution points at exist: `layer4cdp.mjs`(+lib), `layer35.mjs`,
`dev-rig.mjs`, `e2e-host-sweep.mjs`(+lib), `e2e-dictation-sweep.mjs`(+lib),
`l4-ledger.mjs`(+lib), `store-status.mjs`, `release.mjs`.
- `layer4cdp` subcommands `seed | diagnose | verify | self-test` all implemented and
aliased (`scripts/layer4cdp.mjs:338-341`). ✅
- Host-sweep targets exactly pi.ai / claude.ai / chatgpt.com
(`e2e-host-sweep-lib.mjs:7-11`). ✅
- Dictation-sweep `TARGETS` = `fixture` (:27) / `mistral` (:38) / (and `grok`,
unchanged structure), matching the doc's target table. ✅
- Ledger/caps wiring: verified once under lens (a); env-var overrides referenced in
all three real-host entry points. ✅
- Release/ops tooling: `release:status`/`release:freshness --json` both degrade
cleanly to per-store `SKIPPED` with a named reason, exit 0, no credentials present
(as expected in this worktree); `doc/release/publishing-credentials.md` carries the
rotation section `doc/release/README.md` points to; `doc/post-release-monitoring.md`'s
cited PromQL queries resolve as documented. ✅

No real-host harness was run and no live chat host was messaged, per the hard rule.

## Drift found → issues filed

**One new issue: #671** — the required `e2e` check fails on every PR because GitHub's
`ubuntu-latest` runner now resolves to Ubuntu 26.04, unsupported by the pinned
`@playwright/test@1.60.0`. Diff-independent, confirmed on this audit's own PR. Filed
`awaiting-founder` since the fix edits a founder-gated workflow file — see the Lens
(b) addendum above for the full account. The only known gap carried over from October
(#665, `release:plan` baseline-without-tags) remains open and accurate on
re-verification — not re-filed, per the dedup rule. #654 (the other known October gap)
is now fixed and closed (PR #668) — confirmed above, not reopened.

## Fixed inline in the audit PR

None needed — #671 is a live CI-environment regression unrelated to anything this
audit PR's diff touches, so it's filed as its own issue rather than patched inline
(and the fix is founder-gated regardless).

## In-flight (correctly awaiting founder — listed, NOT drift)

`awaiting-founder` queue at audit time (issues only; 0 open PRs carry the label).
**#671 is this cycle's new addition** (filed above as drift, now also queued for the
founder decision it needs); the rest is identical to the October queue:

- **#671** — Required `e2e` check fails on every PR (`ubuntu-latest` → Ubuntu 26.04
vs pinned Playwright 1.60.0); needs a founder decision on the fix approach and
`founder-approved` sign-off on whichever PR implements it.
- **#652** — Release candidate v1.15.0 ready for founder review (Stage 1 automated
prep; opened 2026-09-07; still carries the #665 baseline-correction caveat noted in
its own comments).
- **#617** — Reconcile the running audio provider when SayPi authentication changes
(opened 2026-09-05; auth/JWT-adjacent, so any fix remains subject to the existing
high-blast-radius review + sign-off rule).
- **#556** — Build the Stage-2 release workflow (one-click store release via
environment-scoped CI secrets); blocked on #554's promotions per its own
acceptance criteria.
- **#554** — Promote `path-guard` and `secret-scan` to required branch-protection
checks once each hits 5 consecutive first-attempt-green runs on distinct
non-docs-only PRs; gates Stage 2/3 of the release-delegation criteria.

## Next audit

First Monday of December 2026 (2026-12-07), 06:30 UTC via the cloud routine
(`doc/scheduled-routines.md`).
Loading