CoS

A governance layer for running multiple AI coding agents against one workspace.

Context regeneration, cross-tool rule sync with load-banner observability, commit-time secret scanning, and LLM-judge compliance evals. Extracted from six months of daily operation running Claude Code, Codex CLI, and GitHub Copilot CLI against a single workspace.

Reference implementation. It runs my machine every day; adapt the constants to run yours. macOS-flavored (launchd), single-operator by design.

Table of contents

• The problem
• How it works
• Components
• Requirements
• Quick start
• Configuration
• Scheduling
• Testing
• Design principles
• License

The problem

Run several AI coding agents from different vendors against one workspace for long enough and three failures show up:

1. Stale context. An agent only knows what it reads at session start. Workspace docs described tools that were already dead, so agents kept confidently using them.
2. Silently ignored rules. Each vendor loads instructions from a different location, so a rule written for one tool doesn't exist for the others — and even a loaded rule sometimes isn't followed. Both failures are invisible.
3. Leaks. An unsupervised agent that commits code will eventually commit something secret.

CoS attacks each failure with a mechanism that fires automatically — cron, launchd, or git hook — never with a doc that asks anyone to remember anything.

How it works

flowchart LR
    subgraph scheduled [Scheduled - launchd or cron]
        R[cos-refresh.py] -->|regenerates| A[AGENTS.md<br/>LIVE-STACK / DEAD-NAMES<br/>RULES / REGISTRY]
        R --> SR[sync_rules.py] -->|mirrors rules| T[per-tool instruction files]
        E[cos-semantic-eval.py] -->|trap-question probes| AG[fresh agent sessions<br/>claude / copilot]
        D[cos-doctor.py] -->|is the system alive?| A
    end
    subgraph commit [Every commit - git hooks]
        H[hooks/pre-commit] --> S[secret scan]
        H --> J[check_guardrail.py<br/>LLM judge]
    end
    A -->|read at session start| AG

Loading

Every agent reads AGENTS.md at session start; cos-refresh keeps it true. The evals then verify — with fresh sessions and a grader — that the content actually changed agent behavior.

Components

Script	Failure it attacks	What it does
cos-refresh.py	Stale context	Rescans the workspace, regenerates marker-delimited blocks in AGENTS.md: live services (LaunchAgents, agent crons, MCP servers), a DEAD-NAMES reject-list (seeded + auto-detected from observed deletions), rules summary, project registry. Idempotent, atomic writes, dated snapshots.
sync_rules.py	Rules missing per-tool	Mirrors .claude/rules/*.md into each tool's instruction file. Each rule carries a load-banner token the agent must emit in its first response — a missing token tells you which rule didn't load.
hooks/ + check_guardrail.py	Leaks, unenforced conventions	pre-commit blocks secret-shaped strings (keys, tokens, private keys). The guardrail judge (LLM) additionally blocks "notebook-shaped" infrastructure — prose rules with no enforcement mechanism behind them.
cos-semantic-eval.py	Silently ignored rules	Scheduled probes spawn fresh agent sessions across vendors, ask trap questions ("is RetiredTool active?"), grade the answers, log verdicts. Upstream breakage (e.g. an org policy disabling an engine) reports as skipped, not failure.
cos-doctor.py	The meta-failure	Three checks that the system itself is alive: AGENTS.md fresh + markers intact, git hooks wired, no domain silently abandoned. Alerts on new failures only.
cos-eval.py	Content drift	Structural fixtures: does AGENTS.md match reality? Are pinned non-negotiable rules still present in the instruction files after regeneration?
registry_lib.py, scaffold.py, check_os_birth.py	Untracked sprawl	Manifest per project/dataset, registry with drift detection (missing manifests, duplicate purposes), and a scaffolder that makes the valid manifest the path of least resistance.
gh-wrap.py	Off-repo damage	Wraps gh; sends issue/PR drafts to an LLM judge that enforces collaboration rules before anything reaches a shared repo.

Requirements

• macOS (launchd + launchctl for service inventory; the hooks and evals are portable)
• Python 3.10+ (stdlib only — no pip installs)
• git 2.9+ (core.hooksPath)
• Claude Code CLI (claude) — semantic-eval probes
• GitHub Copilot CLI (copilot) — optional second-vendor probes
• gh — only for gh-wrap.py

Quick start

git clone https://github.com/tjp2021/cos.git
cd cos

# 1. Point CoS at your workspace
export COS_WORKSPACE="$HOME/workspace"

# 2. Adapt the constants marked "EXAMPLE" in:
#    cos-refresh.py   -> SEED_DEAD (your decommissioned systems)
#    cos-eval.py      -> GATEWAY_LABEL, CANARY_DEAD_NAME, REQUIRED_RULE_MARKERS
#    cos-semantic-eval.py -> PROBES (write probes for the rules YOU care about)

# 3. First refresh (writes AGENTS.md blocks + a dated snapshot)
python3 cos-refresh.py

# 4. Wire the hooks into your workspace repo
git -C "$COS_WORKSPACE" config core.hooksPath /path/to/cos/hooks

# 5. Verify the system is healthy
python3 cos-doctor.py

# 6. Run the behavioral eval (costs a few cents; see Configuration)
python3 cos-semantic-eval.py

Example eval output:

{
  "ok": true,
  "model": "haiku",
  "passed": 4,
  "skipped": 1,
  "total": 5,
  "results": [
    { "name": "Claude Code: DEAD-NAMES canary rejected", "ok": true, "detail": "all patterns matched" },
    { "name": "Copilot: DEAD-NAMES canary rejected", "ok": true, "skipped": true,
      "detail": "skipped: copilot org policy blocks third-party MCP servers" }
  ]
}

Configuration

All configuration is environment variables. Nothing is hardcoded to a user or machine.

Variable	Default	Used by	Purpose
COS_WORKSPACE	~/workspace	all	Root of the workspace your agents operate on
COS_SNAPSHOT_DIR	$COS_WORKSPACE/datasets/cos-runs	refresh, doctor, registry	Snapshots, state, registry JSONL
COS_LABEL_PREFIXES	com.example.	refresh	Comma-separated LaunchAgent label prefixes that are yours
COS_AGENT_CLI	~/.local/bin/agent	refresh	Optional always-on agent CLI; its cron list feeds LIVE-STACK
COS_EVAL_MODEL	haiku	semantic-eval	Model for probe sessions (claude -p --model …)
OPENROUTER_API_KEY	—	check_guardrail, gh-wrap	LLM judge calls (unset = judge skips, fail-open, loudly)
TELEGRAM_BOT_TOKEN	—	refresh, doctor, semantic-eval	Failure/drift alerts (optional)
COS_NOTIFY_CHAT_ID	—	refresh, doctor, semantic-eval	Telegram chat for those alerts
COS_SKIP_TELEGRAM	—	all alerting	Set 1 to silence alerts (CI, dry runs)

Cost note: cos-semantic-eval.py spawns real agent sessions. With Haiku, a 5-probe run is roughly $0.05–0.20. Each probe is budget-capped (MAX_BUDGET_USD_PER_PROBE). Run it weekly, not on every commit.

Scheduling

launchd (macOS), daily refresh at 07:00 — save as ~/Library/LaunchAgents/com.yourname.cos-refresh.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>Label</key><string>com.yourname.cos-refresh</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/bin/python3</string>
    <string>/path/to/cos/cos-refresh.py</string>
  </array>
  <key>EnvironmentVariables</key>
  <dict><key>COS_WORKSPACE</key><string>/Users/you/workspace</string></dict>
  <key>StartCalendarInterval</key>
  <dict><key>Hour</key><integer>7</integer><key>Minute</key><integer>0</integer></dict>
  <key>StandardErrorPath</key><string>/tmp/cos-refresh.log</string>
</dict></plist>

launchctl load ~/Library/LaunchAgents/com.yourname.cos-refresh.plist

Recommended cadence: cos-refresh daily, cos-doctor daily (offset an hour), cos-semantic-eval weekly. Plain cron works too — the scripts are just Python with exit codes.

Testing

python3 tests/test_check_os_birth.py           # manifest birth-gate fixtures
python3 tests/test_check_guardrail_judge.py    # pinned judge verdicts (needs OPENROUTER_API_KEY)

The judge test runs known-good and known-bad diffs against the live LLM judge — it exists because the judge once wrongly blocked a legitimate commit, and prompt/model drift should be caught on a schedule, not during a commit.

Design principles

1. If it requires remembering, it will atrophy. Everything fires from a scheduler or a git hook. A convention without an enforcement mechanism is exactly what the guardrail judge rejects.
2. Verify delivery, not existence. An earlier auto-injected context block scored 0/22 when fresh sessions were probed for its content. Injecting context ≠ agents receiving it. Every block earns a probe.
3. Alert on changes, not states. A doctor that re-alerts daily on a known failure gets muted, and then there is no doctor. cos-doctor and cos-refresh alert on new failures only.
4. Fail loudly, degrade gracefully. A failed gather preserves the previous block content and reports degraded (exit 2) instead of writing garbage or dying silently.

License

MIT