Skip to content

refactor(config): make the agent config harness-neutral - #144

Merged
ethan-scitix merged 8 commits into
mainfrom
oss-release
Sep 17, 2026
Merged

ethan-scitix merged 8 commits into
mainfrom
oss-release

Conversation

@ethan-scitix

@ethan-scitix ethan-scitix commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Type

  • refactor — code restructuring, no behavior change

Summary

Makes the repo's agent configuration harness-neutral, so opencode gets the
same instructions, skills, and post-edit checks that Claude Code already
had — without duplicating any of it.

  • AGENTS.md becomes the single source of truth for project
    instructions (it was gitignored; now tracked). Root CLAUDE.md
    shrinks to a one-line @AGENTS.md import. Per-layer
    sieval/*/CLAUDE.md files keep their names — that is Claude Code's
    on-demand load mechanism, and renaming them would break it.
  • Harnesses without that mechanism instead get a generated rule map in
    AGENTS.md: scripts/sync_agent_rules.py derives 16 rows from the 6
    layer files plus the paths: front matter of the 10
    .claude/rules/*.md files. Nothing is hand-copied, so a changed glob
    cannot silently drift — check_agent_rule_map (new, 15th preflight
    check) fails if the generator was not re-run.
  • Skills live once at .agents/skills/<name>/SKILL.md, surfaced to
    Claude Code and opencode via symlinks (all four tracked at mode
    120000).
  • The four post-edit grep rules that lived inline in
    .claude/settings.json move to scripts/post_edit_checks.py, shared
    by two callers: Claude Code's hook and .opencode/plugins/post-edit.ts
    (listens on file.edited, zero dependencies). Both pass the edited
    path as argv[1].

No Codex wiring (changed during review)

This branch originally shipped a .codex/config.toml. Review tested it
against a real Codex 0.144.4 and it registered zero hooks, with empty
warnings and errors — silent on every channel. Asked via Codex's own
hooks/list RPC:

this branch's file → {"hooks":[], "warnings":[], "errors":[]}
documented form    → {"hooks":[{... "matcher":"Bash" ...}]}

Three independent causes, each sufficient on its own:

  1. command sat on the matcher table, but ConfiguredHookMatcherGroup
    carries only matcher and hooks — the handler needs a nested
    [[hooks.PostToolUse.hooks]] with type = "command".
  2. matcher = "shell" matches no tool name (shell is Bash, edits are
    apply_patch). A wrong matcher still registers, so this would have
    failed silently a second time even after fixing (1).
  3. Codex delivers the payload as JSON on stdin; post_edit_checks.py
    reads argv. A corrected file would still run the script with no path.

A registered hook also comes back trustStatus: "untrusted" — project
trust does not cover it.

Writing the stdin adapter is the right fix, but no Codex credential
exists in this environment (codex exec → 401), so its payload shape
cannot be observed and the result could not be verified before shipping.
A config that silently does nothing is worse than none: it reads as
wired. The file is removed; AGENTS.md records why, and Codex users run
python scripts/post_edit_checks.py <path> by hand.

Two gaps found while testing (fixed here)

  • .opencode/commands/ was ignored by literal filename, so the next
    personal skill would ship: its symlink points at a gitignored target,
    and a public clone would get a dangling link. Now pattern-based like
    the two blocks above it.
  • sanitize.sh's guideline allowlist was unanchored. It filters git grep
    output lines, so AGENTS.md exempted every line anywhere that merely
    named the file — BAD = "/home/u/x" # see AGENTS.md passed the
    hardcoded-path scan. Anchored to the path:lineno: prefix; CLAUDE.md
    had the same flaw and is fixed with it.

Test Plan

Automated

  • Lint/format clean (ruff check && ruff format --check)
  • Type check clean (ty check)
  • Unit tests pass — 422 passed in tests/unit/scripts
  • Full preflight green — 26 checks, including the new check_agent_rule_map
  • scripts/sanitize.sh clean — no sensitive patterns

Manual

  • Behavior preservation of the extracted checks: select_checks()
    fuzz-tested against a literal transliteration of the four old grep
    regexes across 20,000 random paths — 0 mismatches. This includes the
    deliberate asymmetry where .pyi does not trigger the meta check
    (the old pattern ended in \.py$), which is pinned by a test.
  • Exit-0 guarantee holds on every path probed: missing binary
    (OSError caught), no arguments, unmatched path, and a check that
    actually runs and fails.
  • Claude Code hook fires for real — wrote a file under
    sieval/tasks/ in a live session and captured both sync scripts
    running. Feeding the same payload to the new single-hook command
    gives 2 / 0 / 0 sync invocations for a task file, a scripts/ file,
    and an empty path.
  • opencode end-to-end fires for real — a model-driven edit
    (opencode run, free muse-spark-1.3-contributor-free tier) changed
    the file, and file.edited reached the script: both sync scripts ran
    and the missing-binary path exited 0. Payload shape also matches the
    installed SDK type defs (EventFileEdited.properties.file).
    Confirmed opencode emits file.edited from inside the
    write/edit/patch tools, not from a file watcher — so the plugin
    tracks agent edits rather than any change on disk.
  • Symlinks resolve; a new personal skill is now ignored across all
    three surfaces (probed by creating one), with both tracked symlinks
    still at mode 120000.
  • sanitize.sh catches a self-exempting # see AGENTS.md probe line
    and stays clean on the real tree.
  • Rule-map parser rejects an unreadable paths: key (reverse
    mutation: with the guard deleted, all three new cases fail with
    DID NOT RAISE).
  • Codex is not wired — deliberately, see above. Nothing to verify.

Harness differences measured (documented in AGENTS.md)

Layer and scoped rules are auto-loaded only under Claude Code. Tested
on a synthetic tree: Codex never reads a CLAUDE.md below the repo root —
not even with the working directory inside that folder — and renaming the
same file AGENTS.md loads it immediately; opencode only walks upward
from the working directory. So for those harnesses the 16 layer and scoped
rules are a pointer to follow, not an injection.

Checklist

Required (all PRs)

  • PR title follows conventional format (type(scope): description)
  • No internal paths, credentials, or personal info in committed files
  • AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
  • No new upper-layer dependencies added to core/
  • Deleted code verified — the four inline grep rules have exactly one
    replacement each in post_edit_checks.py, and no other caller read them

🤖 Generated with Claude Code

ethan-scitix and others added 8 commits September 15, 2026 14:03
Codex reads only AGENTS.md and has no import mechanism, so the full text
moves there; CLAUDE.md becomes a one-line @AGENTS.md import, which Claude
Code expands to the same content. Carries over the layer-dependency
diagram that the draft had rewritten as prose, and replaces the draft's
Tool Bridge section — which described a bridge that was never built.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude Code auto-loads sieval/*/CLAUDE.md and .claude/rules/*.md; opencode
and Codex do neither, so they need an explicit index. Generating it from
the rules' own paths: frontmatter keeps the copy from going stale, and
check_agent_rule_map fails CI when it does. Also asserts the 32 KiB budget
Codex silently truncates past.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codex has no project-level commands and reads skills only from
.agents/skills/, so that becomes the source of truth, with symlinks for
Claude Code (.claude/skills/) and opencode (.opencode/commands/ — its
skills are model-invoked only, so the command link is what restores the
slash trigger and argument passing).

Retires 5 unused commands: the three archived ones plus the weekly and
biweekly summaries.

.claude/commands/ keeps its .gitignore entry: sieval-pr-review.md.bak
remains on disk there and still needs hiding, so the directory is not
actually empty.

.gitignore needed two more re-inclusions than the .agents/skills/ block
alone: .claude/skills/* had no negations, which would have left the
Claude Code symlinks themselves untracked (verified in a scratch repo -
git status stayed silent on them even after `git add -A`), and opencode's
own sieval-pr-review.md command-symlink pointed at the gitignored
personal skill and had no rule of its own, so it would have shipped as a
dangling symlink. Both are fixed the same way: negate the two team-skill
names, leave the personal one covered by the trailing `*`.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three harnesses need the same four checks after an edit. Inlining them per
tool would duplicate behavior, and drift there is silent — a one-character
difference in a path pattern means a file class stops being checked with no
error anywhere. Each harness now dispatches to scripts/post_edit_checks.py.

Also adds .opencode/node_modules/ to the root .gitignore: the existing
.opencode/.gitignore excludes itself (its own last line is `.gitignore`), so
it is never committed and gives a fresh clone no protection.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The `.claude/skills/` negations must stay slash-free because they name
symlinks; a trailing-slash negation matches only real directories and
would silently untrack both. The neighbouring `.agents/skills/` block
does carry slashes, so the two look inconsistent and invite a cleanup
that breaks tracking with no error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…an gaps

Follow-up to the harness-neutral config work, from testing all three
harnesses against their real binaries rather than their docs.

Remove `.codex/config.toml`. Asked Codex 0.144.4 what it actually loaded
via its own `hooks/list` RPC: the file registered zero hooks, with empty
`warnings` and `errors` — silent on every channel. Three independent
causes, each sufficient: `command` sat on the matcher table, but
`ConfiguredHookMatcherGroup` only carries `matcher` and `hooks`, so the
handler needs a nested `[[hooks.PostToolUse.hooks]]`; `matcher = "shell"`
matches no tool name (shell is `Bash`, edits are `apply_patch`), and a
wrong matcher still registers, so it would fail silently a second time;
and Codex delivers the payload as JSON on stdin while
`post_edit_checks.py` reads `argv`, so even a corrected file would run
the script with no path. A registered hook also comes back
`trustStatus: "untrusted"` — project trust does not cover it.

Writing the adapter is the right fix, but no Codex credential exists
here, so its stdin payload shape cannot be observed and the result could
not be verified before shipping. A config that silently does nothing is
worse than none: it reads as wired. `AGENTS.md` now records why the
wiring is absent and how to run the checks by hand under Codex.

Make the `.opencode/commands/` ignore pattern-based, matching the two
blocks above it. It listed one personal skill by filename, so the *next*
one shipped: its symlink points at a gitignored target, and a public
clone would get a dangling link. Verified a fresh personal skill is now
ignored across all three surfaces while both tracked symlinks stay at
mode 120000 — the negations stay slash-free for the reason the
neighbouring comment gives.

Anchor the two guideline entries in `sanitize.sh`. The allowlist filters
git grep output lines, so an unanchored `AGENTS.md` exempted every line
anywhere that merely named the file: `BAD = "/home/u/x"  # see AGENTS.md`
passed the hardcoded-path scan. Now caught, with the guideline files
themselves still exempt. `CLAUDE.md` had the same flaw and is fixed with
it.

Also document what the harnesses really do with layer rules, measured on
a synthetic tree: Codex never reads a `CLAUDE.md` below the root, not
even with the working directory inside that folder (renaming it
`AGENTS.md` loads it immediately), and opencode only walks upward. The 16
layer and scoped rules are therefore automatic under Claude Code alone;
elsewhere the generated map is a pointer, not an injection.

Validation: 418 tests in tests/unit/scripts, full preflight green (25
checks), sanitize clean, ruff check + format clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Trim the comments and docstrings added across this branch, keeping the
reason each one records — every block here documents a trap that cost a
debugging session — while cutting restatement and hedging.

Two were also stale after the Codex hook wiring came out: the
`post_edit_checks.py` docstring still named three harnesses, and the
opencode plugin comment pointed at a Codex that no longer shares it. The
plugin comment now says why `file.edited` is the right event — opencode
emits it from inside the write/edit/patch tools, so it tracks agent edits
rather than any change on disk.

Validation: 418 tests in tests/unit/scripts, full preflight green (26
checks, no FAIL/WARN), sanitize clean, ruff check + format clean. Both
behavioural fixes re-verified after the trim: a fresh personal skill is
still ignored across all three surfaces with both tracked symlinks at
mode 120000, and the self-exempting `# see AGENTS.md` probe line is still
caught.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The frontmatter parser returned [] for anything that was not a block
list, so `paths: ["a/**"]` — valid YAML, and the form someone reaching
for brevity would write — silently dropped that rule from the map and
rendered it as `_(always)_`.

`check_agent_rule_map` could not catch it: the check re-parses the same
frontmatter the same wrong way, so both sides agree and it passes. A
generator whose whole premise is that an omission cannot happen silently
had one failure mode where it does.

Raise instead, naming the file. An absent `paths:` key still means
"applies everywhere" — that is a real choice, and it keeps working; only
a key that is present and unreadable is an error.

Verified by reverse mutation: with the guard deleted, all three new
parametrised cases fail with DID NOT RAISE.

Left alone: the layer glob stays one level (`sieval/*/CLAUDE.md`).
Missing a nested layer file is a conservative failure, the six layer
directories are a fixed convention, and `**` would sweep in vendored
`sieval/community/<vendor>/CLAUDE.md` files.

Validation: 422 tests in tests/unit/scripts, full preflight green (26
checks), ruff check + format clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ethan-scitix
ethan-scitix merged commit 06888ee into main Sep 17, 2026
9 checks passed
@ethan-scitix
ethan-scitix deleted the oss-release branch September 18, 2026 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant