Skip to content

test(mcpverbscheck): pin the exact advertised MCP verb roster — constant-count swap tripwire - #291

Closed
pt-act wants to merge 1 commit into
redhat-et:mainfrom
pt-act:hoplite/zankle-messana-42e1edc0
Closed

pt-act wants to merge 1 commit into
redhat-et:mainfrom
pt-act:hoplite/zankle-messana-42e1edc0

Conversation

@pt-act

@pt-act pt-act commented Sep 19, 2026 •

Copy link
Copy Markdown

…stant-count swap must not slip a verb in

Adds arms (6b)/(6c) to test/mcpverbscheck.sh (L4 section, after the pack_task-absent
assertion — reuses the in-scope l4_field/LIST_OUT2; no second server spawn).

The gap

L4_COUNT==31 fails closed on a verb added, and the per-name arms pin the L4/field-notes
verbs individually — but a rename or swap that keeps the count at 31 and touches a verb no
arm names (analyze, grep, the edit trio, …) passes every existing arm. "Same count + these
names present" is strictly weaker than "the roster is exactly this set". A verb that reaches a
subprocess (a hypothetical run_trace MCP twin of the CLI-only --run-trace) could replace an
unnamed read verb with every existing arm green.

The arms

  • (6b) pins the full sorted advertised roster. ANY add/remove/rename reddens until a human
    updates EXPECTED_VERBS consciously — the point being that a new MCP verb, above all one that
    reaches an exec, is signed for, never a silent drift. The one-line ride-along on every
    verb-changing PR is deliberate: it is the sign-off.
  • (6c) is the mutation control (a synthetic run_trace injected into the live list must
    disturb the comparison — guards the vacuous-arm shape, per §2's "an arm that cannot fail")
    plus a named tripwire asserting no shell/exec-shaped verb is advertised (--run-trace stays
    CLI-only; runtracecheck.sh owns the CLI side).

New arm in an existing gate file, per §6.4 — no new gate, so no regression.sh loop move, no
gatecount_build.py ride-along (verified: manifestcheck rc=0, gatecountcheck rc=0).

Verification (real build, g++ 13.3 / cmake 4.4, ubuntu-24.04)

  • Live tools/list from the built binary matches the pinned set exactly (31 verbs, sorted,
    byte-for-byte).

  • Plant (the gap, demonstrated, not argued): renamed lego → run_trace in
    kMcpVerbTable + the tools/list stanza (constant-count swap) and rebuilt:

    PASS tools/list shows exactly 31 verbs <- the old count arm alone stays green
    FAIL (6b) advertised roster DRIFTED from the pinned set — …
    FAIL (6c) an MCP verb named like a shell/exec entry point is advertised — --run-trace must stay CLI-only

    Reverted the plant; gate ALL PASS with the new arms green.

Context

This came out of a security audit of the MCP surface. Both audit findings closed clean and
neither is this PR: the HTML-report XSS candidate was verified not-a-bug end-to-end
(jsonesc::escapeHtml escapes at emission; all 8 innerHTML sinks escape repo-derived values
or interpolate integers/constexpr vocab), and --run-trace's MCP reachability was confirmed
absent (no such verb; from_trace parses pasted text and executes nothing). This PR is the
durable artifact of that audit: invariant-hardening so the second property stays true by gate,
not by absence.

Summary by CodeRabbit

  • Tests
    • Added coverage to verify that the advertised MCP tool list exactly matches the expected roster.
    • Added checks to detect unexpected additions, removals, or renames in the tool list.
    • Added safeguards confirming that CLI-only commands are not exposed as MCP tools.

…stant-count swap must not slip a verb in

Adds (6b)/(6c) to the L4 section. L4_COUNT==31 fails closed on a verb ADDED
and the per-name arms pin the L4/field-notes verbs individually, but a rename
or swap that keeps the count at 31 and touches a verb no arm names (analyze,
grep, the edit trio, ...) passes every existing arm — CONTRIBUTING §2 shape 7
('true but narrower'). (6b) pins the full sorted advertised roster so ANY
add/remove/rename reddens until a human signs for it; (6c) is the mutation
control proving (6b) can see a new verb, plus a named tripwire asserting no
shell/exec-shaped verb is advertised (--run-trace stays CLI-only).

Context: security-audit follow-up. Both audit findings closed clean — M1
(report XSS) not-a-bug, verified end-to-end: jsonesc::escapeHtml \uXXXX-escapes
<>& at emission and all 8 innerHTML sinks escape repo-derived values or
interpolate integers/constexpr vocab. L1 (--run-trace reachable from MCP) was
never a vuln: no such verb exists. This arm is invariant-hardening so that
stays true, not a vuln fix.

Verified on a real build: planted lego->run_trace in kMcpVerbTable + the
tools/list stanza (constant-count swap) — L4_COUNT==31 stayed green, (6b)
reddened with the diff, (6c) tripwire reddened; reverted, gate ALL PASS.
New arm in an existing gate file — no regression.sh or gate-count ride-along.
@coderabbitai

coderabbitai Bot commented Sep 19, 2026 •

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: redhat-et/ripwire/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: f116ef6b-3962-4533-b8ba-3e4e448cffe1

📥 Commits

Reviewing files that changed from the base of the PR and between 57d713d and f43ffe1.

📒 Files selected for processing (1)
  • test/mcpverbscheck.sh

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.


📝 Walkthrough

Walkthrough

The MCP verb check now pins the 31 advertised tools, compares them with the live tools/list response, tests mutation detection, and rejects trace-related advertised names.

Changes

MCP roster validation

Layer / File(s) Summary
Roster and liveness checks
test/mcpverbscheck.sh
Adds the sorted EXPECTED_VERBS roster, exact live-list comparison, a synthetic run_trace mutation test, and a check that run_trace, run, shell, and exec are not advertised.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Suggested reviewers: joyful-ii-v-i

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the test change: pinning the exact advertised MCP verb roster and detecting constant-count swaps. It is specific and related to the main changes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@joyful-ii-V-I

Copy link
Copy Markdown
Collaborator

Hi @pt-act — thank you for this, and for the writeup. The failure-mode analysis is exactly right: an
"advertise the same count under a different name" swap is real gap in the existing L4 arm, and I like
that (6c) proves (6b) can fail (own words: "prove the arm can fail") rather than just asserting it can.

What's good, with evidence I re-derived myself (not just re-reading your PR description):

  • I independently planted all three mutations in a scratch worktree — add, remove, and the
    count-preserving rename — against the actual tools/list JSON stanza in src/mcp.h (not the
    kMcpVerbTable mirror near the top of that file, which is a documentation/grouping table and is NOT
    what gets served over the wire — worth knowing if you touch this again). Renaming lego→run_probe
    while holding the count at 33 leaves the pre-existing tools/list shows exactly N verbs arm green and
    (6b) red, with a clean symmetric diff (< lego / > run_probe) — precisely the gap you're closing, and
    it fails loudly with a specific message, not an opaque mismatch.
  • (6c)'s mutation control is real — it stayed correctly green (i.e., correctly detected the synthetic
    injection) across every genuine mutation I tried, so it isn't a shape-3 (empty==empty) arm wearing a
    disguise.
  • No new gate file, so no regression.sh/gatecount_build.py registration needed, and I confirmed that
    rather than take the PR's word for it: manifestcheck, gateexitcheck, gatecountcheck all rc=0 on
    your head commit.
  • No conflict with the count-only pin in test/mcpcontractcheck.sh:116 — different file, different
    granularity (count vs. named roster), both stay useful; nothing here needs removing.

Needed before merge:

  1. EXPECTED_VERBS is stale — it pins the pre-train-10 31-verb roster. Since this PR branched, train
    10 (PR Integration train 10: affected and rank_by over MCP #300) landed two new MCP verbs, affected and rank_by, taking the live roster to 33. Against
    current main this PR's own gate fails:
    FAIL  (6b) advertised roster DRIFTED from the pinned set ...
          0a1
          > affected
          25a27
          > rank_by
    
    The fix is additive and small — insert affected (alphabetically first) and rank_by (after
    quality_delta, before replace_symbol_body) into the EXPECTED_VERBS heredoc-style string. Once
    that's done the gate is clean on current main — I verified this locally (33/33 match, rc=0) before
    writing this up, so it's a one-hunk fix, not a rebase fight.
  2. Please rebase (or merge main) so the branch actually builds/tests against current main — right now
    it's 44 commits behind and CI's --quality-delta will show a large, misleading blast radius (trains
    6-10's unrelated work) purely from the base distance, not from anything in this diff. Once EXPECTED_VERBS
    is fixed and the branch is current, I'd expect it clean — a same-population run I did locally (merging
    main into your branch) shows this diff's own quality-delta is regressions=0 gating=0.

Smaller / non-blocking:

  • Once you touch this file for the rebase, it might be worth a one-line comment above EXPECTED_VERBS
    saying where the authoritative source of truth lives (src/mcp.h's tools/list stanza, not
    kMcpVerbTable) — I mention this because it tripped me up for a few minutes during verification, and a
    future maintainer updating this pin will hit the same table first (it's the one with the friendlier
    name and sits right at the top of the file).

What happens next: once EXPECTED_VERBS picks up affected/rank_by and the branch is rebased onto
current main, this is good to merge from my side — the mechanism itself needed no changes, just the
data it pins. Happy to help push that fix if useful.

Dogfood gaps

This was a diff-review + shell-gate task (which file changed, does it pass, plant three mutations,
compare git ref state) — git diff/merge-base/gh pr view and the gate script's own PASS/FAIL/diff
output answered every question directly and were the right tool for each. ./build/ripwire --edit-check
doesn't apply (no C++ symbol changed); --situ/--affected would answer "what should I run for a src/
change" and this diff touches only test/mcpverbscheck.sh, so the gate-file grep I actually did
(grep -rln ... test/*.sh, to find the sibling mcpcontractcheck.sh count pin) is the closer analogue —
tried ripwire . --grep=tools/list mentally but a plain grep -rn "tools/list" test/*.sh was faster
here since I needed cross-file text hits annotated by filename, not by enclosing symbol, and I already
knew the search string verbatim. None where a ripwire verb would have been strictly better.

@joyful-ii-V-I

Copy link
Copy Markdown
Collaborator

Hi @pt-act — thank you again, and sorry for the wait. Your (6b)/(6c) arms were exactly right and needed
no design changes; the only thing stale was the pinned roster itself, which train 10 moved out from
under you (31→33 verbs, affected/rank_by) while this was in review.

What we did: re-derived EXPECTED_VERBS from a live tools/list call on current main rather than
typing the two new names by hand, then re-proved your arm's whole point before landing it — planted an
add, a remove, and the count-preserving rename (lego→run_probe) directly in src/mcp.h's actual
served stanza, and watched all three redden (6b) while the rename case, exactly as you designed it,
left the old count-only check green. That's the gap you closed, still closed.

Landed on lane/t13-contrib-finish, credited to you in the commit body. The orchestrator will close
#291 with this credit once the lane merges — thank you for a mechanism that needed zero rework, only a
data refresh.

Landed in #306 (integration train 13), now on main as 27dedb1. Closing this PR as completed — thank you.

pt-act pushed a commit to pt-act/ripwire that referenced this pull request Sep 21, 2026
…s roster

PR redhat-et#291 (@pt-act) pinned the pre-train-10 31-verb MCP roster in
test/mcpverbscheck.sh's (6b)/(6c) arms. Train 10 (redhat-et#300) added two MCP
verbs, affected and rank_by, taking the live roster to 33, which reddened
the PR's own gate on rebase.

EXPECTED_VERBS is by design a hand-maintained pin, but the VALUE was
derived from the live binary rather than typed from memory: built HEAD,
called tools/list over --mcp, and took the sorted 33-name output verbatim.

Reproved the arm's whole point on this binary before committing: planted
an add, a remove, and a count-preserving rename (lego -> run_probe) in
src/mcp.h's tools/list stanza (not the kMcpVerbTable mirror near the top
of that file, which is not what's served over the wire) and confirmed all
three redden (6b), with the rename case also proving the pre-existing
count-only arm stays green while (6b) catches it — the gap this PR exists
to close.

Credit: @pt-act (PR redhat-et#291, mechanism and (6b)/(6c) arms).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pt-act pushed a commit to pt-act/ripwire that referenced this pull request Sep 21, 2026
…fest pin on the merged tree

fix 3 re-captures a Java/Kotlin `import` as @reference.import, which removes the only
reference train 12's redhat-et#60 module-scope mint had to own in a .kt file. kotlincheck's §1
counts fall back to 3/31/14, its census row to symbols="23", and §1a inverts: the arm
that pinned "exactly ONE module-scope owner, owning the import line" now pins ZERO
owners, the import still VISIBLE on --uses with role="import", and no <file-scope>
in_id anywhere in the fixture. §7a's mutation arm asserts --callers=square falls 1 -> 0
again, keeping train 12's named-edge form, which is the better arm either way.

queries/java/tags.scm's conflict resolved toward fix 3 with train 12's redhat-et#60 consequence
note kept as the record of the call this answers.

qschemetrip.hash re-derived with UPDATE_GOLDEN=1: kParserVer 118 -> 119 and
computeDelta's two new clone-group demotions both move manifest text. NO
kQSnapCacheScheme bump, measured rather than argued -- a blob written by ae6e3e7 is
refused by this build cold and warm ("cache ...-lean.bin: parser-version -- not used"),
and both runs answer 31/14 against the base's 32/15.

printf_parity.manifest: help_all only (UPDATE_GOLDEN_EXPECT matched), from fixes 1 and
4's --help lines. docs/COMMANDS.md regenerated through docs/docs_commands_build.py.

CHANGELOG: the four honesty fixes, and the three contributions credited to @pt-act
(redhat-et#291), @s0undt3ch (redhat-et#298) and @llvm-x86 (redhat-et#302).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@usehoplite
usehoplite Bot deleted the hoplite/zankle-messana-42e1edc0 branch September 21, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants