Summary
Stand up an optional CI job (and local make target) that exercises MCPDriver
end-to-end against real reference MCP servers — at minimum the official
"everything" test server — covering discovery, annotation inference, invocation,
errors, and large outputs through the firewall.
Why this matters
MCP is the kernel's most strategically important driver, and the MCP SDK and
protocol evolve quickly. Current tests exercise the driver against mocks
(tests/test_mcp_driver.py), which cannot catch protocol-version drift, SDK
behavior changes, or annotation-shape changes server-side — the failure mode
adopters actually hit. An interop matrix converts "works with MCP" from a claim
into a tested property, and it directly supports the safety-classification work
(ISSUE 9) with real annotation data.
Current evidence
drivers/mcp.py + drivers/mcp_support.py implement discovery/invocation; tests/test_mcp_driver.py uses in-process fakes only.
pyproject.toml mcp extra pins the SDK range; nothing validates real-server behavior across SDK upgrades.
- The CI Examples step never starts an MCP server; no interop coverage exists anywhere.
External context
The MCP project publishes reference servers (e.g., the "everything" server designed
for client testing) runnable via npx/uvx — purpose-built for exactly this kind of
matrix.
Proposed implementation
- Add
tests/integration/test_mcp_interop.py gated by a marker
(-m mcp_interop) and skipped unless the server toolchain is present.
- CI: a separate non-blocking job installs
.[mcp] + Node, launches the
reference server, runs the marked tests; promote to blocking once stable.
- Cover: tool discovery (count/annotations), safety-class inference outcomes,
a successful invoke through firewall transform, server-error mapping to
DriverError, oversized output meeting budgets.
- Pin server versions; refresh deliberately (pairs with ISSUE 55 Dependabot).
AI-agent execution notes
- Inspect first:
drivers/mcp.py (transport setup), tests/test_mcp_driver.py (fixtures to reuse), .github/workflows/ci.yml.
- Keep the job isolated: regular
make ci must not require Node or network.
- Edge cases: server startup latency (health-check loop); stdio vs SSE transport coverage; flaky-network policy (retries, non-blocking until proven stable).
- Do not loosen driver code to accommodate server quirks silently — file findings.
Acceptance criteria
- Marked interop tests pass locally against the reference server with documented setup.
- CI runs the matrix on a schedule or per-PR (non-blocking initially).
- At least discovery, invoke, error, and large-output paths are covered.
Test plan
The matrix is the test plan; plus a README-in-tests note for local running.
Run make ci (unchanged) and the new job.
Documentation plan
docs/integrations.md MCP section gains a "tested against" matrix; CHANGELOG
Added (CI).
Migration and compatibility notes
Not expected to require migration.
Risks and tradeoffs
External-process tests bring flakiness — isolation, pinning, and non-blocking
status manage it. Node dependency in one CI job is contained cost.
Suggested labels
ecosystem, testing, integrations, reliability
Summary
Stand up an optional CI job (and local make target) that exercises
MCPDriverend-to-end against real reference MCP servers — at minimum the official
"everything" test server — covering discovery, annotation inference, invocation,
errors, and large outputs through the firewall.
Why this matters
MCP is the kernel's most strategically important driver, and the MCP SDK and
protocol evolve quickly. Current tests exercise the driver against mocks
(
tests/test_mcp_driver.py), which cannot catch protocol-version drift, SDKbehavior changes, or annotation-shape changes server-side — the failure mode
adopters actually hit. An interop matrix converts "works with MCP" from a claim
into a tested property, and it directly supports the safety-classification work
(ISSUE 9) with real annotation data.
Current evidence
drivers/mcp.py+drivers/mcp_support.pyimplement discovery/invocation;tests/test_mcp_driver.pyuses in-process fakes only.pyproject.tomlmcpextra pins the SDK range; nothing validates real-server behavior across SDK upgrades.External context
The MCP project publishes reference servers (e.g., the "everything" server designed
for client testing) runnable via npx/uvx — purpose-built for exactly this kind of
matrix.
Proposed implementation
tests/integration/test_mcp_interop.pygated by a marker(
-m mcp_interop) and skipped unless the server toolchain is present..[mcp]+ Node, launches thereference server, runs the marked tests; promote to blocking once stable.
a successful invoke through firewall transform, server-error mapping to
DriverError, oversized output meeting budgets.AI-agent execution notes
drivers/mcp.py(transport setup),tests/test_mcp_driver.py(fixtures to reuse),.github/workflows/ci.yml.make cimust not require Node or network.Acceptance criteria
Test plan
The matrix is the test plan; plus a README-in-tests note for local running.
Run
make ci(unchanged) and the new job.Documentation plan
docs/integrations.mdMCP section gains a "tested against" matrix; CHANGELOGAdded(CI).Migration and compatibility notes
Not expected to require migration.
Risks and tradeoffs
External-process tests bring flakiness — isolation, pinning, and non-blocking
status manage it. Node dependency in one CI job is contained cost.
Suggested labels
ecosystem, testing, integrations, reliability