Skip to content

Add an MCP interoperability test matrix against reference servers #173

Description

@dgenio

Summary

Stand up an optional CI job (and local make target) that exercises MCPDriver
end-to-end against real reference MCP servers — at minimum the official
"everything" test server — covering discovery, annotation inference, invocation,
errors, and large outputs through the firewall.

Why this matters

MCP is the kernel's most strategically important driver, and the MCP SDK and
protocol evolve quickly. Current tests exercise the driver against mocks
(tests/test_mcp_driver.py), which cannot catch protocol-version drift, SDK
behavior changes, or annotation-shape changes server-side — the failure mode
adopters actually hit. An interop matrix converts "works with MCP" from a claim
into a tested property, and it directly supports the safety-classification work
(ISSUE 9) with real annotation data.

Current evidence

  • drivers/mcp.py + drivers/mcp_support.py implement discovery/invocation; tests/test_mcp_driver.py uses in-process fakes only.
  • pyproject.toml mcp extra pins the SDK range; nothing validates real-server behavior across SDK upgrades.
  • The CI Examples step never starts an MCP server; no interop coverage exists anywhere.

External context

The MCP project publishes reference servers (e.g., the "everything" server designed
for client testing) runnable via npx/uvx — purpose-built for exactly this kind of
matrix.

Proposed implementation

  1. Add tests/integration/test_mcp_interop.py gated by a marker
    (-m mcp_interop) and skipped unless the server toolchain is present.
  2. CI: a separate non-blocking job installs .[mcp] + Node, launches the
    reference server, runs the marked tests; promote to blocking once stable.
  3. Cover: tool discovery (count/annotations), safety-class inference outcomes,
    a successful invoke through firewall transform, server-error mapping to
    DriverError, oversized output meeting budgets.
  4. Pin server versions; refresh deliberately (pairs with ISSUE 55 Dependabot).

AI-agent execution notes

  • Inspect first: drivers/mcp.py (transport setup), tests/test_mcp_driver.py (fixtures to reuse), .github/workflows/ci.yml.
  • Keep the job isolated: regular make ci must not require Node or network.
  • Edge cases: server startup latency (health-check loop); stdio vs SSE transport coverage; flaky-network policy (retries, non-blocking until proven stable).
  • Do not loosen driver code to accommodate server quirks silently — file findings.

Acceptance criteria

  • Marked interop tests pass locally against the reference server with documented setup.
  • CI runs the matrix on a schedule or per-PR (non-blocking initially).
  • At least discovery, invoke, error, and large-output paths are covered.

Test plan

The matrix is the test plan; plus a README-in-tests note for local running.
Run make ci (unchanged) and the new job.

Documentation plan

docs/integrations.md MCP section gains a "tested against" matrix; CHANGELOG
Added (CI).

Migration and compatibility notes

Not expected to require migration.

Risks and tradeoffs

External-process tests bring flakiness — isolation, pinning, and non-blocking
status manage it. Node dependency in one CI job is contained cost.

Suggested labels

ecosystem, testing, integrations, reliability

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions