Skip to content

fix(mcp): stop unsandboxed tests writing the real needs-auth cache - #994

Merged
sudomaggie merged 1 commit into
developfrom
fix/mcp-needs-auth-cache-test-isolation
Aug 26, 2026
Merged

fix(mcp): stop unsandboxed tests writing the real needs-auth cache#994
sudomaggie merged 1 commit into
developfrom
fix/mcp-needs-auth-cache-test-isolation

Conversation

@Harry19081

Copy link
Copy Markdown
Member

Problem

specialization::mcp::needs_auth_cache::tests::concurrent_sets_all_persist
fails intermittently in CI — most recently on
#992 (3180 passed, 1 failed,
panicking at needs_auth_cache.rs:280), a PR that changed zero Rust
files. Re-running the job passed, which is the signature of a race
rather than a broken assertion.

The "concurrent" test was never at fault. Its own eight writes are
already serialized by the module's write_lock(), so they cannot lose
each other. The interference came from outside the test.

specialization::mcp::manager::tests::mark_and_clear_needs_auth_round_trip
calls mgr.mark_needs_auth(...) / mgr.clear_needs_auth(...), which go
through needs_auth_cache::set_entry / remove_entry and write
mcp-needs-auth-cache.json under ORGII_HOME. It held no
test_env::sandbox() guard, violating the invariant that
test_helpers::test_env documents in its module docs:

Every test that mutates ORGII_HOME or HOME MUST go through
sandbox() (or lock_home() if it only needs the lock). Having any
test mutate these env vars without the lock re-introduces the race.

ORGII_HOME is process-global, so under cargo test's parallelism that
unlocked test wrote the key "srv" into whichever sandboxed test's
tempdir the variable happened to point at. When that was
concurrent_sets_all_persist, its assert_eq!(cache.len(), 8) saw nine
entries and failed.

There is a second consequence that never showed up as a test failure:
when ORGII_HOME was unset, cache_path() fell back to
dirs::home_dir(), so running the test suite wrote to and deleted the
developer's real ~/.orgii/mcp-needs-auth-cache.json
.

Solution

Fixed at the boundary that resolves the path, not at the assertion that
noticed the symptom.

  • needs_auth_cache::cache_path() now panics under #[cfg(test)] when
    ORGII_HOME is unset, instead of silently reaching for the real home.
    A future test cannot recreate the invalid state without failing loudly
    and with a message that names the fix. This is the regression guard:
    it makes the producing condition impossible rather than asserting on
    its downstream effect.
  • mark_and_clear_needs_auth_round_trip takes a
    test_helpers::test_env::sandbox() guard.

concurrent_sets_all_persist is deliberately left exactly as it was. It
was correct; weakening its assertion or retrying it would have hidden
the defect instead of removing it.

Production behavior is unchanged: the cfg(not(test)) arm keeps the
existing ~/.orgii fallback verbatim, and #[cfg(test)] applies only to
agent-core's own unit-test build — integration tests under tests/
compile against the non-test build and are unaffected.

Potential risks

  • The cfg(test) panic could surface other unsandboxed callers. That
    is the intent, but it does mean a test elsewhere in agent-core reaching
    an MCP lifecycle path without a sandbox would now fail loudly instead
    of silently touching the real home. The full agent-core suite was run
    to bound this: 3181 passed, 0 failed, and the new panic fired on
    nothing else. The blast radius is exactly the one test fixed here.
  • Scope of the assertion is narrow by construction. It is
    #[cfg(test)], so it cannot affect the shipped binary or any
    integration test.
  • Cross-crate callers are unaffected. cache_path() is
    pub(crate), so no other crate can reach it.
  • This does not make every ORGII_HOME-backed module safe. It fixes
    the needs-auth cache specifically. The same class of bug — a test
    touching process-global env state without lock_home() — could exist
    in other modules; this PR does not audit for that.

Verification

Run in a clean worktree off develop (Darwin, cargo test -p agent_core):

  • Full agent-core suite: 3181 passed, 0 failed, 2 ignored. CI's
    failing run was 3180 passed + 1 failed = the same 3181 tests, so the
    counts line up.

  • MCP subset (cargo test -p agent_core specialization::mcp): 103
    passed, 0 failed.

  • cargo clippy -p agent_core --all-targets -- -D warnings: clean.
    (The unrelated pre-existing block v0.1.6 future-compat dependency
    warning is not introduced here.) The pre-commit hook's scoped clippy
    run also passed.

  • Root cause proven, not assumed. Temporarily removing only the new
    sandbox guard makes the offender fail deterministically and
    immediately:

    test specialization::mcp::manager::tests::mark_and_clear_needs_auth_round_trip ... FAILED
    panicked at crates/agent-core/src/specialization/mcp/needs_auth_cache.rs:60:9:
    needs-auth cache resolved without ORGII_HOME — the calling test must hold a
    `test_helpers::test_env::sandbox()` guard
    

    This confirms that test really was resolving the cache path outside any
    sandbox. The guard was restored before committing.

Not run: the original flake is probabilistic, so "the suite passed once"
is not by itself proof it is gone. The deterministic demonstration above
is the load-bearing evidence — it identifies the specific writer and
shows the enforcement catches it at the moment it happens, rather than
relying on repeated green runs.

`mark_and_clear_needs_auth_round_trip` drove `mark_needs_auth` /
`clear_needs_auth`, which write `mcp-needs-auth-cache.json` under
`ORGII_HOME`, without holding a `test_env::sandbox()` guard. That
violates the invariant `test_helpers::test_env` documents: every test
mutating or reading `ORGII_HOME`-backed state must go through the
canonical home lock.

Two consequences, one visible and one not:

- Under `cargo test`'s parallelism the unlocked test wrote "srv" into
  whichever sandboxed test's tempdir `ORGII_HOME` pointed at. When that
  happened to be `concurrent_sets_all_persist`, its
  `assert_eq!(cache.len(), 8)` saw a foreign entry and failed. This is
  the intermittent CI failure at needs_auth_cache.rs:280 — the
  "concurrent" test was never at fault; its own writes are already
  serialized by `write_lock()`.
- With `ORGII_HOME` unset, `cache_path()` fell back to `dirs::home_dir()`,
  so running the suite wrote to and deleted the developer's real
  `~/.orgii/mcp-needs-auth-cache.json`.

Fixes both at the boundary that resolves the path: `cache_path()` now
panics under `#[cfg(test)]` when `ORGII_HOME` is unset instead of
silently reaching for the real home, so a future test cannot recreate
the invalid state without failing loudly. The offending test takes a
sandbox guard.

Production behavior is unchanged — the `cfg(not(test))` arm keeps the
existing `~/.orgii` fallback, and `#[cfg(test)]` covers only agent-core's
own unit-test build.
@sudomaggie
sudomaggie merged commit 1b991ca into develop Aug 26, 2026
6 checks passed
@Harry19081 Harry19081 added bug Something isn't working agent Agent runtime, behavior, memory, providers, or orchestration dev-tooling Developer tooling, build, CI, tests, diagnostics, or release labels Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Agent runtime, behavior, memory, providers, or orchestration bug Something isn't working dev-tooling Developer tooling, build, CI, tests, diagnostics, or release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants