Repository navigation
feat(mcp): propose isolated external tool support - #1732
Draft
JacobWeyer wants to merge 292 commits into
Draft
JacobWeyer wants to merge 292 commits into
JacobWeyer wants to merge 292 commits into
Conversation
Use the exact ordinary policy shape and reject managed deny data before serialization. Require protocol 43 and authenticated pinned TLS for all dedicated managed calls; ordinary registration remains compatible. Pre-commit review: invalid policy bytes advanced the session fence before rejection. Validate those bytes first and cover both RPC directions and managed marker failures with predecessor fixtures and real transport. Document separate TLS activation and managed rollback limits.
Emit MCP cancellation and deferred events only for MCP calls. Keep exact current-run identities while excluding ordinary calls from mixed ledgers. Preserve the previous ordinary question, plan and disconnect behavior. Pre-commit review: correct the stale permission, timeout, request-state and request-emission descriptions. The complete file pass found no other scoped runtime defect.
Exercise actual socket framing with the predecessor Hello shape and the current reply. Verify that version mismatch selects replacement before policy use. These tests characterize existing behavior; they do not claim Linux process replacement or VM survival. Pre-commit review: retain the existing JSON protocol and ownership path; no production protocol change is needed.
Skip only the unmeasured RAM check, then enforce any reported CPU budget. The public preview regression covers six measured and unmeasured cases. Pre-commit review: correct bundle ranking and snapshot-version comments and remove the misleading best-fit explanation from the affinity test. No metadata-store or SQL semantics change.
Preserve the historical recursion failure and its unchanged assertion. Record the successful current full suite using fresh application and DBOS test databases, without filters or a concurrency override. Pre-commit review: correct the reversed historical error description and state the existing worker CI wiring. A later pass is not a causal repair of the intermittent failure or physical acceptance.
Project the pool reference before internal reinspection. Keep strict validation of external pool configuration and preserve inventory checks. Add a regression for the offline reinspection path and extra-field rejection at the public configuration boundary.
Bound readiness transport retries by the startup deadline. Test delayed listener readiness, authentication and identity failures, and single delivery of fixture control writes.
Accept the SELinux label option that Docker derives for the authorized Firecracker host PID container. Require no-new-privileges and the exact capability set. Test rejection for other roles and extra security options.
Require the fixed production rules and the exact loopback resolver graph. Check matching TCP and UDP listener ports before accepting Docker DNS jumps. Add regression cases for changed rules and qualification reads.
Select the coordinator bridge address for Firecracker host RPC DNS and the exact destination rule. Keep VZ on the native host address. Test both backends and document their native DNS setup.
Keep pool and backing identity checks exact while allowing the allocated block count to change during normal writes. Require fully allocated backing files at each check.
Set and verify the AppArmor profile only for the trusted Firecracker host container. Keep worker restrictions, default seccomp and exact capabilities unchanged. A paired native probe reproduces the mount failure and verifies the single-option correction.
Include sysctl and nft for private managed namespace setup. Check the required network commands during the image build so a missing runtime package fails before deployment.
The old chain name is an nft keyword. Use proxy_redirect without changing packet actions or hook priorities. The installed namespace test and the full Rust workspace gate pass.
Allow the trusted host to set namespace controls and bind private DNS. Verify the actual system path state, effective capabilities and kernel mounts. Keep default path protection for other fixture roles.
Contributor
|
definitely something we have planned on the roadmap! will review and get back to you on this |
Author
Please don't do anything too in depth yet; I just passed virtualized local testing yesterday and now I'm running tests against an actual AWS firecracker VM. Then I'm going to do another pass to try and simplify. |
Create disk exports with mode 0600 instead of umask-dependent permissions. Keep exclusive creation, byte integrity and durable sync behavior. Extend the existing export test to check private permissions. Validation: nine managed disk tests; just check (3275 passed, 370 skipped); aarch64 Linux musl all-target Clippy.
Record each failed case and continue only after its cleanup succeeds and the shared stack passes existing health checks. Stop on unsafe cleanup, setup, transport, unknown errors and total deadline expiry. Keep the first case failure as the final result and prevent a failed run from claiming physical acceptance. Validation: 179 runner tests, both TypeScript checks, formatting and independent review. Earlier unchanged proxy probe timeouts remain recorded; later passes do not prove a causal fix.
Use a longer-lived fixture token after the short-expiry, rotation, account and scope checks. Keep the reconnect and credential-leak assertions unchanged. The failed physical probe started with about one second of token validity and returned an upstream failure just after expiry; the exact upstream HTTP status was not retained. Validation: 187 script tests, 187 platform tests with OpenSSL 3, three type checks, formatting and independent review.
Run normal and error reads in one session for each protocol and HTTP response mode. Keep all sixteen calls, exact per-call proofs, acknowledgments and native turn completion checks. Reduce fidelity VM starts from sixteen to eight. Allow ninety minutes for the full matrix across both harnesses. Keep the thirty-second RPC, two-minute step and sixty-second tool limits. Failed cases still prevent physical acceptance. Validation: eight batching regressions, 187 script tests, 187 platform tests, three type checks, formatting and independent review.
Integrate lazy session-disk restore and the shared fill engine. Preserve managed disk-only preparation, protected volumes, policy fences and pinned TLS on the listener bound before host registration. Regenerate the workspace dependency manifest with Hakari and reconcile source-bound acceptance docs. Pre-commit review corrected removed materialization and manifest APIs, backend fixture capabilities and stale lifecycle comments. Complete manifest equality covers lost-reply recovery, including chunks and annotations. The managed DNS shim uses a new materializer layout version and measured pin. The independent boundary review found no remaining merge defect. Validation: Rust workspace format, Clippy and Hakari pass; all 3414 workspace tests pass (382 skipped) with fail-fast disabled. Both Linux all-target Clippy checks pass. Orchestrator typecheck and 96 focused tests, 46 materializer tests, protobuf lint/regeneration, and workflow checks pass.
Require one suite invocation in the unconditional detection job. A second invocation duplicated the same tests and is already absent from the workflow. Pre-commit review confirmed that the three exact packaged-helper targets, artifact checksums and required VZ lane remain enforced. Validation: all 73 MCP CI-script tests pass on Python 3.12.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Claude and Codex sandboxes need organization-managed external MCP tools. The guest must not receive organization credentials or decide which remote service, account, or tool it may use.
This disabled implementation is open for architecture and scope feedback. The contribution guide asks for an issue before a large change; the design discussion can move to an issue if preferred.
Fix
Remote HTTPS MCP calls use an isolated worker and the existing connection, credential, profile-grant, and
ENGRAM_TOOLSmodels.flowchart LR S[Claude or Codex sandbox] --> C[Coordinator] C --> O[Orchestrator: grants, approvals, journal] O -->|mTLS and bound intent| W[Isolated MCP worker] W -->|HTTPS and organization credential| M[Approved MCP server]Reviewed immutable catalogs and session bindings limit exposed tools. Restricted transport validates destinations and keeps organization credentials outside guests. Interactive writes require bound approvals; unattended writes need explicit administrator grants. A durable dispatch barrier prevents automatic replay of uncertain writes. Both harnesses preserve text, structured JSON, and MCP error results.
Managed boot and disk-only recovery keep the guest closed until the exact owner and network policy are verified. Direct guest access to managed service authorities is denied. Ordinary checkpoint, eviction, resume, and migration remain closed for managed sessions; a lost native owner is not adopted.
The feature stays disabled by default. Guest stdio servers, package installation, personal credentials, resources, prompts, sampling, elicitation, MCP Apps, and asynchronous MCP Tasks are outside this release. Deterministic MCP and OAuth services are the acceptance targets.
Suggested review units:
Keep each boundary with its contracts, migrations, and tests. A scope and simplification review has already removed unused credential mutators and duplicate CI execution. This change adds one full Drizzle snapshot; required migration history stays intact. No applied migration is edited, and there are no SQLx cache changes relative to the base.
Upstream integration
Published head
c0602535includes upstream6bd86c29through signed merge4bee6e4a. The additional signed CI test correction isc0602535. The branch has no remaining local merge conflict and includes every upstream-main commit fetched for this integration.The merge uses upstream's shared FillEngine and session-disk flow. Managed cold disks are fully materialized before network and VM creation; protected managed volumes do not enter ordinary settle drivers. The host binds its listener before registration and preserves pinned managed TLS on that listener. Removed upstream manifest and test-fixture APIs have been updated. Materializer layout version 2 records the changed managed-DNS init bytes and keeps an exact packed-image fingerprint; existing stored disks are preserved. Complete manifest equality covers lost-reply reconciliation, including chunks and annotations.
Ordinary sessions retain wire version 38 and source-pinned predecessor compatibility tests. All eight additive managed host RPCs separately require managed protocol 43 and pinned mTLS. No peer-version negotiation or plaintext managed fallback was added. ADR 0127 records the trust boundary and deferred scope.
Test
Merged-source local checks passed:
just check: formatting, workspace/all-target Clippy, generated Hakari dependency check, and 3,414 tests passed, 382 skipped, zero failures. The final complete test invocation used--no-fail-fast; no selected case was left unreached.Physical evidence is source-specific:
c5dc8e7bb75b0d7fThe Firecracker terminal verifier returned
FULL_24_PASS. Matrix checks, pinned physical reads, boot coverage, native-process closure, final pool metadata ownership, and cleanup passed. The final archive and selected member hashes were verified locally. All 28 build pins were retained. The temporary cloud instance, disks, network resources, access key, expiry schedule, and role were removed and their absence verified.That Firecracker run includes the scanner-driven rebind repair: the earlier three failing revocation/OAuth recovery cases all passed in the coherent 24-case run. Earlier failures remain recorded; test assertions were not weakened or omitted. The historical sandbox recursion failure and its later passing results also remain documented.
The earlier full orchestrator result (3,248 passed, 18 skipped), 1,116 web tests, worker tests, and installed storage checks retain their original source scope. They are not a current-head CI result.
Remaining CI and release gates
Current-head upstream CI Gate remains unproven. The signed merge is published; workflow admission and qualified runner availability still need confirmation.
The workflow requires dedicated self-hosted runners with these labels:
self-hosted, Linux, X64, engram-mcp-firecracker.self-hosted, macOS, ARM64, engram-mcp-vz.The runner operator must install the documented configuration, qualified storage and network paths, and pinned artifact access. This account receives HTTP 403 from the upstream runner-list API, so runner registration and availability cannot be confirmed. Fork workflow approval and runner access require upstream maintainer/operator authority. The manual acceptance result does not satisfy these CI jobs.
The upstream merge changes shared storage and host behavior, so both physical jobs must qualify the new source. They remain dependencies of
CI Gate; no checks were removed or bypassed.Mixed-build deployment, managed activation, upgrade/recovery, rollback, and proxy replacement with VM survival remain separate release gates. Source-level predecessor compatibility does not prove a deployed mixed fleet. The PR remains draft and the feature remains disabled while those limits and current-head CI remain open.