Repository navigation
docs(mcp): document write confirmations and add sandbox evals - #509
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Shadow risk calibration
Calibration result: agreement. Shadow risk never blocks or enables merge. Keep it in observation mode until roughly 50 pull requests have human-reviewed labels, then calibrate before considering any policy use. |
fe9973a to
0e71be1
Compare
0f38bda to
15fd483
Compare
0e71be1 to
b0a9a1e
Compare
15fd483 to
d668b04
Compare
d668b04 to
d6f0d45
Compare
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
| // Only a stop this eval made needs undoing; a lost response keeps | ||
| // the default, because the stop may have happened. | ||
| needsResume = structured(stopped).status === 'stopped'; |
There was a problem hiding this comment.
Unclear responses skip cleanup
EvalClient.callTool returns normally for an unreadable HTTP response, with no payload. If the stop succeeded but a gateway error replaced its response, this assignment sets needsResume to false. The next assertion fails and finally skips the resume, leaving the sandbox shipment stopped. Keep cleanup armed for unknown outcomes and disarm it only when the response proves no stop occurred.
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/mcp/eval/writes.eval.ts
Line: 98-100
Comment:
**Unclear responses skip cleanup**
`EvalClient.callTool` returns normally for an unreadable HTTP response, with no payload. If the stop succeeded but a gateway error replaced its response, this assignment sets `needsResume` to false. The next assertion fails and `finally` skips the resume, leaving the sandbox shipment stopped. Keep cleanup armed for unknown outcomes and disarm it only when the response proves no stop occurred.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| The confirmation step makes sure you see the current state before a destructive change. Terminal49 still decides what your credential can do. | ||
| </Note> | ||
|
|
||
| Write tools return these results with `isError: false`, so the AI client can explain them: `rate_limited` when Terminal49 limits the request rate, with `retry_after_s` when Terminal49 says how long to wait, `not_permitted` when the change needs your approval and the server has no secret to sign approvals with, and the codes listed for each tool. Each of these results says that nothing was changed. |
There was a problem hiding this comment.
This guide says every rate_limited result means nothing changed, but batch writes can return that result after other records were saved. Callers following this advice may retry a partly saved batch without checking its state. Tell them to inspect updated, failed, and not_attempted before retrying. The same claim appears in TOOLS_OVERVIEW.md.
Knowledge Base Used: MCP shipping and container tools
Prompt To Fix With AI
This is a comment left during a code review.
Path: docs/mcp/home.mdx
Line: 936
Comment:
**Partial writes look untouched**
This guide says every `rate_limited` result means nothing changed, but batch writes can return that result after other records were saved. Callers following this advice may retry a partly saved batch without checking its state. Tell them to inspect `updated`, `failed`, and `not_attempted` before retrying. The same claim appears in `TOOLS_OVERVIEW.md`.
**Knowledge Base Used:** [MCP shipping and container tools](https://app.greptile.com/-/switch-tenant?tenant=terminal49&next=%2Fcustom-context%2Fknowledge-base%2Fterminal49%2Fapi%2F-%2Fdocs%2Fmcp-tools.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| The cases skip when the gateway does not list the write tools, which happens | ||
| when its toolsets leave out `operations`. They also skip when the sandbox key |
There was a problem hiding this comment.
Wrong toolset skips write checks
These cases require tracking_lifecycle, not operations. The default server already includes operations but omits the lifecycle tools, so following this guidance can leave both write cases skipped. Name tracking_lifecycle and explain that the deployment must explicitly enable it.
| The cases skip when the gateway does not list the write tools, which happens | |
| when its toolsets leave out `operations`. They also skip when the sandbox key | |
| The cases skip when the gateway does not list the lifecycle tools. Enable | |
| `tracking_lifecycle` in the deployment's `T49_MCP_TOOLSETS` to serve them. | |
| They also skip when the sandbox key |
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/mcp/eval/README.md
Line: 84-85
Comment:
**Wrong toolset skips write checks**
These cases require `tracking_lifecycle`, not `operations`. The default server already includes `operations` but omits the lifecycle tools, so following this guidance can leave both write cases skipped. Name `tracking_lifecycle` and explain that the deployment must explicitly enable it.
```suggestion
The cases skip when the gateway does not list the lifecycle tools. Enable
`tracking_lifecycle` in the deployment's `T49_MCP_TOOLSETS` to serve them.
They also skip when the sandbox key
```
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
* feat(mcp): add opt-in reference number tools add_reference_numbers (tier A) and remove_reference_numbers (tier C) in a new opt-in reference_numbers toolset, ported from #452 onto the write kit that landed in #506-#509. Drops the set_party_role party_id hint from the server instructions: it repeats the tool's own description, and with every toolset served the instructions otherwise exceed Claude Code's 2 KB limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(mcp): list the reference number tools in the agent trust feature map Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Document the confirmed-write behavior, deployment secret, token/elicitation channels, party-ID guidance, partial rate-limit results, and the opt-in lifecycle toolset. Update tool inventories and directory submission copy without advertising lifecycle tools on the default hosted server. Add explicitly gated sandbox lifecycle evals with stop/resume cleanup.
Final part of #451 / DEV-14372, based on #508. Original author: Akshay Dodeja. Stack: #506 → #507 → #508 → this PR. The top branch contains all four layers and is the combined preview QA target.
The combined tree preserves all 71 original changed paths after reconciliation with current main. Deliberate differences are the batch-429 fix and its regression tests, updated instruction wording that retains newer main guidance under the 2 KB client limit, a tracking audit test adaptation, and corresponding docs/snapshots.
Validation: full repository verifier and eval typecheck. Live staging preview QA passed all six protocol handshakes, 30 default/33 opt-in/27 read-only tool inventories, additive custom-field and party writes, token-approved overwrite/clear/remove, actual form elicitation accept/decline, forged-token and argument-substitution rejection, stale-preview rejection, and a confirmed stop/resume round trip. Temporary QA records were cleaned up and tracking restoration was verified directly. These checks use a delegated user credential; OAuth login itself and irreversible live cancellation were not exercised. Fixture tests remain the evidence for forced rate limits, transport failures and cross-account rejection.
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.Review order: #506 → #507 → #508 → #509. Each diff is against its predecessor; #509 is the combined QA target.
The exact combined revision is also pushed to
feature/dev-14372-write-stack-qato run the normal CI workflow, whose branch filters otherwise skip stacked PR bases. All four layers pass the full local verifier; the combined suite has 1,002 MCP tests, plus 88 explicitly configured gateway tests.Follow-up review: Clarifies client-owned approval, quota/billing consequences, non-atomic preview/recheck safeguards, and conservative batch-token retry behavior. Batch 4xx failures do not universally preserve approval; documented safe release is narrower.
The PR appears safe to merge, with non-blocking fixes needed in sandbox cleanup and documentation.
Fix with agent prompt
Summary
Adds confirmed writes, opt-in tracking lifecycle tools, and sandbox lifecycle evals.
Acknowledged limitations:
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Write call] --> B[Read current records] B --> C{Would data be removed or replaced?} C -->|No| W[Write planned changes] C -->|Yes| P[Return preview] P --> U[User approves through client or token] U --> R[Read records again] R --> S{Approval and state still match?} S -->|No| N[Refuse or request a new preview] S -->|Yes| W W --> O[Return per-record results and audit event]Reviews (1) · Last reviewed commit: "docs(mcp): clarify client approvals, bil..." · Reviewed by Greptile