Skip to content

docs(mcp): document write confirmations and add sandbox evals - #509

Merged
alosec merged 9 commits into
mainfrom
alex/dev-14372-write-docs-evals
Nov 6, 2026
Merged

alosec merged 9 commits into
mainfrom
alex/dev-14372-write-docs-evals

Conversation

@alosec

@alosec alosec commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Document the confirmed-write behavior, deployment secret, token/elicitation channels, party-ID guidance, partial rate-limit results, and the opt-in lifecycle toolset. Update tool inventories and directory submission copy without advertising lifecycle tools on the default hosted server. Add explicitly gated sandbox lifecycle evals with stop/resume cleanup.

Final part of #451 / DEV-14372, based on #508. Original author: Akshay Dodeja. Stack: #506 → #507 → #508 → this PR. The top branch contains all four layers and is the combined preview QA target.

The combined tree preserves all 71 original changed paths after reconciliation with current main. Deliberate differences are the batch-429 fix and its regression tests, updated instruction wording that retains newer main guidance under the 2 KB client limit, a tracking audit test adaptation, and corresponding docs/snapshots.

Validation: full repository verifier and eval typecheck. Live staging preview QA passed all six protocol handshakes, 30 default/33 opt-in/27 read-only tool inventories, additive custom-field and party writes, token-approved overwrite/clear/remove, actual form elicitation accept/decline, forged-token and argument-substitution rejection, stale-preview rejection, and a confirmed stop/resume round trip. Temporary QA records were cleaned up and tracking restoration was verified directly. These checks use a delegated user credential; OAuth login itself and irreversible live cancellation were not exercised. Fixture tests remain the evidence for forced rate limits, transport failures and cross-account rejection.


Show it working
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Review order: #506 → #507 → #508 → #509. Each diff is against its predecessor; #509 is the combined QA target.

The exact combined revision is also pushed to feature/dev-14372-write-stack-qa to run the normal CI workflow, whose branch filters otherwise skip stacked PR bases. All four layers pass the full local verifier; the combined suite has 1,002 MCP tests, plus 88 explicitly configured gateway tests.

Follow-up review: Clarifies client-owned approval, quota/billing consequences, non-atomic preview/recheck safeguards, and conservative batch-token retry behavior. Batch 4xx failures do not universally preserve approval; documented safe release is narrower.

RetriggerView in GreptileConfidence Score: 4/5

The PR appears safe to merge, with non-blocking fixes needed in sandbox cleanup and documentation.

Fix All in CodexFindings

  1. P2 Unclear responses skip cleanup ▶
  2. P2 Partial writes look untouched ▶
  3. P2 Wrong toolset skips write checks ▶
Fix with agent prompt
### Issue 1
packages/mcp/eval/writes.eval.ts:98-100
`EvalClient.callTool` returns normally for an unreadable HTTP response, with no payload. If the stop succeeded but a gateway error replaced its response, this assignment sets `needsResume` to false. The next assertion fails and `finally` skips the resume, leaving the sandbox shipment stopped. Keep cleanup armed for unknown outcomes and disarm it only when the response proves no stop occurred.

### Issue 2
docs/mcp/home.mdx:936
This guide says every `rate_limited` result means nothing changed, but batch writes can return that result after other records were saved. Callers following this advice may retry a partly saved batch without checking its state. Tell them to inspect `updated`, `failed`, and `not_attempted` before retrying. The same claim appears in `TOOLS_OVERVIEW.md`.

### Issue 3
packages/mcp/eval/README.md:84-85
These cases require `tracking_lifecycle`, not `operations`. The default server already includes `operations` but omits the lifecycle tools, so following this guidance can leave both write cases skipped. Name `tracking_lifecycle` and explain that the deployment must explicitly enable it.

```suggestion
The cases skip when the gateway does not list the lifecycle tools. Enable
`tracking_lifecycle` in the deployment's `T49_MCP_TOOLSETS` to serve them.
They also skip when the sandbox key
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds confirmed writes, opt-in tracking lifecycle tools, and sandbox lifecycle evals.

  • MCP record edits show a preview before they can remove or replace data.
  • Tracking lifecycle tools can stop, resume, or cancel tracking requests.
  • Write activity is audited while confirmation tokens are scrubbed from telemetry.
  • Rate-limit responses retain their wait time and tracking writes keep their result.

Acknowledged limitations:

  • alosec explicitly describes client-owned approval and non-atomic preview/recheck safeguards as intentional.
  • alosec explicitly notes that batch 4xx failures do not universally preserve approval. The narrower release behavior is intentional.
  • The documented per-process limit on remembering used stop confirmations is treated as a known limitation, not a new finding.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Write call] --> B[Read current records]
  B --> C{Would data be removed or replaced?}
  C -->|No| W[Write planned changes]
  C -->|Yes| P[Return preview]
  P --> U[User approves through client or token]
  U --> R[Read records again]
  R --> S{Approval and state still match?}
  S -->|No| N[Refuse or request a new preview]
  S -->|Yes| W
  W --> O[Return per-record results and audit event]
Loading

Reviews (1) · Last reviewed commit: "docs(mcp): clarify client approvals, bil..." · Reviewed by Greptile

@vercel

vercel Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
api Ready Ready Preview Nov 6, 2026 8:31pm UTC

Request Review

@linear-code

linear-code Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

DEV-14372

@github-actions github-actions Bot added the risk-shadow:high Shadow-only risk signal: high label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Shadow risk calibration

Signal Result Authority
Path classifier risk:high Deterministic advisory policy
heuristic fallback risk-shadow:high Shadow only

Calibration result: agreement.

Shadow risk never blocks or enables merge. Keep it in observation mode until roughly 50 pull requests have human-reviewed labels, then calibrate before considering any policy use.

@alosec
alosec force-pushed the alex/dev-14372-tracking-lifecycle branch from 0e71be1 to b0a9a1e Compare November 6, 2026 20:28
@alosec
alosec force-pushed the alex/dev-14372-write-docs-evals branch from 15fd483 to d668b04 Compare November 6, 2026 20:28
@alosec
alosec force-pushed the alex/dev-14372-write-docs-evals branch from d668b04 to d6f0d45 Compare November 6, 2026 20:30
@alosec
alosec changed the base branch from alex/dev-14372-tracking-lifecycle to main November 6, 2026 20:52
@mintlify

mintlify Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
terminal49 🟢 Ready View Preview Nov 6, 2026, 8:52 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@alosec
alosec marked this pull request as ready for review November 6, 2026 20:53
@github-actions github-actions Bot added the risk:high Advisory path risk; current-head human review needed label Nov 6, 2026
@alosec
alosec merged commit c33c1cd into main Nov 6, 2026
33 checks passed
@alosec
alosec deleted the alex/dev-14372-write-docs-evals branch November 6, 2026 20:55
Comment on lines +98 to +100
// Only a stop this eval made needs undoing; a lost response keeps
// the default, because the stop may have happened.
needsResume = structured(stopped).status === 'stopped';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Unclear responses skip cleanup

EvalClient.callTool returns normally for an unreadable HTTP response, with no payload. If the stop succeeded but a gateway error replaced its response, this assignment sets needsResume to false. The next assertion fails and finally skips the resume, leaving the sandbox shipment stopped. Keep cleanup armed for unknown outcomes and disarm it only when the response proves no stop occurred.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/mcp/eval/writes.eval.ts
Line: 98-100

Comment:
**Unclear responses skip cleanup**

`EvalClient.callTool` returns normally for an unreadable HTTP response, with no payload. If the stop succeeded but a gateway error replaced its response, this assignment sets `needsResume` to false. The next assertion fails and `finally` skips the resume, leaving the sandbox shipment stopped. Keep cleanup armed for unknown outcomes and disarm it only when the response proves no stop occurred.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Comment thread docs/mcp/home.mdx
The confirmation step makes sure you see the current state before a destructive change. Terminal49 still decides what your credential can do.
</Note>

Write tools return these results with `isError: false`, so the AI client can explain them: `rate_limited` when Terminal49 limits the request rate, with `retry_after_s` when Terminal49 says how long to wait, `not_permitted` when the change needs your approval and the server has no secret to sign approvals with, and the codes listed for each tool. Each of these results says that nothing was changed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Partial writes look untouched

This guide says every rate_limited result means nothing changed, but batch writes can return that result after other records were saved. Callers following this advice may retry a partly saved batch without checking its state. Tell them to inspect updated, failed, and not_attempted before retrying. The same claim appears in TOOLS_OVERVIEW.md.

Knowledge Base Used: MCP shipping and container tools

Prompt To Fix With AI
This is a comment left during a code review.
Path: docs/mcp/home.mdx
Line: 936

Comment:
**Partial writes look untouched**

This guide says every `rate_limited` result means nothing changed, but batch writes can return that result after other records were saved. Callers following this advice may retry a partly saved batch without checking its state. Tell them to inspect `updated`, `failed`, and `not_attempted` before retrying. The same claim appears in `TOOLS_OVERVIEW.md`.

**Knowledge Base Used:** [MCP shipping and container tools](https://app.greptile.com/-/switch-tenant?tenant=terminal49&next=%2Fcustom-context%2Fknowledge-base%2Fterminal49%2Fapi%2F-%2Fdocs%2Fmcp-tools.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Comment on lines +84 to +85
The cases skip when the gateway does not list the write tools, which happens
when its toolsets leave out `operations`. They also skip when the sandbox key

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Wrong toolset skips write checks

These cases require tracking_lifecycle, not operations. The default server already includes operations but omits the lifecycle tools, so following this guidance can leave both write cases skipped. Name tracking_lifecycle and explain that the deployment must explicitly enable it.

Suggested change
The cases skip when the gateway does not list the write tools, which happens
when its toolsets leave out `operations`. They also skip when the sandbox key
The cases skip when the gateway does not list the lifecycle tools. Enable
`tracking_lifecycle` in the deployment's `T49_MCP_TOOLSETS` to serve them.
They also skip when the sandbox key
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/mcp/eval/README.md
Line: 84-85

Comment:
**Wrong toolset skips write checks**

These cases require `tracking_lifecycle`, not `operations`. The default server already includes `operations` but omits the lifecycle tools, so following this guidance can leave both write cases skipped. Name `tracking_lifecycle` and explain that the deployment must explicitly enable it.

```suggestion
The cases skip when the gateway does not list the lifecycle tools. Enable
`tracking_lifecycle` in the deployment's `T49_MCP_TOOLSETS` to serve them.
They also skip when the sandbox key
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

@github-actions github-actions Bot added risk:unknown Advisory evidence unavailable or changed; no merge permission and removed risk:high Advisory path risk; current-head human review needed labels Nov 6, 2026
@mintlify

mintlify Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
terminal49 🟡 Building – Nov 6, 2026, 8:52 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

alosec added a commit that referenced this pull request Nov 6, 2026
* feat(mcp): add opt-in reference number tools

add_reference_numbers (tier A) and remove_reference_numbers (tier C) in a
new opt-in reference_numbers toolset, ported from #452 onto the write kit
that landed in #506-#509.

Drops the set_party_role party_id hint from the server instructions: it
repeats the tool's own description, and with every toolset served the
instructions otherwise exceed Claude Code's 2 KB limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(mcp): list the reference number tools in the agent trust feature map

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
staging - docs — d6f0d456 Deployed Nov 6, 2026 by mintlify[bot]
Preview — d6f0d456 Deployed Nov 6, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:unknown Advisory evidence unavailable or changed; no merge permission risk-shadow:high Shadow-only risk signal: high

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant