Repository navigation
Defend against prompt injection in untrusted tool content - #79
Merged
Merged
Conversation
Add Provenance and ToolOutcome::External. webfetch, websearch, MCP tool calls, the browser and MCP resources (tool and @server:uri mentions) declare provenance; the registry projection truncates, exposes hidden characters, neutralizes frame tags and wraps the result in <untrusted_content>, with the truncation notice after the closing tag. The webfetch summarizer gets the page in the same frame. Overflow files of framed results record provenance in a sidecar, and read/grep of the managed output directory is re-framed, failing closed to source=overflow.
A retryable failure (transport, idle timeout, in-stream overload, rate limit) after partial text, reasoning or tool calls now discards the attempt and re-sends the same request within the existing retry budget, instead of settling the Turn as interrupted. Replay ends at protocol completion; tool calls only dispatch after the stream completes. - cyber-llm: open_with_retry returns a replaying stream that yields LlmEvent::Discarded before the new attempt; retries before and after output share one budget. collect drops discarded output, keeps usage. - Drain records reasoning items and tool calls with the text when the stream ends, so nothing of a discarded attempt reaches history. Usage a discarded attempt reported is recorded as a failed turn usage.recorded.1 unless a reservation already billed it. - session.retry carries message_id and discarded; the TUI and web dashboard clear the partial answer, exec stream-json prints a retry line. - OpenSpec change retry-interrupted-streams (Retry policy, eager execution durability, exec stream-json schema), docs and ROADMAP.
Add the versioned injection rule table and detector in cyber-core (NFKC, lower case, hidden characters removed, whitespace collapsed; seven kinds; ten recorded hits with redacted excerpts, all counted). It lives in cyber-core so the runtime can scan admitted input. The registry scans each framed result's model-facing text and adds a signals attribute; a webfetch summary carries the page's signals too. Cross-session message text, teammate idle status lines and any channel-sourced admission are scanned at admission. With permissions.untrusted_content.scan "all", read/grep/bash/powershell output is scanned without being rewritten. session.tool.settled.1 and session.prompt.admitted.1 gain an optional untrusted record (also projected on CallState and InboxRow); old events read as no signals. OpenAPI and the TypeScript SDK are regenerated. A std-only cargo bench checks the 5 ms / 50 KiB target in CI with a 2x alarm.
…ashboard Shared notice wording and per-Mode consequence live in cyber-core (untrusted::notice). The TUI draws the warning line under a tool result or promoted message, with excerpts while tool details are expanded. cyber exec text mode writes a stderr warning (suppressed by --quiet); stream-json tool_result carries the untrusted record under schema version 1. The dashboard renders the line and excerpts on tool cards and messages with textContent. Promoted messages carry their admission's record on the message listing; OpenAPI, the SDK and web/dist are regenerated. Clients state the consequence for the default taint/bypass settings until the config loader exposes them (task 5.1).
… actions Fold the run's taint into Session state from settled calls and promoted inbox rows: records since the last source "user" prompt, so it survives restarts and compaction. Children inherit it on their prompt admission (spawn, resume, teammates); foreground agent results, background handbacks and Workflow summaries hand a tainted child's record back. In a tainted run, egress (webfetch, websearch, MCP calls, cross-machine messages, network commands, unparseable or inline-shell commands), external writes and secret-path reads turn every allow into ask: config, Session and saved rules, Mode grants, hook allows and auto always_allow. allow_when_tainted keeps its allow; denies and ceilings are unchanged. dont-ask and unattended asks deny as not pre-approved; bypass runs as before unless bypass is "deny". A classifier allow still asks for risky actions and non-read-only commands, with its reason in the prompt; permission.asked carries metadata.taint and an "always" reply does not exempt later tainted calls. The auto classifier gets the taint evidence and a trusted system line, and permission.auto_decided records tainted. Add permissions.untrusted_content settings (taint, bypass, scan, allow_when_tainted) with strict validation; project layers and project-defined profiles may only tighten them, with one warning per file and key.
…narios and docs Add the untrusted-content topic to the base prompts (version 3), a labeled injection/benign fixture set with a recall and false-positive gate, scripted end-to-end scenarios linked from the recovery manifest, and document the frame, signals, taint and settings. Narrow the email transfer verb so sign-in form labels are not secret_access hits, and update the Copilot test marker for the framed summarizer prompt.
- Keep lower layers' permissions.untrusted_content when a project layer replaces permissions with a string, array or null. - Egress: wrappers (timeout, busybox, ...), non-literal program words, git value options, publish after options, PowerShell backslash paths and quoting, collab writes, computer.control, MCP resource reads. - Secret reads: symlinks, shell globs, $env:USERPROFILE, grep matches inside secret files in a tainted run. - @server:uri and template @ references carry their untrusted record on the prompt admission; a user prompt clears taint, then notes it. - "always" no longer cascades onto pending taint asks. - history_search, shell reads of overflow files and workflow direct tools.read/grep are framed and recorded. - Detector v2: agent-aimed command requests, transfer destinations, data-named path placeholders, companion pipe-to-shell rules, wider invisible-character set, char-offset gaps; benign install/API samples. - Sessions report the effective notice policy; TUI and dashboard use it and share a notice fixture. - Typed taint decision source, authorize_as split, Windows CI filters.
The bash half of the scan-all test starts a real shell, which the Windows runner cannot spawn (msys NtCreateDirectoryObject denied); it now lives in its own #[cfg(unix)] test like the file's other shell test. read_sse_until returned as soon as the needle appeared, so a chunk that ended mid-line left a truncated data line for the context-meter test to parse. It now waits for the blank line ending the needle's event.
This was referenced Oct 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
External tool content now reaches the model in one escape-safe untrusted-content frame, with deterministic injection signals and taint-aware permission checks. Fetched pages, search, MCP/browser results and managed overflow rereads carry provenance; workspace file text retains its own representation.
Behavior
OpenSpec:
add-untrusted-content-defense, originally specified in merged #77. This head includes #78 and merged #81 to verify the combined runtime and generated API. #78 has merged.Verification at c6beff9
Local checks pass: 31 core untrusted tests, nine detector tests, three evaluation tests, 12 framing/overflow tests, 19 tool-level injection/taint tests, 51 dashboard tests, typecheck, scoped cyber-tools Clippy with warnings denied, Rust formatting, OpenAPI generation, SDK generation/check, and all 113 strict OpenSpec items. Spec lint reports zero errors and 25 warnings. The long-whitespace regression executed and failed before its correction. All 38 current-head hosted checks now pass, including the retried macOS observation job and macOS packaging checks. The prospective merge tree equals the tested head tree.
This is defense in depth with a deterministic detector, not a guarantee against every prompt injection. The documented signal classes, mode behavior and configuration remain the feature contract. There are no provider-key live tests in this verification.