Feat/the detector knows icons and url elicitation - #289
Merged
Merged
Conversation
…reads it Tool poisoning hides directives in what an LLM is shown before it calls anything. _collect_text named seven keys and icons was not one of them, so a directive carried in an icon reached no matcher at all -- not for want of a pattern, but because the text was never collected. The probe had been returning the field since it dumps by alias with exclude_none; the scanner was the blind half. icons arrived with revision 2025-11-25 and is server-controlled: src is a URL the client is expected to fetch, mimeType and sizes are free strings rendered beside the tool name. Measured in the installed SDK: mcp-types 2.0.0 declares Icon(src, mime_type, sizes, theme) and carries icons on Tool, Resource, Prompt and Implementation. The test asserts the channel, not the verdict. Its clean half runs first and must stay clean, so a flag in the second half can only have come from the field under test; without that control the same assertion would pass with the channel still blind. Mutation: removing icons from the collected keys kills exactly this test and nothing else, as predicted. The file's own strict baseline was eight errors and none of them were checked by CI. Touching it means owning it: seven missing return annotations annotated, and the method-assign on the logging double named with a coded ignore rather than left bare.
An icon fetched from a CDN is how the field is meant to work, and this matcher reads flattened text, so it cannot tell the server's own origin from anyone else's. A pattern on 'src is remote' would flag the ordinary case and measure nothing. What is scored instead is content the client executes or renders as markup beside the tool's name, before the user has agreed to call anything: icon_active_scheme for javascript:, vbscript:, data:text/html and data:image/svg, icon_executable_carrier for a declared image/svg+xml or an .svg name. They go to MCP_POISONING_PATTERNS rather than INJECTION_PATTERNS. The eval corpus is scored by l1_scorer, which reads the generic list only, and a JSON icons key already appears in one benign sample; a generic pattern would carry that risk into recon output for a signal that belongs to tool metadata. The declared pattern count and the three documents pinned to it are therefore untouched. False positives measured before the patterns were written, on three populations: the eight tools this repository's own server advertises, the 45 benign corpus samples, and hand-built negatives including a PNG data URI. Zero on all three -- with the limit stated plainly: none of those populations contains an icons field at all, so the repository holds no negative control for this category yet. Recorded as a tail. Four mutants, four kills. Removing either category kills the test. Removing either half of the carrier alternation kills it too, but only after the first pass caught a defect in the test rather than the code: one fixture carried a declared type and an .svg name at once, so the extension half was unfalsifiable and its mutant survived. Split into two fixtures, the second with a version query a CDN would append.
The probe has recorded capabilities since it was written and no stage read it: poisoning, overprivilege, exposure and trust all take tools, attestation took transport and a flag. So a server's declared surface -- experimental blocks, protocol extensions, task support -- was collected and dropped. It belongs in the attestation summary, whose own standing reason says MCP has no in-protocol capability attestation: a claim about a set nobody reported. Names only; the values are free-form per the spec and stay in the raw dump. URL-mode elicitation turned out not to be a scanner category at all. Measured in mcp-types 2.0.0: ServerCapabilities has eight fields and elicitation is not among them. It is a client capability -- the client says it is willing to open a URL, and the server demands one with error -32042 in reply to a call. A passive probe calls nothing, so there is nothing on the target to detect. The risk points the other way: a scanner that advertises the capability has agreed to follow the links of the endpoint it is scanning. Today it does not, and that was held by structure alone. The second test pins it, with the control as the point of the file: the SDK builds ElicitationCapability(form=..., url=...) from one callback with no separate switch, so someone adding a form prompt would turn on URL mode in the same line. Read off a built session rather than off our source, because our source is not what decides this. Four mutants, four kills, two of them only after the tests were fixed. Both survivors were wiring: dropping the argument at the stage call site, and dropping it in run(), each left the whole suite green because every existing test entered below that layer. Now one test per layer. agent.py's single strict error was Returning Any -- the untyped coroutine runner, not the probe, decided the type; annotating the result at the call site fixed it and made the module clean, which the drift step then reported as an undeclared clean module. Declared: scope 99 -> 100, errors 284, drift none, crossings unchanged at 22 reaching 29 (agent.py became a crosser as mcp_scan/__init__.py stopped being one). Page and badges follow, by script. Verified in the CI's own shape -- bare mypy on a cold cache -- after a full-package run's cache made a file-list run report 55 errors from seven modules it was not asked about.
…emise The register said no test mentions either term and a tree-wide search returns nothing. Half of that is now false and the other half was never the right question. Icons: the field sat outside the metadata whitelist, so no pattern could have reached it, and two tests hold the channel and the categories. Elicitation: ServerCapabilities has eight fields and none of them is elicitation, so a scanned server cannot advertise URL mode at all. What a target does with it arrives as error -32042 in reply to a tool call, and this scanner calls nothing -- which is why the row stays partly rather than closed. The exposure that does exist runs toward us, and its test is named too. The corpus is deliberately untouched, and that is a measurement rather than a shortcut. Samples are scored by l1_scorer, which reads the generic pattern list only; a tool-metadata category lives in the MCP list, so an icon sample would be scored by nothing, land in the blind subclasses and move the pinned 32/51 without measuring anything. The plan for today assumed these categories would go in the generic detector. The false-positive measurement said otherwise: one benign sample already carries a JSON icons key, and a generic rule would carry that into recon output. The scanning page gains the two categories in its stage table, the declared capability set beside the attestation row, and a section stating both directions of elicitation -- so the behaviour is described where a reader looks for it rather than only in a commit message.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
How it was measured
Checklist
ruff format --check cyberai/ tests/andruff check cyberai/ tests/passpytest -W ignore::DeprecationWarning -m "not slow and not smoke"passes