Repository navigation
Onboarding on the product's own scanner, projects, tokens and enforcer (fixes for #122) - #123
Merged
Merged
Conversation
…n its own branch off main
…he backend fixes land - What the redesign dropped from main and should restore (Home, agent summary, lineage, sub-agents on the map, every-step and done-check rules, nav entries, severity colours, setup banner). - What to fix (mobile sidebar, redirect loop, attack panel errors and polling, shield and wording, Observe scoping, theme scope, dev env names). - What waits on the backend (install line, key rotation, proposed agents, projects as systems, error counts, surface-aware checks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An output redact/mask/tokenize verdict was evaluated and recorded under auto(), but only block raised and nothing rewrote the response, so the caller got the text the policy said to hide. The enforcer's rewritten text now replaces the assistant text in the client's own response object (OpenAI/LiteLLM choices[0].message.content, Anthropic text blocks, LangChain AIMessage.content), sync and async, as the gateway proxy and LangGraph guard already substitute it. Same mode rule as a block: enforced verdict in policy mode, effective verdict in enforce mode, never in observe. A response that cannot be rewritten is refused. A streamed response cannot be rewritten after its chunks were yielded, so an applied redaction raises Blocked at the end of the stream.
…ixes, rules matched by kind and surface - catalog.Feed / feeds_for: one entry per way a check reports a finding; the workspace's custom models and rules spelled out with their own entity and surfaces. - describe() adds finds_all (custom.models: the prefixes its models were registered with); finds stays a string for compatibility. Also on /api/features checks. - can_fire / checks_for_rule: the join the dashboard did, now with surfaces, exact matching for custom entities, and 'any' for a detection rule naming neither.
… the checks that can fire it
- Each decision counts once, on the stage owning its surface; stats across stages add
up to the new top-level 'decisions'. prompt/response are derived: true, with an
overlapping lineStats; reasoning belongs to 'From the LLM'. memory_write is no
longer part of 'To the LLM'.
- Rules on a derived stage are copies flagged sharedWith; checks there that only
run on other stages are listed in detectorsSharedWith.
- Every rule carries surfaces (null: every surface), checks (on for this agent and
able to fire it) and checksOff; finds is 'any' for a detection rule naming neither.
- GET /api/policies/{key}/rules/{rule_id}/checks?agent= for the rule page.
- Models on the map: only the agent's own uses_model edges, seen in the window or
declared.
… are not outages
The judgment posture's fail_closed ("On a judge outage: deny the request") was
stored and displayed but never read. Wire it into main's existing degraded path
instead of a parallel mechanism:
- JevUnavailable carries a cause: outage, refused (EgressRefused, e.g.
pii_egress=block on text with personal data), or not_configured. The panel
reports refused only when every tier declined; any tier that was asked and
failed makes it an outage. Judge detectors put the cause and the entities
they would have produced in DetectorResult.raw.
- The pipeline degrades a judge outage (unavailable/outage, error, timeout,
skipped for budget) and records it in PipelineResult.judge_outage with its
entities. Deployment and pack fail_mode therefore treat "unreachable" and
"timed out" alike. Non-judge detectors are unchanged.
- The posture's fail_closed is a third, judge-only source: in enforce mode it
blocks as pipeline.judge_outage only when an enforcing pack has an enabled,
non-allow detection rule on this surface that the judge's entities could
fire. It can add a block, never remove one.
- Validator judges (validator.<judge> and judge rules in custom.validators)
report outages and refusals the same way; one rule's judge being down no
longer drops the other rules' findings.
- judgment_fail_closed (default posture and ceiling) now defaults to false,
like fail_mode, so a workspace can toggle it; true still forces it on.
Tests: tests/runtime/test_judge_outage.py (refusal with a healthy judge does
not block, no relying rule does not block, timeout == unavailable verdicts,
observe never blocks, closed deployment still blocks judges and non-judges,
validator judges) and posture default/ceiling tests. Docs and CHANGELOG
updated.
On OpenAI's customer-service agents demo the static scan found 2 of 8 agents, no tools and no hand-offs. It now finds all 8, each agent's tools and models, and the 15 hand-offs. Everything is added inside discovery's existing design: import-gated constructors, file-scoped resolution with the unique cross-file fallback, typed edges. - Typed constructors (Agent[Context](...)) for every recognised constructor. - Hand-offs and tools added after construction (= / append / extend / insert / +=, handoff(agent=x, ...)), also from a wiring module; added, never replacing. - Tool lists from a helper's return, tools or ALL_TOOLS, A if c else B, a name bound to a helper's list, and a list or helper exactly one other file defines. An unreadable list adds nothing instead of a tool named after the helper. - Tools made by a call (function_tool(fn), function_tool(...)(fn), imported tool(fn), StructuredTool.from_function), named as the code names them, in agent tool lists and repo tool sites; repo sites honour name_override. - Agent name= from a module constant. - Pre-1.0 LangChain: create_*_agent, initialize_agent, AgentExecutor (one agent). - A hand-written OpenAI/Anthropic/LiteLLM function-calling loop, only when the repository has no framework agent. - AgentSpec gains model (literal, constant, f-string of constants, getenv default, chat-model object; None otherwise), ref (module, name, kind) and tool_details (description, argument names). ScanReport gains unassigned_tools. The submission payload is unchanged. - Test, mock and fixture files are left out of agent and tool attribution; the repo scan's findings still cover them.
…cable, owner checked at use)
…ts and POST /api/projects/{project}/approve
…n leased rows per project; validated payloads; race-free first load; basic guardrails through simulate-before-enforce
…e the model provider in errors
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it needs infer_impact and classify_tool matched substrings, so 'pay' was in display_seat_map, 'ship' in get_membership_status, and transfer_to_human moved money. Both now read a name as tokens and verb phrases (impact.tokens / actions / handoff): a noun-verb after a read verb is what is read, 'and'/'then' start a new phrase, and a hand-off to a person is not a transfer. issue_* at the head of a name pays out instead of reading issues; start_return (refund in its description), list_and_refund move money; cancel/book act irreversibly; fetch_and_email_report sends out. Discovery gains what agentfox init's map needs, so the onboarding mapper can stop walking the AST itself: AgentSpec.named and .instructions (never serialised), a module-level 'agent = build_agent()' reached as the object, docstrings from a plain module of functions another file wraps as tools, and RECOGNISED (what it reads, per framework).
…napshot
A project is a system: a root agent and its delegates_to closure of any kind, across
repositories (registry.projects.systems_of). A root is an agent nothing hands work
to; in a cycle (the airline desk, where everyone hands back to triage) the one that
hands off to the most others, ties by name. A system approved from a repository scan
and one reported by `agentfox init` are the same thing. agents.project is only
provenance: it lists a system's repos and resolves the names init and older links
use.
* GET /api/projects: one entry per system, key = root slug, plus root, name,
members (depth, relation, parent, repo, status) and repos. Existing fields kept.
* GET /api/projects/{p}/map: the closure, sub-agents and parents from other
repositories included, with each agent's depth/relation/repo and hand-off
relations; a repository name resolves to the systems with its agents; an unknown
name is a 404 instead of an empty map.
* Init never proposes over another repository's draft or disconnected agent, so two
repos with a "Triage Agent" are two agents.
* DELETE /api/projects/{p} acts on the members from the same repository, keeps
agents.project, keeps hand-offs whose other end is another project's, records
each agent's live guardrail layers (agents.guardrails_snapshot) before emptying
them, and touches no other agent's layers. POST /api/agents/{slug}/restore-
guardrails re-binds the recorded versions after a reconnect (watching; blocking
again only with policy_production and a recorded simulation).
* /api/agents leaves disconnected agents out unless include_removed; the policy agent
tree and the map's sub-agents/parents skip them too.
* talks_to / touches edges keep their detail and resource kind in the new
lineage_edges.attrs (migration d7a4c2e9f1b6, re-runnable, moves old rows); kind is
the delegation relation only. Input boxes are no longer in an agent's blast radius.
* register_structure records each hand-off's relation as discovery read it, and the
registry impact of a mapped tool is the classifier's reading raised to its risk's
floor; a declared impact is still never replaced.
…gone map_repository(root) is now a thin adapter over discovery.repo.scan: agents, tools, models, hand-offs (every relation, kept in handoffs[].relation), runner refs (module/attr/kind) and coverage (code files read, unparseable files, tools given to no agent, skipped extensions) come from the same scanner `agentfox scan` uses. Tool risk and the system a tool touches are discovery's classify_tool + infer_impact said in the onboarding's words; there is no third classifier. Systems are matched on whole words. Agents are ordered entry first (registry.projects.systems_of), so the agent a test talks to is the system's root. The init CLI's summary reads that coverage, and its "Missing something?" lists what discovery recognises (discovery.agents.RECOGNISED). The ported tests assert through the adapter. Two encoded the old mapper's bugs and now say the right thing: examples/ is read, and an agent whose model is chosen at run time has no model rather than a repository-wide guess.
…ems; UI follow-ups
dry_run_check runs one surface (input, tool_call, tool_result, output) through NativeRedTeamRunner.run_probes for an agent's bound policies, with a deployment-derived probe (provision=False) inside the runner's rolled-back savepoint. Probe outcomes now carry the applied verdict, reason and the rewrite. Guard results add would_be_content: the redaction an observe-mode policy asks for.
The suite is built per agent from tool roles (read, action, outside, hand-off), argument names and descriptions instead of retail Bazaar templates: faked reads are shaped from the tool, only action tools are ever forbidden, hand-offs never are, a set needing a missing role is reported not run, and every set carries benign controls. Results carry outcome, agent, stoppedBy, control and reason; runs report gotThrough, stopped, resisted, errored, notRun, over-blocked controls, a false-positive rate and clean. The simulated BASIC_TEST_GUARDS ids are gone.
While a suite runs, every LangChain tool class's _run/_arun (including classes defined mid-run), the Agents SDK's FunctionTool.on_invoke_tool and a function loop's tool functions go to the recorder, so class-based, factory-built, async and external-package tools never run. A tool that can't be faked means the agent isn't run, and an audit hook refuses file writes, deletes and processes for the suite's duration. The runner drives LangChain/LangGraph, the Agents SDK and plain function loops for every agent in the project, from discovery's refs, and restores cwd and sys.path after loading. An after test sends input, each tool call and result, and the reply to the server's dry-run check and applies the verdict (block stops, escalate holds, redact rewrites); the runner's own guard simulation is gone.
POST /api/onboarding/runner/runs/{id}/guard checks one surface of an attack
against the agent's real policies via redteam.dry_run_check: only for a running
test of the key's own project (or its user's wizard test) and only that test's
agents; nothing is persisted. Jobs say policies=true for an after test and no
longer carry simulated guard ids; a runner that doesn't speak protocol 2 isn't
handed an after test. /guardrails stays for the wizard's display only.
Marker-file tests for module-global, class-based, async, factory-built, late-defined and external-package LangChain tools, Agents SDK function tools and a plain function loop; the audit-hook backstop; HR, ITSM, read-only, retail and multi-agent suites; honest outcomes; the guard route's scope and that it persists nothing. LangChain and Agents SDK tests skip without those packages.
…ts branches Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The "after" attack test failed every check with "database is locked" on SQLite: the runner sends checks several at once and each dry run writes inside its rolled-back savepoint. On SQLite the checks now queue, each in its own transaction. Postgres is unchanged. - An attack AgentFox itself couldn't check was blamed on "your agent's model"; the message now says the AgentFox server couldn't check it. - A test where nothing ran (the agent couldn't be loaded) came back done, clean, "0 attacks got through". It is now failed with the reason, never clean, and the CLI says no attack could be run. - Projects from one repository are listed with its main system first, so the airline repo opens on the 6-agent Triage system, not a one-agent helper. - TODO: an injection false positive the harmless controls found. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…threads Detectors run on pool threads, which don't inherit context variables, so every check saw the deployment's judgment posture and no tenant. A workspace's pii_egress=block was read as the deployment default (redact), so personal data reached a judge the workspace had said must never receive it; judge tiers turned on only for a workspace looked 'not configured'. - The pipeline submits each detector in its own copy of the request's context (one copy per submit: a context can't be entered by two threads at once). - Custom models called side by side do the same. - Tests: a light and a heavy detector, and several custom models, see the workspace's posture and tenant; a reused pool thread doesn't carry one request's context into the next. They failed before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…packs, and survive a disconnect
- /libraries/basic/apply goes through the onboarding card's path
(service.add_basic_guardrails with may_enforce): watching unless the caller
may change production policy and these exact rules were simulated. It
crashed with a TypeError before. deps.may_enforce is the one place that
says who may make a policy block; onboarding, restore and libraries use it.
- /libraries/{id}/mode is gated like POST /api/policies/{key}/mode: a
policy_production role either way (403), and to block, a recorded
simulation of each copy's live rules (409 "simulate first: ..."), checked
for every copy before any switches; each switch audited. 'basic' is a
library id here too (agent.<slug>.basic).
- POST /libraries/{id}/simulate {agents, project?} replays each agent's
traffic against its copies with main's simulate + record_simulation, so the
gate then passes; per agent: blocked / held / hidden, examples, simulation
ids. SimulationDiff gains would_hide (redactions the candidate would add).
- Each version a library writes records its source in its notes
([from <library> <policy> <rules hash>]); status reports outdated /
outdatedAgents, and applying again updates outdated copies in their mode
(publish_in_current_mode). An emptied copy is filled again on apply.
'blocking' reads the project's copies, not an observing workspace pack.
- Disconnect snapshots agent.<slug>.* library copies with the other layers
(projects.own_keys), and restore-guardrails brings them back under the same
gate.
- Drafts from agentfox init take libraries (they apply once added);
/api/policies/applied counts them, marks rule agents draft, and adds
agentList. Library routes accept project and refuse agents outside it
(400), unknown projects/agents (404, tenant-scoped), and anything but a
shipped pack id (404).
- Her four onboarding tests follow #123: drafts, root-slug project keys,
watching until simulated, the setup key reused across inits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…locking - Act asks in the page instead of window.confirm, which the in-app browser suppresses (the button did nothing). Every Act confirm benefits. - Basic guardrails say they watch first and block only after a simulation by someone allowed; the card shows watching/blocking from the status and gets the same Simulate / Start blocking / Back to watching as the other libraries. Outdated copies show 'update available' and an Update button. Requests carry the project. Server refusals (403/409) show beside the button. - 'Would block · watching' (Would ask / Would hide) in Applied rules and the map's line panel via vocab.wouldDo; ruleTitle keeps PII, API, URL, SQL, ID, LLM, MCP, AI, EU upper case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… gates as the rest
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- 'Start blocking' refused without a simulation said to POST an API path; it now says to click Simulate. - A successful simulation resets the Start blocking button, so the earlier refusal doesn't stay on screen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Fixes on top of #122 (onboarding, project map, Observe pages). It targets
onboarding-guardrails-map, so the diff shows only the fixes, not #122 itself.A review of #122 found that next to the UI it added backend pieces that worked alongside the product's existing ones instead of building on them: a second code scanner, a second meaning of "project", a second setup path, its own imitation of the guardrails, and an attack corpus written for one shop. Each handled its own case and missed others the product already handled. This PR keeps everything new and moves it onto the existing parts.
The UI is not changed here. The new design stays. Every UI follow-up the review found (what to restore from main, what to fix, and what these backend changes now need) is in
PRODUCT_TODO.md§6.Changes
Finding agents in code. The onboarding mapper is now a thin adapter over main's scanner (
capabilities/discovery), and its own code reader is deleted.Agent[Ctx])function_tool(fn))AgentExecutor,create_tool_calling_agent)agentfox scan,up, Add all) now finds 8 of 8 agents, their tools, models and 15 hand-offs. Before, it found 2 agents and nothing else.classify_toolandinfer_impactnow read tool names as words, sotransfer_to_humanis no longer "moves money" and "ship" inside a longer name no longer means shipping.Projects are PR #112's systems: a root agent plus everything it hands work to, across repos.
agents.projectis kept only as a record of which repo an agent came from.POST /api/agents/{slug}/restore-guardrailsbrings them back.Setup key (
afi_…) is built on main's API tokens.POST /api/projects/{p}/approveadds them.Attack test
POST /api/onboarding/runner/runs/{id}/guard). Each stop names its rule and whether that rule is only watching.Routes
Detection and the map
GET /api/policies/{key}/rules/{rule_id}/checks).Fixes that apply to everyone, not just onboarding
judgment_fail_closednow defaults to off and can be changed per workspace.auto(): actually hides personal data in replies, where before it only recorded that it should have. A streamed reply is refused at the end when hiding applies.Checks
d7a4c2e9f1b6); upgrade → downgrade → upgrade works on a fresh SQLite database.Defaults chosen (change any you disagree with)
auto()is refused at the end when hiding applies.Known gaps left open
auto()and the guard endpoint don't.PRODUCT_TODO.md§6.Update: Garima's latest commits merged, and libraries on the same rules
Merges
onboarding-guardrails-mapup to3fb6f7c2: Applied rules, Libraries (the packs, applied per project), and the shorter nav. On top of it:would_hideadded to it).agentfox init, and Applied rules lists them as drafts. Only shipped libraries can be named (404 otherwise), and agents must belong to the project and the workspace.Checks:
test_breaker) that passes 3 out of 3 on its own.