Repository navigation
Live suite: plans, previews, tool drift and decoys against scute-api-v2 - #8
Merged
Merged
Conversation
The API has them now (v26), so the examples run instead of being skipped, and check the answers' shapes as the TypeScript harness expects them: - A plan for two calls: its id, say line and steps (permission, resource, details, needs), approved once; each step runs once, the steps show as used, and the extra call's own approval request is denied so nothing is left pending. - A preview answers like a check without using anything up: not a verification, and not the budget (previews on the agent already over its budget leave it active). - Tool drift: the baseline, a repeat, a new tool, a changed one. - A decoy call is refused, closes the run and pauses the agent. 82 examples, 0 failures, 1 pending (scute-ruby has no local decision engine).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The API now has plans, dry-run checks, tool reports and decoys (scute-api-v2 v26), so the live examples written in #7 run instead of being skipped. They check the answers' shapes against what the TypeScript harness expects. No SDK change; every answer matched.
What the examples check
run.request_plan,run.plan_status):id,status: "pending"and asayline.permission,resource,details(the call's arguments) andneeds: "approval".plan_statusshows both asused.run.preview):run.report_tools), on a fresh agent:{known: 1, new: [], changed: []};{known: 1, new: [], changed: []};{known: 1, new: ["void_invoice"], changed: []};{known: 0, new: [], changed: ["refund_invoice"]}.Scute::Guards.decoy), on a fresh agent: the call is refused (decoy_called, "I can't continue with this. A person will follow up."), the run closes, and Scute suspends the agent. The example then resumes it.Results
bundle exec rake liveagainst scute-api-v2 v26: 82 examples, 0 failures, 1 pending. The pending one is a skip: scute-ruby has no local decision engine for the policy snapshot.Cleanup removes the drift and decoy agents with the rest. After the run the suite's app has no users, agents, properties or policy, its settings are back to their defaults, and no access request is left pending. Approved plans and used or denied requests stay, because the API has no way to delete them.