feat(generation): grounded scenario generation with deterministic checks and live voice wiring - #57
Draft
KarthikAvinashFI wants to merge 56 commits into
Draft
Conversation
…enerate, verify loops)
…r guidance channel, critic catch-verification
…ous token budgets
…brics yes-means-met
…ates; final-action closure rule
…untime-mock-to-verdict path
…ds in checkpoints
…sip' into feat/scenario-generation-local
…carried and required
…extra repair attempt
…ge scenario counts
…terministic near-dup filter
…ution check, purpose register
…hing, thread-safe metering
…ntract's per-arg valid sets
…call_nth honored)
…eld merge, 2-subgoal floor, rubric specificity
…eplenishment, explicit exhaustion verdict
…t silently dropped
…ness needs two empty stances
…ith provenance, traces first then coverage
…venance, baseline coverage fills the rest
…rer completeness challenge restored
…r, failure amplification
…s stop failing their own validator
…ines it, else args_present
…ndering the deterministic checkpoint
…rless tools to be asserted by the call
…dles a scenario's own mocks create
…rdings that were never supplied
…g inferred from an absent field
…ll the model its real shape
…r prompt with disclosure rules
…o when one is supplied
…g a scenario live
…es its checks and writes the trace
…ead of hardcoding one agent
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Scenario generation for the agent-learning-kit: point the harness at an agent and it produces test
scenarios that can be staged and graded, rather than prose that has to be turned into tests by hand.
Each generated scenario carries three separated parts: what the simulated caller is told and what
they know (with rules about when they may reveal it), the mocked world the agent acts on, and the
checks that decide pass or fail. Most checks are settled by comparing recorded tool calls, final
state, or a value the agent had to say, in plain Python. A model is asked for a judgement only where
none of those can witness the outcome. On the reference run, 69 percent of checks were code-graded.
How it is kept honest
Generation is a fixed pipeline with four gates, three of which involve no model at all:
contradicts itself is caught before anything else runs
Anything a gate objects to is sent back as a specific instruction, at most three times, and a
rewrite that returns the identical objection stops immediately.
Grounding in real conversations
Given a folder of recorded conversations, the harness works out its layout on its own, judges each
recording's outcome from its content, and prioritises the ones that went wrong. Each failure becomes
a scenario that recreates it plus a small neighbourhood of scenarios that can fail for the same
underlying reason by a different route, so a suite fences the class rather than pinning one case.
Every scenario records where it came from, and a scenario citing a recording that was never supplied
is discarded rather than presented as grounded in production.
Running one live
simulate_bridgeturns a generated scenario into the simulated caller, including the disclosurerules that make an elicitation test meaningful.
vapi_liveserves the scenario's mocks over HTTP,records every tool call with its arguments, and builds a provider assistant from the extracted
contract. Setting
ALK_SCENARIOmakes the existing acceptance runner use a generated scenario inplace of its built-in persona.
This was verified end to end on a live web call: the generated caller held the conversation, the
agent's tool calls were served and recorded, and the scenario's checks graded the result. The run
reported
completedwhile the deterministic checks failed it, which is the distinction the checksexist to make.
Scope
voiceandchatare supported; anything else is refused by name,because a scenario is only worth generating when a runtime can stage and grade it.
scenarios exist for the agent.
Tests
tests/test_generation_pipeline.py, 39 tests, fully offline against a fake model. Each encodes adefect that actually occurred during development.
Notes for review
environment and never reach a generated artifact.
oss/simulation-acceptance/voice_cases.pyis the only existing file touched: it gains an optionalbranch that uses a generated scenario when one is supplied, and is unchanged otherwise.