Skip to content

feat(generation): grounded scenario generation with deterministic checks and live voice wiring - #57

Draft
KarthikAvinashFI wants to merge 56 commits into
feat/platform-scenarios-livekit-sipfrom
feat/scenario-generation-local
Draft

feat(generation): grounded scenario generation with deterministic checks and live voice wiring#57
KarthikAvinashFI wants to merge 56 commits into
feat/platform-scenarios-livekit-sipfrom
feat/scenario-generation-local

Conversation

@KarthikAvinashFI

Copy link
Copy Markdown
Contributor

What this adds

Scenario generation for the agent-learning-kit: point the harness at an agent and it produces test
scenarios that can be staged and graded, rather than prose that has to be turned into tests by hand.

Each generated scenario carries three separated parts: what the simulated caller is told and what
they know (with rules about when they may reveal it), the mocked world the agent acts on, and the
checks that decide pass or fail. Most checks are settled by comparing recorded tool calls, final
state, or a value the agent had to say, in plain Python. A model is asked for a judgement only where
none of those can witness the outcome. On the reference run, 69 percent of checks were code-graded.

How it is kept honest

Generation is a fixed pipeline with four gates, three of which involve no model at all:

  • validators reject any tool, argument or identifier that does not exist in the agent's own code
  • an oracle runs each scenario against the run it itself predicts, so a scenario that
    contradicts itself is caught before anything else runs
  • deterministic deduplication compares every candidate against every accepted one
  • an independent reviewer asks whether a competent agent could actually fail the test

Anything a gate objects to is sent back as a specific instruction, at most three times, and a
rewrite that returns the identical objection stops immediately.

Grounding in real conversations

Given a folder of recorded conversations, the harness works out its layout on its own, judges each
recording's outcome from its content, and prioritises the ones that went wrong. Each failure becomes
a scenario that recreates it plus a small neighbourhood of scenarios that can fail for the same
underlying reason by a different route, so a suite fences the class rather than pinning one case.
Every scenario records where it came from, and a scenario citing a recording that was never supplied
is discarded rather than presented as grounded in production.

Running one live

simulate_bridge turns a generated scenario into the simulated caller, including the disclosure
rules that make an elicitation test meaningful. vapi_live serves the scenario's mocks over HTTP,
records every tool call with its arguments, and builds a provider assistant from the extracted
contract. Setting ALK_SCENARIO makes the existing acceptance runner use a generated scenario in
place of its built-in persona.

This was verified end to end on a live web call: the generated caller held the conversation, the
agent's tool calls were served and recorded, and the scenario's checks graded the result. The run
reported completed while the deterministic checks failed it, which is the distinction the checks
exist to make.

Scope

  • Environments are a closed set. voice and chat are supported; anything else is refused by name,
    because a scenario is only worth generating when a runtime can stage and grade it.
  • Requesting a count returns exactly that count, or an explicit statement that fewer distinct
    scenarios exist for the agent.
  • Spend is metered against a ceiling, and artifacts are written after every accepted scenario.

Tests

tests/test_generation_pipeline.py, 39 tests, fully offline against a fake model. Each encodes a
defect that actually occurred during development.

Notes for review

  • No credentials are read or written by anything in this change; provider keys stay in the
    environment and never reach a generated artifact.
  • oss/simulation-acceptance/voice_cases.py is the only existing file touched: it gains an optional
    branch that uses a generated scenario when one is supplied, and is unchanged otherwise.

…r guidance channel, critic catch-verification
…eld merge, 2-subgoal floor, rubric specificity
@KarthikAvinashFI KarthikAvinashFI self-assigned this Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant