feat(harness): agent source registry and the understand-agent stage

[KarthikAvinashFI]

Stacked on feat/platform-scenarios-livekit-sip. Draft: the run stage is not the focus yet.

What this is

A harness you point at an agent. It reads that agent's source, builds a real environment its tools
act on, and writes test scenarios that are each proved before they are kept. It is driven as a
conversation, in a terminal or on a web page, and nothing in it is written for a particular agent.

The problem it solves is that mocked tool responses answer every call the same way. An agent that
cancels an order that was never placed is told it succeeded, and the test meant to catch that
passes. Here the environment answers truthfully, including a truthful refusal, so the agent has to
cope with the same world a user would put it in.

Nothing under fi/simulate is touched. Generated worlds subclass EnvironmentAdapter, so the
runners that already exist can drive them.

Trying it

Full instructions are in src/fi/alk/harness/README.md (setup from nothing) and
harness-ui/README.md (the web page). The short version:

uv sync --extra livekit --group dev
set -a; . ./.env.acceptance; set +a          # Vertex service account, see the README
export CLOUD_ML_REGION=global ALK_HARNESS_MODEL=claude-sonnet-4-6

.venv/bin/python harness-ui/server.py        # server and page, both on :8777

Open http://localhost:8777, press + new, and say what you want tested:

i want to test my voice ordering agent. the code is at /absolute/path/to/the/agent

One message is enough to start. There is no separate front end to build: the page is a static file
the same server hands out, talking to its own JSON endpoints.

The stages

Each is a model session with a small tool surface and its method in a markdown skill. The model
decides what to do; code decides what is true. Nothing reaches disk except through a tool that
checked it first.

Stage	Produces
understand	contract.json: the tools with exact argument names and permitted values, the hard rules, the real data, and what the agent depends on
build	the world its tools act on, one handler per tool, the simulator prompt, and the sub-goal catalogue with each check written as code
scenarios	one folder per scenario, each proved by three gates before it is kept
run	grades from world state plus every tool call, locally or against a hosted agent

A scenario owns a folder, and the code in it is code rather than strings inside JSON:

scenarios/<name>/
    scenario.json     the instruction, the reference solution, which sub-goals it names
    setup.py          def setup(world)
    ready.py          def ready(world)
    checks/<goal>.py  def check(world, calls), runnable on its own against a finished run

Three gates, all pure code, no model involved:

1. ready: reset, apply setup, run ready. The world must hold what the scenario presumes.
   Otherwise the agent fails for a precondition we got wrong and it reads as the agent's fault.
2. solvable: replay the reference solution and run the checks. They must pass, or either the
   scenario cannot be passed or a check is wrong.
3. not vacuous: reset, set up again, run nothing, run the same checks. They must fail. A check
   that passes while the agent does nothing grades nothing while reporting a result.

Only a scenario clearing all three is kept. The reference solution is kept with it and is never run
against the agent under test.

One conversation is one folder. artifacts/sessions/<id>/ holds the chat and every artifact, so
refreshing the page, restarting the server or coming back tomorrow all resume by reading the folder.
There is nothing held in memory that is not also on disk.

Verification

Two agents, deliberately unalike, both driven entirely through the chat with no code changes
between them.

	voice ordering agent	retail support agent (third-party benchmark)
modality	voice	chat
tools	5	16, all handled
world	3 tables, 95 rows	11 tables, normalized past the contract's 3
probes	25/25, score 1.00	passing, world saved
sub-goals	7, 6 settled by code	10, 8 settled by code
scenarios	5 of 5 green on all three gates	10 of 10 green on all three gates

The second one is the useful test: an agent nobody here wrote, whose data lives in JSON files, in a
different modality, with three times the tools. The contract recorded the files as dependencies and
sampled their contents; the build stage then produced a relational schema and sixteen working
handlers from that alone.

Both worlds refuse correctly, verified by hand: an unavailable item, an item that is not on the
menu, a size the tool does not accept, and an identifier that was never created.

Offline tests: 130, no model calls, no network, no credentials.

What the gates caught, in practice

A world built during scenario writing generated order_id as "O_" + COUNT(*)+1, so after a
removal the count dropped and the next insert reused a live id. It surfaced as an IntegrityError,
a crash rather than a refusal, which is the distinction the design rests on: refusals are the world
working, crashes are ours.

Gate 2 rejected two scenarios whose reference solutions did not pass their own checks, and the stage
corrected both before saving.

Gate 3 has a deliberate softening: vacuity is judged on every check passing, because one check
surviving an empty run ("no unavailable item was ordered") is legitimate. A single check that
survives is still named, because sub-goals are shared and such a check would roll up as a pass for
an agent that did nothing. That reporting found a real one: an authentication sub-goal used by seven
scenarios asserted "no modification happened before authentication", which is true by default when
nothing happened at all.

Known limits

• A tool that talks to a service cannot yet have that service built. The contract records the
  dependency and the skill says to stand one up, but no tool writes one.
• A file-backed agent gets a sampled world, not its full dataset, so it is not benchmark fidelity.
• Asking for 100 scenarios does not work. Ten to twenty in one pass does. Beyond that needs saving
  progress and generating in waves.
• A stage cannot hand back to an earlier stage on its own. Moving back is a person clicking the
  roadmap.
• Browser worlds are registered but stubbed. The judge runs on the local path only.