FE-1574: Let Voice speak through canonical Brunch conversations - #9528
FE-1574: Let Voice speak through canonical Brunch conversations#9528lunelson wants to merge 31 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Merging this PR will not alter performance
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
as_constant |
< 1 ns | < 1 ns | N/A | |
constant_equal |
< 1 ns | < 1 ns | N/A | |
constant_not_equal |
< 1 ns | < 1 ns | N/A | |
access |
< 1 ns | < 1 ns | N/A | |
runtime_equal |
< 1 ns | < 1 ns | N/A | |
runtime_not_equal |
< 1 ns | < 1 ns | N/A |
Comparing ln/fe-1574-direct-voice-flue (a69c2dd) with main (9c21578)
b0cdf33 to
07ed83f
Compare
State the release note, no-engineer demo script on the local deployment posture, and previously-impossible delta (Stop that really stops; one shared typed/spoken conversation), pin completion to the contract stratum rather than the first green tracer, and mark the single-route consolidation as internal sequencing rather than the visible advance.
Name Mission 5's surface as the Petrinaut Brunch panel's typed and Voice route and record that the live branch adopted the litmus on restack.
Now that the panel composer is itself Flue transport, let Voice enter either through it (preferred, one visible store) or by a direct send(); reword proof leaf 3 to forbid only non-Flue submission and record the choice and its fallback criterion on the fog-line.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Update the living Petrinaut integration spec and topology reference to name the guarded Flue conversation mount, browser ChatTransport projection, and current transport-package dependency boundary. Narrow the Mission 5 route scan claim to its actual production paths while preserving superseded /api/chat references as historical provenance.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…n projections Review of PR #9528 read the ten bot findings as samples of three faults. Stop was not bound to the turn it targeted: a Stop during an in-flight admission now waits for that admission before the conversation-wide abort, a durable Stop result that lands after a newer turn started is ignored, and a Voice turn whose submission settles aborted closes its Realtime call without speaking, driven by Flue's settlement index because the AI SDK reports ready between a step and its automatic client-tool follow-up. The live-stream and snapshot projections encoded the same rules twice: the snapshot keeps providerExecuted on a running server tool and folds a client-tool continuation into the assistant message it resumed; the transport reports the resumed assistant id and the tracker keeps the originating submission for it; one client-tool catalog feeds the panel transport and the history projection; host history hydrates only once it carries every locally streamed reply. A retained Voice input can be withdrawn through an AbortSignal, a consumer stream cancel no longer writes to a closed controller, and first-canonical-text latency is recorded when the first completed text block appears. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A client-tool continuation is projected onto the assistant message it resumes, so one message can be written by several submissions, and either side may have admitted the continuation: the panel after a readPetrinautDoc call, or Voice when it answers a pending brunch_ask. Keeping only the first submission served one direction and broke the other. The tracker now records every submission per message, segments carry that set, and the bridge matches by membership while excluding segments that predate the answer. The Stop generation guard also covers the rejection path, so a durable Stop that fails after a newer turn started no longer records an error on that turn. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ld it The AI SDK reports ready between a step that ended in client tool calls and the follow-up it sends automatically. Hosts read that gap as the end of the turn: the Voice bridge closed its Realtime call on the step's partial text, and a Stop whose durable abort landed already-settled did nothing while the follow-up went out anyway. The panel now keeps its composer status busy until the follow-up starts, and a Stop pressed during the step withholds the follow-up and marks the response stopped.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit acc97ae. Configure here.
kostandinang
left a comment
There was a problem hiding this comment.
Approving this PR, but noticed three correctness issues, which we may defer to #9537 or the reconciliation one #9564 (review).
-
Prevent replay of earlier client-tool results. Continuations are folded into the same assistant message. The transport currently collects every completed client-tool result from that message, so a later continuation can resend both the new result and results already delivered in previous steps. It should send only newly completed results or track delivered tool-call IDs.
-
Terminate Voice correctly when a continuation is withheld. If Stop is pressed after a tool-only step has completed, the panel suppresses the automatic follow-up. Flue reports the completed step rather than an aborted submission, so the Voice bridge does not recognize it as terminal and can leave the Realtime call waiting indefinitely. The withheld continuation needs an explicit terminal outcome.
-
Scope Stop so a delayed abort cannot affect a newer turn. Stop currently issues a conversation-wide asynchronous abort. The original turn can finish and a new turn can be admitted before that abort reaches Flue, allowing the stale abort to cancel the newer turn. New submissions should remain blocked until Stop settles, or cancellation should target a specific submission.
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1002 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1527 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 108 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 27 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |

🌟 What is the purpose of this PR?
Make the mounted Flue conversation route the only product door into a Brunch conversation, and make Voice a faithful audio projection of that one canonical conversation. In the Petrinaut Brunch panel you can type or speak to Brunch in one conversation; what you hear is exactly what Brunch wrote; Stop asks Brunch to stop rather than only hiding its answer; and reopening the panel rehydrates the same canonical conversation from Flue without re-sending or replaying anything.
The production throughline is the local Petrinaut surface under
yarn dev:brunch: the typed panel'suseChatand the Voice control both reach one browser Flue client at the same-origin-proxied/agents/chat/:instanceIdroute, behind the single ownership guard, into the mountedChatAgent. The former Brunch/api/chathandler, projector, and proxy are deleted.What the proof establishes: one typed turn and one finalized spoken turn each cross the Flue route exactly once, the typed turn returns as one finite AI SDK stream, the Voice turn returns as visible text and TTS of the same canonical words, Stop uses Flue's durable abort, and reopen rehydrates canonical messages from Flue observation. What it does not claim: trusted production authentication or authorization, remote deployment, structured
brunch_askquestions (that capability stays suspended), Petrinaut mutation tools, faithful reopen presentation, or Mission 5 product acceptance.Landing decision. This PR lands as the transport tracer. The 2026-09-04 human witness (recorded in the evidence README and
MISSION.md) passed typed and Voice admission, exactly-one visible Voice input, spoken canonical output, barge-in, and explicit durable Stop, but failed the faithful-reopen gate: canonical messages returned after reload, yet per-message typed/Voice provenance was absent and the stopped assistant entry rehydrated as ordinary truncated content. Rather than expand this branch, the owner deferred that remediation to Mission 6 (#9537, stacked on this PR), which owns preserving both distinctions across its second-tab resume proof. Mission 5 acceptance (proof leaf 8) therefore remains open inMISSION.mdand is not a gate for merging this transport change.🔗 Related links
libs/@hashintel/brunch-agent/MISSION.mddocs/evidence/implementations/mission-5-direct-voice-flue/README.md🚫 Blocked by
main. The proof leaf 8 acceptance gate is deferred to Mission 6 as described above, not carried as a blocker here.🔍 What does this change?
@hashintel/brunch-agent-transport-aisdkbecomes the browser-side adapter over public@flue/sdk: aChatTransportthat admits onesend()per turn, projects thewait()event stream intoUIMessageChunks, and projects Flue history snapshots into AI SDK messages. It no longer contains a server handler./api/chat; the local launcher proxies/agents/chat/*instead. Ownership is derived from principal plus logical conversation id in one guard.requestStop.submissionId, and speaks exactly the canonical text.requestStopso Stop can become durable before the local stream is cancelled (changeset:.changeset/durable-brunch-stop.md).flue.tsentrypoints; YAML plugin machinery is removed.🏗️ Agent notes
Mission authority is
libs/@hashintel/brunch-agent/MISSION.md(live as of 2026-09-04). The six sections, condensed:Imperative. One Flue route is the only product door into a Brunch conversation; Voice is an audio projection of that conversation. Now, because Mission 4 fixed the canonical agent composition while two transports still reached the same conversation.
Throughline. Petrinaut typed panel (
useChatwith a host-supplied browserChatTransport) and Petrinaut microphone (OpenAI Realtime, one validated finalizedcontinue_interviewanswer) → one browsercreateFlueClient()per principal + logical conversation id → onesend()at the same-origin-proxied/agents/chat/:instanceId→agentOwnershipGuard→createAgentRouter(ChatAgent)withuseBrunchAgent()+useSdcpnPlugin()→wait(admission, { onEvent })for the finite per-turn stream,observe({ live: "sse" })for canonical state and reopen → parts correlated bysubmissionId→ panel projector terminated onsubmission-settled; Voice speaks canonical completed text unchanged.Proof. Nine leaves with named oracles: (1) typed panel over the browser transport, (2) direct finalized admission, (3) no server-side AI SDK door, (4) canonical visible and TTS output, (5) cancellation and abort stay distinct, (6) reopen resumes without replay, (7) architecture and boundary integrity, (8) real Voice witness and retained proof bundle, (9) focused repository verification and truthful docs. Leaves 1 to 7 and 9 have automated evidence recorded in the evidence README; leaf 8 ran on 2026-09-04 and failed its reopen gate, with remediation deferred to Mission 6.
Constraints. Flue history is the sole canonical record;
@flue/sdkis used directly with no re-specified offsets, retries, or recovery; one product route; the panel stays onuseChat; only the validated finalized answer is submitted and never automatically retried; Brunch owns response content; the four cancellation kinds stay distinguishable; the local principal is a discriminator, not authentication; the transport package depends only onaiand@flue/sdk;brunch_askis not mounted.Fog-line. How Voice enters Flue (shape B, via the panel transport, is what shipped); the home of the identity and header contract; whether
wait()alone gives a clean per-turn stream or reconnect needsobserve(); whetherreconnectToStreamresumes; which parts of #9496/#9507/#9512 to port; speech selection for multi-block responses; whether one Stop affordance can express local interruption and durable abort; whether the same-origin proxy carries every SDK route.Stop or reorient. Stop on a second conversation authority, any non-Flue turn route, provisional STT submission, automatic retry of ambiguous admission, canonical text rewritten by another model, hand-rolled recovery, restored stub agent, or activated
brunch_ask; stop if client-tool resume semantics need a server helper; stop if local cancellation aborts durable work or Stop only cancels the browser request.Deferred items are recorded in
MISSION.md§Deferred andMISSION.next.md.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
@hashintel/petrinaut:.changeset/durable-brunch-stop.mdand.changeset/voice-input-withdrawal-and-stop-correlation.md; the@hashintel/brunch-agent*packages are private)📜 Does this require a change to the docs?
The changes in this PR:
libs/@hashintel/petrinaut/docs/ai-assistant.md, Brunch evidence and mission records)🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
turbo.json's have been updated to reflect this (plugin packages gainedbuild,lint:*, andtest:unitscripts;apps/brunch-agent, core, and plugin-dafnyturbo.jsonupdated)The ten review-bot findings on this PR were read as samples of three underlying faults and all are addressed in the review-fix commit, with a failing-first test for each:
send()was still in flight now waits for that admission before Flue's conversation-wide abort, soaborted: falseno longer reads as "already settled" while the admitted turn keeps running. A durable Stop result or failure that lands after a newer turn started is ignored. A Voice turn whose submission settlesabortedcloses its Realtime function call without speaking, driven by Flue's settlement index rather than chat status, because the AI SDK reportsreadybetween a step and its automatic client-tool follow-up.providerExecutedon a server tool that is still running and folds a client-tool continuation back into the assistant message it resumed, matching the live stream. The transport reports the resumed assistant id to the host, and the tracker records every submission that wrote a message, so Voice correlates by membership and speaks a reply whether the continuation was admitted by the panel after areadPetrinautDoccall or by Voice answering a pendingbrunch_ask. One catalog (brunch-client-tools.ts) now feeds the panel transport and the history projection, and a test asserts every registered interactive widget answers a tool in it. Host-owned history is hydrated only once the snapshot carries every locally streamed reply.AbortSignal; Petrinaut drops it from the queue and rejects it when the signal fires.stream.cancel()no longer enqueues on a closed controller, andfirst-canonical-textlatency is recorded when the first completed canonical text block appears rather than at settlement.brunch_askremains suspended on the server. Its browser paths are now consistent but carried as latent code, not as a claim.Second review round (four further Bugbot findings): two did not reproduce (the abort race attaches a handler to the losing branch through
Promise.race, so no rejection is unhandled; an empty host snapshot never latches hydration because the panel returns before latching when both lists are empty). Two were real and share one root: the AI SDK reportsreadybetween a step that ends in client tool calls and the follow-up it sends automatically, and hosts read that gap as the end of the turn. The Voice bridge closed its Realtime call on the step's partial text, and a Stop whose durable abort landedalready-settleddid nothing while the follow-up went out anyway. The panel now keeps its composer status busy until the follow-up starts, and a Stop pressed during that step withholds the follow-up and marks the response stopped, each with a failing-first test.Final unresolved review thread: conversation-owned state could leak across an explicit conversation switch. Commit
2697d0a834keys the stateful panel by conversation identity souseChat, pending continuation/error/stopped state, late callback refs, and pending input recovery are replaced together. A failing-first panel test holds one conversation in a client-tool continuation, switches identity, and requires the replacement to render ready.🐾 Next steps
🛡 What tests cover this?
libs/@hashintel/brunch-agent/packages/transport-aisdk/test/chat-transport.test.ts,transcript.test.ts,ui-stream.test.tsapps/petrinaut-website/src/main/app/voice-interview/realtime-brunch-bridge.test.ts,voice-interview-control.test.tsx,voice-turn-controller.test.ts,canonical-speech.test.tsapps/petrinaut-website/src/main/app/local-storage-demo/brunch-panel-transport.test.ts,local-storage-demo-app.test.tsxlibs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel.test.tsxapps/brunch-agentboundary and ownership testsyarn exec turbo run lint:tsc lint:eslint test:unit build --filter @apps/brunch-agent --filter @apps/petrinaut-website --filter @hashintel/petrinaut --filter @hashintel/brunch-agent --filter @hashintel/brunch-agent-plugin-sdcpn --filter @hashintel/brunch-agent-transport-aisdk❓ How to test this?
yarn dev:brunchand open the Petrinaut Brunch preview.📹 Demo
No recording. The 2026-09-04 human witness is recorded in prose in the evidence README; its artifact bundle was not retained after the reopen gate failed.