FE-1580: Reconcile Voice turn behavior on the shared Brunch conversation - #9564
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
PR SummaryHigh Risk Overview Brunch question marking moves to a new core
Local Reviewed by Cursor Bugbot for commit a0ca1cb. Bugbot is set up for automated code reviews on this repo. Configure here. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## ln/fe-1575-resumable-workpiece-petrinaut #9564 +/- ##
=============================================================================
+ Coverage 15.41% 66.32% +50.90%
=============================================================================
Files 208 1773 +1565
Lines 6557 191593 +185036
Branches 1141 7835 +6694
=============================================================================
+ Hits 1011 127077 +126066
- Misses 5445 63035 +57590
- Partials 101 1481 +1380 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
🔵 Needs a closer look
It spans transport durability, Voice state, browser tools, and public UI behavior while required human microphone, reload, and latency gates remain open.
Pull request overview
Reconciles Petrinaut Voice with the canonical, resumable Brunch/Flue conversation path.
Changes:
- Adds half-duplex Voice handoff, replay controls, compact docks, and transcript authority.
- Strengthens transport idempotency, error handling, history projection, and stopped/Voice metadata.
- Adds question markers, persistent diagnostics, extensive tests, documentation, and mission evidence.
File summaries
| File | Description |
|---|---|
package.json |
Prebuilds Brunch dependencies for local development. |
.../ai-assistant-panel/types.ts |
Extends message metadata for Voice origins and stopped responses. |
.../voice-dock/transcription-icon.tsx |
Removes the obsolete transcription icon. |
.../voice-dock/playback-menu.tsx |
Adds Voice replay actions. |
.../ai-assistant-contents/voice-dock.tsx |
Adds collapse, replay, handoff, and notice controls. |
.../ai-assistant-contents/tool-list.tsx |
Displays complete tool errors inline. |
.../defer-voice-messages.ts |
Removes deferred transcript behavior. |
.../defer-voice-messages.test.ts |
Removes obsolete deferral tests. |
.../ai-assistant-contents.stories.tsx |
Adds compact and collapsed Voice stories. |
.../components/voice-session-labels.ts |
Adds labels for new Voice controls. |
.../types/ai-assistant-composer-control.ts |
Extends public Voice controls and stopped state. |
.../voice-session/use-voice-session.ts |
Adds granular Voice capability hooks. |
.../voice-session/types.ts |
Extends Voice session state. |
.../voice-session/store.ts |
Extends Voice actions. |
.../notifications/toaster.tsx |
Adds persistent, detailed, copyable notifications. |
.../notifications/provider.tsx |
Supplies notification details and persistent errors. |
.../notifications/provider.test.tsx |
Tests notification duration and detail behavior. |
.../notifications/context.ts |
Adds notification detail support. |
.../panda-preset.ts |
Removes an unused Voice animation. |
.../docs/ai-assistant.md |
Documents the revised Voice and error UX. |
.../transport-aisdk/test/ui-stream.test.ts |
Tests hidden tools and error projection. |
.../transport-aisdk/test/transcript.test.ts |
Tests durable Voice and stopped metadata. |
.../transport-aisdk/test/chat-transport.test.ts |
Tests admission identity, failures, and callbacks. |
.../transport-aisdk/src/ui-stream.ts |
Hides internal tools and preserves error details. |
.../transport-aisdk/src/transcript.ts |
Reconstructs history metadata and hides marker tools. |
.../transport-aisdk/src/index.ts |
Adds typed admission failures and deterministic delivery. |
.../transport-aisdk/src/error-text.ts |
Adds bounded error serialization. |
.../core/vite.config.ts |
Builds the question-marker entry point. |
.../core/test/question-marker.test.ts |
Tests marker schemas, tool behavior, and prompt instructions. |
.../core/src/question-marker.ts |
Defines the durable question-marker contract. |
.../core/src/prompts/SYSTEM.md |
Instructs Brunch to mark direct questions. |
.../core/src/index.ts |
Exports question-marker APIs. |
.../core/src/flue.ts |
Mounts the question-marker tool. |
.../core/package.json |
Exposes the question-marker subpath. |
.../MISSION.next.md |
Updates the mission dependency and planning record. |
.../voice-resumable-reconciliation/verification.md |
Records verification results and remaining gates. |
.../voice-resumable-reconciliation/main-restack.md |
Records the restack and route verification. |
.../voice-resumable-reconciliation/import.md |
Records imported source provenance. |
.../mission-5-voice-safety-parity/witness-blocker.md |
Documents the outstanding human witness. |
.../mission-5-voice-safety-parity/provenance-blocker.md |
Documents the direct-user provenance limitation. |
.../mission-5-voice-safety-parity/donor-behavior-matrix.md |
Records adopted Voice behavior and acceptance status. |
.../mission-5-question-marker-and-provenance-decision-2026-09-04.md |
Records question-marker and provenance decisions. |
.../server/voice/openai-voice-policy.ts |
Makes Realtime transcription-only and half-duplex. |
.../server/voice/openai-voice-policy.test.ts |
Tests the revised Realtime policy. |
.../server/voice/openai-realtime-call.test.ts |
Tests forwarding the half-duplex policy. |
.../voice-interview/voice-session-state.ts |
Projects replay, handoff, and notice state. |
.../voice-interview/voice-session-state.test.ts |
Tests the new projected state. |
.../voice-interview/voice-interview-control.test.tsx |
Tests admission, setup, microphone, and controls. |
.../voice-interview/voice-browser-tools.integration.test.tsx |
Exercises the combined Voice/browser-tool lifecycle. |
.../voice-interview/canonical-speech.ts |
Selects exact canonical response and question speech. |
.../voice-interview/canonical-speech.test.ts |
Tests exact marker-based question selection. |
.../local-storage-demo/use-flue-chat-history.ts |
Applies hidden-tool and history projection rules. |
.../local-storage-demo/use-flue-chat-history.test.ts |
Tests persisted Voice origins across reopen. |
.../local-storage-demo/local-storage-demo-app.tsx |
Wires Voice lifecycle tracking and removes brunch_ask. |
.../local-storage-demo/local-storage-demo-app.test.tsx |
Tests production Voice registration and durable Stop. |
.../local-storage-demo/brunch-panel-transport.ts |
Tracks response lifecycle and admission failures. |
.../local-storage-demo/brunch-panel-transport.test.ts |
Tests tracker events and failure propagation. |
apps/petrinaut-website/README.md |
Documents current Voice behavior and limitations. |
apps/brunch-agent/test/petrinaut-chat.test.ts |
Verifies marker persistence and hiding. |
apps/brunch-agent/test/petrinaut-chat.integration.ts |
Extends the real Flue integration scenario. |
apps/brunch-agent/test/petrinaut-chat-result.ts |
Extends integration result types. |
apps/brunch-agent/test/local-dev-origins.test.ts |
Verifies launcher dependencies and API plugins. |
apps/brunch-agent/test/architecture/boundaries.integration.ts |
Registers the new package boundary. |
apps/brunch-agent/petrinaut-local.vite.config.ts |
Preserves website plugins while merging local config. |
.changeset/flue-voice-safety.md |
Records the Petrinaut patch release. |
Review details
- Files reviewed: 79/79 changed files
- Comments generated: 0
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Merging this PR will not alter performance
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
as_constant |
< 1 ns | < 1 ns | N/A | |
constant_equal |
< 1 ns | < 1 ns | N/A | |
constant_not_equal |
< 1 ns | < 1 ns | N/A | |
access |
< 1 ns | < 1 ns | N/A | |
runtime_equal |
< 1 ns | < 1 ns | N/A | |
runtime_not_equal |
< 1 ns | < 1 ns | N/A |
Comparing ln/fe-1580-reconcile-voice-resumable-workpiece (a0ca1cb) with main (9c21578)1
Footnotes
2db4790 to
132831f
Compare
976bb1c to
ce0ac40
Compare
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Import 58f7584..be56a18 from PR #9531, preserving its original branch and evidence. Retain the repaired fixture catalogue, input mapping, admission namespaces and parent continuation/Stop behavior. Necessary join resolutions are recorded in the Mission 6b import evidence. Co-authored-by: Kostandin Angjellari <kostandinangjellari@gmail.com>
355edd8 to
a0ca1cb
Compare
386ff76 to
275d1f3
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a0ca1cb. Configure here.
| this.#handleConnectionFailure("invalid-response", "connection"); | ||
| return; | ||
| } | ||
| this.#emit({ |
There was a problem hiding this comment.
Silent TTS completion sticks the microphone
Medium Severity
A completed canonical response stays in #authorizedResponseIds until output_audio_buffer.stopped or output_audio_buffer.cleared. #syncMicrophoneTrack treats any remaining authorized id as assistant-owned, so a completed TTS that never emits those buffer events leaves capture closed and the turn appearing still in progress.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit a0ca1cb. Configure here.
| if ( | ||
| functionCalls.length > 1 || | ||
| (functionCalls.length > 0 && this.#canonicalResponseIds.has(responseId)) | ||
| ) { |
There was a problem hiding this comment.
Late output after cancel drops session
Medium Severity
#handleResponseDone deletes a cancelled response from #cancelledCanonicalResponseIds as soon as the terminal event arrives. #handleOutputBufferEvent still uses that set to ignore leftover output_audio_buffer.started events, so a started event after the cancelled response.done looks unauthorized and tears down the Voice connection.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit a0ca1cb. Configure here.
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1002 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1527 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 108 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 27 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |



🌟 What is the purpose of this PR?
Make Voice's completed-transcript, half-duplex experience work on the same canonical Brunch conversation that Mission 6 already uses for a resumable workpiece and browser mutation. In the local Petrinaut Brunch panel you can type, then speak, let Brunch change the prepared net, use Your turn or Stop, and reopen the same conversation without replaying speech or duplicating the change.
This imports KA's Voice hardening onto Mission 6's repaired fixture path and reconciles the joins that neither parent owned: deferred browser-tool execution, continuation ownership, Stop withholding, canonical stopped-entry projection, and surviving Voice origins after history fold.
What the accepted proof establishes: scoped automated suites plus the 2026-09-07 owner witness cover half-duplex admission, a causally necessary spoken browser mutation, causal per-step client results, Your turn, coherent Tab-B resume, active-submission durable Stop, Tab-C stopped-entry recovery, playback controls, and no duplicate/autoplay. It does not claim direct spoken-user attribution after hydration, durable recovery of locally withheld work after a settled tool-call step, comparative latency, or remote deployment.
🔗 Related links
libs/@hashintel/brunch-agent/MISSION.mddocs/evidence/implementations/voice-resumable-reconciliation/verification.mddocs/evidence/implementations/voice-resumable-reconciliation/import.md🚫 Blocked by
🔍 What does this change?
useChat/ Flue browser transport as the only Voice admission door; Realtime has no tools and cannot submit.🏗️ Agent notes
Mission authority is
libs/@hashintel/brunch-agent/MISSION.md(accepted with explicit limitations on 2026-09-07). This is the explicit exception to one new issue per mission: the owner directed a PR that references FE-1580 without rewriting that issue.Imperative. Make KA's completed-transcript, half-duplex Voice experience work safely over Mission 6's resumable browser mutations and coherent workpiece/document recovery. Preserve both capabilities. Distinguish committed prose, submission settlement, pending browser work, coherent document settlement, and terminal provider output at the shared boundaries.
Throughline. Completed current-turn microphone transcript → shared panel
submitVoiceInputWithAdmission/useChat→ browserChatTransportover the memoized Flue client → same-origin/agents/chat/:instanceIdand mounted BrunchChatAgent→ committed canonical prose, hidden server question marker, browser-tool requests → existing canonical browser validation and effects → original call-id outputs resume the same conversation → canonical speech queue and acknowledged cancellation → coherent workpiece/document settlement → canonical history reopen and another real turn.Proof. The accepted owner witness used the real microphone and prepared fixture. An
SDCPNnegative control performed one read and no mutation; an explicit spoken confirmation then produced oneaddArc, a separate verification read, two model workpiece revisions, coherent revision 2, one audible reply, Tab-B continuation, canonical durable abort and Tab-C stopped-entry recovery. The witness first exposed cumulative cross-step client results and a non-causal prepared answer; both received failing-first regressions and repairs. Direct spoken-user hydration attribution, post-settlement durable withholding and comparative latency were explicitly deferred with no claim.Constraints. One conversation/log and the mounted Flue route; no direct Voice send or live
brunch_ask. Realtime hastool_choice: noneandcreate_response: false. Only durably completed, submission-correlated canonical assistant prose may speak. Local playback cancellation, local withheld browser work, and durable Flue abortion stay distinct. Preserve Mission 6 fixture identity, scoped catalogue, no-op honesty, and prior-coherent-bundle refusal. KA's original branch/PR stay untouched.Fog-line. Question-marker compliance remains a model limitation. Direct-user attribution after hydration and comparative latency are explicitly deferred. Stop is durable for active Flue submissions; locally withheld browser work after a settled tool-call step may reappear as pending after reopen. The negative-control answer was too verbose and durable Stop was poorly discoverable while Voice remained active.
Stop or reorient. Stop if another conversation route appears, admission auto-retries, speech is rewritten, Stop lets withheld work execute, failures disappear, or coherent settlement is falsely reported. Reorient rather than invent a durable marker, forge aborted settlements, or disable ordinary pending-tool recovery to paper over the local-Stop/reopen gap.
Deferred. Mission 7 consumes this local reconciliation and must re-pin the prompt/tool baseline before paid runs; it cannot inherit acceptance. Retirement of KA's original PR needs separate authorization.
Implementation record
58f7584080..be56a18ff0, imported with attribution; seeimport.md.docs/evidence/implementations/voice-resumable-reconciliation/main-restack.md.already-settled. The panel can withhold pending browser work locally, but a fresh process cannot infer that withholding from the snapshot, so reopen can recover the tool as pending work. User docs warn about this. It is an owner reorientation point, not a solved projection case.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
@hashintel/petrinaut:.changeset/flue-voice-safety.md; the@hashintel/brunch-agent*packages are private)📜 Does this require a change to the docs?
The changes in this PR:
libs/@hashintel/petrinaut/docs/ai-assistant.md,apps/petrinaut-website/README.md, and the Brunch reconciliation evidence)🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
dev:brunchnow also builds@apps/brunch-agentfirst; noturbo.jsonchange)metadata.stoppedand are skipped; this local-withholding case is not the same thing.🐾 Next steps
🛡 What tests cover this?
voice-browser-tools.integration.test.tsx.❓ How to test this?
yarn dev:brunch.http://127.0.0.1:4915/?brunch-fixture=crew-reservation-v1.This local demo and its acceptance gates, not merely green unit tests, define the visible advance.
📹 Demo
The accepted owner witness records the real microphone/browser path, canonical summary, screenshots, explicit deferrals, and evidence-bundle limitation. The complete pre-registered telemetry/network/latency bundle was not retained and is not inferred.