Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
091ddd6
Recut Voice interruption mission from deployed main
kostandinang Sep 9, 2026
ecf0027
Allow Voice interruption by speaking
kostandinang Sep 6, 2026
3df3a38
Reject false Voice interruption transcripts before admission
kostandinang Sep 6, 2026
2ad49c4
Fix Voice interruption retention and VAD eagerness
kostandinang Sep 7, 2026
4227825
Preserve retained answers after transcript failures
kostandinang Sep 8, 2026
97d6f7c
Adapt Voice transcription import to NodeNext
kostandinang Sep 9, 2026
6cc0240
Add durable Voice continuity proof
kostandinang Sep 9, 2026
f0e9fcc
Record direct Voice provenance disposition
kostandinang Sep 9, 2026
a888f6c
Align continuity proof with supported provenance
kostandinang Sep 9, 2026
268303c
Fix Voice interruption review blockers
kostandinang Sep 9, 2026
a16c19e
Release interrupted Voice output ownership
kostandinang Sep 9, 2026
301ceb1
Merge remote-tracking branch 'origin/main' into kostandin/fe-1604-rec…
kostandinang Sep 9, 2026
863ee19
Fix Voice checks and explicit handoff
kostandinang Sep 9, 2026
bc9fc13
Correct Voice continuity proof scope
kostandinang Sep 9, 2026
a1940d8
Address Voice interruption review blockers
kostandinang Sep 9, 2026
5db7859
Preserve Voice input after speech stops
kostandinang Sep 10, 2026
940bba4
Merge remote-tracking branch 'origin/main' into kostandin/fe-1604-rec…
kostandinang Sep 10, 2026
0d2dbd9
Preserve active input across voice turn races
kostandinang Sep 10, 2026
4acd4d6
Record FE-1604 continuity witness deferral
kostandinang Sep 10, 2026
129ff8a
Preserve voice input ordering across transcript races
kostandinang Sep 10, 2026
b99a98c
Settle voice input boundaries across cancellation races
kostandinang Sep 10, 2026
556e5f1
Fix voice lifecycle race conditions
kostandinang Sep 10, 2026
c3272da
Settle voice response collision races
kostandinang Sep 10, 2026
350f717
Merge remote-tracking branch 'origin/main' into kostandin/fe-1604-rec…
kostandinang Sep 10, 2026
62a41b1
Drain retained voice answers after empty replies
kostandinang Sep 10, 2026
4e38d31
Merge remote-tracking branch 'origin/main' into kostandin/fe-1604-rec…
kostandinang Sep 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/petrinaut-voice-interruption.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@hashintel/petrinaut": patch
---

Allow Voice users to interrupt assistant playback by speaking, with a browser-saved preference and optional manual handoff.
30 changes: 23 additions & 7 deletions apps/petrinaut-website/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,8 @@ The text composer remains available. Sending typed text ends Voice mode first,
then submits the draft exactly once through the same conversation; a failed
handoff restores the draft. Closing the assistant pauses capture and speech
before hiding it. Reopening preserves the mounted session in **Paused** state.
The dock exposes **Your turn** while canonical audio owns the turn. That action
With **Interruption by speaking** disabled, the dock exposes **Your turn** while
canonical audio owns the turn. That action
clears pending input and output, waits for the provider's matching
acknowledgements and response terminal event, and only then opens the
microphone for fresh capture. Its playback menu offers **Repeat question** and
Expand All @@ -160,7 +161,8 @@ action stays disabled rather than guessing from the final segment.
The browser sends its SDP offer to this app; the server initializes a trusted
`gpt-realtime-2` audio-input/audio-output session through OpenAI's unified
Realtime call endpoint. The provider key, model, instructions, language, and
vocabulary policy stay server-side. Realtime exposes no tools, uses
session configuration stay server-side; the transcription vocabulary is shared
with the browser's local admission filter. Realtime exposes no tools, uses
`tool_choice: "none"`, and configures semantic VAD to detect an input boundary
without creating a model response.

Expand All @@ -174,21 +176,35 @@ Petrinaut's shared composer path. Connection epoch, item id, and content index
form its stable identity. Duplicate, empty, failed, unavailable, and over-limit
transcripts never submit; recoverable failures leave a not-heard or too-long
notice in the dock. Provisional transcription remains display-only.
Only interruption-originated completions receive local prompt-regurgitation
and self-echo checks before admission or pending-answer retention. Comparison
uses NFKC, lowercase, punctuation removal, and whitespace collapse. Exact
normalized active-playback echoes are rejected at any length; fuzzy comparison
requires at least 80% ordered bigram overlap and minimum lengths of eight tokens
for the vocabulary prompt or six for active canonical playback. The playback
reference is captured when interruption starts, excludes queued speech and
history, and is released on completion or lifecycle cleanup. Rejections produce
only content-free diagnostics; they create no answer, error, or pending-answer
notice. Short novel answers remain valid and the admitted payload keeps its
original casing and punctuation.

The bridge waits for the correlated Brunch turn before returning canonical
speech segments to Realtime. It instructs Realtime to speak only those
segments. Generated audio is not a verbatim recording: canonical Brunch text
remains visible and authoritative. Voice is half-duplex: the physical
microphone is closed while the interviewer speaks, while Brunch is working, and
through cancellation. Audio captured before a **Your turn** handoff is
discarded and cannot become a later answer.
remains visible and authoritative. **Interruption by speaking** is enabled by
default: speech detection immediately cancels generation and clears output audio,
never the input buffer. The completed answer waits if Brunch is still busy.
False speech detection may still stop playback even if the transcript is later
discarded. Disable this browser-saved preference for half-duplex capture: the
microphone closes during assistant output, and audio captured before a completed
**Your turn** handoff cannot become a later answer.

The local Brunch preview reaches the mounted route through its same-origin, protocol-preserving proxy; this does not establish remote authentication or public ingress. Denying microphone permission leaves the text composer available and submits nothing to Brunch. When Voice mode cannot continue, the inline recovery state distinguishes microphone, connection, and other Voice failures, explains the next action, and offers **Reconnect** where appropriate. Sanitized error codes and diagnostic references remain collapsed under **Technical details**.

Realtime connection, transcription, and canonical speech timings use random
request IDs, and the existing Brunch transport provides its own request
correlation. Browser and server diagnostics report only operation, stage,
outcome, duration, request ID, and—where applicable—status or a sanitized error
outcome, duration, request ID, and—where applicable—status, rejection reason, or a sanitized error
code. Voice responses also expose privacy-safe `Server-Timing` metrics. These
diagnostics never record audio, SDP, transcript or prompt contents, canonical
speech text, credentials, or provider response bodies. Production Voice remains
Expand Down
Loading
Loading