Skip to content

Media overlays: opt-in "Enable interactive links" (links stay clickable while the readaloud is playing or paused) - #3908

Open
parallel-systems11 wants to merge 4 commits into
edrlab:developfrom
parallel-systems11:feature/media-overlays-interactive-links
Open

parallel-systems11 wants to merge 4 commits into
edrlab:developfrom
parallel-systems11:feature/media-overlays-interactive-links

Conversation

@parallel-systems11

@parallel-systems11 parallel-systems11 commented Sep 25, 2026 •

Copy link
Copy Markdown

What

A new readaloud setting, Enable interactive links (mediaOverlaysInteractiveLinks, default off), shown beside "Disable continuous play" when a publication has media overlays.

Two behaviours, both only when the setting is on:

  1. Links stay clickable. While media overlays are playing or paused, the webview's document click handler currently swallows every click (preventDefault + stopPropagation), links included. With this setting on, a click whose target sits inside an a[href] is let through instead. Nothing else is needed: the navigator already interrupts the readaloud (staying active) when it loads the link target, and once the new document reports its reading location the readaloud resumes there, exactly as it does today when continuous play crosses into the next spine item.
  2. The readaloud waits at the end of a document. When the last text/audio pair of a document ends, mediaOverlaysNext() currently stops (staying active) and turns to the next spine item. With this setting on it pauses instead, because in an interactive book the end of a chapter is where the reader chooses, and the next spine item in reading order is not that choice.

Everything else is unchanged, and with the setting off nothing changes at all.

Plus two general fixes, each in its own commit (not tied to the setting; happy to split them into separate PRs if you prefer):

  • A reading-location report that lands inside the text/audio pair already playing or paused no longer restarts that pair. The document re-scrolls after a window resize, a font size change or a layout switch and reports its location each time; today mediaOverlaysHandleIpcMessage answers every such report with mediaOverlaysInterrupt() + playMediaOverlaysForLink(), so the current pair starts over (and a paused pair starts playing). With sentence-sized pairs that is a few seconds; with paragraph or chapter-sized pairs it is the whole passage. The built-in TTS is not affected by these reports, so this brings pre-recorded narration in line with it. A report that lands in a different pair behaves exactly as before.
  • The highlight keeps in step with the audio at fine granularities. The step from one pair to the next ran only on the browser's timeupdate event (about four times a second) and moved along by one pair per event, so with a pair per word the highlight fell behind on every word shorter than a tick, and further at faster playback rates. While the audio plays, the same check now runs on every animation frame, so the highlight lands on the pair the clock is in within a frame at any rate. One step per pair, guarded by the pair's end; the timeupdate/ended handlers stay for the end of the audio.

Why

Interactive books (branching stories, "choose your path" adventures) are EPUBs whose choices are links at the end of each scene, and whose narration is a media overlay per scene. With "Disable continuous play" the readaloud pauses at the choices as intended, but the reader then has to press Stop before any choice can be clicked, and press Play again on the next scene, on every scene. With this setting on the loop becomes: listen, click a choice, listen.

How

Mirrors the existing ttsClickEnable plumbing end to end:

  • src/common/models/reader.ts, src/common/redux/states/reader.ts: mediaOverlaysInteractiveLinks: boolean (default false).
  • src/renderer/reader/components/ReadingAudio.tsx: the checkbox, pushed only when useMO.
  • src/renderer/reader/components/Reader.tsx: applies the flag once the navigator is up, beside ttsAndMediaOverlaysManualPlayNext (the ttsOverlayEnableNeedsSync block).
  • src/renderer/reader/redux/sagas/readerConfig.ts: re-applies it when the setting changes, like the other readaloud flags.
  • r2-navigator-js/electron/common/events.ts: R2_EVENT_MEDIA_OVERLAY_INTERACTIVE_LINKS + payload.
  • r2-navigator-js/electron/renderer/media-overlays.ts: mediaOverlaysEnableInteractiveLinks(doEnable) stores the flag on the host window and sends it to the active webviews; mediaOverlaysNext() pauses at the end of a document when the flag is set.
  • r2-navigator-js/electron/renderer/dom.ts: host default, and the flag is re-sent to each webview on dom-ready like the TTS flags.
  • r2-navigator-js/electron/renderer/webview/state.ts, preload.ts: the webview keeps the flag; the click handler lets a link click through when it is set (no other event is sent).
  • src/resources/locales/en.json + the generated typings: the two strings.

Tested

Windows, built from the v3.5.1 source, with a media-overlay EPUB (one overlay per chapter, the chapter's choices as links, 124 chapters).

  • Setting on, "Disable continuous play" off: play, the readaloud pauses at the end of the chapter, click a choice, the target chapter opens and reads on its own. Repeated across many chapters.
  • Setting on, "Disable continuous play" on: same loop (the pause then comes from the existing setting).
  • Setting off: the click handler takes the stock path (clicks swallowed while playing or paused).
  • The same-pair fix: window resize, full screen on and off, repeatedly, during playback: the narration keeps going without a restart. Before the fix the chapter started over on every resize.

🤖 Generated with Claude Code

@parallel-systems11
parallel-systems11 force-pushed the feature/media-overlays-interactive-links branch from e2910b4 to bfcb6d0 Compare September 25, 2026 03:21
@parallel-systems11

Copy link
Copy Markdown
Author

Pushed one refinement to the "wait at the end of a document" half of the setting: it now waits only at the end of a document whose overlay is labelled as body matter (or not labelled at all), and flows on from front and back matter — a title page, a preamble, a cast page — where there is no choice to wait for. The label is the SMIL epub:type (seq/body), which the parser already puts on the node's Role: anything in {cover, titlepage, frontmatter, backmatter, preamble, toc, landmarks, colophon, dedication, epigraph, acknowledgments, copyright-page, imprint, foreword, preface, afterword, appendix, bibliography, glossary, index} with nothing in {bodymatter, chapter, part, volume, prologue, epilogue} flows on. Tested on Windows with the same media-overlay EPUB: the title page's narration plays through to How to Read on its own, chapters still wait at their choices.

@parallel-systems11

Copy link
Copy Markdown
Author

Pushed a fourth commit: with a pair per word (the same EPUB rebuilt at word granularity, ~16,600 pairs) the highlight fell behind the audio, one pair per timeupdate tick, and further at 2x. The step now also runs on every animation frame while the audio plays, guarded by the current pair's end, so the highlight sits on the spoken word at any playback rate. Tested on Windows at 1x and 2x. The description above is updated to list it.

ps and others added 4 commits September 25, 2026 15:28
A new readaloud setting, shown beside "Disable continuous play" when a
publication has media overlays (default off). With it on:

- links in the text stay clickable while media overlays are playing or
  paused: the webview's document click handler lets a click inside an
  a[href] through instead of swallowing it. The navigator already
  interrupts the readaloud (staying active) when it loads the link target
  and resumes it once the new document reports its reading location;
- the readaloud waits at the end of a document instead of turning to the
  next spine item, because in an interactive book the end of a chapter is
  where the reader chooses, and the next spine item in reading order is
  not that choice.

For interactive books (branching stories) whose choices are links at the
end of each scene and whose narration is one media overlay per scene.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e the current pair

The document re-scrolls after a window resize, a font size change or a
layout switch, and reports its reading location each time. The host
answered every report with an interrupt and a fresh play of the located
pair, so the pair already playing started over, and a paused pair started
playing. When the located pair is the current one, do nothing; a report
that lands in a different pair behaves as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…apters wait

The end-of-document wait applies to body matter only. A document whose
overlay is labelled front or back matter (the SMIL epub:type on seq/body,
parsed into the node's Role) and nowhere labelled body has no choice to
wait for, so the readaloud turns the page as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nularities

The step from one text/audio pair to the next ran only on the browser's
"timeupdate" event, about four times a second, and moved along by one pair
per event. With a pair per word the highlight fell behind on every word
shorter than a tick, and further at faster playback rates. While the audio
plays, the same check now runs on every animation frame, so the highlight
lands on the pair the clock is in within a frame at any rate. One step per
pair, guarded by the pair's end; the "timeupdate"/"ended" handlers stay
for the end of the audio.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@parallel-systems11
parallel-systems11 force-pushed the feature/media-overlays-interactive-links branch from c80b38b to e970178 Compare September 25, 2026 21:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant