Bound video call rendering and recover dropped streams

[jlucaso1]

Summary

Fix sustained video-call resource growth and redundant GUI work on native and web, and consume the merged RTP orientation correction from oxidezap/whatsapp-rust#1463.

• Retire decoded-video atlas entries with scene ownership, including cached paint and shared views/windows.
• Cache the chat/settings/status body independently of call pictures, preserving input invalidation and deferred focus/blur handoff.
• Coalesce video readiness in a capacity-one lane with bounded fairness. Preserve the first picture when its enabling call state is still queued.
• Retire decoders and delayed publications per direction. Turning one camera off preserves the opposite reference chain.
• Route identity-bound, rate-limited keyframe requests over the originating connection after GUI decoder loss. Refresh both directions after daemon lag.
• Bound browser call output to one active readback and one replaceable pending frame, reuse JS buffers, reject stale output early, and convert unrotated pixels in place.
• Add GPUI atlas/render/focus tests, real H.264 lifecycle tests, and nine browser tests using production code and real VideoFrame.copyTo. Run browser readback tests in CI.

Evidence

The supplied production trace attributes 42.5% of the main-thread sampled timeline to copyTo. Main-thread task occupation rises from 75.9% to 91.3% between equal five-second windows. Sampling has substantial overhead; these are not direct per-function CPU measurements.

The atlas reproduction retained eight obsolete entries before the fix. Regression coverage now keeps only the current entry across 120 replacements. Sixty steady-state call notifications cause zero body rebuilds. Browser mutation checks detect broken pending replacement, buffer reuse, and stale-output rejection.

Removing one Rust pixel allocation and reusing one JS destination each avoids 70.31 MiB/s of allocation demand for a stable 720p20 stream. This is arithmetic, not a measured FPS or CPU improvement. No post-fix live call was recorded.

See docs/call-performance.md for measurements, limitations, and follow-up candidates. No private trace, capture artifact, or personal log data is included.

Dependencies And Protocol

• All WhatsApp crates remain on the same main source, locked to merged rotation fix 077cc3fccaba3e9752252dc998937b63ec63bbf5. The upstream real-WASM oracle ran without skipping.
• gpui-component uses the 0.5 compatibility branch at 970c6069a540390f874396d859930aa61f1ad76c containing the input invalidation fix proposed in input: Avoid notifying unchanged scroll and paint state longbridge/gpui-kit#2988. This avoids unrelated upstream 0.6 API changes.
• IPC protocol advances to 28 for directional keyframe requests. Deploy daemon and frontend together.

Verification

• Workspace formatting and Clippy, all targets/features with warnings denied, passed.
• Full workspace tests passed with 1,324 unit/integration tests and one doctest before the final state-ordering regression. The final GUI suite passed with 351 tests and seven existing ignored tests.
• Nightly wasm GUI/session checks passed with existing warnings.
• All nine browser readback tests passed in Chrome 151 with matching driver and wasm-bindgen runner 0.2.127.
• Locked all-target dependency resolution passed without unrelated lockfile-edge changes.

Remaining Gaps

Live browser-to-Android orientation, sustained call FPS/CPU, shared-memory browser runtime, and native GPU-memory behavior still need a production-like retest. The browser fixture tests readback and conversion, not hardware decoding or GPUI painting. AudioWorklet migration, capture timestamp preservation across drops, and slow transport/prefetch isolation are not part of this patch.

---

Summary by cubic

Bounds video call rendering so sustained calls stop accumulating memory and redundant GUI work, and recovers dropped streams by re-requesting keyframes instead of leaving the picture blank.

Bug Fixes

• Decoded-video atlas entries now retire when GPUI releases the scene that painted them, including cached paint and shared views.
• The chat/settings/status body caches independently of call pictures; sixty steady-state call notifications cause zero body rebuilds.
• Video readiness is coalesced into a capacity-one lane, and turning one camera off preserves the opposite direction's decoders.
• Browser call output is bounded to one active readback and one replaceable pending frame, reusing buffers and converting unrotated pixels in place.
• Dropped GUI decoders now send rate-limited keyframe requests over the originating connection for both call directions.
• Reusing one JS destination and dropping one Rust pixel allocation avoids 70.31 MiB/s of allocation demand on a stable 720p20 stream.
• Nine browser tests exercise production readback and conversion code and run in CI.
• Unused web decode code and the oxidezap-plugin workspace dependency are removed.

Migration

• IPC protocol advances to 28 for directional keyframe requests; deploy daemon and frontend together.
• gpui-component tracks the fix/input-scroll-notify-0.5 branch pending an upstream input-invalidation fix.
• whatsapp-rust is pinned to the merged RTP rotation fix.

^{Written for commit 2bfb971. Summary will update on new commits.}

Summary by CodeRabbit

• New Features

  • Video calls can now automatically request missing keyframes, helping recover from frozen or incomplete video.
  • Improved handling of local and remote video streams across calls and reconnects.
  • Video frames are rendered more reliably, including improved rotation and texture management.

• Bug Fixes

  • Reduced dropped video frames during busy sessions.
  • Prevented stale frames from appearing after calls or streams change.
  • Improved recovery from decoder errors and interrupted video.

• Tests

  • Added comprehensive browser-based WebCodecs and video recovery coverage.

• Documentation

  • Updated video performance findings and troubleshooting guidance.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-07T08:19:20.199964Z	2bfb971	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Team

Run ID: 14a273ae-94dd-4dff-b58b-98a0c275c654

📥 Commits

Reviewing files that changed from the base of the PR and between bf958fb and 2bfb971.

⛔ Files ignored due to path filters (2)

• Cargo.lock is excluded by !**/*.lock
• crates/gui/tests/webcodecs/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (43)

• .github/workflows/ci.yml
• AGENTS.md
• Cargo.toml
• crates/daemon/src/server/mod.rs
• crates/daemon/src/session_bridge/act.rs
• crates/gui/Cargo.toml
• crates/gui/src/app/body.rs
• crates/gui/src/app/calls_ctl.rs
• crates/gui/src/app/mod.rs
• crates/gui/src/components/call_card/video.rs
• crates/gui/src/components/media_viewer.rs
• crates/gui/src/components/message_bubble/media.rs
• crates/gui/src/components/status/view.rs
• crates/gui/src/platform/fonts.rs
• crates/gui/src/session/attach.rs
• crates/gui/src/session/embedded.rs
• crates/gui/src/session/frames.rs
• crates/gui/src/session/mod.rs
• crates/gui/src/session/native.rs
• crates/gui/src/session/recovery.rs
• crates/gui/src/session/sink/mod.rs
• crates/gui/src/session/sink/native.rs
• crates/gui/src/session/sink/web.rs
• crates/gui/src/session/tab.rs
• crates/gui/src/session/web.rs
• crates/gui/src/video/call_native.rs
• crates/gui/src/video/call_web.rs
• crates/gui/src/video/geometry.rs
• crates/gui/src/video/mod.rs
• crates/gui/src/video/recovery.rs
• crates/gui/src/video/texture.rs
• crates/gui/src/video/web.rs
• crates/gui/src/video/webcodecs.rs
• crates/gui/tests/webcodecs/Cargo.toml
• crates/gui/tests/webcodecs/README.md
• crates/gui/tests/webcodecs/fixture.js
• crates/gui/tests/webcodecs/lib.rs
• crates/gui/tests/webcodecs/readback.rs
• crates/ipc/src/protocol.rs
• crates/ipc/src/transport.rs
• crates/session/src/whatsapp/calls/registry.rs
• docs/call-performance.md
• docs/gotchas.md

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

This change adds directional video-keyframe recovery, per-stream decoder retirement, dedicated frame readiness delivery, cached GUI rendering, texture lifetime management, serialized WebCodecs readback, browser tests, and related CI and documentation updates.

Changes

Video recovery and frame delivery

Layer / File(s)	Summary
Directional keyframe recovery crates/ipc/*, crates/session/..., crates/daemon/..., crates/gui/src/session/*, crates/gui/src/video/recovery.rs	Adds directional keyframe requests, protocol version 28 handling, daemon forwarding, synchronous call lookup, bounded recovery sinks, and per-stream throttling.
Per-stream frame delivery and retirement crates/gui/src/session/frames.rs, crates/gui/src/session/sink/*, crates/gui/src/video/call_*.rs, crates/gui/src/app/calls_ctl.rs	Tracks local and remote streams independently, rejects stale generations, preserves frame readiness through a separate channel, and retires inactive decoders and pictures.

GUI and WebCodecs

Layer / File(s)	Summary
Texture-backed video rendering and cached app body crates/gui/src/video/texture.rs, crates/gui/src/app/*, crates/gui/src/components/*	Adds atlas texture reuse and deferred release. Caches the application body and limits rebuilds from call notifications.
WebCodecs readback and rotation handling crates/gui/src/video/webcodecs.rs, crates/gui/src/video/geometry.rs, crates/gui/tests/webcodecs/*	Serializes readback per decoder, replaces pending frames, reuses buffers, handles stale generations and failures, and adds browser coverage for lifecycle and pixel conversion behavior.
Build integration and behavioral documentation .github/workflows/ci.yml, Cargo.toml, AGENTS.md, docs/*, crates/gui/Cargo.toml	Adds the WebCodecs browser test to CI, updates development dependencies and workspace metadata, and records the revised video behavior and measurements.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: ⚪ Minimal · up to 2bfb9

No actionable merge-blocking risk is established for the current changes.

Poem

A rabbit watched the keyframes hop,
And kept stale pictures from the crop.
Textures slept when panes let go,
Readbacks queued in steady flow.
The browser tests now guard the gate,
While GUI frames arrive in state.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 62.99% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 154 functions across 35 files. (8 skipped…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes: limiting video call rendering and recovering dropped streams.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 62.99% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 154 functions across 35 files. (8 skipped: 8 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a07ab2f2f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".