Repository navigation
feat: prepare initial publications before device capture (CLT-3610) - #2292
xianshijing-lk wants to merge 1 commit into
Conversation
|
This PR bumps
|
size-limit report 📦
|
| await negotiate(); | ||
| } | ||
|
|
||
| prepared?.assertActive(); |
There was a problem hiding this comment.
🔴 Cancelled fallback publication leaves media sending
During fallback negotiation, cancel() completes without removing the ordinary sender. The later assertActive() rejects attachment before registering its publication. Media keeps sending without a publication that normal unpublication can find.
Learn more
Fallback attachment uses the ordinary AddTrack and sender-negotiation path. Cancellation only reconciles the reserved sender in removeInitial, which returns immediately when no reservation exists. Once ordinary negotiation finishes, this cancellation check throws before the publication enters the participant's maps. Neither the failure handler in attach nor cancellation removes that ordinary sender, so the negotiated track remains live but untracked.
Example: Connect with singlePeerConnection: false, attach a microphone, and cancel the preparation while its AddTrack response or SDP answer is pending. After both responses arrive, attachment rejects, but the ordinary sender still carries the microphone and no local publication exists for it.
Recommended fix: Give the preparation ownership of fallback publication until commit too. On cancellation or failure, remove any ordinary sender created by that attempt, cancel its pending registration, detach publication listeners, and reconcile negotiation without stopping caller-owned media. Add a deferred-negotiation regression test that cancels attachment and verifies both sender removal and absence of a published track.
Was this helpful? React with 👍 or 👎 to provide feedback.
| const transceiver = await this.pcManager!.addPublisherTransceiverOfKind( | ||
| plan.kind === Track.Kind.Audio ? 'audio' : 'video', | ||
| { | ||
| direction: 'sendonly', | ||
| streams: [stream], | ||
| sendEncodings: plan.encodings, | ||
| }, | ||
| ); |
There was a problem hiding this comment.
🔴 Prepared microphones lose negotiated encoder settings
Before capture, sender.track stays null for reserved microphones. placeholderMidsFromTransceivers therefore misclassifies them, and applyRemoteDescription replaces their Opus parameters with receive-section defaults. Requested stereo and DTX settings disappear before attachment.
Learn more
The first answer arrives before capture can attach media. applyInitialPublisherLayout creates receive-only sections before the reserved send sections. After setting the initial local description, applyRemoteDescription classifies sections using placeholderMidsFromTransceivers, which tests only whether the sender has a track. Reserved senders therefore enter the placeholder set alongside receive-only sections. With every audio section classified as a placeholder, conformBundledCodecFmtp selects the first receive section's parameters and overwrites the reserved microphone's server-provided encoder parameters. Attaching media does not renegotiate those parameters.
Example: Prepare a stereo microphone and capture only after connection. An answer with default Opus parameters on the earlier receive section and stereo=1;usedtx=1 on the microphone section loses those microphone parameters during answer processing.
Recommended fix: Distinguish reserved outgoing sections from receive placeholders even while their senders have no track. Preserve that distinction in both offer and answer conformance, and test an initial answer containing receive-only placeholders plus a null-track send-only microphone with distinct Opus parameters.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if ( | ||
| !['vp8', 'h264'].includes(opts.videoCodec) || | ||
| opts.scalabilityMode || | ||
| (typeof opts.backupCodec === 'object' && opts.backupCodec.codec !== opts.videoCodec) |
There was a problem hiding this comment.
🟡 Default video backup codec silently omitted
With H264 and backupCodec: true, prepareInitial accepts a configuration requiring a VP8 backup. Ordinary prepare expands that shorthand, but reserved attachment skips it. The publication advertises only H264 and loses its VP8 fallback.
| (typeof opts.backupCodec === 'object' && opts.backupCodec.codec !== opts.videoCodec) | |
| (opts.backupCodec === true && opts.videoCodec !== defaultVideoCodec) || | |
| (typeof opts.backupCodec === 'object' && opts.backupCodec.codec !== opts.videoCodec) |
Was this helpful? React with 👍 or 👎 to provide feedback.
| const opts: TrackPublishOptions = { | ||
| ...this.roomOptions.publishDefaults, | ||
| ...specification.publishOptions, | ||
| source: specification.source, | ||
| }; |
There was a problem hiding this comment.
🟡 Caller mutations change prepared video encodings
Changing videoEncoding.maxBitrate after preparation alters non-simulcast encodings when format.frameRate is absent. computeVideoEncodings retains that object, but layer metadata and start bitrate were already computed. JOIN then uses inconsistent publication settings.
Learn more
The prepared-publication rule requires publication intent to be snapshotted. These spreads retain nested objects from the caller and room defaults. In the non-simulcast case, computeVideoEncodings returns videoEncoding itself, and prepareInitial leaves that reference intact when no format frame rate is supplied. Layer metadata and start bitrate are calculated immediately, but join consumes the encoding object later.
Example: Prepare a 1280×720 camera with simulcast: false, no format.frameRate, and a reused videoEncoding containing maxBitrate: 500000. Set that object's bitrate to 1500000 before connecting. The sender reserves 1.5 Mbps while its prepared layer bitrate and start-bitrate target still describe 500 kbps.
Recommended fix: Copy nested publication settings before deriving the plan, including encoding objects, presets, backup settings, and layer arrays. Ensure the stored encodings cannot alias caller-owned objects. Add a regression test that mutates the original options between preparation and connection and verifies unchanged encoding and layer metadata.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
|
||
| ### Initial prepared publications | ||
|
|
||
| `Room.preparePublication()` snapshots device-free intent; the V1 offer builder derives each |
There was a problem hiding this comment.
in our last discussion it sounded like you didn't want to expose an API like preparePublications directly to users because it might be confusing.
I agree with this concern, can you elaborate on why you chose to expose it as a user facing API now?
What and why
CLT-3610. Session startup should be able to join and dispatch an agent while microphone/camera permission and device initialization run independently. Currently, a late capture still needs standalone track registration and SDP negotiation after JOIN.
Add experimental, additive SDK APIs for declaring publication intent without accessing a device:
The V1 JOIN offer reserves trackless senders and carries AddTrack requests, using their actual SDP MSID identities. Response waiters exist before signaling opens. Attachment reuses normal publication finalization and events, replacing the negotiated sender's track. Shared SDP processing preserves codec consistency and the connection's video start-bitrate policy.
Reservations are room-owned until commit or cancellation. Disconnect/disposal, retries, connection changes, concurrent attachment, and cancellation reconcile pending state. Preconnect buffering starts when an early microphone is supplied. Early subscription notifications survive device permission delays, and buffer listeners are installed before publication notification replay and removed on subscription/unpublication.
Single-PC V1 offer-with-JOIN is the optimized path. Dual-PC/V0, Firefox, React Native, E2EE, frame metadata, SVC/additional-codec configurations, and unsupported capabilities defer to ordinary publication. Actual format or server codec changes also reconcile and fall back. Existing publish APIs remain unchanged. Companion: livekit/components-js#1523 feature-detects these APIs; its public Session API stays unchanged.
Validation
check:breaks, core imports/types, published consumer declarations, Throws checks, build, formatting, lint, compatibility and publint completed. Lint/compatibility/packaging report warnings, with no errors.Example single-PC run with 100 ms injected signaling RTT, measuring publication completion after capture:
The JOIN finished before delayed capture in every matrix case. This removes the separate registration/negotiation exchanges; it is not a promise of exactly two RTTs of elapsed improvement because ordinary publication already overlaps them. Real media arrival is measured separately by the harness. Cross-browser device testing beyond Chrome remains to be done.
Reproduction and lifecycle contract:
examples/prepared-publications/README.md; real-SFU harness:scripts/benchmark-prepared-publication.mjs(also supportsPREPARED_EXPECT_CODEC_FALLBACK=1).Session UX benchmark
A separate feature A/B measures the actual React Session hook and UI with a real SFU. Across 100 measured runs (ten per variant per scenario, plus ten excluded warmups), the user's signaling and ICE/DTLS/media pass through latency relays. The warm fixture agent connects directly to the SFU. Both variants use the same SDK build, isolating the old versus prepared Session flow.
P50 seconds from the Connect click, existing → prepared, at 150 ms network RTT:
The first response proxy improves 1.265 → 1.138 s in the 800 ms case and 2.472 → 2.310 s in the two-second case. Fast capture and one-second credentials show little meaningful median gain: normal publication overlaps transport setup. The credential stage remains 1.004 s in both variants.
The main perceived benefit is an early welcome while device access is pending. JOIN publisher intent can make the remote participant visible before capture, interacting with the SFU's subscriber-only update batching. This greeting gain is not purely a two-RTT saving and depends on the agent's greeting policy. The existing
session.isConnectedstate can render Connected early; an app awaiting all ofstart()still waits for its devices.Full report, reproduction, p95s and all 100 raw samples. Audio measurements use real decoded waveforms as a playout proxy; these are controlled Chrome/warm-agent results, with no production authentication, page-load, agent cold-start, STT/LLM/TTS or physical-speaker measurement.
Checklist