Skip to content

feat: prepare initial publications before device capture (CLT-3610) - #2292

Open
xianshijing-lk wants to merge 1 commit into
mainfrom
sxian/CLT-3610/fast-track-publication-session-api
Open

xianshijing-lk wants to merge 1 commit into
mainfrom
sxian/CLT-3610/fast-track-publication-session-api

Conversation

@xianshijing-lk

@xianshijing-lk xianshijing-lk commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

What and why

CLT-3610. Session startup should be able to join and dispatch an agent while microphone/camera permission and device initialization run independently. Currently, a late capture still needs standalone track registration and SDP negotiation after JOIN.

Add experimental, additive SDK APIs for declaring publication intent without accessing a device:

const microphone = room.preparePublication({
  source: Track.Source.Microphone,
  format: { channelCount: 1 },
  publishOptions: { audioPreset: { maxBitrate: 32000 }, preConnectBuffer: true },
});
const camera = room.preparePublication({
  source: Track.Source.Camera,
  format: { width: 1280, height: 720, frameRate: 30 },
  publishOptions: { videoCodec: 'h264', simulcast: false },
});
const connecting = room.connect(url, token, { initialPublications: [microphone, camera] });
// Capture independently; attachTrack waits for connection if necessary.
const publication = await microphone.attachTrack(capturedMicrophone);
// Cancel unused handles when capture/startup fails. Successful attachment makes cancel a no-op.
await camera.cancel();
await connecting;

The V1 JOIN offer reserves trackless senders and carries AddTrack requests, using their actual SDP MSID identities. Response waiters exist before signaling opens. Attachment reuses normal publication finalization and events, replacing the negotiated sender's track. Shared SDP processing preserves codec consistency and the connection's video start-bitrate policy.

Reservations are room-owned until commit or cancellation. Disconnect/disposal, retries, connection changes, concurrent attachment, and cancellation reconcile pending state. Preconnect buffering starts when an early microphone is supplied. Early subscription notifications survive device permission delays, and buffer listeners are installed before publication notification replay and removed on subscription/unpublication.

Single-PC V1 offer-with-JOIN is the optimized path. Dual-PC/V0, Firefox, React Native, E2EE, frame metadata, SVC/additional-codec configurations, and unsupported capabilities defer to ordinary publication. Actual format or server codec changes also reconcile and fall back. Existing publish APIs remain unchanged. Companion: livekit/components-js#1523 feature-detects these APIs; its public Session API stays unchanged.

Validation

  • SDK suite: 1,171 passed, 1 skipped; public consumer tests, TypeScript, check:breaks, core imports/types, published consumer declarations, Throws checks, build, formatting, lint, compatibility and publint completed. Lint/compatibility/packaging report warnings, with no errors.
  • Real Chrome 158 + LiveKit server 1.13.10: microphone, simulcast camera, and concurrent audio/video, with/without preparations and with both PC modes. Subscriber audio samples/video decoding are verified, not only publication acknowledgments. Compatible attachment emits no AddTrack or offer; legacy fallback delivers media normally.
  • A second SFU restricted to Opus/H264 verifies VP8 reservations can join and then fall back to H264, including concurrent publication and both PC modes.
  • Actual components startup helper + agent peer: delayed capture allows JOIN first; early capture delivers a nonempty preconnect buffer and both live tracks. Cancellation and delayed capture cleanup also have regression tests.

Example single-PC run with 100 ms injected signaling RTT, measuring publication completion after capture:

Capture Ordinary Prepared
Microphone 146 ms 2 ms
Camera 154 ms 3 ms
Both concurrently 141 ms 4 ms

The JOIN finished before delayed capture in every matrix case. This removes the separate registration/negotiation exchanges; it is not a promise of exactly two RTTs of elapsed improvement because ordinary publication already overlaps them. Real media arrival is measured separately by the harness. Cross-browser device testing beyond Chrome remains to be done.

Reproduction and lifecycle contract: examples/prepared-publications/README.md; real-SFU harness: scripts/benchmark-prepared-publication.mjs (also supports PREPARED_EXPECT_CODEC_FALLBACK=1).

Session UX benchmark

A separate feature A/B measures the actual React Session hook and UI with a real SFU. Across 100 measured runs (ten per variant per scenario, plus ten excluded warmups), the user's signaling and ICE/DTLS/media pass through latency relays. The warm fixture agent connects directly to the SFU. Both variants use the same SDK build, isolating the old versus prepared Session flow.

P50 seconds from the Connect click, existing → prepared, at 150 ms network RTT:

Case Connected UI Microphone-ready UI First greeting audio ready
Fast capture 0.446 → 0.445 0.468 → 0.453 0.523 → 0.530
+800 ms capture 0.443 → 0.442 1.114 → 0.908 1.161 → 0.538
+2 s capture/permission 0.451 → 0.451 2.320 → 2.112 2.368 → 0.528

The first response proxy improves 1.265 → 1.138 s in the 800 ms case and 2.472 → 2.310 s in the two-second case. Fast capture and one-second credentials show little meaningful median gain: normal publication overlaps transport setup. The credential stage remains 1.004 s in both variants.

The main perceived benefit is an early welcome while device access is pending. JOIN publisher intent can make the remote participant visible before capture, interacting with the SFU's subscriber-only update batching. This greeting gain is not purely a two-RTT saving and depends on the agent's greeting policy. The existing session.isConnected state can render Connected early; an app awaiting all of start() still waits for its devices.

Full report, reproduction, p95s and all 100 raw samples. Audio measurements use real decoded waveforms as a playout proxy; these are controlled Chrome/warm-agent results, with no production authentication, page-load, agent cold-start, STT/LLM/TTS or physical-speaker measurement.

Checklist

  • This PR only fixes typos or documentation and needs no accepted issue.
  • An AI tool wrote or helped write this change. Tool: Codex.

@github-actions

Copy link
Copy Markdown
Contributor

This PR bumps

  • livekit-client: minor

@github-actions

Copy link
Copy Markdown
Contributor

size-limit report 📦

Path Size
dist/livekit-client.esm.mjs 120.97 KB (+2.16% 🔺)
dist/livekit-client.umd.js 130.55 KB (+2.05% 🔺)
dist/core/core.mjs 82.25 KB (+3.05% 🔺)

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 4 potential issues.

Devin Review

await negotiate();
}

prepared?.assertActive();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Cancelled fallback publication leaves media sending

During fallback negotiation, cancel() completes without removing the ordinary sender. The later assertActive() rejects attachment before registering its publication. Media keeps sending without a publication that normal unpublication can find.

Learn more

Fallback attachment uses the ordinary AddTrack and sender-negotiation path. Cancellation only reconciles the reserved sender in removeInitial, which returns immediately when no reservation exists. Once ordinary negotiation finishes, this cancellation check throws before the publication enters the participant's maps. Neither the failure handler in attach nor cancellation removes that ordinary sender, so the negotiated track remains live but untracked.

Example: Connect with singlePeerConnection: false, attach a microphone, and cancel the preparation while its AddTrack response or SDP answer is pending. After both responses arrive, attachment rejects, but the ordinary sender still carries the microphone and no local publication exists for it.

Recommended fix: Give the preparation ownership of fallback publication until commit too. On cancellation or failure, remove any ordinary sender created by that attempt, cancel its pending registration, detach publication listeners, and reconcile negotiation without stopping caller-owned media. Add a deferred-negotiation regression test that cancels attachment and verifies both sender removal and absence of a published track.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/room/RTCEngine.ts
Comment on lines +423 to +430
const transceiver = await this.pcManager!.addPublisherTransceiverOfKind(
plan.kind === Track.Kind.Audio ? 'audio' : 'video',
{
direction: 'sendonly',
streams: [stream],
sendEncodings: plan.encodings,
},
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Prepared microphones lose negotiated encoder settings

Before capture, sender.track stays null for reserved microphones. placeholderMidsFromTransceivers therefore misclassifies them, and applyRemoteDescription replaces their Opus parameters with receive-section defaults. Requested stereo and DTX settings disappear before attachment.

Learn more

The first answer arrives before capture can attach media. applyInitialPublisherLayout creates receive-only sections before the reserved send sections. After setting the initial local description, applyRemoteDescription classifies sections using placeholderMidsFromTransceivers, which tests only whether the sender has a track. Reserved senders therefore enter the placeholder set alongside receive-only sections. With every audio section classified as a placeholder, conformBundledCodecFmtp selects the first receive section's parameters and overwrites the reserved microphone's server-provided encoder parameters. Attaching media does not renegotiate those parameters.

Example: Prepare a stereo microphone and capture only after connection. An answer with default Opus parameters on the earlier receive section and stereo=1;usedtx=1 on the microphone section loses those microphone parameters during answer processing.

Recommended fix: Distinguish reserved outgoing sections from receive placeholders even while their senders have no track. Preserve that distinction in both offer and answer conformance, and test an initial answer containing receive-only placeholders plus a null-track send-only microphone with distinct Opus parameters.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

if (
!['vp8', 'h264'].includes(opts.videoCodec) ||
opts.scalabilityMode ||
(typeof opts.backupCodec === 'object' && opts.backupCodec.codec !== opts.videoCodec)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Default video backup codec silently omitted

With H264 and backupCodec: true, prepareInitial accepts a configuration requiring a VP8 backup. Ordinary prepare expands that shorthand, but reserved attachment skips it. The publication advertises only H264 and loses its VP8 fallback.

Suggested change
(typeof opts.backupCodec === 'object' && opts.backupCodec.codec !== opts.videoCodec)
(opts.backupCodec === true && opts.videoCodec !== defaultVideoCodec) ||
(typeof opts.backupCodec === 'object' && opts.backupCodec.codec !== opts.videoCodec)

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +651 to +655
const opts: TrackPublishOptions = {
...this.roomOptions.publishDefaults,
...specification.publishOptions,
source: specification.source,
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Caller mutations change prepared video encodings

Changing videoEncoding.maxBitrate after preparation alters non-simulcast encodings when format.frameRate is absent. computeVideoEncodings retains that object, but layer metadata and start bitrate were already computed. JOIN then uses inconsistent publication settings.

Learn more

The prepared-publication rule requires publication intent to be snapshotted. These spreads retain nested objects from the caller and room defaults. In the non-simulcast case, computeVideoEncodings returns videoEncoding itself, and prepareInitial leaves that reference intact when no format frame rate is supplied. Layer metadata and start bitrate are calculated immediately, but join consumes the encoding object later.

Example: Prepare a 1280×720 camera with simulcast: false, no format.frameRate, and a reused videoEncoding containing maxBitrate: 500000. Set that object's bitrate to 1500000 before connecting. The sender reserves 1.5 Mbps while its prepared layer bitrate and start-bitrate target still describe 500 kbps.

Recommended fix: Copy nested publication settings before deriving the plan, including encoding objects, presets, backup settings, and layer arrays. Ensure the stored encodings cannot alias caller-owned objects. Add a regression test that mutates the original options between preparation and connection and verifies unchanged encoding and layer metadata.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread AGENTS.md

### Initial prepared publications

`Room.preparePublication()` snapshots device-free intent; the V1 offer builder derives each

@lukasIO lukasIO Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in our last discussion it sounded like you didn't want to expose an API like preparePublications directly to users because it might be confusing.
I agree with this concern, can you elaborate on why you chose to expose it as a user facing API now?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants