Repository navigation
feat: prepare initial publications before device capture (CLT-3610) #2292
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
xianshijing-lk
wants to merge
1
commit into
main
Choose a base branch
from
sxian/CLT-3610/fast-track-publication-session-api
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| --- | ||
| 'livekit-client': minor | ||
| --- | ||
| Add experimental device-free publication preparation so initial microphone and camera senders can join with the room and attach captured media without another publication negotiation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| # Prepared publications (experimental) | ||
|
|
||
| Declare publication intent before requesting device access. The SDK snapshots publication defaults and options, chooses a codec and encoding envelope, and can register trackless senders in the initial JOIN offer. Opening a device never blocks JOIN. | ||
|
|
||
| ```ts | ||
| const microphone = room.preparePublication({ | ||
| source: Track.Source.Microphone, | ||
| format: { channelCount: 1 }, | ||
| publishOptions: { audioPreset: { maxBitrate: 32000 }, preConnectBuffer: true }, | ||
| }); | ||
| const camera = room.preparePublication({ | ||
| source: Track.Source.Camera, | ||
| format: { width: 1280, height: 720, frameRate: 30 }, | ||
| publishOptions: { | ||
| videoCodec: 'h264', | ||
| simulcast: false, | ||
| videoEncoding: { maxBitrate: 1_500_000 }, | ||
| }, | ||
| }); | ||
|
|
||
| const connecting = room.connect(url, token, { | ||
| initialPublications: [microphone, camera], | ||
| }); | ||
| const capturing = room.localParticipant.createTracks({ | ||
| audio: { channelCount: 1 }, | ||
| video: { resolution: { width: 1280, height: 720, frameRate: 30 } }, | ||
| }); | ||
|
|
||
| const publishing = capturing.then(async (tracks) => { | ||
| try { | ||
| // attachTrack waits for connection independently and starts the requested | ||
| // microphone preconnect buffer as soon as the track is supplied. | ||
| return await Promise.all( | ||
| tracks.map((track) => | ||
| (track.kind === Track.Kind.Audio ? microphone : camera).attachTrack(track), | ||
| ), | ||
| ); | ||
| } catch (error) { | ||
| // This also handles capture resolving after connection failed or was cancelled. | ||
| tracks.forEach((track) => track.stop()); | ||
| throw error; | ||
| } | ||
| }); | ||
| try { | ||
| await Promise.all([connecting, publishing]); | ||
| } catch (error) { | ||
| await room.disconnect(); | ||
| throw error; | ||
| } finally { | ||
| await Promise.all([microphone.cancel(), camera.cancel()]); | ||
| } | ||
| ``` | ||
|
|
||
| The handle belongs to its room and must appear once in that room's next `connect()` call. Preparation is synchronous and only allowed while disconnected. A handle accepts one track; it never opens devices or stops caller-owned tracks. Cancel an unused handle when capture fails or startup is abandoned. Disconnect and disposal cancel outstanding handles. Cancelling after successful attachment does nothing; ordinary publication and unpublication APIs own the track from then on. | ||
|
|
||
| For compatible capture, attachment reuses the negotiated sender with `replaceTrack` and returns a normal `LocalTrackPublication`. No new AddTrack request or offer is required. The JOIN carries codec/layer/encoding intent; mute status and capture-dependent audio processing features are reconciled when media attaches. No placeholder audio/video frames are generated. | ||
|
|
||
| The optimized path currently requires the SDK's V1 offer-with-JOIN path (single PC on supported non-Firefox browsers). Legacy dual PC, Firefox, React Native, unavailable codec capabilities, SVC/additional-codec configurations, E2EE, and frame-metadata configurations use normal publication after connection. A different actual capture format or server-selected codec also reconciles the reservation and falls back. This preserves join-first behavior, but those fallback cases do not save the publication exchanges. Screen sharing remains on its existing user-gesture capture path. | ||
|
|
||
| The exchange reduction and elapsed-time improvement are separate: bundling avoids the standalone registration/negotiation exchanges, while current ordinary fast publication already overlaps those exchanges. Do not assume every session improves by exactly two RTTs. Capture, ICE/DTLS, server support, media pacing, and the first decoded frame still affect perceived latency. | ||
|
|
||
| Run `pnpm build`, start `livekit-server --dev`, then run: | ||
|
|
||
| ```sh | ||
| LIVEKIT_URL=ws://127.0.0.1:7880 node scripts/benchmark-prepared-publication.mjs | ||
| ``` | ||
|
|
||
| The real-browser harness injects 50 ms each direction at SDK signal dispatch, delays capture until after JOIN, verifies subscriber audio samples/video decoding, and asserts no AddTrack/offer on compatible attachment. It covers prepared and ordinary publication with both PC modes, including concurrent audio/video publication. Set `CHROMIUM_EXECUTABLE_PATH` to use an installed Chrome. Keys default to the local dev server's `devkey`/`secret`; other servers require `LIVEKIT_API_KEY` and `LIVEKIT_API_SECRET`. | ||
|
|
||
| For codec substitution coverage, point the same harness at a server whose `room.enabled_codecs` contains only `audio/opus` and `video/h264`, and set `PREPARED_EXPECT_CODEC_FALLBACK=1`. Its VP8 reservations must join successfully, reconcile, and publish H264 to the subscriber through the ordinary path. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,234 @@ | ||
| // Run after `pnpm build` against a real SFU: LIVEKIT_URL=ws://localhost:7880 node scripts/benchmark-prepared-publication.mjs | ||
| // CHROMIUM_EXECUTABLE_PATH optionally selects an installed Chrome. Devices are browser test devices. | ||
| import { AccessToken } from 'livekit-server-sdk'; | ||
| import assert from 'node:assert/strict'; | ||
| import { readFile } from 'node:fs/promises'; | ||
| import { createServer } from 'node:http'; | ||
| import { chromium } from 'playwright'; | ||
|
|
||
| const url = process.env.LIVEKIT_URL ?? 'ws://127.0.0.1:7880'; | ||
| // For a server restricted to H264, verify VP8 reservations fall back instead of blocking JOIN. | ||
| const expectCodecFallback = process.env.PREPARED_EXPECT_CODEC_FALLBACK === '1'; | ||
| const apiKey = process.env.LIVEKIT_API_KEY ?? 'devkey'; | ||
| const apiSecret = process.env.LIVEKIT_API_SECRET ?? 'secret'; | ||
| const bundle = await readFile(new URL('../dist/livekit-client.esm.mjs', import.meta.url)); | ||
| const server = createServer((request, response) => { | ||
| response.setHeader('Content-Type', request.url === '/sdk.js' ? 'text/javascript' : 'text/html'); | ||
| response.end(request.url === '/sdk.js' ? bundle : '<!doctype html><body></body>'); | ||
| }); | ||
| await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); | ||
| let browser; | ||
| try { | ||
| browser = await chromium.launch({ | ||
| executablePath: process.env.CHROMIUM_EXECUTABLE_PATH, | ||
| headless: true, | ||
| args: [ | ||
| '--autoplay-policy=no-user-gesture-required', | ||
| '--use-fake-ui-for-media-stream', | ||
| '--use-fake-device-for-media-stream', | ||
| ], | ||
| }); | ||
| const rows = []; | ||
| for (const singlePeerConnection of [true, false]) { | ||
| for (const prepared of [false, true]) { | ||
| for (const kind of ['audio', 'video', 'both']) { | ||
| const roomName = `prepared-${Date.now()}-${Math.random().toString(36).slice(2)}`; | ||
| const tokens = await Promise.all( | ||
| ['publisher', 'subscriber'].map(async (identity) => { | ||
| const token = new AccessToken(apiKey, apiSecret, { identity }); | ||
| token.addGrant({ room: roomName, roomJoin: true }); | ||
| return token.toJwt(); | ||
| }), | ||
| ); | ||
| const page = await browser.newPage(); | ||
| try { | ||
| await page.goto(`http://127.0.0.1:${server.address().port}`); | ||
| const row = await page.evaluate( | ||
| async ({ url, tokens, singlePeerConnection, prepared, kind }) => { | ||
| const { Room, RoomEvent, Track, createLocalTracks } = await import('/sdk.js'); | ||
| const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); | ||
| const waitFor = async (predicate) => { | ||
| const deadline = performance.now() + 10000; | ||
| while (!(await predicate())) { | ||
| if (performance.now() > deadline) throw new Error('real media did not arrive'); | ||
| await delay(20); | ||
| } | ||
| }; | ||
| const subscriber = new Room({ singlePeerConnection }); | ||
| const publisher = new Room({ singlePeerConnection }); | ||
| const remotes = new Map(); | ||
| subscriber.on(RoomEvent.TrackSubscribed, (track) => { | ||
| remotes.set(track.kind, track); | ||
| const element = track.attach(); | ||
| element.muted = true; | ||
| document.body.appendChild(element); | ||
| void element.play(); | ||
| }); | ||
| const trace = []; | ||
| // Inject 50 ms each direction in the SDK signal dispatch only (100 ms RTT). | ||
| // RTP/ICE are undelayed; this isolates the publication signaling critical path. | ||
| const client = publisher.engine.client; | ||
| const send = client.sendRequest.bind(client); | ||
| const receive = client.handleSignalResponse.bind(client); | ||
| const pending = new Set(); | ||
| client.sendRequest = async (message, ...args) => { | ||
| trace.push(message.case); | ||
| await delay(50); | ||
| return send(message, ...args); | ||
| }; | ||
| client.handleSignalResponse = (message) => { | ||
| const task = delay(50).then(() => receive(message)); | ||
| pending.add(task); | ||
| void task.finally(() => pending.delete(task)); | ||
| }; | ||
| let tracks = []; | ||
| try { | ||
| await subscriber.connect(url, tokens[1]); | ||
| const sources = | ||
| kind === 'both' | ||
| ? [Track.Source.Microphone, Track.Source.Camera] | ||
| : [kind === 'video' ? Track.Source.Camera : Track.Source.Microphone]; | ||
| const publicationOptions = (source) => | ||
| source === Track.Source.Camera | ||
| ? { videoCodec: 'vp8', simulcast: true } | ||
| : { red: false }; | ||
| const reservations = new Map( | ||
| prepared | ||
| ? sources.map((source) => [ | ||
| source, | ||
| publisher.preparePublication({ | ||
| source, | ||
| ...(source === Track.Source.Camera | ||
| ? { format: { width: 640, height: 480, frameRate: 30 } } | ||
| : {}), | ||
| publishOptions: publicationOptions(source), | ||
| }), | ||
| ]) | ||
| : [], | ||
| ); | ||
| const started = performance.now(); | ||
| // Permission/device initialization completes late, while JOIN runs independently. | ||
| const capture = (async () => { | ||
| await delay(650); | ||
| return await createLocalTracks({ | ||
| audio: kind !== 'video' && { channelCount: 1 }, | ||
| video: kind !== 'audio' && { | ||
| resolution: { width: 640, height: 480, frameRate: 30 }, | ||
| }, | ||
| }); | ||
| })(); | ||
| const connecting = publisher.connect(url, tokens[0], { | ||
| initialPublications: [...reservations.values()], | ||
| }); | ||
| await connecting; | ||
| const connectedMs = performance.now() - started; | ||
| const initialVideoStartBitrate = publisher.engine.pcManager.publisher | ||
| .getLocalDescription() | ||
| ?.sdp.includes('x-google-start-bitrate='); | ||
| tracks = await capture; | ||
| const captureReadyMs = performance.now() - started; | ||
| const traceStart = trace.length; | ||
| await Promise.all( | ||
| tracks.map((track) => { | ||
| const source = | ||
| track.kind === Track.Kind.Video | ||
| ? Track.Source.Camera | ||
| : Track.Source.Microphone; | ||
| const reservation = reservations.get(source); | ||
| return reservation | ||
| ? reservation.attachTrack(track) | ||
| : publisher.localParticipant.publishTrack(track, publicationOptions(source)); | ||
| }), | ||
| ); | ||
| const publishedMs = performance.now() - started; | ||
| await waitFor(async () => { | ||
| if (remotes.size !== sources.length) return false; | ||
| const received = await Promise.all( | ||
| [...remotes.values()].map(async (remote) => { | ||
| if (!remote.receiver) return false; | ||
| const stats = await remote.receiver.getStats(); | ||
| return [...stats.values()].some( | ||
| (stat) => | ||
| stat.type === 'inbound-rtp' && | ||
| (remote.kind === Track.Kind.Video | ||
| ? stat.framesDecoded > 0 | ||
| : stat.totalSamplesReceived > 0), | ||
| ); | ||
| }), | ||
| ); | ||
| return received.every(Boolean); | ||
| }); | ||
| const mediaMs = performance.now() - started; | ||
| const publicationSignals = trace | ||
| .slice(traceStart) | ||
| .filter((type) => type === 'addTrack' || type === 'offer'); | ||
| return { | ||
| singlePeerConnection, | ||
| prepared, | ||
| kind, | ||
| connectedMs: Math.round(connectedMs), | ||
| captureReadyMs: Math.round(captureReadyMs), | ||
| publishAfterCaptureMs: Math.round(publishedMs - captureReadyMs), | ||
| mediaAfterCaptureMs: Math.round(mediaMs - captureReadyMs), | ||
| publicationSignals, | ||
| videoStartBitrate: | ||
| kind === 'audio' || | ||
| initialVideoStartBitrate || | ||
| publisher.engine.pcManager.publisher | ||
| .getLocalDescription() | ||
| ?.sdp.includes('x-google-start-bitrate='), | ||
| }; | ||
| } finally { | ||
| await publisher.disconnect(); | ||
| await subscriber.disconnect(); | ||
| tracks.forEach((track) => track.stop()); | ||
| await Promise.all(pending); | ||
| await publisher.dispose(); | ||
| await subscriber.dispose(); | ||
| } | ||
| }, | ||
| { url, tokens, singlePeerConnection, prepared, kind }, | ||
| ); | ||
| console.log(JSON.stringify(row)); | ||
| assert.ok( | ||
| row.connectedMs < row.captureReadyMs, | ||
| 'JOIN must finish without device acquisition', | ||
| ); | ||
| if (prepared && singlePeerConnection && !(expectCodecFallback && kind !== 'audio')) | ||
| assert.deepEqual( | ||
| row.publicationSignals, | ||
| [], | ||
| 'late attachment must not register or renegotiate', | ||
| ); | ||
| else | ||
| assert.ok( | ||
| row.publicationSignals.includes('addTrack') && | ||
| row.publicationSignals.includes('offer'), | ||
| ); | ||
| assert.ok(row.videoStartBitrate, 'video must preserve its start-bitrate hint'); | ||
| rows.push(row); | ||
| } finally { | ||
| await page.close(); | ||
| } | ||
| } | ||
| } | ||
| } | ||
| for (const row of rows.filter( | ||
| (row) => | ||
| row.prepared && row.singlePeerConnection && !(expectCodecFallback && row.kind !== 'audio'), | ||
| )) { | ||
| const baseline = rows.find( | ||
| (other) => | ||
| !other.prepared && | ||
| other.kind === row.kind && | ||
| other.singlePeerConnection === row.singlePeerConnection, | ||
| ); | ||
| assert.ok( | ||
| row.publishAfterCaptureMs < baseline.publishAfterCaptureMs, | ||
| 'reservation must reduce publication latency', | ||
| ); | ||
| } | ||
| } finally { | ||
| await browser?.close(); | ||
| await new Promise((resolve) => server.close(resolve)); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in our last discussion it sounded like you didn't want to expose an API like
preparePublicationsdirectly to users because it might be confusing.I agree with this concern, can you elaborate on why you chose to expose it as a user facing API now?