Skip to content

Add frame metadata for video tracks - #1073

Open
davidliu wants to merge 1 commit into
mainfrom
dl/frame_transformer
Open

davidliu wants to merge 1 commit into
mainfrom
dl/frame_transformer

Conversation

@davidliu

@davidliu davidliu commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds per-frame metadata for video tracks: a capture timestamp, a frame ID and user data supplied by the app. It's sent in the packet trailer (LKTS) that the JS and Rust SDKs use, and subscribers look up a frame's values by its RTP timestamp.

Depends on , which adds the trailer transformer and its Android bindings. This PR doesn't bump webrtc in libs.versions.toml yet. It won't build until a webrtc release with those APIs is out and the version is bumped here.

API

// Publish
room.localParticipant.publishVideoTrack(
    track,
    VideoTrackPublishOptions(frameMetadata = FrameMetadataPublishOptions(timestamp = true, frameId = true)),
)
// Optional: supply your own values, such as user data
track.frameMetadataProvider = FrameMetadataProvider { frame, current -> current.copy(userData = bytes) }

// Subscribe
remoteVideoTrack.addRenderer { frame ->
    val metadata = remoteVideoTrack.lookupFrameMetadata(frame.rtpTimestamp)
}
  • frameMetadata on VideoTrackPublishOptions and VideoTrackPublishDefaults turns frame metadata on, sets which fields the SDK fills in, and declares them to subscribers. The SDK fills in the capture time as wall-clock microseconds and a per-track frame ID.
  • These are @Beta: FrameMetadata, FrameMetadataProvider, LocalVideoTrack.frameMetadataProvider, RemoteVideoTrack.lookupFrameMetadata, TrackPublication.frameMetadataFeatures and FrameMetadataFeature. The publish options aren't, so copy() on the option classes still works without an opt-in.

How it works

  • Publish: a stamping processor sits at the end of every SDK-created camera and screencast track's processor chain. While the track is published with an option on, it fills in the defaults and calls the provider. FrameMetadataManager creates the send transformer and links it to the video source, which passes each frame's values to it. Backup codec senders share the transformer, and restartTrack relinks it to the new source.
  • Subscribe: a track whose publisher declared features gets a receive transformer. It strips the trailer and keeps the values of the last 300 frames for lookups. A resubscribe disposes the previous subscription's transformer.
  • Capability: the client always advertises CAP_PACKET_TRAILER, so the server forwards trailers instead of stripping them.
  • E2EE: FrameTransformerInstaller installs a track's cryptor and metadata transformer as one chain, in the right order for each direction. E2EEManager now creates cryptors without installing them.
  • Threading: code that uses a transformer holds the lock its disposal waits on: the track's lock for send, and FrameMetadataManager's for receive. An unpublish or a participant leaving, handled on another thread, can't dispose it mid-use. The KDoc on FrameMetadataManager and LocalVideoTrack.withFrameMetadataTransformer gives the lock order.

Behavior changes

  • E2EEManager.addPublishedTrack, addSubscribedTrack, removePublishedTrack and removeSubscribedTrack are now internal. They no longer install the cryptor, so an app calling them directly would send unencrypted media.
  • restartTrack now moves backup codec senders to the new track. Before, backup codec video stopped after a restart.
  • Every SDK-created camera and screencast track gets the stamping processor. When frame metadata is off, it passes frames through unchanged.

Known limitations

  • User data isn't encrypted, even with E2EE. The KDoc says so.
  • AV1: Android and Rust carry the trailer in a metadata OBU, while JS appends it raw. AV1 metadata doesn't cross between JS and Android.
  • If an E2EE cryptor is disposed between its creation and its install, the install can still throw. It's a narrow race; will fix it separately in E2EEManager.
  • Found while testing, but preexisting bug that happens on main: restartTrack races RTCMetricsManager and can crash (2 of 6 runs of the restart test). Will be fixed in a separate PR.

Testing

  • Unit tests: FrameMetadataMockE2ETest (27) and FrameMetadataStampingTest (16).
  • On a device: instrumented tests in e2e-android against a LiveKit server, on a Pixel 6 (Android 16). They cover:
    • VP8, H.264, VP9 and AV1 (L1T3)
    • E2EE with VP8 and H.264
    • user data at the size limit, oversized, and forming a start code
    • video processors and the provider
    • simulcast layer switches
    • VP9 SVC with a VP8 backup codec
    • restartTrack, republishing, and full reconnects on either side
  • By hand:
    • interop with JS and Rust in both directions, with and without E2EE

Changeset: .changeset/frame-metadata.md (minor).

🤖 Generated with Claude Code

Publishers can send a capture timestamp, a frame ID and their own user
data with each video frame, in the packet trailer the JS and Rust SDKs
use. Subscribers look the values up by RTP timestamp. This needs a
webrtc-sdk build with the frame metadata APIs; the webrtc version isn't
bumped here.

API:
- `frameMetadata` (FrameMetadataPublishOptions) on
  VideoTrackPublishOptions and VideoTrackPublishDefaults turns frame
  metadata on, picks the fields the SDK fills in, and declares them in
  the AddTrackRequest's packet_trailer_features.
- Beta: FrameMetadata, FrameMetadataProvider,
  LocalVideoTrack.frameMetadataProvider,
  RemoteVideoTrack.lookupFrameMetadata,
  TrackPublication.frameMetadataFeatures and FrameMetadataFeature.
- E2EEManager's add/remove track methods are now internal. They no
  longer install the cryptor, so calling them directly would send
  unencrypted media.

Publishing:
- A stamping processor at the end of every SDK-created camera and
  screencast track's processor chain fills in the defaults (capture
  time as wall-clock microseconds, a per-track frame ID) and calls the
  provider, while the track is published with an option on.
- FrameMetadataManager creates the send transformer and links it to
  the video source. Backup codec senders share it, and restartTrack
  relinks it to the new source.
- restartTrack now also moves backup codec senders to the new track.
  Before, backup codec video stopped after a restart.

Subscribing:
- A track whose publisher declared features gets a receive
  transformer, which strips the trailer and keeps the values of the
  last 300 frames. A resubscribe disposes the previous subscription's
  transformer.
- The client always advertises CAP_PACKET_TRAILER, so the server
  forwards trailers instead of stripping them.

E2EE: FrameTransformerInstaller installs a track's cryptor and
metadata transformer as one chain, in the right order for the
direction. E2EEManager now creates cryptors without installing them.

Threading: code that uses a transformer holds the lock its disposal
waits on (the track's lock for send, FrameMetadataManager's for
receive). An unpublish or participant leave on another thread
therefore can't dispose it mid-use. E2EE cryptors aren't covered yet.

ScaleCropVideoProcessor keeps a frame's metadata.

Tests: FrameMetadataMockE2ETest (27) and FrameMetadataStampingTest
(16). Verified on a device with instrumented tests covering VP8,
H.264, VP9, AV1, E2EE, simulcast, a backup codec, restartTrack,
republishing and reconnects. Interop with the JS and Rust SDKs tested
by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 825315d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
client-sdk-android Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 4 potential issues.

Devin Review

Comment on lines +101 to +103
if (publication.trackInfo?.packetTrailerFeaturesList.isNullOrEmpty()) {
return null
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Old subscriptions retain metadata transformers

When a resubscribed track drops trailer features, addSubscribedTrack leaves its previous transformer registered. The new subscription replaces the track object, so unsubscribing it cannot release the old transformer or chain.

Learn more

A receive transformer is registered under the subscribed video track object. addSubscribedMediaTrack creates a new object on every subscription and replaces the publication's current track. The early return skips the same-SID cleanup below it, so the old entry stays registered when the new subscription no longer declares metadata. Later unpublishTrack reports only the replacement track; removal cannot find the old entry.

Example: Subscribe to a video SID with PTF_FRAME_ID, update its track info to an empty feature list, then subscribe again and unpublish it. The first transformer and its chain remain registered until room disconnect.

Recommended fix: Remove previous entries for the same SID before testing packetTrailerFeaturesList, including when the new subscription has no features. Keep the removal scoped to the same publisher if SIDs are not globally unique.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

sender: RtpSender,
cryptor: FrameCryptor?,
metadataTransformer: FrameMetadataTransformer?,
): FrameTransformerChain? = install(first = cryptor, second = metadataTransformer) { sender.setFrameTransformer(it) }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Transformer installation bypasses the RTC thread

Publishing or subscribing invokes setFrameTransformer on callback threads without the required RTC executor. Other peer-connection operations use that executor, leaving transformer installation outside the repository's WebRTC threading contract.

Learn more

Room callbacks reach installSendTransformers and installReceiveTransformers directly, without switching to the SDK's serialized RTC executor. The repository's WebRTC threading rule applies to both sender and receiver transformer installation. The new source-link, factory, lookup, and disposal calls also need an audit for the same rule. These methods currently hold track or manager locks during installation, so simply adding a blocking executor call inside those locks can deadlock against RTC callbacks.

Example: A room subscription callback installs a receive transformer while an SDP operation runs on the RTC executor; these WebRTC API calls no longer share the mandated serialization.

Recommended fix: Route native transformer operations through executeBlockingOnRTCThread with the owning track's RTCThreadToken, and restructure the lock boundaries so they never block waiting for the RTC executor.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +1670 to +1673
/**
* Installs a published track's cryptor and frame metadata transformer, whichever it has, on its
* sender in a single install.
*/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Frame metadata lacks entry-point example

The new frame metadata API has no short usage example on Room or LiveKit. Consumers cannot discover its publish-and-lookup flow from the required entry-point documentation.

Learn more

The repository's AGENTS.md requires short code examples for new APIs at an entry point, such as Room. The new public publish options and lookupFrameMetadata work together, but no Room or LiveKit KDoc shows that workflow.

Example: A user reading Room documentation cannot find how to set VideoTrackPublishOptions(frameMetadata = ...) and then call lookupFrameMetadata(frame.rtpTimestamp) in a video sink.

Recommended fix: Add a concise publishing and subscribing code example to the Room or LiveKit class KDoc, with links to the new options and lookup API.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

sender: RtpSender,
cryptor: FrameCryptor?,
metadataTransformer: FrameMetadataTransformer?,
): FrameTransformerChain? = install(first = cryptor, second = metadataTransformer) { sender.setFrameTransformer(it) }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟥 Video metadata bypasses end-to-end encryption

When E2EE publishers set frame userData, installSend appends those bytes after encryption. Network observers can read the metadata despite the video track's encryption.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@davidliu davidliu Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: Rust and JS packet trailers are unencrypted as well. E2EE not working here is a documented limitation.

@xianshijing-lk

Copy link
Copy Markdown
Contributor

Hi @chenosaurus and @changt , could you please help review this PR ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants