Repository navigation
examples/camera_livekit_bridge: publish receiver frames to LiveKit - #1478
Open
gareth-morgan-nv wants to merge 10 commits into
Open
gareth-morgan-nv wants to merge 10 commits into
gareth-morgan-nv wants to merge 10 commits into
Conversation
Start a downstream consumer of camera_receiver's CUDA IPC output: a client that connects to the publisher's socket, subscribes, opens each slot's IPC handle and reads frame notices with their metadata, and a probe that prints them with a checksum of the first pixels. Pixels stay on the GPU. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
gareth-morgan-nv
requested review from
farbod-nv,
jsepulveda-nvidia,
nv-jakob and
tengw-nv
November 11, 2026 04:35
This was referenced Nov 11, 2026
…m to LiveKit Add the rest of the bridge: each camera stream's frames are copied on the GPU from the CUDA IPC slot into an NVENC input surface, encoded to H.264 and submitted to a LiveKit track as pre-encoded access units, with the frame's sequence, its sensor start time as the application timestamp and the sensor counter as user data. Frames reach the encoder without leaving the GPU. The NVENC encoder and the LiveKit session are small modules; the session also answers the clock-sync requests that arrive as data messages. Each stream runs on its own thread and encodes the newest picture, skipping older ones. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…oding Add a per-stream integer divisor (--stream NAME=TRACK:DIV) that averages each block of the NV12 picture on the GPU between the IPC slot and the encoder input, so a large camera stream can be published at a size the consumers can handle without the pixels leaving the GPU. This is also the point where a GPU operation on the picture belongs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Describe how the bridge moves frames from the IPC slot to a LiveKit track, how to build and run it, the per-stream shrink option, and the capacity measured against egocap's receiver and recording checks. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…the pipeline Add unit tests for the GPU downscale (exact against a CPU reference), the clock-sync wire format (golden bytes) and the NVENC encoder (Annex-B output, SPS/PPS and IDR, pool behaviour, decodable by ffmpeg), a test that reads known GPU patterns through the IPC client and probe from a separate publisher process, and an end-to-end test of mock sender, receiver, bridge, a local LiveKit server and a subscriber that checks track names, sizes, frame ids, capture-time age and user data. Each skips with a stated reason when a prerequisite is missing. The probe's checksum now reads at most the slot's size instead of always 4096 bytes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Add --timestamp-basis: capture_unix_us (default, the sensor start-of-frame as Unix time), capture_monotonic_us (the same instant on this host's monotonic clock, derived as now minus the frame's age) and monotonic_us (the time the frame is published). A frame with no usable sensor start-of-frame carries no timestamp on the capture bases. The conversion is a small header-only helper with its own test. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…te unqualified capture time Take the frame-ready instant when the picture is on the encoder's input, before encoding, so the monotonic timestamp does not include encode delay. Use the sender's sensor start-of-frame only when the receiver reports a realtime (Unix) sender clock; any other domain publishes no capture time instead of labelling it Unix. Refuse --timestamp-basis capture_monotonic_us: it moved the sensor time onto the host's monotonic clock by assuming the sender's wall clock agrees with the host's, with no bound on that error. It needs a measured sender-to-host clock mapping, which does not exist yet. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Start each stream's encoder at 2 Mbps and apply the target bitrate that the LiveKit congestion controller reports for the track (10% headroom, capped at --bitrate-mbps, floor 0.5 Mbps) by reconfiguring NVENC without a restart or an IDR. A fixed high rate sent into a connection whose bandwidth is not yet known queued frames behind the sender's pacer, making the first frames 1-2 s old and growing. With the adaptive start they are about 60-150 ms old and settle near 80 ms. The stats line reports the current rate, the transport target and the number of changes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
… the rate honest, add timing tests Report the sensor start-of-frame to frame-ready time in the stats line, from frames whose sensor time is on a realtime sender clock, so the latency before a consumer's own frame-ready clock starts can be added to what it reports. If NVENC rejects a bitrate change the encoder keeps its old rate, so the reported rate and the settings for the next change are restored too. Add tests: the bitrate setter and getter, zero rejected, no forced IDR after a reconfigure, smaller packets at a lower rate, and a stream that stays decodable across changes; and a timing test that runs egocap's clock-sync client and recording feed against the bridge on monotonic_us, refuses capture_monotonic_us, and checks that egocap's loader rejects capture_unix_us under its timing branch. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…e been overwritten The receiver overwrites a slot without notice once its ring wraps. After the GPU copy completes, the bridge now checks for a newer notice for the same slot, or as many newer notices as the ring has other slots, and drops the picture instead of encoding a possibly torn one. Dropped pictures are counted as slot overruns in the stats line. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
gareth-morgan-nv
force-pushed
the
gmorgan/CameraStreamingLiveKit
branch
from
November 11, 2026 05:40
83b23e2 to
d950e1f
Compare
gareth-morgan-nv
had a problem deploying
to
dev
November 11, 2026 06:18 — with 
GitHub Actions
Failure
GitHub Actions
Failure
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
examples/camera_livekit_bridge, a standalone C++ bridge from the receiver's CUDA IPC frames to LiveKit: GPU copy or box downscale, NVENC H.264 (low latency, bitrate following the transport's target), pre-encoded publish with frame metadata, and a clock-sync responder. Pixels never reach host memory.--timestamp-basischooses what the frame timestamp means (capture_unix_usor the frame-readymonotonic_us). Stacked on the mock and receiver-fix PR.Known limitation
The receiver overwrites an IPC slot without notice once its ring wraps. After the GPU copy the bridge drops the picture if a newer notice for the same slot arrived, or as many newer notices as the ring has other slots (counted as "slot overruns"). This is a best-effort check, not exclusive slot ownership: the receiver queues notices asynchronously and may drop them, so a wrap that overlaps the copy can be undetected. No test forces a wrap; the drop path is exercised only by reading the code. A real fix needs a protocol change (reservation or a read guard), which this PR does not attempt.
Testing
ctest (gpu_ops, nvenc, clock_sync, timestamp_basis) and the Python IPC-client and mock-to-LiveKit pipeline tests. With the mock on an RTX PRO 5000: head 2560x1984 at 8 Mbps median capture-to-receive 75-83 ms, no missing frames, 0 slot overruns at 30 fps. Not run against real cameras.
🤖 Generated with Claude Code