Skip to content

sensing: send the camera video to the PC relay over a direct link - #1468

Closed
gareth-morgan-nv wants to merge 1 commit into
gmorgan/EgoCapVideoStage/Relayfrom
gmorgan/EgoCapVideoStage/Producer
Closed

gareth-morgan-nv wants to merge 1 commit into
gmorgan/EgoCapVideoStage/Relayfrom
gmorgan/EgoCapVideoStage/Producer

Conversation

@gareth-morgan-nv

Copy link
Copy Markdown
Contributor

Summary

The Sensing producer sends its camera video to the PC relay over a direct link instead of publishing to LiveKit.

Deployment change: an Orin on this build needs a running relay on the PC, and the LiveKit SDK is no longer fetched or installed for the producer. The stream option is --add-stream=sensor=N,name=NAME,direct=HOST:PORT; the LiveKit options become --fps, --scale-div and --bitrate.

  • Frame correlation: each encoded unit is matched to its capture by the stamp the encoder returns (the driver is asked to copy input timestamps). A unit that matches none is sent without a capture time. --frame-correlation=fifo keeps by-order matching for previews, not for recording.
  • Encoder: writes a VUI so a receiving decoder does not hold back pictures.
  • Timestamp basis: monotonic only, the clock the link carries.

Validation

  • This code compiled and linked on a Jetson Orin (SIPL build, JetPack with CUDA) as part of the head of the producer-control PR above this one; it has not been built on its own or with the Argus backend, and it has not been run.
  • A PC-only check: every header and CMake path this change references resolves, and clang-format-14 and pre-commit are clean.
  • Not run: any stream from a real camera through the direct link, and any latency measurement.

Stacked on the relay PR.

🤖 Generated with Claude Code

The producer no longer publishes to LiveKit. Each stream's encoded H.264 units go over a
direct TCP link to the PC relay (video_stage_worker), which publishes them. Streams are named
(--add-stream=sensor=N,name=NAME,direct=HOST:PORT), and the LiveKit options become --fps,
--scale-div and --bitrate.

Each unit is matched to its capture by the stamp the encoder returns (the driver is asked to copy
input timestamps), and a unit that matches none is sent without a capture time; --frame-correlation=fifo
keeps the old by-order matching for previews. The encoder now writes a VUI so a receiving decoder does not
hold back pictures. The timestamp basis is monotonic only, since that is the clock the link carries.

This is a deployment change: an Orin on this build needs a running relay on the PC, and the
LiveKit SDK is no longer fetched or installed for the producer.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
@coderabbitai

coderabbitai Bot commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@github-actions

Copy link
Copy Markdown
Contributor

✅ Preview deployed: https://NVIDIA.github.io/IsaacCapture/preview/pr-1468/

Built from commit 3e9aa42. It may take a minute for the updated preview to appear.

@gareth-morgan-nv

Copy link
Copy Markdown
Contributor Author

Closing as superseded: the camera path is now the camera_streamer architecture (SIPL/Argus sender, AVS1 over TCP, PC receiver and LiveKit bridge; see #1473-#1479). Not merged; the branch and commits are kept.

🤖 Generated with Claude Code

github-actions Bot added a commit that referenced this pull request Nov 11, 2026
@gareth-morgan-nv

Copy link
Copy Markdown
Contributor Author

This PR is superseded by the camera_streamer PRs:

🤖 Generated with Claude Code

This branch was successfully deployed

1 active deployment
dev — 3e9aa424 Deployed Nov 10, 2026 by gareth-morgan-nv via publish-wheel #6740
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants