sensing: publish Orin cameras to LiveKit with H.264 encode

.pre-commit-config.yaml

@@ -14,6 +14,8 @@ repos:
     rev: v6.0.0
     hooks:
       - id: trailing-whitespace
+        # NVIDIA's copied sources keep their upstream bytes (see the directory's README.md).
+        exclude: ^src/plugins/sensing/third_party/jetson_multimedia_api/(classes|include)/
       - id: check-symlinks
       - id: destroyed-symlinks
       - id: check-added-large-files
@@ -25,7 +27,7 @@ repos:
       - id: check-toml
       - id: end-of-file-fixer
         # These LFS SVGs are assets; rewriting their final byte creates new LFS objects.
-        exclude: ^(docs/source/_static/camera-viz-controls|examples/vehicle_teleop/assets/vehicle_teleop_flow)\.svg$
+        exclude: ^(docs/source/_static/camera-viz-controls|examples/vehicle_teleop/assets/vehicle_teleop_flow)\.svg$|^src/plugins/sensing/third_party/jetson_multimedia_api/(classes|include)/
       - id: check-shebang-scripts-are-executable
       - id: detect-private-key
       - id: debug-statements

CMakeLists.txt

@@ -71,6 +71,7 @@ option(BUILD_PLUGIN_NOITOM_MOCAP "Build Noitom mocap plugin (downloads MocapApi
 option(BUILD_PLUGIN_OGLO "Build OGLO tactile glove plugin (BLE, Linux only; fetches SimpleBLE + nlohmann/json)" OFF)
 option(BUILD_PLUGIN_WUJI_GLOVE "Build Wuji glove plugin (requires the wuji_sdk C SDK)" OFF)
 option(BUILD_PLUGIN_SHARPA_AVATAR "Build Sharpa AvatarExo Glove plugin (Linux; skips if the Avatar SDK is missing)" OFF)
+option(BUILD_PLUGIN_SENSING "Build SENSING GMSL camera plugin (Jetson only; requires SIPL, NVENC and the LiveKit SDK)" OFF)
 option(BUILD_EXAMPLES "Build examples" ON)
 option(BUILD_EXAMPLE_TELEOP_ROS2 "Build only the teleop_ros2 ROS 2 reference integration (e.g. for Docker)" OFF)
 option(BUILD_TESTING "Build unit tests" ON)

README.md

@@ -91,6 +91,9 @@ To get started, please refer to Isaac Lab's [Installation](https://isaac-sim.git
 The default license is [Apache-2.0](LICENSE.md).
 EgoPost is licensed under [OpenMDW-1.1](LICENSES/LicenseRef-OpenMDW-1.1.txt).
 Third-party materials may carry their own licenses.
+The Jetson encoder classes under `src/plugins/sensing/third_party/jetson_multimedia_api` are NVIDIA's
+[BSD-3-Clause](src/plugins/sensing/third_party/jetson_multimedia_api/LICENSE) sample sources, copied from the Jetson
+Multimedia API; see that directory's README.
 
 ### Separately retrieved optional materials
 

REUSE.toml

@@ -58,6 +58,20 @@ precedence = "override"
 SPDX-FileCopyrightText = "Copyright 2015 The Chromium Authors"
 SPDX-License-Identifier = "BSD-3-Clause"
 
+# NVIDIA's Jetson Multimedia API encoder classes, copied from the nvidia-l4t-jetson-multimedia-api package (see its
+# README.md). Each file keeps its own BSD-3-Clause header; files whose header has no SPDX tag are covered here.
+[[annotations]]
+path = "src/plugins/sensing/third_party/jetson_multimedia_api/**"
+precedence = "closest"
+SPDX-FileCopyrightText = "Copyright (c) 2016 NVIDIA CORPORATION. All rights reserved."
+SPDX-License-Identifier = "BSD-3-Clause"
+
+[[annotations]]
+path = "src/plugins/sensing/third_party/jetson_multimedia_api/LICENSE"
+precedence = "override"
+SPDX-FileCopyrightText = "Copyright (c) 2016-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved."
+SPDX-License-Identifier = "BSD-3-Clause"
+
 # EgoPost's default never overrides upstream headers or sidecar licenses.
 # The license-only scaffold has no covered files yet; the canonical text is reserved for imports.
 [[annotations]]

docs/source/device/camera/sensing.rst

@@ -155,3 +155,72 @@ on the side that fails for a read-only runtime report:
    * - Low FPS or dropped frames
      - Check the Jetson power mode. Maximum performance can be selected with
        ``sudo nvpmodel -m 0 && sudo jetson_clocks``.
+
+Build and run the LiveKit publisher
+-----------------------------------
+
+``camera_plugin_sensing`` runs on the Orin. It captures the GMSL cameras with
+SIPL, encodes H.264 on the hardware encoder and publishes one LiveKit video
+track per camera. CaptureServer, the egocap backend, recording and the browser
+run on the PC.
+
+Prerequisites: an AGX Orin on JetPack 7.2.1 / L4T R39.2.1 with the SENSING
+drivers installed and the cameras working, CUDA, CMake, and network access to
+the LiveKit server. The build also needs two header trees; neither is fetched
+by ``setup.sh``:
+
+- ``JETSON_MMAPI_DIR``: the Jetson Multimedia API tree, containing
+  ``include/nvbufsurface.h``. The encoder classes are copied into the repository
+  and are not taken from it. CMake searches ``/usr/src/jetson_multimedia_api``.
+- ``SIPL_API_DIR``: the Jetson SIPL API tree, containing
+  ``sipl/include/NvSIPLCamera.hpp`` (the ``Jetson_SIPL_API_R<release>_aarch64.tbz2``
+  package published with the L4T release). CMake searches ``/usr/src/jetson_sipl_api``,
+  ``/opt/nvidia/jetson_sipl_api`` and ``$SIPL_API_DIR``.
+
+.. code-block:: bash
+
+   # Build. The LiveKit SDK v1.12.4 prebuilt is downloaded and hash-checked; use
+   # -DLIVEKIT_SDK_DIR=... for a local copy (it needs LICENSE and NOTICE files in that
+   # directory, or -DLIVEKIT_SDK_LICENSE_FILE / -DLIVEKIT_SDK_NOTICE_FILE).
+   cmake -B build -DBUILD_PLUGIN_SENSING=ON -DBUILD_VIZ=OFF -DBUILD_EXAMPLES=OFF \
+     -DBUILD_TESTING=OFF -DBUILD_PYTHON_BINDINGS=OFF \
+     -DSIPL_API_DIR=... -DJETSON_MMAPI_DIR=...
+   cmake --build build --target camera_plugin_sensing -j
+
+   # Check the cameras are visible.
+   build/src/plugins/sensing/app/camera_plugin_sensing --list-sensors
+
+   # Publish. The token file is a LiveKit token for the room and publisher identity
+   # named in the PC's streams file.
+   build/src/plugins/sensing/app/camera_plugin_sensing \
+     --livekit-url=ws://LIVEKIT_HOST:7880 --livekit-token-file=join.token \
+     --add-stream=sensor=0,livekit=sensing-left --add-stream=sensor=1,livekit=sensing-right
+
+The ``livekit=`` names must equal the ``track`` names in the PC's streams file.
+Stop with Ctrl-C. ``--help`` lists every option. The example uses plain ``ws://``
+for a LiveKit server on a trusted LAN; the join token is sent in the clear, so
+use a ``wss://`` endpoint on any other network.
+
+``SENSING_TSC_PROBE=1`` logs the SIPL capture counter beside the host clocks twice
+a second, as separate uncalibrated samples. The counter's units are unconfirmed, so
+do not subtract or fit them.
+
+Cross-compile on a PC
+---------------------
+
+``scripts/setup_orin_crosscompile.py`` builds the producer for the Orin on an x86 Ubuntu PC, without sudo and
+without touching the Orin. Everything goes under one directory (``--root``, default ``~/orin-cross-39.2.1``).
+
+.. code-block:: bash
+
+   src/plugins/sensing/scripts/setup_orin_crosscompile.py doctor   # what is present, what is missing
+   src/plugins/sensing/scripts/setup_orin_crosscompile.py fetch    # sysroot and aarch64 compiler
+   src/plugins/sensing/scripts/setup_orin_crosscompile.py syntax   # type-check every source in seconds
+   src/plugins/sensing/scripts/setup_orin_crosscompile.py build    # camera_plugin_sensing for aarch64
+
+``fetch`` unpacks the L4T libraries from the Jetson Linux archive, and the SIPL API, from the SDK Manager
+download directory (``--sdk-downloads``). It takes the Multimedia API from there or from NVIDIA's apt repository,
+the C library and EGL/V4L2 libraries from Ubuntu 24.04, spdlog at the revision the repository pins, and Arm's
+GNU toolchain; each download is checked against the publisher's SHA-256. ``build`` also needs the CUDA toolkit
+with its aarch64 cross target (``cuda-cross-sbsa``) and uses that toolkit's own ``nvcc``. The result is the same
+binary a build on the Orin produces, but it has not been run: deploy it and test it on the Orin.

src/plugins/CMakeLists.txt

@@ -44,3 +44,7 @@ endif()
 if(BUILD_PLUGIN_SHARPA_AVATAR)
     add_subdirectory(sharpa_avatar)
 endif()
+# Gated: Jetson only -- SIPL and the Tegra NVENC entry point are L4T-only.
+if(BUILD_PLUGIN_SENSING)
+    add_subdirectory(sensing)
+endif()