Skip to content

sensing: send the camera video to the PC relay over a direct link #1468

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gareth-morgan-nv wants to merge 1 commit into from
Closed

sensing: send the camera video to the PC relay over a direct link #1468

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ option(BUILD_PLUGIN_NOITOM_MOCAP "Build Noitom mocap plugin (downloads MocapApi
option(BUILD_PLUGIN_OGLO "Build OGLO tactile glove plugin (BLE, Linux only; fetches SimpleBLE + nlohmann/json)" OFF)
option(BUILD_PLUGIN_WUJI_GLOVE "Build Wuji glove plugin (requires the wuji_sdk C SDK)" OFF)
option(BUILD_PLUGIN_SHARPA_AVATAR "Build Sharpa AvatarExo Glove plugin (Linux; skips if the Avatar SDK is missing)" OFF)
option(BUILD_PLUGIN_SENSING "Build SENSING GMSL camera plugin (Jetson only; requires SIPL, NVENC and the LiveKit SDK)" OFF)
option(BUILD_PLUGIN_SENSING "Build SENSING GMSL camera plugin (Jetson only; requires SIPL and NVENC)" OFF)
option(BUILD_VIDEO_STAGE "Build the video stage: the Orin-to-PC video transport and the PC relay" OFF)
option(BUILD_EXAMPLES "Build examples" ON)
option(BUILD_EXAMPLE_TELEOP_ROS2 "Build only the teleop_ros2 ROS 2 reference integration (e.g. for Docker)" OFF)
Expand Down
35 changes: 17 additions & 18 deletions docs/source/device/camera/sensing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,17 +156,18 @@ on the side that fails for a read-only runtime report:
- Check the Jetson power mode. Maximum performance can be selected with
``sudo nvpmodel -m 0 && sudo jetson_clocks``.

Build and run the LiveKit publisher
-----------------------------------
Build and run the camera producer
---------------------------------

``camera_plugin_sensing`` runs on the Orin. It captures the GMSL cameras with
SIPL, encodes H.264 on the hardware encoder and publishes one LiveKit video
track per camera. CaptureServer, the egocap backend, recording and the browser
run on the PC.
SIPL, encodes H.264 on the hardware encoder and sends each camera's video over a
direct TCP link to the PC video stage (``video_stage_worker``), which publishes it
to LiveKit. The producer has no LiveKit client of its own. CaptureServer, the
egocap backend, recording and the browser run on the PC.

Prerequisites: an AGX Orin on JetPack 7.2.1 / L4T R39.2.1 with the SENSING
drivers installed and the cameras working, CUDA, CMake, and network access to
the LiveKit server. The build also needs two header trees; neither is fetched
drivers installed and the cameras working, CUDA, CMake, and a network path to the
PC running ``video_stage_worker``. The build also needs two header trees; neither is fetched
by ``setup.sh``:

- ``JETSON_MMAPI_DIR``: the Jetson Multimedia API tree, containing
Expand All @@ -179,9 +180,7 @@ by ``setup.sh``:

.. code-block:: bash

# Build. The LiveKit SDK v1.12.4 prebuilt is downloaded and hash-checked; use
# -DLIVEKIT_SDK_DIR=... for a local copy (it needs LICENSE and NOTICE files in that
# directory, or -DLIVEKIT_SDK_LICENSE_FILE / -DLIVEKIT_SDK_NOTICE_FILE).
# Build.
cmake -B build -DBUILD_PLUGIN_SENSING=ON -DBUILD_VIZ=OFF -DBUILD_EXAMPLES=OFF \
-DBUILD_TESTING=OFF -DBUILD_PYTHON_BINDINGS=OFF \
-DSIPL_API_DIR=... -DJETSON_MMAPI_DIR=...
Expand All @@ -190,16 +189,16 @@ by ``setup.sh``:
# Check the cameras are visible.
build/src/plugins/sensing/app/camera_plugin_sensing --list-sensors

# Publish. The token file is a LiveKit token for the room and publisher identity
# named in the PC's streams file.
# Send. Each stream's direct= is the PC address and port its video_stage_worker
# --stream=PORT:TRACK listens on; start the worker first.
build/src/plugins/sensing/app/camera_plugin_sensing \
--livekit-url=ws://LIVEKIT_HOST:7880 --livekit-token-file=join.token \
--add-stream=sensor=0,livekit=sensing-left --add-stream=sensor=1,livekit=sensing-right
--add-stream=sensor=0,name=sensing-left,direct=PC_HOST:5600 \
--add-stream=sensor=1,name=sensing-right,direct=PC_HOST:5601

The ``livekit=`` names must equal the ``track`` names in the PC's streams file.
Stop with Ctrl-C. ``--help`` lists every option. The example uses plain ``ws://``
for a LiveKit server on a trusted LAN; the join token is sent in the clear, so
use a ``wss://`` endpoint on any other network.
The ``direct=`` ports must match the ``--stream`` ports of the PC's
``video_stage_worker``, whose ``TRACK`` names must equal the ``track`` names in
the PC's streams file. Stop with Ctrl-C. ``--help`` lists every option. The direct
link is plain TCP: use a wired link or a trusted network.

``SENSING_TSC_PROBE=1`` logs the SIPL capture counter beside the host clocks twice
a second, as separate uncalibrated samples. The counter's units are unconfirmed, so
Expand Down
12 changes: 5 additions & 7 deletions src/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,17 +25,15 @@ if(BUILD_VIZ)
add_subdirectory(viz)
endif()

# The LiveKit SDK and publisher library: the PC relay publishes to LiveKit; the camera producer still does until it sends
# direct to the relay.
if(BUILD_VIDEO_STAGE OR BUILD_PLUGIN_SENSING)
add_subdirectory(livekit_publisher)
endif()

# The LiveKit SDK and publisher library: the PC relay publishes to LiveKit. The camera producer on the Orin does not use
# LiveKit; it sends over its direct links.
if(BUILD_VIDEO_STAGE)
add_subdirectory(livekit_publisher)
add_subdirectory(video_stage)
elseif(BUILD_PLUGIN_SENSING)
# The LiveKit library uses the record codec and clock-exchange helpers.
# The camera producer sends AVS1 records to the video stage and needs only its record codec and sender.
add_subdirectory(video_stage/wire)
add_subdirectory(video_stage/sender)
endif()

if(BUILD_PLUGINS)
Expand Down
4 changes: 2 additions & 2 deletions src/plugins/sensing/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
# SPDX-License-Identifier: Apache-2.0

# ==============================================================================
# SENSING Camera Plugin: Jetson SIPL capture, hardware H.264 encode, LiveKit publication
# SENSING Camera Plugin: Jetson SIPL capture, hardware H.264 encode, direct streaming to the PC video stage
# ==============================================================================
# Two leaf targets, one per directory (cmake/cmake-structure.md rule 1); the LiveKit wrapper is src/livekit_publisher:
# Two leaf targets, one per directory (cmake/cmake-structure.md rule 1):
# core/ sensing_core -- SIPL capture, encoder, conversion, track output
# app/ camera_plugin_sensing
#
Expand Down
10 changes: 5 additions & 5 deletions src/plugins/sensing/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,11 @@ before replacing it; do not hand-edit the board wiring. Its CSI ports, I2C
buses, deserializer addresses, GPIOs, and link modes must match the installed
SIPL drivers.

## Orin: LiveKit publisher
## Orin: camera producer

`camera_plugin_sensing` (built from [`app/`](app/), [`core/`](core/) and
[`livekit/`](livekit/)) captures the cameras with SIPL, encodes H.264 on the
hardware encoder and publishes one LiveKit video track per camera. The
`camera_plugin_sensing` (built from [`app/`](app/) and [`core/`](core/)) captures the cameras with SIPL, encodes H.264
on the hardware encoder and sends each camera's video over a direct TCP link to the PC video stage (`src/video_stage`),
which publishes it to LiveKit. The producer has no LiveKit client of its own. The
[SENSING user guide](https://nvidia.github.io/IsaacCapture/main/device/camera/sensing.html)
has the build and run steps.

Expand All @@ -53,4 +53,4 @@ guessing). SIPL is built by default (`-DSENSING_WITH_SIPL=OFF` for an Argus-only
(`-DSENSING_WITH_ARGUS=ON`, Argus device index with `--add-stream=sensor=N,...`, `--argus-mode=K`) captures NV12 into
DMA-BUFs that the hardware encoder reads without a copy. **The Argus code has never been built or run: there is no
Argus SDK or Argus camera available to the author.** Argus DMA-BUF input cannot be combined with
`--livekit-scale-div` above 1. An Argus-only build (`-DSENSING_WITH_SIPL=OFF -DSENSING_WITH_ARGUS=ON`) also builds on JetPack 6.2 / L4T R36.4.3.
`--scale-div` above 1. An Argus-only build (`-DSENSING_WITH_SIPL=OFF -DSENSING_WITH_ARGUS=ON`) also builds on JetPack 6.2 / L4T R36.4.3.
10 changes: 0 additions & 10 deletions src/plugins/sensing/app/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,16 +24,6 @@ install(DIRECTORY "${SENSING_CONFIGS_DIR}/"
DESTINATION plugins/sensing_camera/configs
)

# The plugin links the LiveKit runtime, whose RUNPATH points into the build tree; ship it beside the
# binary, where INSTALL_RPATH=$ORIGIN (and liblivekit.so's own $ORIGIN) finds it.
install(FILES
"${SENSING_LIVEKIT_SDK_DIR}/lib/liblivekit.so"
"${SENSING_LIVEKIT_SDK_DIR}/lib/liblivekit_ffi.so"
DESTINATION plugins/sensing_camera
)
install(FILES "${SENSING_LIVEKIT_LICENSE_FILE}" "${SENSING_LIVEKIT_NOTICE_FILE}"
DESTINATION plugins/sensing_camera/licenses/livekit-sdk
)
# The vendored encoder classes are compiled into the binary, so their license text ships with it.
install(FILES "${SENSING_MMAPI_VENDORED_DIR}/LICENSE"
DESTINATION plugins/sensing_camera/licenses/jetson-multimedia-api
Expand Down
124 changes: 67 additions & 57 deletions src/plugins/sensing/app/main.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ void signal_handler(int signal)
// Argument parsers
// =============================================================================

/// "sensor=N,livekit=NAME" -> one sensor and the track that carries it.
/// "sensor=N,name=NAME[,direct=HOST:PORT]" -> one sensor and the stream that carries it.
static StreamConfig parse_stream_arg(const std::string& arg)
{
StreamConfig cfg{};
Expand All @@ -68,18 +68,28 @@ static StreamConfig parse_stream_arg(const std::string& arg)
cfg.sensor_id = static_cast<uint32_t>(std::stoul(val));
has_sensor = true;
}
else if (key == "livekit")
else if (key == "name")
{
cfg.livekit_track = val;
// What the PC, the management channel and the logs call this stream.
cfg.name = val;
}
else if (key == "direct")
{
// HOST:PORT of this stream's PC video-stage port; the producer connects and reconnects.
const auto colon = val.rfind(':');
if (colon == std::string::npos || colon == 0 || colon + 1 == val.size())
throw std::runtime_error("--add-stream direct= must be HOST:PORT");
cfg.direct_host = val.substr(0, colon);
cfg.direct_port = static_cast<uint16_t>(std::stoul(val.substr(colon + 1)));
}
else
{
throw std::runtime_error("Unknown key in --add-stream: '" + key + "'");
}
}

if (!has_sensor || cfg.livekit_track.empty())
throw std::runtime_error("--add-stream requires sensor=<camera-index>,livekit=<track>");
if (!has_sensor || cfg.name.empty())
throw std::runtime_error("--add-stream requires sensor=<camera-index>,name=<stream name>");

return cfg;
}
Expand Down Expand Up @@ -125,13 +135,15 @@ static std::string default_platform_config()
static void print_usage(const char* program_name)
{
std::cout
<< "Usage: " << program_name << " --livekit-url=URL --livekit-token-file=PATH --add-stream=... [options]\n"
<< "Usage: " << program_name << " --add-stream=... [options]\n"
<< "\n"
<< "Publishes each SENSING sensor as a LiveKit camera track: SIPL or Argus capture, hardware H.264,\n"
<< "sent as it is.\n"
<< "Captures each SENSING sensor (SIPL or Argus), encodes it as H.264 on the hardware encoder and sends\n"
<< "it over TCP to the PC video stage, which publishes it onwards.\n"
<< "\n"
<< "Streams (repeat for each sensor):\n"
<< " --add-stream=sensor=N,livekit=NAME camera N (SIPL or Argus device), published as track NAME\n"
<< " --add-stream=sensor=N,name=NAME[,direct=HOST:PORT]\n"
<< " camera N (SIPL or Argus device), called NAME;\n"
<< " direct= is where the PC video stage listens for it\n"
<< " --list-sensors print the cameras, then exit\n"
<< " --camera-backend=auto|sipl|argus auto picks the one backend that matches (default);\n"
<< " (a backend must also be built in)\n"
Expand All @@ -144,18 +156,18 @@ static void print_usage(const char* program_name)
<< " --link-masks='M M' one mask per deserializer (default: '0x0000 0x1100')\n"
<< " --nito-dir=DIR ISP tuning directory (default: /var/nvidia/nvcam/settings/sipl)\n"
<< "\n"
<< "LiveKit:\n"
<< " --livekit-url=URL server, e.g. wss://livekit.example.org\n"
<< " --livekit-token-file=PATH file holding the join token; it names the room and the\n"
<< " participant identity\n"
<< " --livekit-fps=N publication rate cap (default: 30)\n"
<< " --livekit-scale-div=N reduce every track by this integer divisor (default: 1)\n"
<< "Encoding:\n"
<< " --fps=N frame rate cap of each stream (default: 30)\n"
<< " --scale-div=N reduce every stream by this integer divisor (default: 1)\n"
<< " --timestamp-basis=B what each frame's timestamp is: monotonic_us (default) is this host's\n"
<< " monotonic clock when the frame was ready to encode; capture_monotonic_us\n"
<< " (UNVERIFIED on hardware) is the sensor's capture event on that clock and fails\n"
<< " the stream if it cannot be placed; processing_unix_us is Unix processing time.\n"
<< " the stream if it cannot be placed. The direct link carries this clock only.\n"
<< " The streams file on the PC must declare the same basis.\n"
<< " --livekit-bitrate=N target bitrate of each track's encoder in bps (default: 8000000)\n"
<< " --frame-correlation=C how an encoded unit is matched to its capture: stamp (default) by the stamp the\n"
<< " encoder returns, sending a unit that matches none without a capture time; fifo\n"
<< " assumes one unit per input in order (approximate; not for recording)\n"
<< " --bitrate=N target bitrate of each stream's encoder in bps (default: 8000000)\n"
<< " --gop=N IDR period in frames (default: fps*5)\n"
<< "\n"
<< "Other:\n"
Expand All @@ -165,9 +177,8 @@ static void print_usage(const char* program_name)
<< " --swap-uv swap the chroma planes\n"
<< "\n"
<< "Example:\n"
<< " " << program_name << " --livekit-url=wss://livekit.example.org \\\n"
<< " --livekit-token-file=join.token --add-stream=sensor=0,livekit=sensing-left \\\n"
<< " --add-stream=sensor=1,livekit=sensing-right\n";
<< " " << program_name << " --add-stream=sensor=0,name=sensing-left,direct=192.168.1.205:5600 \\\n"
<< " --add-stream=sensor=1,name=sensing-right,direct=192.168.1.205:5601\n";
}

// =============================================================================
Expand All @@ -179,7 +190,6 @@ try
{
SensingConfig camera_config;
std::map<uint32_t, StreamConfig> stream_map;
std::string livekit_token_file;
bool list_sensors = false;
CameraBackend requested_backend = CameraBackend::Auto;

Expand Down Expand Up @@ -232,41 +242,45 @@ try
}
requested_backend = *parsed;
}
else if (arg.find("--livekit-url=") == 0)
{
camera_config.livekit.url = arg.substr(14);
}
else if (arg.find("--livekit-token-file=") == 0)
else if (arg.find("--fps=") == 0)
{
livekit_token_file = arg.substr(21);
}
else if (arg.find("--livekit-fps=") == 0)
{
camera_config.livekit.fps = static_cast<uint32_t>(std::stoul(arg.substr(14)));
camera_config.encode.fps = static_cast<uint32_t>(std::stoul(arg.substr(6)));
}
else if (arg.find("--timestamp-basis=") == 0)
{
const std::string basis = arg.substr(18);
if (basis == "monotonic_us")
camera_config.livekit.timestamp_basis = TimestampBasis::FrameReadyMonotonic;
camera_config.encode.timestamp_basis = TimestampBasis::FrameReadyMonotonic;
else if (basis == "capture_monotonic_us")
camera_config.livekit.timestamp_basis = TimestampBasis::SensorCaptureMonotonic;
else if (basis == "processing_unix_us")
camera_config.livekit.timestamp_basis = TimestampBasis::ProcessingUnix;
camera_config.encode.timestamp_basis = TimestampBasis::SensorCaptureMonotonic;
else
{
std::cerr
<< "Error: --timestamp-basis must be monotonic_us or capture_monotonic_us: the direct link carries the sender's monotonic clock only."
<< std::endl;
return 1;
}
}
else if (arg.find("--frame-correlation=") == 0)
{
const std::string how = arg.substr(20);
if (how == "stamp")
camera_config.encode.correlation = FrameCorrelation::Stamp;
else if (how == "fifo")
camera_config.encode.correlation = FrameCorrelation::Fifo;
else
{
std::cerr << "Error: --timestamp-basis must be monotonic_us, capture_monotonic_us or processing_unix_us."
<< std::endl;
std::cerr << "Error: --frame-correlation must be stamp or fifo." << std::endl;
return 1;
}
}
else if (arg.find("--livekit-bitrate=") == 0)
else if (arg.find("--bitrate=") == 0)
{
camera_config.livekit.bitrate_bps = static_cast<uint32_t>(std::stoul(arg.substr(18)));
camera_config.encode.bitrate_bps = static_cast<uint32_t>(std::stoul(arg.substr(10)));
}
else if (arg.find("--livekit-scale-div=") == 0)
else if (arg.find("--scale-div=") == 0)
{
camera_config.livekit.scale_div = static_cast<uint32_t>(std::stoul(arg.substr(20)));
camera_config.encode.scale_div = static_cast<uint32_t>(std::stoul(arg.substr(12)));
}
else if (arg.find("--gop=") == 0)
{
Expand Down Expand Up @@ -405,22 +419,18 @@ try
}

// Check the settings that need no hardware or network before anything is opened.
if (camera_config.livekit.url.empty() || livekit_token_file.empty())
throw std::runtime_error("--livekit-url and --livekit-token-file are required");
if (camera_config.livekit.fps == 0)
throw std::runtime_error("--livekit-fps must be at least 1");
if (camera_config.livekit.bitrate_bps < 100'000)
throw std::runtime_error("--livekit-bitrate must be at least 100000");

// The token is read from a file so it never appears in a process listing, and is not printed.
std::ifstream token_file(livekit_token_file);
if (!token_file)
throw std::runtime_error("cannot read --livekit-token-file " + livekit_token_file);
std::getline(token_file, camera_config.livekit.token);
const auto last = camera_config.livekit.token.find_last_not_of(" \t\r\n");
camera_config.livekit.token.erase(last == std::string::npos ? 0 : last + 1);
if (camera_config.livekit.token.empty())
throw std::runtime_error("--livekit-token-file " + livekit_token_file + " is empty");
if (camera_config.encode.fps == 0)
throw std::runtime_error("--fps must be at least 1");
if (camera_config.encode.bitrate_bps < 100'000)
throw std::runtime_error("--bitrate must be at least 100000");
{
const bool any_direct = std::any_of(
stream_configs.begin(), stream_configs.end(), [](const StreamConfig& s) { return !s.direct_host.empty(); });
if (!any_direct)
std::cerr << "Warning: no stream has a direct=HOST:PORT destination: nothing will receive the video. "
"Give --add-stream a direct=HOST:PORT."
<< std::endl;
}

std::signal(SIGINT, signal_handler);
std::signal(SIGTERM, signal_handler);
Expand Down
Loading
Loading