Skip to content

examples/camera_streamer: carry a qualified sensor capture time to LiveKit - #1479

Open
gareth-morgan-nv wants to merge 2 commits into
gmorgan/CameraStreamingLiveKitfrom
gmorgan/CameraStreamingTiming
Open

gareth-morgan-nv wants to merge 2 commits into
gmorgan/CameraStreamingLiveKitfrom
gmorgan/CameraStreamingTiming

Conversation

@gareth-morgan-nv

Copy link
Copy Markdown
Contributor

Summary

Receiver, mock and bridge side of capture-time alignment. Stacked on the bridge PR.

  • Wire: AVS1 bundle flag 0x0008: one 24-byte record per sample with the sensor capture instant on the sender's monotonic clock, its stated uncertainty, anchor age and a validity bit. The receiver parses and validates it and publishes the fields in the IPC metadata.
  • Mock: emits it from a synthetic sender clock (offset, drift, steps), writes the true capture instant to a ground-truth file, and serves the sender's stateless UDP clock responder (13-byte request, 29-byte reply) with mute, loss and reorder faults.
  • Bridge: measures the sender clock over that responder and publishes capture_monotonic_us only for frames that qualify: valid record, current mapping with warm-up, mapping plus sensor uncertainty within the budget, possible age. Otherwise no capture time is published and frame-ready time is never substituted. A reply counts only from the sender's address, of the exact size and tag, matching the outstanding request and the current session generation.

Limitations

A step of the sender clock shows at the next exchange, so frames in between (at most one exchange interval) carry the old offset; a learned age level cannot tell a step from a stall, so none is used. The sender side (real Orin) is not part of this PR and has not been qualified against it.

Testing

ctest 7/7 (clock-estimator golden vectors from egocap's estimator, qualifier, UDP client against a fake sender, plus the existing targets). A scenario test through the real receiver, bridge and a local LiveKit server (accuracy, step, restart, stale mapping, loss and reorder, invalid records, refusal) passed twice; with a 1.2 s offset and 50 ppm drift the published capture time is within about 100 us of ground truth. A take saved by the unchanged egocap timing recorder matched ground truth within 0.84 ms. The scenario test was reformatted after those runs and not rerun.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

gareth-morgan-nv and others added 2 commits November 11, 2026 00:37
…erve a clock responder

The AVS1 bundle gains an optional timing extension (flag 0x0008): one 24-byte
record per sample with the sensor capture instant on the sender's monotonic
clock, the sender's stated uncertainty, the anchor age and a validity bit. The
receiver parses and validates it and publishes the fields in the IPC metadata.

The mock sender emits it from a synthetic sender monotonic clock with its own
offset, drift and steps, writes the true capture instant on this host's clock to
a ground-truth file, and serves the sender's stateless UDP clock responder
(13-byte Q, 29-byte R) with mute, loss and reorder faults for tests.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
…ground truth

Scenarios through the real receiver, bridge and a local LiveKit server with a
sender clock offset and drift: accuracy, a clock step, a session restart, a
stale clock mapping, lost and reordered replies, invalid timing records, and
refusal without a sender clock. The mock's lint fix rides along.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>

This branch had an error being deployed

1 failed deployment
dev — eecc167f Deployed Nov 11, 2026 by gareth-morgan-nv via publish-wheel #6782
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants