tam interface instantiation - #330
Merged
juelg merged 19 commits intoAug 21, 2026
Merged
Conversation
Lets the 1 kHz control tick read the newest few TAM history samples without copying the whole buffer.
- src/hw/simadaptor.h: self-contained Eigen implementation of the TAM adaptor MLP (jointwise AdaLN blocks + input normalization), parsing the binary exported by SimAdaptorInference.export_simadaptor_weights_cpp; adds LoadFromMemory next to LoadFromFile. - set_tam_mlp_weight now parses the packed binary (one byte per float64 element) off the control thread and publishes the parsed model; it throws on malformed input instead of failing silently at 1 kHz. - tam_forward conditions on q/dq and the last history_steps commanded torques in the ideal-model (gravity-included) space: current tick from the controller arguments, older ticks from the TAM history buffer, so the signature gains robot_state and gravity. The residual ramps in over 1 s once weights + latent are available and is clipped per joint by the new FrankaConfig.tam_residual_clip (default 10/10/10/10/2/2/2 Nm; the wrist joints have a 12 Nm limit). - History samples now record the accumulated controller time (period.toSec is the 1 ms delta, which broke the encoder's stream continuity), reuse the tick's gravity vector, and the buffer holds 4 s instead of 0.2 s so a late encoder poll cannot lose samples; buffer and ramp state reset at controller start so restarts never mix streams.
run_history_encoder streams the controller's TAM history through the torque-adaptation-module RealTimeHistoryAdaptor (~5 Hz) and pushes each latent to the controller; the adaptor binary is exported from the checkpoint at startup and sent through set_tam_mlp_weight. Handles controller restarts (timestamp reset -> encoder reset) and rejects base_tam_fusion checkpoints, which need history streams this integration does not record. TAM is opt-in via InferenceConfig.tam_ckpt; without it the example behaves as before.
InferenceConfig.tam is the single switch: the default checkpoint is fetched once via simadaptor.assets.fetch_checkpoint (cached under ~/.cache/simadaptor) and the ideal-model MJCF comes packaged with the torque-adaptation-module dependency; tam_ckpt/tam_xml stay as overrides.
FrankaConfig.rt_priority>0 elevates the osc/joint/zero-torque control thread to SCHED_FIFO from inside the thread — best-effort, with a warning fallback when the rtprio rlimit forbids it. Unlike ignore_realtime=false this works on stock kernels; with ignore_realtime=true libfranka performs no elevation at all, and the TAM residual's ~0.1-0.3 ms per tick then misses 1 kHz deadlines on a loaded machine (observed communication_constraints_violation, success rate 0.63-0.77). Default 0 (unchanged behavior); the TAM example sets 80.
The encoder's own allocations are small; JAX's default ~75% device-memory grab would starve every other GPU user on the control machine.
The 4 s attention window is the maximum context, not a minimum: the streaming encoder emits its first latent when the first 400 ms patch completes. Also correct the gain-switch note: the last latent is kept across the restart, so the residual resumes immediately (with the ramp).
When SCHED_FIFO is denied (no rtprio rlimit), ask RealtimeKit for SCHED_RR — the same zero-configuration mechanism the desktop audio stack uses. rtkit caps the priority (typically 20), which still preempts every normal thread, and requires a finite RLIMIT_RTTIME; the control thread blocks every millisecond so the runtime cap is never approached. On a desktop session TAM therefore gets real-time scheduling out of the box; headless/SSH setups still need the one-time rtprio rlimit (polkit denies rtkit requests from non-active sessions).
History timestamps now come from a robot-lifetime monotonic clock instead of controller_time, so a controller restart (e.g. a pd_mode gain switch) no longer restarts the encoder's timeline — it is just a short hole in a continuous 1 kHz stream. The history buffer is kept across restarts, the encoder bridges the hole with masked padding rows (pad_history_gaps + keep_mask; gaps over 2 s still reset), and the latent persists throughout. tam_forward holds the residual until the MLP window has refilled with post-restart rows (freshness guard) and re-ramps over 1 s. Verified against the real encoder: latents continue across a 120 ms hole with no stream reset.
RealTimeHistoryAdaptor now handles it all internally (overlap dedup, short-hole bridging on its dense grid, auto-reset on backwards timelines and over-long gaps), so the loop is back to poll -> push_window -> set_tam_latent.
The pad_history_gaps removal also swallowed the config constants (ROBOT2IP, camera dicts, control-mode settings) and InferenceConfig's dataclass decorator. Rebuilt from the pre-refactor file with only the intended change: the encoder loop relies on RealTimeHistoryAdaptor's internal stream handling.
RealTimeHistoryAdaptor.from_checkpoint / history_torque_mode / adaptor_weight_bytes replace the manual asset resolution, private-attr mode detection, and temp-file weight export. The applied-only check stays here as one line: it is this integration's constraint (single recorded torque stream), not the runtime's.
InferenceConfig keeps only tam: bool — checkpoint, MJCF, attention window and latent rate all use the package defaults (override by editing _init_tam if ever needed). The cached adaptor binary lives on the runtime (adaptor_weight_bytes), so the example no longer mirrors it.
SimAdaptor::forward_stream owns the generic model work — input assembly from history_steps StreamRows, latent/shape validation, the guarded forward, and the per-joint clip (bit-identical to the previous inline assembly). Franka::tam_forward keeps only the robot glue: thread-safe model/latent loads, building the rows from the history buffer (tau_cmd + gravity, freshness guard) and the activation ramp.
The control thread is the only writer of the TAM history at that point in the tick, so it keeps its own small ring of the newest samples and tam_forward reads its MLP window without touching the shared buffer's mutex. This also drops the last_n addition to the core ThreadSafeFixedBuffer — the branch no longer modifies rcs core at all.
…imadaptor.h rt_priority (now default 80 — the 1 kHz loop always benefits, and the elevation degrades gracefully when denied) and tam_residual_clip are plain C++ config defaults: the Python bindings and example wiring are gone. The real-time elevation helper moves next to the rest of the TAM machinery in simadaptor.h.
juelg
merged commit Aug 21, 2026
3c436ff
into
RobotControlStack:feat/tam-integration
5 of 6 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.