How to run (order, commands, which page has the details): RUN.md.
Reads the PICO controllers, turns hand motion into follower joint angles (IK), validates every packet,
and sends it to the existing follower receiver openarm_leader_remote (lerobot fork) on port 5701
(right) / 5702 (left). The video path and the receiver are not rebuilt here.
Goal and gates: goal.md. Why each choice was made: log/log.md. Background: notion/.
PICO ─Wi-Fi─ PC Service :63901 ─ xrobotoolkit_sdk :60061 ─ pico_teleop (one reader) ─ZMQ 5701/5702─ openarm_leader_remote ─ follower
| Gate | State | Still needed before hardware |
|---|---|---|
| 1 input-only diagnostic | PASS live 2026-10-31, all 5 phases (log-007, evidence/gate1) |
— (finding: a hidden controller is not detectable) |
| 2 offline mapping / IK | done offline (tests + evidence/gate2, all PASS) |
measure the real grasp point (--tcp-offset-m); view a run in Meshcat |
| 3 fake receiver | done offline, and online PASS with live PICO → real port 5701 → real receiver (no robot, log-009) | — |
| 4 recording dry run | PASS 2026-10-31: lerobot-record with the PICO, chest + left wrist, follower tracks the actions (log-020) | a leader-arm recording with the same command for the side-by-side schema check |
| 2 axis signs (no arm) | PASS both arms 2026-10-31 (log-008) | — |
| 2b wrist rotation | offline + dry run PASS (log-015/016); real arm: left roll/pitch/yaw 1:1 (log-019, log-027), right roll left (log-029); with walls no lock (log-034) | right-arm wrist with the wall version; pitch right arm |
| pilot (leader vs PICO) | leader × 5 (log-021); PICO 11-01: 2 good episodes with joint_1 120° (log-025), then locked → wall fix (log-030/033), wall check PASS on the left arm (log-034) | new start-pose set kitakyushu-wood-l1-pico-v1 (log-046, joint_7 gap gone); re-record the leader block, then PICO block × 5 on the same set; wall/slip rule built (flag only, log-047) |
| 5 hardware | PASS 10-31 left arm (log-010/011/014); 11-01 right arm first session position PASS (log-029, Kento-san / Suraj); wall check left (log-034) | wall check on the right arm; gripper force feedback; follower-feedback patch (Kento) |
First real-arm sessions: 2026-10-31, left arm, supervised (log-010).
- Offline: finish the replays old vs new (today's locked runs + session 4 + pilot redo): no edge locks, no creep
- If replays still lock on
IK jumpedat an edge: treat it as a WALL there, test, replay again - log-030 + commit (deferred: limit avoidance in wrist mode; not possible: headset wall signal / vibration)
- Elephant + Goldfish review of the day (log-035): code fixes applied; process gap: steps 5 ran minutes after the commit, before this review
- Arm: wall check PASS on the left arm (log-034: 94 walls, no lock): joint-limit, drift and rotation walls seen; stretched-arm / out-of-reach / joint-jump walls not yet seen on the robot
- Repeat R2 right wrist (roll right, pitch) with the pre-move (Enter)
- PICO block × 5 (left) with the wall; wall time + slip per episode: page notes /
tools/wall_report.py(log-047) - Compare with the leader block (after Kento's joint_7 offset fix and a leader redo)
Side task: PICO video (FPV) with an on/off button on the controller (log-031): built, not used (Wi-Fi load).
Jerk (log-032):
HZ=60 ACC=300not yet on the arm (the wall checks ran with it off): needs its own supervised check before recording with it. Before PICO data collection (log-035): written approval (120° j1, right arm, wall behaviour), right-arm wall check, 60 Hz check, episode rule for wall time / slip, Kento's joint_7 offset.
- joint_7 start offset (Kento): in leader-arm mode the follower starts with joint_7 ≈ +25°, the pose set (and the PICO) at −63.4° (~89° apart, log-021). Check the leader↔follower joint_7 calibration; redo pilot block 1.
- Block-1 outcomes per episode (grasp / place) from the operator.
- Ask Kento also about the joint_1 start: leader episodes start at +51…+54°, the PICO (pose set) at +64.4°.
- Right arm: the 120° joint_1 drift limit applies to it too; review after its zero/sign check.
Left controller drives the left arm, right controller the right arm. Stand behind the robot,
facing the same way it faces, then recenter the headset (forward for you = forward for the arm).
Start each squeeze facing roughly forward: the sender refuses to START when the headset faces
more than 75° away from the robot's forward. Once engaged you may turn your head to watch the arm
(the motion comes from the controller's position, not the head). The status line shows facing ±N deg.
| Input | Effect | When |
|---|---|---|
| Grip, squeeze and keep holding | after 0.3 s of holding (the terminal prints ENGAGED): ENGAGE, the arm follows your hand's motion from where it is now, at half scale. The arm does not jump to your hand. A short accidental squeeze (e.g. while pulling the trigger) does nothing. |
only from HOLD, only with the link up (not waiting/LINK DOWN), and only after the grip was seen open once since start (grip already held at start = nothing happens) |
| Grip, open fully (< 0.4) | HOLD: no new targets are sent; the arm finishes the last small step (≤ 0.9° per joint, one 20 ms cycle) and stays there. Settling on the real arm: to be measured in the first 2 cm test. Your hand can move freely. | ENGAGED |
| Grip half-squeezed (0.4 to 0.6) | nothing changes (keeps the current state) | always |
| Trigger | gripper: released = open (-58°, 2° short of the measured stop at -60.1°), fully pulled = closed (0°). Gentle sync: after each engage the gripper moves slowly (30°/s) toward where the trigger says until they match, then follows at normal speed. Example: at start the gripper is closed and the trigger released, so after engaging it opens slowly over ~2 s; release the grip to stop it. | ENGAGED only; in HOLD and FAULT the gripper keeps its angle |
Wrist rotation (only with --orientation) |
turning your hand turns the gripper about the same axis by the same angle (1:1; position stays half scale). Relative: the controller's tilt at the moment you squeeze counts as zero; turning 20° from there turns the gripper 20°; release and squeeze again re-zeroes it. The gripper turns at most 90°/s, so after a fast hand turn it keeps turning briefly (a 45° snap ≈ 0.5 s) until it catches up. Wrist turns count toward the 60° posture limit. | ENGAGED only; off by default; used on the arm 10-31 and 11-01 (log-019/027/029/034) |
| A/B/X/Y, menu, sticks | left Y held 1 s = PICO video on/off only when started with --video-device and never while engaged (log-031). A toggles sending in the PICO app: pressing it stops all input (the arm FAULTs). B: the app's video 2D/3D view. |
— |
| Ctrl-C on the sender PC | stop sending. The receiver holds the last pose after 250 ms (the arm stays powered). Before starting the sender again: restart the follower (runbook R2). | always |
There is no stop button on the controllers. To stop the arm moving: open the grip. To stop everything: the spotter presses the emergency stop / switches off the arm PSU (safety protocol §1).
Keep both controllers in front of you while engaged; open the grip before a hand leaves the view. A hidden controller is not flagged by the PICO (its position just stays put). When it is seen again it jumps: the sender FAULTs on jumps faster than 3 m/s, and on a desired TCP more than 15 cm ahead of the arm target. A slower drift below both limits would be followed (speed-limited); not measured live.
FAULT: the arm freezes and ignores your hand. You cannot see this inside the headset (no vibration or sound yet), so the spotter watches the sender terminal and calls out "FAULT". To recover: open the grip fully, bring the hand back to where it was when you engaged (or closer to your body), hold the grip 0.3 s again. If the same FAULT repeats: stop and report.
| Message in the terminal | Plain meaning | What to do |
|---|---|---|
input stale (> 0.4 s without new PICO data) / pose zero / pose frozen / grip/trigger out of range |
tracking or the headset stream was lost | check the headset is on and the controller is in view; release, squeeze |
operator faces … deg from the robot's forward … when starting |
at the moment you squeezed, the headset looked more than 75° away from the robot's forward (e.g. your body is turned toward the PC) | turn your body to face the robot's forward (recenter if needed), then squeeze; once engaged you may look at the arm |
WALL: … the arm stops here, move back (not a FAULT) |
log-030: the move would go deeper into the workspace edge: a joint within 3° of its limit, the arm stretched (sigma < 0.03), a joint past its session drift limit (120° joint_1 / 90°), a joint > 60° from where you squeezed, or a target out of reach. The arm holds there, stays ENGAGED and your hand slips (pushing further does nothing); moving back follows at once. No FAULT, does not count toward LOCKED. Logged per cycle (wall in the JSONL) and summed at exit. |
move back; or release and squeeze again |
WALL: hand turned faster than the gripper … / WALL: joints would jump … / WALL: wrist cannot keep your hand's rotation here |
log-030b/033: the extra hand turn beyond 40° ahead is dropped (the gripper keeps turning ≤ 90°/s for a moment); a big joint jump is never sent; a move that would tilt the gripper away from your hand is held | turn slower / move back |
near singular (sigma … < 0.02) |
backstop only: the stretched-arm WALL (below 0.03) stops a deeper move first, so this is not reached in normal use | bring the hand back toward you |
hand jumped at … m/s |
the controller position jumped (tracking lost and caught up) | keep the controller in view; release, squeeze |
hand rotation jumped … deg in one update or … at … deg/s |
the controller's rotation changed > 30° in one PICO update or faster than 720°/s: only a tracking catch-up does that | keep the controller in view; release → "ready" → squeeze |
hand is … cm ahead of the arm target |
your hand moved faster than the arm may follow; the 15 cm is measured at the gripper tip (hand × 0.5) | release, move slower, squeeze |
LINK DOWN |
no echo from the arm PC for 1 s (it may be saving an episode). The arm holds; the sender keeps sending the held pose for 5 s, then stops and exits (SENDING STOPPED) |
if link back comes within 5 s: release and squeeze on "ready". After SENDING STOPPED: restart both follower and sender (runbook R2) |
echo of packets we did not send |
another sender (e.g. leader_agent) uses the same arm port | spotter: emergency stop first (the other sender may still drive the arm), then find and stop the other sender |
LOCKED: the same FAULT … 2x within 30 s or LOCKED: 3 FAULTs within 60 s |
repeated problems: engaging is locked for the rest of this sender run | stop: release, esc the follower, then restart follower and sender (R2) |
validator: … |
a packet was unsafe (non-finite, over a limit, too big a step) and was not sent | stop the session and report it |
The sender prints the state of each arm once per second and every event as it happens.
bash ~/projects/PICO-controller/scripts/gui.sh # then open http://localhost:8090 on this PC
Full guide: docs/collection-website.md (setup, people, cards, stages, both arms, refusals, recovery, data locations);
PICO connection: docs/pico-working-setup.md. Restart the page after every code update.
cd ~/projects/PICO-controller
uv venv --python 3.10 .venv && uv pip install --python .venv/bin/python "pin==4.1.0" "numpy<2.3" pyzmq pyyaml pytest matplotlib "nicegui>=2,<3"
bash scripts/test.sh -q # offline tests, no hardwareThe SDK is the py3.10 build in ~/Desktop/XRoboToolkit-PC-Service-Pybind (commit 75cb113).
scripts/pico_env.sh puts it on the path and refuses to run if the PC Service is down.
Preflight (all must be true):
- The follower receiver (
run_teleop_remote_*.sh) and anyleader_agent/ PICO sender are not running. The diagnostic refuses to start if it finds one, but also check the arm PC by eye. - Only one program reads the PICO: a second reader silently gets zeros (Kento's request §8).
- PC Service (no sudo). If
ss -tln | grep -E '63901|60061'shows both ports it is running: skip the start. Otherwise start it:Close its 3D demo (it is a second reader):cd /opt/apps/roboticsservice && \ LD_LIBRARY_PATH=/opt/apps/roboticsservice:/opt/apps/roboticsservice/lib:/opt/apps/roboticsservice/SDK/x64 \ nohup ./RoboticsServiceProcess > /tmp/roboticsservice.log 2>&1 &
pgrep -af RobotLinuxDemomust print nothing; if it does, close that window orkill <pid>. - Network: switch the PC to Wi-Fi
orbohand read its192.168.0.xaddress fromhostname -I(other addresses there are docker/USB; ignore them). The PICO must be onorbohtoo. - App: "PC Service:" = that
192.168.0.xaddress → wait forStatus: WORKING→ tick Controller → turn sending on (Send in the app). Confirm with the 3-s check below. From then on never press A (A toggles sending off). Never short-press Home (leaves the app); recentering is a long press of Home (PICO default; confirm on your unit) and is the only Home use allowed. - Check data (prints
rate ~80 Hz | R ok L okwhen good):cd ~/projects/PICO-controller && bash scripts/pico_env.sh python -c "import time;from pico_teleop.xr_input import SdkSource,pose_status;s=SdkSource();t=time.monotonic();n=set() while time.monotonic()-t<3: f=s.read();n.add(f.t_device_ns);time.sleep(0.005) s.close();print('rate ~%.0f Hz'%(len(n-{0})/3),'| R',pose_status(f.right.pose),'L',pose_status(f.left.pose))" 2>&1 | grep rate
rate ~0 Hz/zero: sending is off (step 3), the demo is open (step 1), or the headset slept. Still zero after all three: restart the service (pkill -f RoboticsServiceProcess, then step 1) and reconnect the app. This fixed it on 2026-10-31. - Keep the headset on from here (off for about 8 s = sleep = stream stops). A second person reads the phase text below aloud and runs the commands; the wearer only says "go".
- One phase per run (8 s lead-in after the data check, then the phase):
left/right phase, in this order: 10 s moving about 10 cm forward/back, left/right, up/down → grip fully twice → trigger fully twice → X then Y (left) / B only (right).
D="bash scripts/pico_env.sh python -m pico_teleop.diag_input --no-prompt --lead-in-s 8" $D --phase left --phase-seconds 30 # left only; right lies still $D --phase right --phase-seconds 30 # right only; left lies still $D --phase left_loss --phase-seconds 20 # hold left in view 5 s, behind your back 7 s, back in view $D --phase right_loss --phase-seconds 20 # same with right $D --phase headset_off --phase-seconds 20 # headset on 4 s, off and laid down 10 s, back on
- Result: each run ends
GATE 1 (LIVE phases <name>, …): PASS|FAIL. Gate 1 = all five PASS. Expected PASS lines: left 8 checks, right 7, each loss phase 1 ("stream stayed ≥ 30 Hz": the loss itself is only reported, because a hidden controller is not detectable), headset_off 1 ("stream went stale": the stop is the expected result).
On FAIL: moved too small → move further; swept too small → open fully, squeeze fully; stayed still → the other controller moved or sides are swapped (report it); no gap > 0.25 s → Wi-Fi gap, rerun
once. Re-score any file: bash scripts/pico_env.sh python -m pico_teleop.diag_input --replay runs/<file>.
After the runs (checklist): gzip the passing files into evidence/gate1/ · write the next
log/log-NNN.md (command, git log -1 --oneline, the PASS lines, file names) · add its row to
log/log.md · update the README status row. Sign-off for gate items: Suraj.
The sender in --dry-run creates no socket at all (nothing can reach 5701/5702) and skips the
receiver-echo gate, so you can engage with no follower. Preflight anyway: ss -tln | grep -E ':570[12]'
prints nothing, and no run_teleop_remote_* runs on any PC. Recenter facing the direction that will be
"robot forward". Then, one arm at a time:
cd ~/projects/PICO-controller && bash scripts/pico_env.sh python -m pico_teleop.sender --dry-run --arms left(no --host, no --confirm…). Squeeze the left grip and hold it; move the hand 10 cm, hold still 2 s,
read the once-per-second line TCP moved x… y… z… cm (robot frame: x forward, y left, z up; half scale):
| Hand move (10 cm) | Expected on that line |
|---|---|
| forward (away from you) | x ≈ +5, y and z within ±1 |
| left | y ≈ +5, x and z within ±1 |
| up | z ≈ +5, x and y within ±1 |
Release between moves (it re-anchors) and release before bringing the hand back. Repeat with
--arms right. Left/right words are easy to confuse: if a sign looks wrong, check the geometry first
(hold both controllers in front of you: the right one must read at XR +x of the left one, log-008) and
never change the frames.py matrix from a left/right report alone. Ctrl-C ends it. Any wrong sign or
axis → stop and report; do not go near an arm. Log it like gate 1 (runbook item 2 → PASS, owner Suraj).
cd ~/projects/PICO-controller && bash scripts/pico_env.sh python -m pico_teleop.sender --dry-run --arms left --orientationSqueeze and hold, turn the hand about 20° on ONE axis, hold 2 s, read the line's rot x… y… z… deg
(robot frame: x forward, y left, z up), release, turn back. Position (TCP moved) should stay within ±1 cm.
| Hand turn (about 20°) | Expected rot |
|---|---|
| roll: twist the wrist so the top of the controller tips to your right (clockwise as you look forward) | x ≈ +20, y and z within ±3 |
| pitch: tip the front of the controller down | y ≈ +20, x and z within ±3 |
| yaw: turn the front of the controller to your left (counter-clockwise seen from above) | z ≈ +20, x and y within ±3 |
| Do it twice per arm: once with the hand neutral at engage, once with the hand held tilted (45-90°) | |
| at engage (this is what checks the quaternion convention; the offline tests cannot). Repeat with | |
--arms right. A wrong sign or axis: stop and report (do not edit frames.py). Log it. |
From notion/OpenArm 安全プロトコル…md (archived copy, 2026-10-30) and Kento's request §9:
an Orboh member beside the arm (the PICO wearer cannot see it); emergency stop confirmed to exist
(protocol TODO#1 is open) and within reach; the arm PSU switches located and labelled (power cycle is
the only sure way to go limp); workspace clear (connect() moves the arm by itself for about 2.2 s);
start pose verified; one arm, small and slow motions first. First supervised left-arm sessions: 2026-10-31 (log-010, log-011); procedure: docs/hardware-runbook.md.