Teach any robot arm a task by demonstrating it once with your hands — no code, no teleop rig, no dataset. The robot narrates what it understood, rehearses it with real physics in Isaac Sim, executes, verifies from the camera, and remembers it as a named skill anyone can teach and recompose by language.
Built in one day at the Physical AI Sprint Hackathon (Nebius · NVIDIA · Antioch · Toloka), San Francisco.
Programs first, policies later.
Robot arms are now cheap. Programming one still costs more than the task it replaces — which kills the long tail: the 90% of small manufacturing, lab, kitchen and warehouse tasks that never justify an integrator. Foundation policies promise generality but need thousands of teleop demos per task and fail silently.
ONCE bets on the other order. For structured tabletop tasks, a symbolic program over spatial relations (put red on blue) generalises from one demonstration — zero training — and executing those programs, verified, is exactly what produces the data a learned policy needs later. The teaching interface is the data engine.
show once → narrate → rehearse (physics) → execute → verify → remember → compose → human-verify → fine-tune
| Beat | What happens | Powered by |
|---|---|---|
| Show | You move blocks with your hands. Overhead camera diffs BEFORE/AFTER into a program of relations. | OpenCV · object-permanence tracker |
| Narrate | "I saw you put green left of blue, then red on blue. Let me rehearse that." | NVIDIA Cosmos-Reason on Nebius (rules planner primary; VLM ghost-gated) |
| Rehearse | Local kinematic ghost (0 ms) → Isaac Sim with real physics on Antioch; refuses if a stack would topple. | Antioch antioch-sim, RTX PRO 6000, live stream |
| Execute | Pick/place on the SO-101; snap-to-cells; safety envelope + e-stop. | Feetech STS3215 driver |
| Verify | Re-checks relations from the camera; retries; teach-by-correction turns a miss into a second learning moment. | relation verifier |
| Remember | Skill saved under the teacher's name; runs/sim/human badges. | skill library |
| Compose | "Do Sarah's skill but with the yellow ones." | Cosmos + rules fallback |
| Flywheel | Every episode → 3-labeler human verdict → fine-tune data. | Toloka Pipeline API · Nebius SkyPilot |
flowchart LR
subgraph EDGE["Laptop · edge (deterministic core, no network on the critical path)"]
CAM[Overhead camera] --> DET[Color / ArUco detector]
DET --> TRK[WorldTracker<br/>object permanence]
TRK --> DIFF[diff_to_program<br/>relations, not coordinates]
DIFF --> PLAN{Planner policy}
PLAN -->|rules primary| PROG[Program DSL<br/>pick · place · on/left_of/…]
PLAN -.->|weak plan → VLM<br/>ghost-gated| VLM
PROG --> GHOST[LocalGhost<br/>kinematic dry-run · bump & topple checks]
GHOST -->|pass| EXEC[Executor → Primitives → SO-101<br/>snap-to-cells · safety · e-stop]
GHOST -->|fail| REFUSE[refuse: “that would land on green — show me again”]
EXEC --> VER[Verifier<br/>relations from camera]
VER -->|ok| SKILL[(Skill library)]
VER -->|miss| CORR[Teach-by-correction<br/>human fixes one block → diff → learn]
SKILL --> COMP[Compose by language]
COMP --> PROG
EXEC -.-> GEST[GestureArm<br/>real arm mirrors every step]
DASH[Projector dashboard<br/>FastAPI + WS · narration hero · ghost canvas · hotkeys]
TTS[Narrator · TTS]
end
subgraph CLOUD["Sponsor cloud (additive, flagged, always with a local fallback)"]
VLM[NVIDIA Cosmos-Reason<br/>served on Nebius Token Factory]
ISAAC[Antioch · Isaac Sim 6.0<br/>once_ghost scenario · physics verdict · livestream]
TOL[Toloka Pipeline API v2<br/>3-labeler YES/NO quorum]
FT[Nebius bucket + SkyPilot<br/>LeRobot fine-tune]
end
GHOST -.->|async physics badge / stage beat| ISAAC
VER --> EP[(Episode store<br/>frames · program · verdicts)]
EP --> TOL --> EP
EP --> FT
MIR[Mirror mode<br/>MediaPipe hand → arm] -.-> EXEC
Design rules that made a one-day build demo-safe:
- One data contract (
once/schema.py):WorldState → Program → Skill → Episode. Rules, VLM, sim, arm and verifier all consume/produce it — swap any box, nothing else changes. - Every model or cloud call has a deterministic twin and a timeout. The demo can never stall on a network.
- Verify-then-act, verify-after-act. Sim gates execution; camera checks the result; both produce a spoken line.
- Object permanence. A top-down camera loses the bottom block of a stack; the tracker remembers hidden objects while something plausibly covers them — found by the mock loop before any hardware existed.
- Reliability is the ticket, recovery is the win. Retry → teach-by-correction → e-stop + replay of a recorded episode through the same UI.
git clone https://github.com/vnmoorthy/once && cd once
uv venv && uv pip install -e ".[dev]" && uv run pytest -q # 10 tests
ONCE_USE_TTS=false uv run once --mock # full loop, no hardware → http://localhost:8000Dashboard keys (press o): d human demo · b learn & do · x scramble · r again · i rehearse in Isaac Sim · c compose · p replay · space E-STOP.
One-command 3-minute show (real arm gestures + Cosmos + Isaac Sim + synthetic blocks):
uv run python scripts/demo3.py --judge <name>uv run python scripts/arm_test.py --scan # finds the SO-101, pings servos 1-6
uv run python scripts/power_check.py # reads servo voltage — STS3215 want 12 V, not USB's 5 V
uv run python scripts/go.py # guided: gripper test → teach 3 points (validated) → camera → first pick/place → launch
uv run python scripts/mirror.py --camera 0 # MediaPipe hand → arm mirrorCamera: scripts/make_markers.py prints 4 ArUco corners; scripts/calibrate.py --aruco needs no clicking; ONCE_ARUCO_LIVE=true self-heals a bumped camera.
cp .env.example .env # paste keys
uv run python scripts/check_sponsors.py --enable # one green/red per sponsor; flips flags on green| Used for | Flag | |
|---|---|---|
| Nebius Token Factory | serves the reasoning model; episode bucket; SkyPilot fine-tune | ONCE_USE_VLM, ONCE_USE_NEBIUS_UPLOAD |
NVIDIA nvidia/Cosmos3-Super-Reasoner |
narration, second-opinion planning, composition by language | via Nebius |
Antioch antioch-sim → Isaac Sim |
my-sim/src/scenarios.py::once_ghost — physics rehearsal, pass/fail per block, livestream |
ONCE_USE_ANTIOCH |
| Toloka Pipeline API v2-beta | every taught episode → 3-labeler "does the sentence match?" verdict | ONCE_USE_TOLOKA |
once/schema.py data contract (WorldState, Program DSL, Skill, Episode)
once/perception/ camera (real + mock), homography/ArUco, detectors, WorldTracker
once/demo/diff.py before/after → Program (rules) ← the deterministic twin of the VLM
once/planner/ vlm.py (Nebius/NVIDIA), compose.py
once/skills/ skill library
once/exec/ arm drivers (Feetech/LeRobot/mock), pose table, cells, safety, primitives, executor, gesture
once/sim/ghost.py LocalGhost + AntiochGhost (SDK/CLI)
once/verify/ relation verifier
once/data/ episodes, Nebius (upload/fine-tune), Toloka (v2-beta)
once/narrator/tts.py the robot's lines
once/dashboard/ FastAPI + WS + projector page
my-sim/ Antioch project: once_ghost scenario + streamed show
scripts/ go · demo3 · calibrate · teach · arm_test · power_check · mirror · wave · check_sponsors · toloka_*
- One-shot teach → narrate → rehearse → execute → verify → remember (mock-verified 90/90 cycles, live on SO-101)
- Isaac Sim physics rehearsal on Antioch with per-block verdicts and livestream
- Cosmos-Reason narration + composition on Nebius; Toloka 3-labeler pipeline
- Verified-episode → LeRobot ACT/SmolVLA fine-tune loop closed end to end
- Multi-arm; non-block objects (ArUco-tagged); learned grasp poses
- Skill sharing across cells — the library becomes the account
Land with one $200 arm + ONCE in an SMB kitting cell; expand by skills, not robots. The moat is the corpus of human-verified, sim-checked teach events with correction diffs — every customer teach makes the next robot need fewer demos. Model vendors have compute; ONCE has the ground truth.
Made with too little sleep in San Francisco · Design doc, decision log and the office-hours review live in the repo history.