[Real-world active learning] PR 5: observation conversion + twin-sync primitives

[amburger66]

Plan PR 5 of claude_plans/BRIDGE_INTERFACE_PLAN.md (in BabyRobotPredicator). Depends on #93 (PR 4, merged).

Lands the conversion and synchronization primitives the real-world wrapper needs — and nothing else. No RealWorldEnv, and nothing new wired into the live execution loop.

Simulator knowledge — on PyBulletEnv

Generic on purpose, so every future PyBullet-backed real environment inherits it rather than reimplementing it.

• sync_to_state(state) — _set_state plus explicit velocity zeroing. Adopting a state from outside the simulation is exactly the case _set_state doesn't cover: it writes poses through update_object → resetBasePositionAndOrientation, which leaves each body holding whatever momentum the previous rollout gave it, so the bodies drift on the next stepSimulation. Robot joints need no equivalent — _set_state routes them through set_joints, which already resets with targetVelocity=0.
• gripper_joint_layout() — the finger indices and open/closed values the action splitter needs, read off the simulated robot.

Domain knowledge — on PyBulletDominoRealEnv

Pure conversions: no hardware, no I/O beyond the scene file, no notion of a robot.

• state_from_observation(obs, prev_state) — capture id → slot via self._scene_ids (which the env has computed since day one and never used; this is what it was for), base → world via the existing pose_base_to_world / domino_upright_yaw. Dominoes the cameras didn't see keep their last known pose, and the robot's entry — including the joint positions _set_state trusts — carries forward untouched.
• task_from_observation(obs, train_or_test) — the same conversion plus goal semantics.
• _build_task_from_scene refactored onto that same path. Both sources normalize to _PerceivedDomino first, so the captured scene and live re-perception cannot drift apart — including the start-domino yaw canonicalization, which is easy to apply in one place and forget in the other. A test asserts the two paths build the same task from the same poses.

Observations are read structurally (obs.dominoes, each with .id / .xyz / .quat_xyzw), so the env still imports no babyrobot and the tests run against a plain stub. They do not skip, and that is asserted.

Two decisions worth a reviewer's eye

1. state_from_observation does not canonicalize the start domino's yaw. That 180° flip exists to orient the opening push, so it belongs to building a task's initial state. Mid-episode the start domino may already have been pushed, and re-canonicalizing would fight what the cameras actually saw. Tested both ways.
2. An observed capture id that isn't in the scene is dropped, with a warning, rather than raising. Perception occasionally reports a spurious marker, and aborting a physical run over one bad detection is worse. Tested.

One same-value refactor in the existing path

_flush_real_actions now calls self.gripper_joint_layout() instead of gripper_joint_layout_from_robot(self._pybullet_robot). Identical value — the method just delegates — and it keeps the new method from landing as dead code. Flagging it because it is the only line this PR touches inside the execution path; happy to revert it if you'd rather PR 5 leave that file's runtime behavior byte-identical.

Known limitation, stated plainly

Like domino_upright_yaw itself, these conversions assume the dominoes they are given are standing. Reading a toppled domino's pose back into the twin is not modeled here. The plan doesn't ask for it at this stage, but it is a real gap for the closed loop and someone should decide whether PR 6 needs it.

How this was tested

• 19 new tests in tests/envs/test_pybullet_domino_real.py — the first tests this env has ever had. Scene load, _domino_role explicit-vs-id, role counts, the base→world transplant against hand-computed numbers, start-yaw canonicalization (default, no-flip, and explicit start_push_dir_base), goal is Toppled(target), id→slot mapping, carry-forward, robot/joint preservation, unknown ids, and both new PyBulletEnv methods.
• Mutation-checked, because one test was lying. The first version of the velocity test passed with sync_to_state's zeroing removed — the domino component zeroes its own blocks at the end of reset_state, masking it. The test now stubs that hook out, and fails when the zeroing is removed. Three more mutations verified: mapping observation slots by position instead of capture id, and rebuilding the state instead of carrying it forward, each fail the corresponding tests.
• tests/envs/ + tests/pybullet_helpers/: 248 passed / 8 skipped without the submodule, 83 passed with it installed.
• mypy . --config-file mypy.ini clean over 700 files; pylint 10.00/10 on the changed files; yapf/isort/docformatter clean; submodule working tree untouched.

No hardware involved — every conversion here is pure, and the one simulator write is exercised against PyBullet in-process.

🤖 Generated with Claude Code

[yichao-liang]

Great work!

[yichao-liang]

This idea of zeroing the velocities looks interesting. I wonder whether we should apply it more generally in _set_state...