Training dexterous robot policies typically relies on real-world demonstrations, which are costly to scale across many tasks. This approach does not scale well, since each new task requires operators, physical robots, and repeated data collection sessions. A recent study addressed this problem by shifting most of the data collection into simulation, then using a small amount of real-world data to adapt the policy to physical hardware.
During the pre-training phase, operators controlled a 56-DoF bimanual dexterous robot inside a physics simulator, with a VR headset tracking their hands and driving the robot's arms and fingers through inverse kinematics. Since collection did not depend on physical hardware, virtual objects could be reset quickly. Five operators collected 75 hours of demonstrations across six scenes and hundreds of object instances, performing open-ended tasks that captured many distinct ways of achieving each goal. This data trained a 222M-parameter diffusion transformer that predicts action chunks from multi-view images and proprioception, using sensorimotor history to maintain reactive and temporally coherent control.
After pre-training, the team brought the policy to physical hardware through a short fine-tuning stage. The hardware setup used two YAM Pro arms, each equipped with a 22-DoF Sharpa Wave dexterous hand, along with three RealSense D405 cameras positioned to capture both a top-down view and each hand's perspective. For each task, they collected one to two hours of real demonstrations using a teleoperation system that mirrored the simulation setup. For this stage, operators wore MANUS gloves for finger tracking, paired with an attached Quest controller for wrist tracking, replacing the headset's built-in hand tracking used in simulation. The team reported that this produced smoother, more precise control, which they found necessary to control the robot through occlusions and at a distance.
The pre-trained policy outperformed an identical policy trained from scratch on the same real demonstrations across all five bimanual tasks (racking plates, stacking cups, playing Jenga, hanging mugs, and tossing bottles into a bin). Its training loss was also lower throughout, a pattern the researchers link to stronger real-world performance. The study demonstrates how large-scale simulation pre-training can reduce reliance on real-world demonstration data when adapting dexterous policies to physical robots. During real-world data collection, MANUS gloves provided finger tracking as part of a teleoperation setup that the researchers reported produced smoother, more precise control, particularly when operating through occlusions and at a distance.
