Robot world models predict how the world changes in response to actions, supporting applications like teleoperation, policy evaluation, and model-based planning. For dexterous robotics, this is complicated by limited robot data and scarce action labels, since collecting robot trajectories is costly and hardware variation restricts scenario coverage.
NVIDIA's DreamDojo addresses this by pretraining a foundation world model on roughly 44,000 hours of egocentric human video, using continuous latent actions as unified proxy labels. This knowledge is then transferred to robot embodiments, including the Fourier GR-1 humanoid, through post-training on target robot data.
As one component of its human‑video data, DreamDojo includes an In‑lab dataset collected in tabletop laboratory settings to validate core modeling designs. In this dataset, collectors wear MANUS gloves paired with the Vive Ultimate Tracker to capture precise hand poses during manipulation tasksinvolving objects and verbs that do not appear in the default GR‑1robot training data.
The hand pose measurements from the MANUS‑based setup are retargeted to GR‑1’s action space, producing GR‑1 actions that match the robot’s degree‑of‑freedom specifications. This configuration provides a high‑precision, ground‑truth action conditioning baseline that represents an ideal scenario where additional motion‑capture equipment supplies fine‑grained action labels for the world model.
DreamDojo’s scalable pretraining method relies on continuous latent actions extracted directly from video, which encode motion between consecutive frames into compact proxy labels and enable learning from large human‑video datasets without motion capture for every recording.
In the In‑lab ablation study, the authors compare models without human‑video pretraining, with action‑free pretraining, with latent action conditioning, and with ground‑truth action conditioning using GR‑1 actions retargeted from MANUS‑captured hand motion via Vive Ultimate Tracker. This setup establishes MANUS gloves as ahigh-precision motion capture reference for validating latent action conditioning and studying retargeting in controlled in-lab demonstrations.
Fourier GR-1 is the primary target embodiment for many DreamDojo experiments, including the In-lab evaluation benchmark. Additional post-training on Unitree G1, AgiBot, and YAM lets DreamDojo simulate diverse contact-rich tasks and counterfactual actions beyond the original robot training distribution.
This foundation supports downstream applications including policy evaluation on AgiBot fruit-packing tasks, model-based planning, and live teleoperation of a virtual G1 robot at real-time speeds after distillation. Across these applications, the In-lab MANUS-based dataset is one evaluated configuration among several, not the sole source of action data.
