Breaking the Sim-to-Real Gap in Robot Hand Training with MANUS Gloves

This use case summarizes and interprets content from the research paper “In-Hand Manipulation of Articulated Tools with Dexterous Robot Hands with Sim-to-Real Transfer”( https://arxiv.org/abs/2509.23075). The technical results and methods described are based on the authors’ published findings.

‍

Robotic Hands Need to Use Human Tools

A major goal in robotics is developing systems that can operate effectively in human-centered environments. To do this, robots must be able to interact with tools designed for human hands. Mastering tool manipulation allows robots to perform versatile tasks in everyday settings and move beyond traditional factory automation.

In this study, UC San Diego explores this challenge by enabling dexterous in-hand manipulation of articulated tools using a robotic hand with fewer degrees of freedom than a human hand.

‍

Challenge

For humans, using scissors or forceps feels effortless. For robots, however, articulated tools with moving joints pose significant challenges due to:

1. Extremely complex contact dynamics

Internal joint friction, stiction, backlash, and small structural gaps are difficult to model accurately, resulting in substantial mismatches between simulation and the real world.

2. Large sim-to-real transfer gap

Policies that perform well in simulation often struggle or fail on physical hardware because small differences in physical properties can lead to large behavioral deviations.

‍

MANUS Gloves as the Solution

To address unstable or inconsistent robot-initialized grasp poses, the UCSD team integrated MANUS Metagloves Pro into their workflow:

• The gloves were used as a hand tracker to provide teleoperated demonstrations.
• They provided the initial grasp configurations for tool manipulation.
• They initialized the articulated object state whenever the robotic hand could not reliably achieve a stable grasp.

‍

Where MANUS Fits in the Three-Stage Training Framework

The research team designed a three-phase training framework, with MANUS gloves contributing to the process by supplying human-demonstrated initial grasps when needed.

Phase 1: Simulation Training and Disturbance Injection

• Train a privileged policy using perfect object pose information.
• Add randomized forces and torques to improve robustness.

Whenever the robot cannot secure the tool autonomously, the team uses MANUS gloves for teleoperated demonstrations to provide a consistent, feasible initial grasp.

Phase 2: Policy Distillation to Proprioception Only

• Distill the privileged policy into a student policy using only joint positions and simple commands.
• Remove privileged information to enable deployment on real robots.

Phase 3: CATFA Tactile Adaptation

• Introduce CATFA (Cross-Attention Tactile Force Adaptation), which fuses tactile signals with motor torque feedback.
• Enable online fine-tuning, disturbance compensation, and stable contact control.

‍

Results

Across surgical forceps, pliers, staplers, laparoscopic instruments, and more, the CATFA-enabled system achieved consistently strong results. Success rates for challenging tools rose from approximately 20 to 30 percent to 100 percent, while tasks that were already performing well gained additional stability and precision.

‍

Future Impact

In this study, MANUS Metagloves Pro supported the training process by providing reliable, human-demonstrated initial grasps whenever the robot struggled to establish them autonomously. This workflow benefits from gloves that can reproduce natural human hand configurations consistently and capture subtle grasp variations required for articulated tools. By using a teleoperated demonstration system that integrates smoothly into robotics pipelines, researchers were able to ensure stable starting states for each trial, which in turn enabled effective learning and sim-to-real transfer.

Beyond this study, the same characteristics that made MANUS suitable for grasp initialization also support broader applications in dexterous robotics, digital human research, VR and AR interaction, and rehabilitation technology. MANUS gloves provide a natural interface for capturing human hand motion and converting it into data that can be used for training, teleoperation, and interactive control.

‍