Achieving human-level dexterity requires precise, low-latency human-to-robot control. By integrating MANUS Quantum Metagloves and Meta Quest 3, the ByteDexter system enables real-time teleoperation of a 20-DoF linkage-driven anthropomorphic robotic hand. The system achieves accurate pinch grips, stable power grasps, and advanced in-hand manipulation, providing high-quality demonstration data for dexterous robotics, teleoperation, and embodied AI research.
Robotic dexterity is difficult to achieve because robots face two key challenges: hardware limitations and software mismatches between human and robot movement.
Building a robotic hand that moves like a human hand is far from simple. Different design approaches each come with trade-offs:
ByteDexter addresses these issues with a hybrid design:
The system is supported by a microsecond-level kinematic solver for real-time control.
Human and robotic hands differ in joint types, ranges of motion, and geometric proportions. These mismatches cause distortion in complex tasks such as pinching, multi-finger coordination, and in-hand manipulation. Conventional fingertip-IK and keypoint-matching methods struggle to maintain high-fidelity reproduction of human motion.
ByteDexter brings hardware and software together into a cohesive, high-performance teleoperation system.
The operator’s finger motion is translated into joint commands for the ByteDexter hand, while the Quest controller provides continuous wrist-pose input. The FR3 arm and ByteDexter hand then reproduce the operator’s motion in real time, creating a natural full-hand control interface.
Instead of directly mapping joint angles or fingertip positions, ByteDexter uses keyvectors, geometric vectors that represent spatial relationships between keypoints of the hand.
These capture:
An adaptive weighting strategy prioritizes pinch-relevant vectors for precise thumb–index or thumb–middle control, separation vectors to reduce collisions, and smoothness terms for continuous, stable motion. This framework effectively reconciles human–robot palm differences and preserves coordinated dexterous motion.
The system delivers tighter, more stable pinch grasps, accurately reproduces coordinated thumb–finger movements, and significantly reduces unintended collisions. It supports dexterous in-hand manipulation, including smooth pinch-to-power transitions, dual-object handling, lid opening through pushing or controlled rotation, and other coordinated multi-DoF actions.
This figure compares fingertip-distance tracking performance during pinch tasks between ByteDexter’s retargeting method (red) and the baseline method DexPilot (blue). ByteDexter’s retargeting produces fingertip trajectories that follow human motion more faithfully, resulting in more accurate and stable pinches.
With MANUS data gloves and the linkage-driven ByteDexter hand, the system reliably replicates human hand motion, maintains coordinated hand–arm control, performs continuous multi-step tasks, and efficiently manipulates diverse objects in real-world settings. This makes the platform well suited for teleoperation, high-quality data collection, imitation learning, and embodied AI development.
