Force-Modulated Dexterity

Unified Compliance for Contact-Rich Manipulation

Ying Yuan1,2 David Held2 Jitendra Malik1 Haozhi Qi1
1Amazon FAR 2Carnegie Mellon University
Paper Code (Coming Soon) Twitter
🔈

Autonomous policy rollout. A visuomotor policy trained entirely on demonstrations collected through our force-modulated teleoperation system — hover to unmute.

Method

Force Modulation of Nominal Actions

The operator’s command is a nominal action. Force feedback rewrites only its contact-relevant components, in coordinates set by task geometry and contact structure.

System overview: a Sharpa glove with Vive tracker produces the human action; the FOMO controller modulates it using force sensing before sending the robot action to a Tianji arm with a Sharpa Wave hand equipped with wrist F/T, finger torque and fingertip tactile sensing.
System. A teleoperator or learned policy provides the nominal action; wrist force-torque, fingertip tactile and joint-torque sensing modulate it before execution on the arm and hand.
FOMO-Dex controller: (a) task-frame directions for unilateral load relief, translational tool compliance and multi-contact force regulation; (b) force error calculation with bilateral and unilateral errors; (c) direction-wise admittance mapped to arm Cartesian offsets or finger-joint corrections.
Controller. Task geometry defines interaction directions; force errors along them drive direction-wise admittance, yielding arm/finger corrections for unilateral load relief, translational tool compliance and multi-contact force regulation.
Results · Human-in-the-loop

Teleoperated Demonstrations

FOMO-Dex, naive compliance and stiff control, teleoperated on the same hardware. Task-frame force modulation yields smoother contact and fewer failed grasps. Hover a clip to hear it.

Card Pickup

Unilateral Load Relief
FOMO-Dex
FOMO-Dex
FOMO-Dex
FOMO-Dex
Naive Compliant
Naive Compliant
Naive Compliant
Naive Compliant
Stiff
Stiff
Stiff
Stiff

Chip Pickup

Unilateral Load Relief
FOMO-Dex
Stiff

Paper Cutting

Translational Tool Compliance
🔈
FOMO-Dex
🔈
Naive Compliant
🔈
Stiff

Bolt–Nut

Multi-Contact Force Regulation
FOMO-Dex
Naive Compliant
Stiff

Screwdriver

Multi-Contact Force Regulation
🔈
Screwdriver AFOMO-Dex
🔈
Screwdriver AFOMO-Dex
🔈
Screwdriver AFOMO-Dex
🔈
Screwdriver AFOMO-Dex
🔈
Screwdriver AFOMO-Dex
🔈
Screwdriver BFOMO-Dex
🔈
Screwdriver CFOMO-Dex
🔈
Screwdriver DFOMO-Dex
🔈
Naive Compliant
🔈
Naive Compliant
🔈
Naive Compliant
🔈
Naive Compliant
🔈
Naive Compliant
🔈
Screwdriver AStiff
🔈
Screwdriver AStiff
🔈
Screwdriver AStiff
🔈
Screwdriver BStiff
🔈
Screwdriver BStiff
🔈
Screwdriver CStiff
🔈
Screwdriver CStiff
🔈
Screwdriver DStiff
🔈
Screwdriver DStiff
Results · Autonomous

Learned Visuomotor Policies

A DINOv2 + Diffusion Policy behavior-cloning model, trained on teleoperated demonstrations including the compliance corrections, rolled out autonomously.

Card Pickup

Autonomous, 1× Speed
BC rollouts
FOMO-Dex
FOMO-Dex
Stiff
Stiff
Stiff

Screwdriver

Autonomous, 1× Speed
BC rollouts
🔈
FOMO-Dex
🔈
FOMO-Dex
🔈
Stiff
🔈
Stiff
🔈
Stiff
🔈
Stiff
🔈
Stiff

Bolt-Nut

Autonomous, 1× Speed
BC rollouts
🔈
FOMO-Dex
🔈
FOMO-Dex
🔈
FOMO-Dex
🔈
Stiff
🔈
Stiff