Synchronized capture of hand kinematics and forces
Reconstructing what a hand does with a biomechanical model takes two kinds of data that are rarely measured together: where the fingers are and how much force they exert. This project built the system that acquires both at once, with a depth camera and fingertip force sensors, and feeds them to a musculoskeletal model of the hand. With two test trajectories it showed that including forces improves the reconstruction when there is contact.
Context
Vision-based hand pose estimation methods deliver positions. Instrumented gloves deliver angles, with calibration and drift problems. Neither reports on contact forces or on the internal state of the hand, that is, joint torques and muscle activations. A biomechanical model can estimate that state if it is given both measurements.
Acquisition system
Kinematics. An OAK-D stereo camera provides color and depth. A detection model based on MediaPipe Hands runs on it, identifying 21 hand landmarks and locating them in three dimensions.
Landmarks detected on the image and their three-dimensional reconstruction
Forces. Force-sensitive resistors at the fingertips, read with an Arduino. Each sensor was calibrated with several values of the reference resistor in the voltage divider to choose the working range.
Calibration curves of the sensors for four reference resistors
Synchronization. The two sources have different rates and latencies. A messaging system based on ZeroMQ publishes each stream with its timestamp, and a server stores them and delivers them aligned to the model.
Publisher-subscriber architecture for coordinates and forces
Reconstruction
The measured points go through inverse kinematics to obtain joint angles of MyoSuite's MyoHand model. An unscented Kalman filter combines angles and forces, and from the estimated state the torques are computed by inverse dynamics and the muscle activations by quadratic programming.
Processing flow, from points and forces to positions, forces and actuators of the model
Results
With a synthetic reference trajectory, four conditions were compared, according to whether external forces are present and whether they are included in the filter state.
Fraction of frames below each error threshold in the four conditions
Without external forces, including them in the state or not gives the same result. With external forces that are not included in the state, the filter does not correct the error. When they are included, the reconstruction improves substantially.
With a real trajectory, in which the hand starts extended and the fingers spread laterally, and with a force close to 5 N applied at the fingertips, the result is consistent with the previous one.
Error with experimental data, without and with external forces
What is missing
The evaluation is limited to two trajectories, so it is not quantitatively comparable with work validated on large datasets. In the experimental test the external force is synthetic, that is, applied in the simulation rather than coming from a measured contact. The camera requires the hand to face it and move slowly.
How it fits in Robiolab
This project extends the lab's wearable sensor fusion line to the hand and is the experimental basis of the Kalman filter estimation developed in collaboration with Politecnico di Milano. Its usefulness to the group is direct: a way of measuring what a hand does, and with what force, is the input needed by prosthesis control, rehabilitation gloves and teleoperation of robotic hands.