Jianyuan Min
minjianyuan@gmail.com

Video-based Hand Manipulation Capture Through Composite Motion Control

ACM Transactions on Graphics. Presented at SIGGRAPH 2013

[download paper] [download video]
This paper describes a novel motion capture method for acquiring physically realistic hand grasping and manipulation data from multiple video streams. The key idea of our approach is to introduce a composite motion control to simultaneously model hand articulation, object movement, and subtle interaction between the hand and object. We formulate video-based hand manipulation capture in an optimization framework by maximizing the consistency between the simulated motion and the observed image data. We search an optimal motion control that drives the simulation to best match the observed image data. We demonstrate the effectiveness of our approach by capturing a wide range of high-fidelity dexterous manipulation data. The system achieves superior performance in our comparison against alternative methods such as marker-based motion capture and kinematic motion tracking. We also show the power of our recovered motion controllers by adapting the captured motion data to new objects with different properties.