In several hand-object(s) interaction scenarios, the change in the objects’ state is a direct consequence of the hand’s motion. This has a straightforward representation in Newtonian dynamics. We present the first approach that exploits this observation to perform model-based 3D tracking of a table-top scene comprising passive objects and an active hand. Our forward modeling of 3D hand-object(s) interaction regards both the appearance and the physical state of the scene and is parameterized over the hand motion (26 DoFs) between two successive instants in time. We demonstrate that our approach manages to track the 3D pose of all objects and the 3D pose and articulation of the hand by only searching for the parameters of the hand motion. In the proposed framework, covert scene state is inferred by connecting it to the overt state, through the incorporation of physics. Thus, our tracking approach treats a variety of challenging observability issues in a principled manner, without the need to resort to heuristics.
As in our earlier approaches for tracking a single hand (BMVC 2011) or two strongly interacting hands (CVPR 2012) the input to the method is visual data acquired with an RGBD camera. You may also be interested in having a look at our related work on full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints (ICCV'2011), where hand-object interaction is observed and tracked based on input provided by a multicamera system.
A video with 3D tracking experiments.
The electronic versions of the above publications can be downloaded from my publications page.