In several hand-object(s) interaction scenarios, the change in the objects’ state is a direct consequence of the hand’s motion. This has a straightforward representation in Newtonian dynamics. We present the first approach that exploits this observation to perform model-based 3D tracking of a table-top scene comprising passive objects and an active hand. Our forward modeling of 3D hand-object(s) interaction regards both the appearance and the physical state of the scene and is parameterized over the hand motion (26 DoFs) between two successive instants in time. We demonstrate that our approach manages to track the 3D pose of all objects and the 3D pose and articulation of the hand by only searching for the parameters of the hand motion. In the proposed framework, covert scene state is inferred by connecting it to the overt state, through the incorporation of physics. Thus, our tracking approach treats a variety of challenging observability issues in a principled manner, without the need to resort to heuristics.
As in our earlier approaches for tracking a single hand (BMVC 2011) or two strongly interacting hands (CVPR 2012) the input to the method is visual data acquired with an RGBD camera. You may also be interested in having a look at our related work on full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints (ICCV'2011), where hand-object interaction is observed and tracked based on input provided by a multicamera system.
Qualitative evaluation of the performance of the method
A video with 3D tracking experiments.
- N. Kyriazis, A.A. Argyros, “Physically Plausible 3D Scene Tracking: The Single Actor Hypothesis”, oral presentation, in Proceedings of the IEEE CVPR 2013, Portland, Oregon, USA, Jun. 25-27, 2013.
- N. Kyriazis, I. Oikonomidis and A.A. Argyros, “Binding vision to physics based simulation: The case study of a bouncing ball”, in Proceedings of the 22nd British Machine Vision Conference, BMVC’2011, University of Dundee, UK, Aug. 29-Sep. 1, 2011.
- N. Kyriazis, I. Oikonomidis, A.A. Argyros, “Tracking the articulated motion of two strongly interacting hands”, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2012, Rhode Island, USA, June 18-20, 2012.
- I. Oikonomidis, N. Kyriazis and A.A. Argyros, “Efficient model-based 3D tracking of hand articulations using Kinect”, in Proceedings of the 22nd British Machine Vision Conference, BMVC’2011, University of Dundee, UK, Aug. 29-Sep. 1, 2011.
- I. Oikonomidis, N. Kyriazis and A.A. Argyros, “Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints”, in Proceedings of the 13th IEEE International Conference on Computer Vision, ICCV’2011, Barcelona, Spain, Nov. 6-13, 2011.
- N. Kyriazis, I. Oikonomidis, A.A. Argyros, “A GPU-powered computational framework for efficient 3D model-based vision”, Technical Report TR420, Jul. 2011, ICS-FORTH, 2011.
The electronic versions of the above publications can be downloaded from my publications page.