Shape from Interaction (SfI)

Brief description

We present ``Shape from Interaction" (SfI), an approach to the problem of acquiring 3D representations of rigid objects through observing the activity of a human who handles a tool. SfI relies on the fact that two rigid objects cannot share the same physical space. The 3D reconstruction of the unknown object is achieved by tracking the known 3D tool and by carving out the space it occupies as a function of time. Due to this indirection, SfI reconstructs rigid objects regardless of their material and appearance properties and proves particularly useful for the cases of textureless, transparent, translucent, refractive and specular objects for which there exists no practical vision-based 3D reconstruction method. Additionally, object concavities that are not directly observable can also be reconstructed. The 3D tracking of the tool is formulated as an optimization problem that is solved based on visual input acquired by a multicamera system. Experimental results from a prototype implementation of SfI support qualitatively and quantitatively the effectiveness of the proposed approach.

Graphical illustration of the proposed method. Shape from Interaction (SfI): The interaction of a knowntool with an object of unknown 3D structure is observed. The accurate and occlusions-tolerant tracking of the tool provides constraints that are enough to reconstruct densely the unknown object. Thus, SfI recovers 3D models of objects that because of their physical properties (non-textured, transparent, translucent, specular, highly concave, etc) cannot be reconstructed by existing vision-based methods.

Sample results

3D reconstruction results for various 3D objects. From left to right SfI, Kinect, Visual Hull, Stereo. From top to bottom: Statue, Spray, Doll, Mug, Mirror 1, Mirror 2, Bowl, Ashtray, Flower. N/A denotes that the extracted foreground masks in individual views were inconsistent and gave rise to a null 3D reconstruction.

A video with sample qualitative results.


Relevant publications

  • D. Michel, X. Zabulis, A.A. Argyros, “Shape from Interaction”, in Machine Vision Applications (MVA) Journal.

The electronic versions of the above publications can be downloaded from my publications page.