Towards Force Sensing from Vision

Brief description

We consider the problem of estimating realistic contact forces during manipulation, backed with ground-truth measurements, using vision alone. Interaction forces are usually measured by mounting force transducers onto the manipulated objects or the hands. Those are costly, cumbersome, and alter the objects’ physical properties and their perception by the human sense of touch. Our work establishes that interaction forces can be estimated in a cost-effective, reliable, non-intrusive way using vision. This is a complex and challenging problem. Indeed, in multi-contact, a given motion can generally be caused by an infinity of possible force distributions. To alleviate the limitations of traditional models based on inverse optimization, we collect and release the first large-scale dataset on manipulation kinodynamics as 3.2 hours of synchronized force and motion measurements under 193 object-grasp configurations. We learn a mapping between high-level kinematic features based on the equations of motion and the underlying manipulation forces using recurrent neural networks (RNN). The RNN predictions are consistently refined using physics-based optimization through second-order cone programming (SOCP). We show that our method can successfully capture interaction forces compatible with both the observations and the way humans intuitively manipulate objects, using a single RGB-D camera.

Visit also:

Sample results

Video with experimental results (IEEE Trans. on PAMI paper).

Video with experimental results (CVPR 2015 paper).


  • T.-H. Pham, A. Kheddar, A. Qammaz, A.A. Argyros
  • This work has been supported by the EU project ROBOHOW.

Relevant publications

  • T.-H. Pham, N. Kyriazis, A.A. Argyros and A. Kheddar, "Hand-Object Contact Force Estimation From Markerless Visual Tracking", IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE, vol. 40, no. 12, pp. 2883-2896, December 2018.
  • T.-H. Pham, A. Kheddar, A. Qammaz, A. A. Argyros, "Towards Force Sensing from Vision: Observing Hand-Object Interactions to Infer Manipulation Forces", IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015), Boston, Massachusetts, June 7-12, 2015.

The electronic versions of the above publications can be downloaded from my publications page.