Controlling a computer mouse through hand gestures

Brief description

We have developed a vision-based interface for controlling a computer mouse via 2D and 3D hand gestures. The proposed interface builds upon our previous work that permits the detection and tracking of multiple hands that can move freely in the field of view of a potentially moving camera system. Dependable hand tracking, combined with finger detection, facilitates the definition of simple and, therefore, robustly interpretable vocabularies of hand gestures that are subsequently used to enable a human operator convey control information to a computer system. Two such vocabularies are defined, implemented and validated. The first one depends only on 2D hand tracking results while the second also makes use of 3D information. As confirmed by several experiments, the proposed interface achieves accurate mouse positioning, smooth cursor movement and reliable recognition of gestures activating button events. Owing to these properties, our interface can be used as a virtual mouse for controlling any Windows application.

Sample results

Download video showing part of a powerpoint presentation, where the computer is controlled through vision based interpretation of hand gestures

Download video showing a video game (Access Invaders) being played while computer is controlled through vision based interpretation of hand gestures.


Antonis Argyros, Manolis Lourakis, Cedric Groyer, Mickael Marechal, Benjamin Marie.

Relevant publications

  • A.A. Argyros, M.I.A. Lourakis, "Binocular Hand Tracking and Reconstruction Based on 2D Shape Matching", in proceedings of the International Conference on Pattern Recognition 2006 (ICPR’06), Hong Kong, China, 20 – 24, August 2006.
  • A.A. Argyros, M.I.A. Lourakis, “Vision-based Interpretation of Hand Gestures for Remote Control of a Computer Mouse”, in proceedings of the HCI’06 workshop (in conjunction with ECCV’06), LNCS 3979, Springer Verlag, pp.40-51, Graz, Austria, May 13th, 2006. Recipient of the “Best Paper Award”.
  • A.A. Argyros, M.I.A. Lourakis, “Tracking Skin-colored Objects in Real-time”, invited contribution to the “Cutting Edge Robotics” book, ISBN 3-86611-038-3, Advanced Robotic Systems International, 2005.
  • A.A. Argyros, M.I.A. Lourakis, “Real time Tracking of Multiple Skin-Colored Objects with a Possibly Moving Camera”, in proceedings of the European Conference on Computer Vision (ECCV’04), Springer-Verlag, vol. 3, pp. 368-379, May 11-14, 2004, Prague, Chech Republic.
  • A.A. Argyros, M.I.A. Lourakis, “Tracking Multiple Colored Blobs With a Moving Camera” in proceedings of the Computer Vision and Pattern Recognition Conference, (CVPR’05), vol. 2, no. 2, p. 1178, San Diego, USA, June 20-26, 2005.
  • A.A. Argyros, M.I.A. Lourakis, “3D Tracking of Skin-Colored Regions by a Moving Stereoscopic Observer”, Applied Optics, Information Processing Journal, Special Issue on Target Detection, Vol. 43, No 2, pp. 366-378, January 2004.
  • K. Sage, J. Howell, H. Buxton, A.A. Argyros, “Learning Temporal Structure for Task-based Control”, Image and Vision Computing Journal (IVC), Elsevier, special issue on Cognitive Systems, Volume 26, Issue 1, Pages 39-52, Elsevier, Jan 2008.
  • S.O. Orphanoudakis, A.A. Argyros, M. Vincze “Towards a Cognitive Vision Methodology: Understanding and Interpreting Activities of Experts”, ERCIM News, No 53, Special Issue on “Cognitive Systems, April 2003

The electronic versions of the above publications can be downloaded from my publications page.