Controlling a computer mouse through hand gestures

Brief description

We have developed a vision-based interface for controlling a computer mouse via 2D and 3D hand gestures. The proposed interface builds upon our previous work that permits the detection and tracking of multiple hands that can move freely in the field of view of a potentially moving camera system. Dependable hand tracking, combined with finger detection, facilitates the definition of simple and, therefore, robustly interpretable vocabularies of hand gestures that are subsequently used to enable a human operator convey control information to a computer system. Two such vocabularies are defined, implemented and validated. The first one depends only on 2D hand tracking results while the second also makes use of 3D information. As confirmed by several experiments, the proposed interface achieves accurate mouse positioning, smooth cursor movement and reliable recognition of gestures activating button events. Owing to these properties, our interface can be used as a virtual mouse for controlling any Windows application.

Sample results


Download video showing part of a powerpoint presentation, where the computer is controlled through vision based interpretation of hand gestures


Download video showing a video game (Access Invaders) being played while computer is controlled through vision based interpretation of hand gestures.


Antonis Argyros, Manolis Lourakis, Cedric Groyer, Mickael Marechal, Benjamin Marie.

Relevant publications

The electronic versions of the above publications can be downloaded from my publications page.