2D tracking of skin-colored regions by a moving camera

Brief description

We have proposed a method for tracking multiple skin colored objects in images acquired by a possibly moving camera. The proposed method encompasses a collection of techniques that enable the modeling and detection of skin-colored objects as well as their temporal association in image sequences. Skin-colored objects are detected with a Bayesian classifier which is bootstrapped with a small set of training data. Then, an on-line iterative training procedure is employed to refine the classifier using additional training images. On-line adaptation of skin-color probabilities is used to enable the classifier to cope with illumination changes. Tracking over time is realized through a novel technique which can handle multiple skin-colored objects. Such objects may move in complex trajectories and occlude each other in the field of view of a possibly moving camera. Moreover, the number of tracked objects may vary in time. A prototype implementation of the developed system operates on 320x240 live video in real time (30Hz) on a conventional Pentium 4 processor.

The proposed 2D tracker has formed a basic building block for tracking multiple skin colored regions in 3D. More specifically, we have developed a method which is able to report the 3D position of all skin-colored regions in the field of view of a potentially moving stereoscopic camera system. The prototype implementation of the 3D version of the tracker also operates at 30 fps.

On top of this functionality, the tracker is able to deliver 3D contours of all skin colored regions; this is performed at a rate of 22 fps.

One of the very important aspects of the proposed tracker is that it can be trained to any desired color distribution, which can be subsequently tracked efficiently and robustly with high tolerance in illumination changes.

Due to its robustness and efficiency, the proposed tracker(s) have already been used as important building blocks in a number of diverse applications. More specifically, the 2D tracker has been employed for:

Moreover, the 3D tracker has been employed as a basic building block in the framework of a cognitive vision system developed within the EU-IST ActIPret project, whose goal is the automatic interpretation of the activities of people handling tools. In fact, the ActIPret project was the one that supported financially this research on 2D and 3D tracking. Although the ActIPret project has been successfully finished, there are still several on-going activities in several aspects of 2D and 3D tracking of skin colored regions.

Sample results

Download video: A video showing the performance of the tracker in varying illumination conditions

2D tracking of human face and hands.

Finger detection on the tracked skin-colored regions.

Download video: A video of a detected and tracked hand he estimated 3D trajectory of the centroid of the hand appearing in the video at the left. The developed hand tracker can also detect and reconstruct the 3D contour of the hand


Download video: A video showing (among other things) the performance of the 3D tracker in the context of the ActIPret demonstrator


Relevant publications

The electronic versions of the above publications can be downloaded from my publications page.