Asai K. (ed.) Human-Computer Interaction. New Developments

Подождите немного. Документ загружается.

Adaptive Real-Time Image Processing for Human-Computer Interaction

331

5.2 Head tracking for human-robot interaction

Let

denote the brightness value at the location },{

x in an image

that was

acquired in time

. Let R be a set of J image locations },...,1|{

)(

=x defining a

template.

},...,2,1|{)(

)(

JjIY

txt

==R is a vector of the brightness values at locations

)( j

the template. We assume that the transformations of the template can be modeled by a

parametric motion model

);(

g ωx , where x denotes an image location and

},...,,{

)()2()1( l

tttt

ωωω

=ω denotes a set of l parameters. The image variations of planar

objects that undergo orthographic projection can be described by a six-parameter affine

motion models (Hager and Belhumeur, 1998):

uxxωx +=

⎥

⎦

⎤

⎢

⎣

⎡

⎥

⎦

⎤

⎢

⎣

⎡

= A

);(

(9)

where

uuedca ),,,,,(

=ω . With these assumptions, the tracking of the object in time

can be achieved by computing

1+t

ω such that )(

));((

ttt

YgY =

ω , where the template

)(

Y is in pose determined by the estimated state.

Given a set

},...,1|,{

)()(

NnwS

== ω of weighted particles, which approximate the

posterior distribution

)|(

:1 tt

Yp ω , the maximum aposteriori estimate (MAP) of the state is

calculated according to the following formula:

)(

maxarg)|(maxarg

tttt

wYp

ωω

ωω ≈=

(10)

The motion parameters in time 1

t take values according to the following formula:

))]

;(()(

[

111 tttttt

gYYA ωωω RR

+++

−+= .

(11)

This equation can be expressed as follows:

111 +++

ttt

A yω . Given N measurements we can

estimate matrix

1+t

A from matrices consisting of adjoined vectors

ω and

y (Horn, 1986):

]

,...,

[

)()()1()1( N

tttt

ωωωω

−−=Δ

]

,...,

[

)()()1()1( N

tttt

YYYYY −−=Δ .

(12)

(13)

Using the least squares (LS) method we can find the solution for

1+t

A (Horn, 1986):

))((

−

ΔΔΔΔ=

ttt

YYYMA

(14)

Singular value decomposition of

yields:

UWVY =Δ . Taking q largest diagonal

elements of

W the solution for

1+t

A is as follows:

qqqtt

UWVMA

−

Δ= . The value of q

Human-Computer-Interaction, New Developments

332

depends on the number of diagonal elements of

W that are below a predefined threshold

value.

In the C

ONDENSATION algorithm we utilized the following motion model:

111

+++

tttt

νωωω ,

(15)

where

1+t

ν is zero mean Gaussian i.i.d. noise, independent of state and with covariance

matrix

which specifies the extent of noise.

When individual measurements carry more or less weight, the individual rows of

yω Δ=Δ A can be multiplied by a diagonal matrix with weighting factors. If the diagonal

matrix is the identity matrix we obtain the original solution. In our approach such row

weighting is used to emphasize or de-emphasize image patches according to number of

background pixels they contain. The background pixels can be detected by a supplementing

tracker, built on different cues, with different failure mode.

5.3 Appearance modeling using adaptive models

Our intensity-based appearance model consists of three components, namely, the W-

component expressing the two-frame variations, the S-component characterizing the stable

structure within all previous observations and F-component representing a fixed initial

template. The model

},,{

tttt

FSWA

represents thus the appearances existing in all

observations up to time

−

t . It is a mixture of Gaussians (Jepson et al., 2001) with centers

},,|{

fswi

, their corresponding variances },,|{

fswi

and mixing probabilities

},,|{

fswim

= .

The update of the current appearance model

A to

1+t

A is done using the Expectation

Maximization (EM) algorithm. For a template

)(

t,R,Y corresponding to the estimated state

we evaluate the posterior contribution probabilities as follows:

⎥

⎦

⎤

⎢

⎣

⎡

−

−=

)(

exp

πσ

(16)

where fswi ,,= and Jj ,...,2,1

. If the considered pixel belongs to background, the

posterior contribution probabilities are calculated using

)(

I :

⎥

⎦

⎤

⎢

⎣

⎡

−

−=

)(

exp

πσ

(17)

This prevents the slowly varying component from updating by background pixels. The

posterior contribution probabilities (with

∑

o 1

)(

) are utilized in updating the mixing

probabilities in the following manner:

Adaptive Real-Time Image Processing for Human-Computer Interaction

333

fswimom

,,|)1(

)(

=−+=

γγ

(18)

where

is accommodation factor. Then, the first and the second-moment images are

determined as follows:

)(

1,1

)1(

IoMM

γγ

+−=

2)(

)(

1,2

)

()1(

IoMM

γγ

+−=

(19)

In the last step the mixture centers and the variances are calculated as follows:

2)(

)(

1,2

)(

1,1

)(

)(,

−==

μσμ

)(

σσμ

)(

σσμμ

(20)

When the considered pixel belongs to background, the mixture center in the component

expressing two-frame variations is updated according to:

)(

(21)

where index

l refers to last non-background pixel.

In order to initialize the model

A the initial moment images are set using the following

formulas:

),(

01,1,1

tImM

and )),((

1,1,1,2

tImM

R+=

. The observation likelihood

is calculated according to the following equation:

∏∑

⎥

⎦

⎤

⎢

⎣

⎡

−

−=

jfswi

1,,

)(

exp

)|(

πσ

(22)

In the particle filter we use a recursively updated mixture appearance model, which depicts

stable structures in images seen so far, initial object appearance as well as two-frame

variations. If a supplementing tracker with different failure mode is used, the update of

slowly varying component is done using only pixels that are labeled as belonging to

foreground. In pairwise comparison of object images we employ only non-background

pixels and in case of background we use the last foreground pixels. Our probabilistic models

differ from those proposed in (Zhou and Chellappa, 2004) in that we adapt models using

information about background.

Human-Computer-Interaction, New Developments

334

5.4 Face tracking using particle filter built on adaptive appearance models

Figure 3. shows tracking the face of a person on images acquired by a camera mounted on

the computer monitor. The experiments have been done in a typical home environment. The

camera was located in front of wooden doors or furniture. In such a scenario a typical color

based tracker can have severe difficulties in following the face because the skin color form

clusters, which overlap with colors of pixels belonging to wood. In the depicted images we

can see that the jitter of the tracking window is relatively low. However, this gray-level

image based tracker failed in frame #36.

Figure 4. depicts some tracking results which have been obtained in experiments with

partial occlusions of the face undergoing tracking. In image #25 we can observe that despite

of the considerable occlusion of the face, the template’s location has not been shifted from

desirable location. However, in images #45, #55 we can notice that in response to the

occlusion of the face the template has been moved to left side. Therefore, the part of the

template learned some background pixels. After the occluding book has been moved to the

right a considerable jitter of the template has been observed in our experiment. Due to the

mentioned above undesirable effects the tracking failed in some of our other experiments.

In Fig. 5. we present some tracking results that were obtained on another test sequence

However, as we can see, the tracker failed in frame #33. In this experiment far larger

tracking window has been utilized in order to cover the area of the face to be tracked.

6. Tracking using Multi-Part Object Representation

6.1 The algorithm

In work (Pérez et al., 2002) it has been demonstrated that multi-part object representation

leads to better tracking. The authors proposed a method consisting in dividing the object to be

Available on: http://research.microsoft.com/vision/cambridge/i2i

Fig. 3. Face tracking using particle filter built on adaptive appearance models. Frames #1,

#20, #25, #30, #35 (from left to right and from top to bottom)

Adaptive Real-Time Image Processing for Human-Computer Interaction

335

tracked into non-overlapping rectangular regions and then computing in each such a sub-

region a color histogram. The observation model has been constructed under assumption

that image data in different sub-regions are independent. The object likelihood was

proportional to the exponential of negative sum of squared distances between histograms. In

(Fieguth et al., 1997), rectangle sub-regions that are defined within the extent of the object to

be tracked are employed to calculate the averaged color of pixels. The object tracking is

achieved through an exhaustive search in a confidence region. An occlusion model has been

developed to discriminate between good and spurious measurements.

Cognitive science states that complex entities are perceived as composition of simple elements.

Objects are represented through such components and the relations between them (Ommer

et al., 2006). One of the disadvantages of color histograms is the loss of spatial information. To

incorporate such information in the object representation we divide the object template into

adjoining cells, regularly spaced within the extent of the target to be tracked. We compute

histograms within such regularly spaced cells using a fast method that has been proposed in

(Porikli et al., 2005). Given the estimated position in the previous frame and the histograms

within object at the estimated position we employ the chi-square test between such

histograms and histograms extracted from cells within the template at candidate position.

Fig. 4. Face undergoing tracking via particle filter built on adaptive appearance models.

Frames #1, #15, #25, #45, #55 (from left to right and from top to bottom)

Fig. 5. Face tracking using particle filter built on adaptive appearance models. Frames #1,

#32, #33 (from left to right and from top to bottom)

Human-Computer-Interaction, New Developments

336

The

X test is given by:

∑

+−=

icieicie

hhhh ))/()((

X , where

and

represent

the number of entities in the i-th bin of the histograms, and a low value for

X indicates a

good match. The distances are transformed into likelihoods through the usage of

exponential function. We seek for the object in the new frame within a search window,

which center is located at the previous object position. At each candidate position we

compare the corresponding histograms and the result is utilized to vote for the considered

position. Every cell votes in its own map. Then we combine the votes in a relevance map,

which encodes hypothesis where the target is located in the image. In order to cope with

partial occlusions we employ a simple heuristics aiming at detecting outliers. If the

difference between corresponding histograms is below a certain level the score in the

relevance map remains unchanged. The level for each cell is determined individually using

the distances between histograms from the initial template and corresponding histograms

from few first frames. Similar test is performed with respect to the actual medians.

6.2 Face tracking using multi-part representation

In Fig. 6 we can observe that this algorithm temporally failed in frame #35 and then

recovered in later images. However, as we can see in frames #53 and #77 a considerable

jitter accompanied the tracking of the face. The tracker is able to track the face under severe

rotations, see frames #134 - #173. However, the tracking of the face ends with considerable

jitter of the window, see frame #175.

Figure 7. shows some tracking results in case of partial occlusions of the face. In image #25

we can see that despite of the occlusion the window of the tracker has not been shifted from

desirable location, see also Fig. 4. However, the occlusion of the face from the opposite side

caused severe shifts of the tracked window, see also frame #48. In consequence, the tracker

temporally failed in frame #55 and then recovered in the next frames.

In the tracking results presented in Fig. 8 we can see that the tracker can fail in some images.

Its advantage is that it can deal with severe rotations of face undergoing tracking. However,

in order to achieve such functionality, relatively large tracking windows should be employed

in the course of the tracking. In the discussed experiments the template has been divided

into four non-overlapping sub-regions. In order to take advantages of the tracker in the last

sequence, the number of sub-regions should be something larger, say eight sub-regions.

7. Combined Face Tracker

Obtaining a collection consisting of both positive and negative examples for on-line learning

is a complex task. In work (Zeki, 2001) the author argues that the human visual system

consists of a number of interacting but still autonomously operating subsystems that process

different object representations. Within such a mechanism, the subsystems can serve

mutually in course of learning and bootstrapping of object representations. This inclined us

to construct an object tracker consisting of two independent trackers with different failure

modes, complementing each other and operating in co-training framework. The co-training

approach has been employed in previous work (Levin et al., 2004), in which a tracker starts

with a small training set and increases it by co-training of two classifiers, operating on

different features.

In the combined algorithm we utilize the trackers presented in the last two Sections. Our

Adaptive Real-Time Image Processing for Human-Computer Interaction

337

Fig. 7. Face tracking using multi-part object representation in case of occlusions.

Fig. 6. Face tracking using multi-part object representation in case of occlusions. Frames #1,

#15, #25, #45, #46, #48, #55, #65, #70 (from left to right and from top to bottom)

Human-Computer-Interaction, New Developments

338

experiments demonstrated that even very simple combined algorithm based on the distance

between the locations of windows, which have been determined via the employed trackers,

leads to considerable improvement of tracking, see Fig. 9 and 10. As we already mentioned in

the previous Section, in our combined tracker the adaptive appearance models are learned

on-line using only foreground pixels, owing to the estimates determined by the complementary

tracker. The information about distance between the locations of windows allows us to

detect such pixels easily. What more, given such information we can accommodate the

histograms of the object parts, which likely belong to the tracked object. In consequence, the

drift of the part-based tracker is smaller. This in turn allows us to accommodate the

histograms with rather lower risk that they will be updated via non-object pixels.

In the results depicted in Fig. 9 the size of the windows of both trackers has been estimated on

the basis of depth map, which has been determined by the stereovision system. The depth

map has also been utilized in labeling of the pixels for the adaptation. Assuming that the face

is relatively flat we exclude pixels too distant from the distance of the template to the camera.

Typically, in the combined face tracker the distance between the windows of the utilized

trackers is small. In the part-based tracker we tried to determine the rotation of the template

though searching for the best one using information on the similarities of the histograms.

However, in the course of the tracking the orientation of the template estimated in such a way

considerably fluctuates about the true template orientation. Therefore, better results were

achieved using the orientation estimated by adaptive appearance model based tracker. This

in turn allowed us to accommodate the histogram in experiments like these depicted in Fig. 11.

8. Conclusions

We have discussed adaptive algorithms for face tracking. Face tracking is one of the most

important research directions in human machine interaction with many possible

applications, for example see work (Tu et al., 2007). The algorithms were validated in

various test sequences and demonstrated their great potential to be used in applications

employing human face to complement the opportunities offered by the computer mouse.

Fig. 8. Face tracking using multi-part representation. Frames #1, #51, #86, #112, #126, #200

Adaptive Real-Time Image Processing for Human-Computer Interaction

339

Fig. 9. Face tracking via combined algorithm. Frames #1, #20, #25, #30, #35, #40 #59, #85, #104

Fig. 10. Tracking of the face undergoing occlusion using combined algorithm. Frames #1,

#15, #25, #45, #55

The distance map provided by stereovision system offers great possibilities to construct robust

algorithms for human-machine interaction. The proposed algorithms for adaptation and

unsupervised learning were tested in real home/laboratory scenarios, and acknowledged

great ability to deal with varying illumination conditions as well as complex background.

Human-Computer-Interaction, New Developments

340

Fig. 11. Face tracking using combined algorithm. Frames #1, #51, #86, #112, #126, #135

9. References

Aggarwal, J. K.; Cai, Q. (1999). Human motion analysis: A review, Computer Vision and Image

Understanding, vol. 73, no. 3, pp. 428-440

Birchfield, S. (1998). Elliptical head tracking using intensity gradients and color histograms,

IEEE Conf. on Computer Vision and Pattern Recognition, pp. 232-237

Bradski, G. (1998). Computer vision face tracking for use in a perceptual user interface, Intel

Technoloy Journal, vol. 2, no. 2, pp. 12-21

Comaniciu, D.; Ramesh, V.; Meer P. (2000). Real-time tracking of non-rigid objects using

Mean Shift, Proc. Int. Conf. on Computer Vision and Pattern Recognition, pp. 142-149

Crowe, E. C.; Narayann, N. H. (2000). Comparing interfaces based on what users watch and

do, Proc. Eye Tracking Research and Applications Symposium, ACM, pp. 29-36

Davis, J.; Vaks, S. A. (2001). Perceptual user interface for recognizing head gesture

acknowledgements, ACM Workshop on Perceptual User Interfaces, pp. 1-7

Doucet, A.; Godsill, A.; Andrieu, Ch. (2000). On sequential Monte Carlo sampling methods

for bayesian filtering, Statistics and Computing, vol. 10, pp. 197-208

Duchowski, A. T. (2002). A breadth-first survey of eye tracking applications, Behavior

Research Methods, Instruments, & Computers, vol. 34, no. 4, pp. 455-470

Emond, B. (2007). Multimedia and human-in-the-loop: interaction as content enrichment,

Proc. of the ACM Int. Workshop on Human-Centered Multimedia, pp. 77-84

Fasel, B.; Luettin, J. (2003). Automatic facial expression analysis: A survey, Pattern

Recognition, vol. 36, pp. 259-275

Fieguth, P.; Terzopoulos, D. (1997). Color-based tracking of heads and other mobile objects

at video frame rates. Proc. IEEE Conf. on Comp. Vision and Patt. Rec., pp. 21-27

Fong, T.; Nourbakhsh, I.; Dautenhahn, K. (2003). A survey of socially interactive robots,

Robotics and Autonomous Systems, vol. 42, pp. 143-166

Gavrila, D. (1999). The visual analysis for human movement: A survey, Computer Vision and

Image Understanding, vol. 13, no. 1, pp. 82-98