Honkela A. Solutions to Selected Problems on Independent Component Analysis

Подождите немного. Документ загружается.

Solutions to selected problems from

Hyv¨arinen, Karhunen, Oja:

Independent Component Analysis

Antti Honkela

June 5, 2002

Antti Honkela ICA: Exercise solutions

Contents

Problems for Chapter 2 3

Problem 2.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Problem 2.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Problem 2.15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Problem 2.18 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Problem 2.21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Problem 2.23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Problems for Chapter 3 7

Problem 3.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Problem 3.9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Problems for Chapter 4 8

Problem 4.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Problem 4.13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Problem 4.17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Problems for Chapter 5 10

Problem 5.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Problem 5.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Problem 5.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Problems for Chapter 6 11

Problem 6.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Problem 6.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Problems for Chapter 7 13

Problem 7.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Problem 7.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Problem 7.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Problems for Chapter 8 14

Problem 8.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Problem 8.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Problem 8.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Problem 8.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Problems for Chapter 9 17

Problem 9.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Problem 9.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

HUT, Neural Networks Research Centre 1

Antti Honkela ICA: Exercise solutions

Problem 9.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Problems for Chapter 11 19

Problem 11.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Problem 11.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Problems for Chapter 12 19

Problem 12.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Problem 12.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Problems for Chapter 13 20

Problem 13.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Problem 13.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

HUT, Neural Networks Research Centre 2

Antti Honkela ICA: Exercise solutions

Problems for Chapter 2

Problem 2.7

The transformed variables are uncorrelated when the correlation E{y

} = 0. The correlation

can be evaluated as

E{y

} = E{(cos(α)x

+ sin(α)x

)(−sin(α)x

+ cos(α)x

)}

= −sin(α) cos(α) E{x

} + (cos

(α) − sin

(α)) E{x

} + sin(α) cos(α) E{x

}

= sin(α) cos(α)(σ

− σ

) + (cos

(α) − sin

(α))ρσ

sin(2α)(σ

− σ

) + cos(2α)ρσ

(1)

Setting this to zero yields

tan(2α) =

2ρσ

− σ

(2)

which implies

α =

arctan

2ρσ

− σ

. (3)

Problem 2.8

2.8.1

The marginal distributions can be evaluated by direct integration of the joint distribution over

extra variables.

(x) =

x,y

(x, y)dy =

+ 3x

)ydy =

+ 3x

), (4)

for x

, x

∈ [0, 1].

(y) =

x,y

(x, y)dx =

+ 3x

)y dx

= y









= y

= 2y,

(5)

for y ∈ [0, 1].

) =

(x)dx

+ 3x

)dx

(2x

+ 3), (6)

for x

∈ [0, 1].

) =

(x)dx

+ 3x

)dx

(1 + 6x

), (7)

for x

∈ [0, 1].

All marginal distributions are equal to 0 outside the speciﬁed intervals.

HUT, Neural Networks Research Centre 3

Antti Honkela ICA: Exercise solutions

2.8.2

Random variables x and y are statistically independent if p

x,y

(x, y) = p

(x)p

(y). In this case

(x)p

(y) =

+ 3x

)

[2y] = p

x,y

(x, y). (8)

For x

and x

, however

) =

(2x

+ 3)

(1 + 6x

) 6= p

, x

), (9)

and the variables are thus not independent.

Problem 2.15

2.15.1

By the linearity of the expectation operator

E{x} = E{3u − 4v} = 3 E{u} − 4 E{v} = −1 (10)

E{y} = E{2u + v} = 2 E{u} + E{v} = 3. (11)

2.15.2

The variance of a linear combination of independent variables satisﬁes

Var{αu + βv} = α

Var{u}+ β

Var{v}. (12)

Thus

Var{x} = Var{3u − 4v} = 3

Var{u} + 4

Var{v} = 25 (13)

Var{y} = Var{2u + v} = 2

Var{u} + 1

Var{v} = 5 (14)

2.15.3

Clearly

3 −4

2 1

¶µ

= A

, where

∼ N

µµ

1 0

0 1

¶¶

By the properties of the gaussian density presented in Section 2.5.1 of the book,

∼ N

, AA

= N

µµ

−1

25 2

2 5

¶¶

(15)

2.15.4

Again by the properties presented in Section 2.5.1,

y|x ∼ N

3 +

(x + 1), 5 −

= N

2x + 77

(16)

HUT, Neural Networks Research Centre 4

Antti Honkela ICA: Exercise solutions

Problem 2.18

2.18.1

Let x be uniformly distributed on interval [−a, a], a > 0.

Now

E{x

} = Var{x} =

(17)

E{x

} =

−a

dx =

. (18)

Thus

kurt{x} = E{x

} − 3(E{x

})

= a

−

= −

< 0 (19)

and the distribution of x is subgaussian.

2.18.2

Let x have a Laplacian distribution, i.e. p

(x) =

exp(−λ|x|).

It is known that

∞

exp(−λx)dx =

n+1

. (20)

Thus

E{x

} =

∞

−∞

exp(−λ|x|)x

dx = λ

∞

exp(−λx)x

dx =

(21)

and similarly

E{x

} =

∞

−∞

exp(−λ|x|)x

dx = λ

∞

exp(−λx)x

dx =

. (22)

These yield

kurt{x} = E{x

} − 3(E{x

})

24 − 3 · 2

> 0 (23)

and the distribution of x is thus supergaussian.

Problem 2.21

2.21.1

Let z = x + y, where x and y are independent random variables. The characteristic function of

the sum z is the product of the characteristic functions of x and y, i.e.

(ω) = ϕ

(ω)ϕ

(ω). (24)

This yields the following relation for the second characteristic functions:

(ω) = ln(ϕ

(ω)) = ln(ϕ

(ω)ϕ

(ω)) = ln(ϕ

(ω)) + ln(ϕ

(ω)) = φ

(ω) + φ

(ω). (25)

HUT, Neural Networks Research Centre 5

Antti Honkela ICA: Exercise solutions

By the linearity of the diﬀerentiation operator,

(z) = (−)

(ω)

dω

ω=0

= (−)

(φ

(ω) + φ

(ω))

dω

ω=0

= κ

(x) + κ

(y). (26)

NB: You must be very careful when evaluating the expectations required for the characteris-

tic function. A casual “proof” of Eq. (24) by ϕ

(ω) = E{exp(ωz)} = E{exp(ω(x + y)} =

E{exp(ωx)}E{exp(ωy)} = ϕ

(ω)ϕ

(ω) is problematic because the expectations are actually

taken over diﬀerent distributions (p

(z) versus p

(x) and p

(y)).

2.21.2

Let z = βx. By Eq. (2.84) of the book, the density of z can now be written as

(z) =

. (27)

Thus

(ω) = ln(E{exp(ωz)}) = ln

exp(ωz)p

(z)dz

= ln

exp(ωz)

= ln

exp(ωβx)p

(x)dx

= φ

(βω)

(28)

Writing this in another way gives

(ω) =

∞

k=0

(z)

(ω)

= φ

(βω) =

∞

k=0

(x)

(βω)

∞

k=0

(x)β

(ω)

(29)

which yields the desired formula.

Problem 2.23

2.23.1

(m) = E{y(n)y(n − m)}

= E{[x(n + k) − x(n − k)][x(n + k − m) − x(n − k − m)]}

= E{x(n + k)x(n + k − m) − x(n + k)x(n − k − m)

− x(n − k)x(n + k − m) + x(n − k)x(n − k − m)}

= r

(m) − r

(m + 2k) − r

(m − 2k) + r

(m)

= 2r

(m) − r

(m + 2k) − r

(m − 2k)

(30)

HUT, Neural Networks Research Centre 6

Antti Honkela ICA: Exercise solutions

2.23.2

(ω) =

∞

m=−∞

(m) exp(−mω)

∞

m=−∞

(2r

(m) − r

(m + 2k) − r

(m − 2k)) exp(−mω)

= 2S

(ω) −

∞

m=−∞

(m + 2k) exp(−mω) −

∞

m=−∞

(m − 2k) exp(−mω)

= 2S

(ω) − exp(2kω)

∞

m=−∞

(m + 2k) exp(−(m + 2k)ω)

− exp(−2kω)

∞

m=−∞

(m − 2k) exp(−(m − 2k)ω)

= 2S

(ω) − exp(2kω)S

(ω) − exp(−2kω)S

(ω)

= S

(ω)[2 − exp(2kω) − exp(−2kω)]

= S

(ω)[2 − 2 ∗ cos(2kω)] = 4S

(ω) sin

(kω)

(31)

Problems for Chapter 3

Problem 3.3

Let w

, . . . , w

be the columns of matrix W. The element (i, j) of A = W

MW can be written

as a

= w

The trace of A is

trace(A) =

i=1

. (32)

By Eq. (3.12) in the book,

∂trace(A)

= Mw

+ M

. (33)

Combining these yields

∂trace(A)

= MW + M

W. (34)

Problem 3.9

The Lagrangian function of the problem is

L(w, λ) = F (w) + λ(w

w − 1). (35)

Its gradient with respect to w is

∂L(w, λ)

∂F (w)

+ 2λw. (36)

Setting this to zero yields

w = −

2λ

∂F (w)

(37)

which proves the claim.

HUT, Neural Networks Research Centre 7

Antti Honkela ICA: Exercise solutions

Problems for Chapter 4

Problem 4.5

Let us equate the sample moments with theoretical ones

i=1

y(i) = Kσ

(38)

i=1

y(i)

= E{x

} = (E{y})

+ Var{y} = (Kσ

)

+ 2Kσ

= (2K + K

)σ

. (39)

The parameters can now be solved by noting that

2K + K

= 1 +

(40)

which implies

K = 2

− 1

−1

= 2

− d

(41)

and

− d

. (42)

Problem 4.7

4.7.1

= (H

−1

(43)

} = E{(H

−1

} = (H

−1

E{Hθ + v

}

= (H

−1

H)θ + 0 = θ

(44)

4.7.2

θ −

= θ − (H

−1

= θ − (H

−1

(Hθ + v

)

= θ − θ − (H

−1

= −(H

−1

(45)

= E{(θ −

)(θ −

)

} = E{(−(H

−1

)(−(H

−1

)

}

= E{(H

−1

H(H

−T

} = (H

−1

E{v

}H(H

−1

= (H

−1

H(H

−1

(46)

4.7.3

If C

= σ

−1

H(H

−1

= σ

−1

. (47)

Problem 4.13

The sum satisﬁes z ∼ N (0, Kσ

HUT, Neural Networks Research Centre 8

Antti Honkela ICA: Exercise solutions

4.13.1

The likelihood function is

p(z|K, σ

) = (2πKσ

)

−1/2

exp(−

2Kσ

) (48)

and thus the log-likelihood

ln p(z|K, σ

) = −

ln(2πσ

) −

ln K −

2Kσ

(49)

Assuming the variance σ

is known, the maximum likelihood estimate can be solved by setting

the derivative of the log-likelihood to zero

∂

∂K

ln p(z|K, σ

) = −

= 0 (50)

which yields

K =

. (51)

4.13.2

As K is not random, the condition for unbiasedness is E{

K|K} = K. Evaluating the expectation

yields

K|K} = E

(

)

Kσ

= K (52)

so the estimator is unbiased.

Problem 4.17

4.17.1

(x) =

(x, y)dy =

8xydy = 4x

(53)

and thus

y|x

(y|x) =

(x, y)

(x)

8xy

, (54)

where 0 ≤ y ≤ x. A plot of the distribution is shown in Fig. 1.

4.17.2

The maximal value of the posterior is the largest possible, i.e. ˆy

MAP

= x.

4.17.3

By Theorem 4.2 of the book, the mean-square error of the estimator is

ˆy

MSE

= E{y|x} =

y|x

(y|x)dy =

dy =

x. (55)

HUT, Neural Networks Research Centre 9