Troyan V., Kiselev Y. Statistical Methods of Geophysical Data Processing

Подождите немного. Документ загружается.

Basic concepts of the probability theory 63

1.9.2 Entropy of the complex system

Let us consider a union of two systems ξ and η with the possible states x

, x

, . . . , x

and y

, y

, . . . , y

correspondingly. The states (x

, y

) of the complex system (ξ, η)

are the complex combinations of x

, y

and together with appropriate probabilities

(Table 1.2).

Table 1.2 Representation of the joint distribu-

tion function for a case of two discrete random

variables x

and y

. . . x

n−1

. . . p

1 n−1

1 n

. . . p

2 n−1

2 n

. · · . . . · ·

. . . p

m n−1

m n

The entropy of the complex system is written as

ξη

= −

i=1

j=1

log p

, H

ξη

= M[−log p(ξ, η)].

For a case of independent random variables we have

p(ξ, η) = p(ξ)p(η)

and, hence,

ξη

= H

+ H

, (1.56)

or extending on a case of an arbitrary number of independent systems, we shall

obtain

,ξ

,...,ξ

= H

+ H

+ ··· + H

in a case of a union of independent systems their entropies are summed.

In case of dependent systems the notion of the conditional entropy of a system

ξ with respect to a state y

of the system η is introduced:

ξ/y

= −

i=1

p(x

) log p(x

), (1.57)

ξ/y

= M[−log p(ξ/y

)]. (1.58)

The mean entropy or the total entropy is deﬁned as

ξ/η

j=1

ξ/y

i=1

j=1

p(x

) log p(x

). (1.59)

Taking into account that p

p(x

) = p

(joint probability), we obtain

ξ/η

= −

i=1

j=1

log p(x

). (1.60)

64 STATISTICAL METHODS OF GEOPHYSICAL DATA PROCESSING

The entropy of a joint system (ξ, η) is determined by the formula

ξη

= H

+ H

η/ξ

, or H

ξη

= H

+ H

ξ/η

, (1.61)

that can be proof using the notion of the entropy and formula for multiplication of

probabilities

p(ξ, η) = p(ξ)p(η/ξ) = p(η)p(ξ/η).

1.9.3 Shannon information (discrete case)

A notion of the amount of information is tightly connected with a notion of the

entropy being measure of the indeterminacy. The deriving of the information is

accompanied by a diminution of the indeterminacy, therefore the amount of the

information can be measured by a diminution of the indeterminacy, i.e. entropy.

Let us ξ is an observed random variable, that describes a physical system. The

entropy before observation (a priori) we shall determine H

apr

, and entropy after

observation (a posteriori) we shall determine H

apost

, then the Shannon information

is deﬁned as follows:

= H

apr

− H

apost

. (1.62)

In that special case, when as a result of the observation ξ the state of a system was

completely spotted, i.e. H

apost

= 0, the information is equal

= H

apr

= −

i=1

log p

= M[−log p(ξ)]. (1.63)

However a direct observation of the random variable ξ frequently appears impos-

sible and the inference about a state of the system described by ξ, is implemented

by observations of another random variable η which is connected with ξ. So for

example, in a radiolocation an operator observes a reﬂected signal η by a screen of

the locator, instead of the direct observation of an airplane ξ. In a seismic explo-

ration the inference about an interior structure of the Earth crust ξ is yielded on

the basis of the observation of a seismic ﬁeld η on original ground. The amount of

the information about the system ξ at the observation of the system η is deﬁned as

a decreasing of the entropy of the system ξ at observation of the system η:

η→ξ

= H

− H

ξ/η

, (1.64)

ξ→η

is called the complete (or mean)information, which contains in the system η

about the system ξ. It is easy to show, that

ξ→η

= H

− H

η/ξ

. (1.65)

Basic concepts of the probability theory 65

Taking into account the equalities

ξη

= H

+ H

η/ξ

= H

+ H

ξ/η

we obtain

ξ→η

= I

η→ξ

= I

η↔ξ

where I

η↔ξ

is called the complete mutual information. If the random variables ξ

and η (that describe the system) are independent variables, then I

η↔ξ

= 0, since

η/ξ

= H

, H

ξ/η

= H

. It is possible to represent the information I

η↔ξ

using the

entropy of the joint system

η↔ξ

= H

+ H

− H

ξη

(1.66)

or, with the help of expectation operator, we obtain

η↔ξ

= M[−log p(ξ) − log p(η) + log p(ξ, η)],

η↔ξ

= M



log

p(ξ, η)

p(ξ)p(η)



, (1.67)

η↔ξ

i=1

j=1

log

ξi

ηj

. (1.68)

In a case of the practical problems solution there can be a necessity for an estimation

of the partial information I

→x

about the system ξ, contained in the individual

message, connected with a speciﬁc state y

, where the complete information I

η↔ξ

is the mathematical expectation of all messages y

j = 1, 2, . . . , m, over partial

information

η↔ξ

j=1

→ξ

, (1.69)

where

→ξ

i=1

p(x

) log

p(x

)

ξi

. (1.70)

The expression (1.70), which deﬁnes the partial information, it is possible to rewrite

as:

→ξ

= M



log

p(ξ/y

)

p(ξ)



, (1.71)

where M

is a conditional expectation operation. It could show, that partial in-

formation I

→ξ

, as the complete information I

η→ξ

, can not be negative one. The

partial information about an event x

, which is obtained with the help of a message

about the other event y

, can be represented as follows

→x

= log

p(x

)

. (1.72)

It is necessary to note, that the such partial information from an event to an other

event, as against entered above the complete I

η↔ξ

and the partial I

→ξ

information,

can be both positive and negative.

66 STATISTICAL METHODS OF GEOPHYSICAL DATA PROCESSING

1.9.4 Entropy and information for systems with a continuous set

of states

The measure of indeterminacy of a system, which is described by a continuous

random variable ξ with the given density function, is the diﬀerential entropy H

deﬁned by the formula

= −

∞

−∞

(x) log f

(x)dx,

= M[−log f

(x)]. (1.73)

In a case of two continuous random variables (or two systems) ξ and η, their

density function f

ξη

(x, y), marginal densities f

(x), f

(y) and conditional densities

η/ξ

(y/x), f

ξ/η

(x/y), it is possible to introduce the partial conditional entropy H

η/x

i.e. the entropy of the system η on condition that the system ξ is in a deﬁnite

condition

H(η/x) = −

∞

−∞

ηξ

(y/x) log f

ηξ

(y/x)dy.

The total or mean conditional entropy is deﬁned as

H(η/ξ) = −

∞

−∞

(x)f

η/ξ

(y/x) log f

η/ξ

(y/x)dxdy,

or, taking into account, that f

ξη

(x, y) = f

(x)f

η/ξ

(y/x), we obtain

H(η/ξ) = −

∞

−∞

ξη

(x, y) log f

η/ξ

(y/x)dxdy.

Similarly, how it was made for a discrete case, it is possible to show, that the

entropy of the system (ξ, η) is equal

H(ξ, η) = H(ξ) + H(η/ξ), (1.74)

and in a case of independent random variables (or independent states of the system)

ξ and η:

H(ξ, η) = H(ξ) + H(η). (1.75)

Example 1.18. In the class of continuous distributions f

(x) which belong to the

random variable ξ, to ﬁnd such, that for the given mathematical expectation m

and the variance σ

provides a maximum of the entropy H

(max H

It is given

∞

−∞

(x)dx = 1,

∞

−∞

(x)dx = m

Basic concepts of the probability theory 67

∞

−∞

(x − m

)

(x)dx = σ

To ﬁnd

(x) ⇒ max





−

∞

−∞

(x) log f

(x)dx





Finding

(x) is reduced to the solution of a variational problem

(x) ⇒ max





∞

−∞

Φ(f

(x), x)dx





with the side conditions

∞

−∞

(x), x)dx = c

(k = 1, 2, 3),

where

Φ(f

(x), x) = −f

(x) log f

(x),

= f

(x), c

= 1,

= xf

(x), c

= m

= (x − m

)

(x), c

= σ

To write down the Euler equations:

∂Φ

/∂f = 0,

(f, x) = −f log f + λ

f + λ

xf + λ

(x − m

)

After the diﬀerentiation we obtain the equation

−1 − log f + λ

+ λ

x + λ

(x − m

)

= 0,

with a solution relatively to f :

(x) = exp(λ

+ λ

x + λ

(x − m

)

− 1).

Using additional conditions

∞

−∞

exp(λ

+ λ

x + λ

(x − m

)

− 1)dx = 1,

∞

−∞

x exp(λ

+ λ

x + λ

(x − m

)

− 1)dx = m

∞

−∞

(x − m

)

exp(λ

+ λ

x + λ

(x − m

)

− 1)dx = σ

68 STATISTICAL METHODS OF GEOPHYSICAL DATA PROCESSING

we obtain

= 1 − ln(

√

2πσ

), λ

= 0, λ

= −(2σ

)

−1

and

(x) =

√

2π

exp



−

(x − m

)

2σ



Thus, the normal distribution corresponds to the peak entropy in a class of distri-

bution with the predetermined ﬁrst two moments.

The expression for the complete mutual information contained in two continuous

systems ξ and η, is similar to expression the (1.68) for a discrete case, but with the

replacement of the probabilities of discrete events by the distribution functions, can

be written as

η↔ξ

∞

−∞

ξη

(x, y) log

ξη

(x, y)

(x)f

(y)

dxdy,

or, using the expectation operator,

η↔ξ

= M [log(f

ξη

)] , (1.76)

where I

η↔ξ

is the nonnegative value, which is equal to zero only for independent

random variables ξ and η.

Example 1.19. Let us consider a simple model

η = ξ + ζ,

where η is a recorded signal, ξ is a transmitted signal with the normal distribution

N(m

= 0, σ

), ζ is an additive noise with the normal distribution N(m

, σ

), where

ξ and ζ are independent random variables.

To ﬁnd an amount of the information I

η↔ξ

about the signal ξ, which is contained

in the recorded signal η.

By deﬁnition we have

η↔ξ

= M



log

ξη



= M



log

η/ξ



= Mκ, κ = log

η/ξ

By problem situation σ

= σ

+ σ

(y) =

√

2π

+ σ

exp

(

−

(σ

+ σ

)

η/ξ

(y/x) =

√

2πσ

√

1 − r

exp

(

−

2(1 − r

)



− r



)

To ﬁnd the correlation coeﬃcient of η and ξ taking into account that η = ξ + ζ:

r =

M(η · ξ) =



M(ξ

) + M (ξζ)



Basic concepts of the probability theory 69

By substituting r into f

η/ξ

(y/x), we obtain

η/ξ

(y/x) =

√

2πσ

exp

(

−

(y − x)

2σ

)

√

2πσ

exp

(

−

2σ

)

We use the obtained relations for ﬁnding the quantity κ:

κ = log

+ σ

ln 2

2σ

−

2(σ

+ σ

)

Now it is necessary to calculate the mathematical expectation

Mκ = I

η↔ξ

= log

+ σ

ln 2

Mζ

2σ

−

Mη

2(σ

+ σ

)

Taking into account that Mζ

= σ

, Mη

= σ

+ σ

, we obtain

η↔ξ

= log

+ σ

So, the information on a signal ξ, contained in the recorded signal η, is equal

to the logarithm of the standard deviation of the signal η (σ

) to the standard

deviation of the noise ζ (σ

1.9.5 Fisher information

Let us a random variable ξ has the density function f

(x, θ), where θ is a parameter.

Let us assume that f

(x, θ) is diﬀerentiable on θ, then the Fisher information about

the unknown parameter θ, which is contained in a random variable ξ or in its density

function f

(x, θ), is treated as the diminution of indeterminacy about the unknown

value θ after an observation of the random variable ξ. The amount of the Fisher

information about θ, contained in a random variable ξ, is deﬁned as

(F )

(θ) = M[(∂ log f

(x, θ)/∂θ)

]. (1.77)

In a case of a random vector ξ = (ξ

, ξ

, . . . , ξ

) and a parametric vector θθ =

(θ

, θ

, . . . , θ

) the Fisher matrix with the dimension [S ×S] and the elements

(F )

(θθ) = M



∂ log f

ξξ

, x

, . . . , x

, θ)

∂θ

∂ log f

, x

, . . . , x

,θθ)

∂θ



(1.78)

is introduced.

The properties of the Fisher information.

(1) Let ξ

and ξ

are independent random variables, I

(F )

(θ), I

(F )

(θ) are the quan-

tities of information about θ, which are contained in the pair (ξ

, ξ

). Then the

following equality is valid

(F )

(θ) = I

(F )

(θ) + I

(F )

(θ).

70 STATISTICAL METHODS OF GEOPHYSICAL DATA PROCESSING

Proof.

(F )

(θ) = M



∂

∂θ

[log f

(x, θ) ·f

(x, θ)]



= M



∂ log f

(x, θ)

∂θ



+ M



∂ log f

(x, θ)

∂θ



+2M



∂ log f

(x, θ)

∂θ

∂ log f

(x, θ)

∂θ



= I

(F )

(θ) + I

(F )

(θ),

since



∂ log f

(x, θ)

∂θ

∂ log f

(x, θ)

∂θ



= M



∂ log f

(x, θ)

∂θ



· M



∂ log f

(x, θ)

∂θ



= 0.

(2) Let ξ

, ξ

, . . . , ξ

are independent, identically distributed random variables and

(F )

(θ) the amount of information about θ, which is contained in an each ran-

dom variable. Then the amount of information in ξ

, ξ

, . . . , ξ

is equal to

(F )

(θ). Analogously to the scalar case it is possible to show:

(1) If I

(F )

(θ) and I

(F )

(θ) are information matrices, which are connected with

the random variables ξ

and ξ

, then the information matrix for the pair of

random variables (ξ

, ξ

) is equal to

(F )

(

θ) = I

(F )

(

θ) + I

(F )

(

θ).

(2) If ξ

, ξ

, . . . , ξ

are independent, random variables with equal distributive

law and I

(F )

(θ

θ) are amount of information about

θ, which are contained in

an each random variable ξ

, then the amount of information contained in

, ξ

, . . . , ξ

is equal to nI

(F )

(θ

θ).

(3) Let us a function T = T (ξ) of a random variable ξ is given, then

(F )

(θ) −I

(F )

(θ) ≥ 0,

i.e. the matrix is nonnegative deﬁned and in a case of a single parameter θ

(F )

≥ I

(F )

i.e. the amount of information about θ which is contained in a random variable,

ever more or equal than the amount of information, which is contained in the

function of the random variable T = T (ξ).

When it is possible by using an observed value of a random variable with the

probability 1 to restore precisely a value of parameter θ, in this case we tell, that

the random variable contains the greatest possible information about parameter.

But if the distribution is a constant at all values of parameter, it is impossible to

Basic concepts of the probability theory 71

judge about the value of θ by the results of observations of the random variable.

The sensitivity of a random variable with respect to a parameter can be measured

by the quantity of the change of the distribution of this random variable at change

of the value of the parameter.

In the case of a vector parameter

θ a distance between distributions at transition

from (θ

, θ

, . . . , θ

) to (θ

+ δθ

, θ

+ δθ

, . . . , θ

+ δθ

) describes by a quadratic

form

(F )

(θ

θ)δθ

δθ

The formula (1.77) for the Fisher information can be rewritten as

(F )

(θ) = M



−

∂

log f

(x, θ)

∂θ



, (1.79)

since

∂

log f

∂θ

= −



∂f

∂θ



∂

∂θ

;



∂f

∂θ





∂ log f

∂θ





∂

∂θ



∞

−∞

(x, θ)

∂

(x, θ)

∂θ

∂

∂θ

∞

−∞

(x, θ)dx = 0.

In the case of the vector parameter (

θ), using analogous relations, we shall obtain

(F )

(

θ) = M



−

∂

log f (x, θ)

∂θ



. (1.80)

Example 1.20. Let us the random variable ξ belongs to the normal distribution

with the known variance σ

and the unknown mathematical expectation m

. To

ﬁnd the amount of information about m

, which is contained in a single observation

of ξ. The Fisher information about m

, which is contained in a single observation

is equal

(F )

) = −

∞

−∞

∂

∂m

−

(x − m

)

2σ

− ln σ

√

2π

√

2πσ

exp

−

(x − m

)

2σ

∞

−∞

√

2πσ

exp

−

(x − m

)

2σ

dx =

72 STATISTICAL METHODS OF GEOPHYSICAL DATA PROCESSING

For the case of n observations (x

, . . . , x

), we have

(F )

,...,ξ

(m) = −

∞

−∞

. . .

∞

−∞

∂

∂m

i=1

−

− m

)

l ln(σ

√

2π)

i=1

√

2πσ

exp

−

− m

)

. . . dx

1.9.6 Exercises

(1) To ﬁnd the entropy of the random variable ξ, which corresponds to a phase of

a seismic signal, on condition that ξ belongs to the uniform distribution at the

interval (−π/2, π/2).

(2) To ﬁnd the entropy of the random variable ξ, which corresponds to an ampli-

tude of a seismic signal, on condition that ξ belongs to the normal distribution

N(0, σ

(3) Let us specify a state of a seismic signal by the three values: an amplitude ξ

belonging to the normal distribution N(0, σ

); a phase ξ

, belonging to the

uniform distribution at an interval (0, π); an arrival time, belonging to the uni-

form distribution (0, T ). To ﬁnd the entropy of the system, which corresponds

to the seismic signal.

(4) At an urn is contained ﬁve white and six black spheres. Five spheres have been

extracted from the urn, they are three black and two white ones. To ﬁnd the

information contained at the observation of an event A

with respect to an

event A

: the next extracted sphere will have a black colour.

(5) At one square of chess-board a chess piece is situated. A priori any arrangement

of the chess piece is supposed. To ﬁnd the information, which contains in a

statement about the location of the chess piece.

1.10 Random Functions and its Properties

The function ξ(t) is called the random function of the argument t, which value at

any value t is a random variable. The argument t is considered as a nonrandom

value.

In many geophysical problems the argument t is the time, for example, at a

record of seismic waves (seismic trace) or record of an electromagnetic ﬁeld in a

given spatial point. Therefore in the further the argument t we shall call as time.

Usually two types of argument of a random function are considered

(1) The argument of a random function t can possess any values in a given interval,

which can be ﬁnite or inﬁnite.

(2) The argument of a random function t can possess only deﬁnite discrete values.