Klir G.J. Uncertainity and Information. Foundations of Generalized Information Theory

Подождите немного. Документ загружается.

less complex than simpliﬁcations Z

, Z

. Among them, simpliﬁcation Z

has

the smallest diagnostic uncertainty, S(Y | X

) = 0.822. The loss of information

with respect to Z is

and with respect to Z

The solution set (including the given system Z) is:

EXAMPLE 9.2. A relation R among variables x

, x

is deﬁned by the set

of possible triples listed in Table 9.2a. Observe that the state sets of the vari-

ables are, respectively, X

= {0, 1} and X

= X

= {0, 1, 2}. The relation may be

utilized for providing us with useful information about the state of any one of

the variables, given a joint state of the remaining two variables. Relevant

amounts of uncertainty for this purpose, which are expressed in this case by

the Hartley functional, are shown in Table 9.2b under the heading R. Also

shown in the table are the associated amounts of information.

The aim of this example is to illustrate the role of the principle of minimum

uncertainty in choosing simpliﬁcations based on the quantization of variables

and x

. Assuming that the states in X

and X

are ordered, each of the sets

can be quantized by using one of the following functions:

When we apply function q

to both X

and X

in relation R, we obtain the sim-

pliﬁed relation R

shown in Table 9.2a, where duplicate states are crossed out.

By applying the other combinations of functions q

and q

, we obtain the other

simpliﬁed relations in Table 9.2a (R

, R

, and R

), where each subscript indi-

cates the combination employed. For each of the four simpliﬁed relations, rel-

evant amounts of uncertainty and information are given in Table 9.2b. Observe

that the simpliﬁcations with the minimum amount of uncertainty (indicated

in the table by bold entries) are also correctly the ones with the maximum

amount of information (or with minimum loss of information with respect to

R). For variable x

, any of relations R

, R

, and R

qualiﬁes as the best sim-

pliﬁcation; for variable x

, R

is the only minimum-uncertainty simpliﬁcation;

for x

, both R

and R

are minimum-uncertainty simpliﬁcations. Observe that

does not preserve any information of R with respect to variable x

00 00

10 11

21 21

ÆÆ

SOL

ZZ Z=

{}

,, .

IY X IY X

0 103

()

= ..

IY X IY X

()

0 155.,

362 9. METHODOLOGICAL ISSUES

Table 9.2. Preferred Simpliﬁcation Determined by the Principle of Minimum Uncertainty (Example 9.2)

001000001000001

002001001001001

010000000010010

011000001010011

020010010010010

021010011010011

100100100100100

110100100110110

111100101110111

112101101111111

(a)

H(X

¥ X

) log

10 = 3.322 log

5 = 2.321 log

6 = 2.585 log

6 = 2.585

H(X

¥ X

) log

5 = 2.321 log

3 = 1.585 log

4 = 2.000 log

4 = 2.000

H(X

¥ X

) log

6 = 2.585 log

4 = 2.000 log

4 = 2.000

H(X

¥ X

) log

8 = 3.000 log

3 = 1.585 log

4 = 2.000 log

4 = 2.000

H(X

¥ X

) 0.322 0.736 0.585 0.585 0.585

H(X

¥ X

) 0.737 0.321 0.585 0.585 0.585

H(X

¥ X

) 1.000 0.736 1.000 0.585 0.585

I(X

¥ X

) 0.678 0.264 0.415 0.415 0.415

I(X

¥ X

) 0.848 0.679 0.415 0.415 0.415

I(X

¥ X

) 0.585 0.264 0.000 0.415 0.415

(b)

363

When departing from the two classical uncertainty theories, several options

open for applying the principle of minimum uncertainty. Using Figure 6.15 as

a guide, the following options can be identiﬁed:

1. To minimize the generalized Hartley functional GH.

2. To minimize the aggregated uncertainty S

3. To minimize either of the total uncertainty components in TU or

TU.

4. To consider both components in TU or in

TU and replace a single uncer-

tainty preference ordering with two preference orderings, one for each

component.

Among these alternatives, the ﬁrst one seems conceptually the most funda-

mental. This alternative, which may be called a principle of minimum non-

speciﬁcity, guarantees that the imprecision in probabilities does not increase

more than necessary when we simplify a system to some given level of com-

plexity. This alternative is also computationally attractive since the functional

to be minimized (the generalized Hartley measure) is a linear functional. The

aim of the following example is to illustrate some of the other alternatives.

EXAMPLE 9.3. Consider a system Z with one input variable, x

, and one

output variable, x

, in which the relationship between the variables is expressed

in terms of the joint interval-valued probability distribution given in Table

9.3a. To illustrate the principle of minimum uncertainty, let us consider two

simpliﬁcations of the systems by quantizing the input variable via either func-

tion q

or function q

introduced in Example 9.2. Interval-valued probability

distributions of the two simpliﬁcations, Z

and Z

, are shown in Table 9.3b.They

are also shown with their marginals in Figure 9.2.Their complete formulations,

including the Möbius representations, are in Table 9.4. Subsets A of the Carte-

sian product {0, 1}

are deﬁned by their characteristic functions. Relevant con-

ditional uncertainties (since x

is an input variable and x

is an output variable)

are given in Table 9.3c. They are calculated by the differences between the

uncertainties on X

¥ X

(shown in Table 9.4) and the uncertainties on X

(shown in Figure 9.2). We can conclude that (a) Z

is preferred according to S

and generalized Shannon (GS); (b) Z

is preferred according to GH; and (c)

both Z

and Z

are accepted in terms of TU =·GH, GSÒ, since they are not

comparable in terms of the joint preference ordering.

9.2.2. Conﬂict-Resolution Problems

Another application of the principle of minimum uncertainty is the area of

conﬂict-resolution problems. For example, when we integrate several overlap-

ping subsystems into one overall system, the subsystems may be locally incon-

sistent in the following sense. An overall system composed of subsystems is

said to be locally inconsistent if it contains at least one pair of subsystems that

share some variables and whose uncertainty functions project to distinct mar-

364 9. METHODOLOGICAL ISSUES

9.2. PRINCIPLE OF MINIMUM UNCERTAINTY 365

ginal uncertainty functions based on the shared variables. An example of

locally inconsistent probabilistic subsystems is shown in Figure 9.3a.

Local inconsistency among subsystems that form an overall system is a kind

of conﬂict. If it is not resolved, the overall system is not meaningful. For

example, the overall system in Figure 9.3a is not meaningful because no joint

probability distribution function exists on X ¥ Y ¥ Z that is consistent with

both probability distribution functions

p of the two given subsystems.

Two attitudes toward locally inconsistent collections of subsystems can be

recognized. According to one of them, such collections should be rejected on

the basis of the fact that they do not represent any overall systems. According

to the other attitude, the local inconsistencies should be resolved by modify-

ing the given uncertainty functions of the subsystems to achieve their consis-

tency. However, this can usually be done in numerous ways. The right way to

do that, on epistemological grounds, is to obtain the consistency with the small-

est possible total loss of information contained in the given uncertainty func-

tions. The total loss of information is expressed by the sum of information

losses for the individual subsystems. Thus, resolving local inconsistency can be

formulated, in generic terms, as the following optimization problem, where

Table 9.3. Illustration to Example 9.3

(a) Given System Z

, x

) m

, x

)

0 1 0.1 0.3

0 1 0.3 0.4

1 0 0.0 0.0

1 1 0.2 0.4

2 0 0.0 0.2

2 1 0.2 0.4

(b) Simpliﬁcations Z

and Z

of System Z

(s) m

(s)

0 1 0.1 0.3 0.1 0.3

0 1 0.5 0.7 0.3 0.4

1 0 0.0 0.2 0.0 0.2

1 1 0.2 0.4 0.4 0.6

and Z

| X

) = 0.814 S

| X

) = 0.871

GH(X

| X

) = 0.200 GH(X

| X

) = 0.158

GS(X

| X

) = 0.614 GS(X

| X

) = 0.713

û) denotes the loss of information when uncertainty function

u is

replaced with uncertainty function

û.

Given a family of subsystems {

Z | s Œ⺞

} whose uncertainty functions,

u (s

Œ⺞

), formalized in uncertainty theory T, are locally inconsistent, determine

locally consistent counterparts of these uncertainty functions,

û (s Œ ⺞

), for

which the functional

reaches its minimum subject to the following constraints:

(a) Axioms of theory T.

(b) Conditions of local consistency of uncertainty functions

û(s Œ ⺞

When for example, this optimization problem is formulated within proba-

bility theory, then

u and

û are probability distribution functions: (a) are

axioms of probability theory, (b) are linear algebraic equations, and

Iuu ux

ss s

log

()

Iuu

()

366 9. METHODOLOGICAL ISSUES

Table 9.4. Complete Formulation of the Two Simpliﬁcations Discussed in Example 9.3

00 01 10 11 m

(A) m

(A)

A: 0 0 0 0 0.0 0.0 0.0 0.0 0.0 0.0

1 0 0 0 0.1 0.3 0.1 0.1 0.3 0.1

0 1 0 0 0.5 0.7 0.5 0.3 0.4 0.3

0 0 1 0 0.0 0.2 0.0 0.0 0.2 0.0

0 0 0 1 0.2 0.4 0.2 0.4 0.6 0.4

1 1 0 0 0.6 0.8 0.0 0.4 0.6 0.0

1 0 1 0 0.1 0.3 0.0 0.1 0.3 0.0

1 0 0 1 0.3 0.5 0.0 0.5 0.7 0.0

0 1 1 0 0.5 0.7 0.0 0.3 0.5 0.0

0 1 0 1 0.7 0.9 0.0 0.7 0.9 0.0

0 0 1 1 0.2 0.4 0.0 0.4 0.6 0.0

1 1 1 0 0.6 0.8 0.0 0.4 0.6 0.0

1 1 0 1 0.8 1.0 0.0 0.8 1.0 0.0

1 0 1 1 0.3 0.5 0.0 0.6 0.7 0.1

0 1 1 1 0.7 0.9 0.0 0.7 0.9 0.0

1 1 1 1 1.0 1.0 0.2 1.0 1.0 0.1

GH(X

¥ X

) = 0.400 GH(X

¥ X

) = 0.358

¥ X

) = 1.785 S

¥ X

) = 1.871

GS(X

¥ X

) = 1.385 GS(X

¥ X

) = 1.513

=·0.4, 1.385Ò TU

=·0.358, 1.513Ò

where X

is the state set of subsystem Z

, is the directed divergence introduced

in Section 3.3 (see Eq. (3.56)).

The appearance of local inconsistencies among subsystems of an overall

system is an indicator that the claims expressed by the uncertainty functions

of the subsystems are not fully warranted under the given evidence.Thus, mod-

ifying them in a minimal way (with minimum loss of information) to achieve

their consistency, as facilitated by the described optimization problems, is an

epistemologically sound conﬂict-resolution strategy.

EXAMPLE 9.4. To illustrate the described optimization problem by a spe-

ciﬁc example, let us consider the two locally inconsistent subsystems described

9.2. PRINCIPLE OF MINIMUM UNCERTAINTY 367

[0.0, 0.2]

[0.1, 0.3]

0 1

1110

0100

[0.7, 0.9]

[0.4, 0.6]

[0.1, 0.3] [0.3, 0.4]

) = 1

GH(X

) = 0.2

GS(X

) = 0.8

[0.4, 0.6]

[0.1, 0.8]

1110

0100

[0.7, 0.9]

[0.2, 0.4] [0.0, 0.2]

[0.1, 0.3] [0.5, 0.7]

S (X

) = S(0.6, 0.4)

= 0.971

GH(X

) = 0.2

GS(X

) = 0.971 – 0.2

= 0.771

[0.6, 0.8]

[0.2, 0.4]

Figure 9.2. Two simpliﬁcations discussed in Example 9.3.

in Figure 9.3a. Denoting, for convenience,

p(x

, y

) =

p(z

, y

) =

pˆ(x

) =

pˆ

, and

pˆ(z

, y

) =

pˆ

for all i, j, k Œ {1, 2}, the optimization problem in

this case has the following form.

Given probabilities

and

(i, j, k Œ {1, 2}), determine the values of

pˆ

and

pˆ

(i, j, k Œ {1, 2}) for which the functional

reaches its minimum under the following constraints:

(c1)

pˆ

= 1

(c2)

pˆ

= 1

(c3)

pˆ

≥ 0 for all i, j Œ {1, 2}

(c4)

pˆ

≥ 0 for all k, j Œ {1, 2}

(c5)

pˆ

(c6)

pˆ

Constraints (c1)–(c4) capture axioms of probability theory; constraints (c5)

and (c6) specify conditions for local consistency of the two subsystems.

jkji

log

====

ÂÂÂÂ

368 9. METHODOLOGICAL ISSUES

p(x

, y

)

0.5 0.1

0.2 0.2

)

0.6

0.65 y

0.4

0.35 y

p(z

, y

)

0.4 0.25 y

0.15 0.2 y

Subsystem Z

Inconsistent marginals

Consistent marginals

(a)

)

) Y

0.625

0.375

),(

0.3846 0.2404

0.1632 0.2118

Subsystem

(b)

, y

)

0.5208 0.1042

0.1875 0.1875

Figure 9.3. Resolving local inconsistency of subsystems (Example 9.4). (a) Locally inconsis-

tent subsystems Z

and Z

; (b) consistent subsystem Z

and Z

obtained by minimizing the loss

of information in given subsystems Z

and Z

The objective functional in the optimization problem, which measures the total

loss of information for each modiﬁcation of the probability distribution func-

tions of subsystems Z

and Z

, is within the given constraints positive (due to

the Gibbs inequality) and convex. Hence, the optimization problem has a

unique solution, which is given in Figure 9.3b.The loss of information in resolv-

ing the local inconsistency in this example is 0.0021.

9.3. PRINCIPLE OF MAXIMUM UNCERTAINTY

The principle of maximum uncertainty allows us to develop epistemologically

sound procedures for dealing with a wide variety of problems that involve

ampliative reasoning—any reasoning in which conclusions are not entailed in

the given premises.

Ampliative reasoning is indispensable to science in a variety of ways.

For example, whenever we utilize a given system for predictions, we employ

ampliative reasoning. Similarly, when we want to estimate microstates from

the knowledge of relevant macrostates and partial information regarding the

microstates (as in image processing and many other problems), we must resort

to ampliative reasoning.The problem of the identiﬁcation of an overall system

from some of its subsystems is another example that involves ampliative

reasoning.

Ampliative reasoning is also common and important in our daily lives,

where, unfortunately, the principle of maximum uncertainty is not always

adhered to. Its violation leads almost invariably to conﬂicts in human com-

munication, as was well expressed by Bertrand Russell in his Unpopular

Essays [1950]:

[W]henever you ﬁnd yourself getting angry about a difference in opinion, be on

your guard; you will probably ﬁnd, on examination, that your belief is getting

beyond what the evidence warrants.

9.3.1. Principle of Maximum Entropy

The principle of maximum uncertainty is well developed and broadly utilized

within classical information theory, where it is called the principle of maximum

entropy. It is formulated, in generic terms, as the following optimization

problem: determine a probability distribution ·p(x) | x ŒX Ò that maximizes

the Shannon entropy subject to given constraints c

, c

,...,c

, which express

partial information about the unknown probability distribution, as well as the

general constraints (axioms) of probability theory.The most typical constraints

employed in practical applications of the maximum entropy principle are

mean (expected) values of one or more random variables or various marginal

probability distributions of an unknown joint distribution.

9.3. PRINCIPLE OF MAXIMUM UNCERTAINTY 369

As an example, consider a random variable x with possible (given) non-

negative real values x

, x

,..., x

. Assume that probabilities p

of values

(i Œ ⺞

) are not known, although we do know the mean (expected) value

E(x) of the variable. Employing the maximum entropy principle, we estimate

the unknown probabilities p

, i Œ ⺞

, by solving the following optimization

problem, in which values x

(i Œ ⺞

) and their expected value are given and

probabilities p

(i Œ ⺞

) are to be determined.

Maximize the functional

subject to the constraints

(9.1)

and

(9.2)

For the sake of simplicity, in this formulation we use the natural logarithm

in the deﬁnition of Shannon entropy. Clearly, the solution to this problem is

not affected by changing the base of the logarithm in the objective functional.

Equation (9.1) represents the available information; Eq. (9.2) represents the

standard constraints imposed on p by probability theory.

First, we form the Lagrange function

where a and b are the Lagrange multipliers that correspond to the two con-

straints. Second, we form the partial derivatives of L with respect to p

(i Œ

⺞

), a, and b, and set them equal to zero; this results in the equations

∂

px i

Ex px

ii n

=- - - - = Œ

=- =

()

for each ⺞

Lpp p pxEx

=- - -

()

===

ÂÂÂ

ln ,ab

111

pi p

in i

≥Œ

()

⺞ ,.

Ex px

()

Sp p p p p

nii

, ,..., ln

()

370 9. METHODOLOGICAL ISSUES

The last two equations are exactly the same as the constraints of the opti-

mization problem. The ﬁrst n equations can be written as

When we divide each of these equations by the sum of all of them (which must

be one), we obtain

(9.3)

for each i Œ ⺞

. In order to determine the value of b, we multiply the ith

equation in Eq. (9.3) by x

and add all of the resulting equations, thus

obtaining

and

Multiplying this equation by e

bE(x)

results in

(9.4)

This equation must now be solved (numerically) for b and the solution

substituted for b in Eq. (9.3), which results in the estimated probabilities

(i Œ ⺞

EXAMPLE 9.5. To illustrate this application of the maximum entropy prin-

ciple, ﬁrst consider an “honest” (unbiased) die. Here x

= i for i Œ ⺞

and

E(x) = 3.5. Equation (9.4) has the form

--- + + + =

---

25 15 05 05 15 25 0

25 15 05 05 15 25

.... . . .

......

eeee e e

bbbbbb

xExe

xEx

()

[]

()

[]

xe E x e

()

ÂÂ

()=

pe e e

-- -

()

-- -

()

-- -

()

9.3. PRINCIPLE OF MAXIMUM UNCERTAINTY 371