Schulz M. Control Theory in Physics and Other Fields of Science: Concepts, Tools, and Applications
Подождите немного. Документ загружается.
7.3 Feedback Control 207
V (Y,τ, T)=V (X, t, T)
X(τ )=Y,u
−
t
τ
dt
∂
∂t
V (X, t
,T)
X(τ )=Y,u
−
t
τ
dt
'
F (X, u, t
)V (X, t
,T)
X(τ )=Y,u
. (7.59)
On the other hand, we may consider a control
u(t
,X)=
u(t
,X)forτ ≤ t
≤ t
u
∗
(t
,X)fort ≤ t
≤ T.
(7.60)
Thus, because of (7.48), we now obtain
J[Y, τ, u, T ]=
t
τ
dt
dXφ(t
,X,u)p
u
(X, t
| Y,τ)
+
T
t
dt
dX
φ(t
,X
,u
∗
)
×
dXp
u
∗
(X
,t
| X, t)p
u
(X, t | Y,τ)
=
t
τ
dt
φ(t
,X,u)
X(τ )=Y,u
+
T
t
dt
φ(t
,X
,u
∗
)
X
(t)=X,u
∗
X(τ )=Y,u
=
t
τ
dt
φ(t
,X,u)
X(τ )=Y,u
+ J[X, t, u
∗
,T]
X(τ )=Y,u
=
t
τ
dt
φ(t
,X,u)
X(τ )=Y,u
+ V (X, t,T )
X(τ )=Y,u
, (7.61)
where we have used in the last step (7.51) and in the previous step (7.48).
Because of the fact that u is not the optimal control, we obtain the inequality
V (Y,τ,T) ≤ J[Y,τ, u, T ]or
V (Y,τ, T) ≤
t
τ
dt
φ(t
,X,u)
X(τ )=Y,u
+ V (X, t,T )
X(τ )=Y,u
. (7.62)
Considering (7.59)weget
208 7 Optimal Control of Stochastic Processes
0 ≤
t
τ
dt
φ(t
,X,u)
X(τ )=Y,u
+
t
τ
dt
∂
∂t
V (X, t
,T)
X(τ )=Y,u
+
t
τ
dt
'
F (X, u, t
)V (X, t
,T)
X(τ )=Y,u
. (7.63)
The equality holds if u = u
∗
. Finally, we write t = τ + ε, divide by ε and take
the limit ε → 0. Thus, we obtain
0 ≤
∂
∂τ
V (Y,τ,T)+φ(τ,Y,u)+
'
F (Y,u, τ)V (Y,τ,T) . (7.64)
Since the first term does not depend on u, the equality is reached if the
sum of the second and third terms reaches its minimum. This is, of course,
equivalent to the requirement u = u
∗
, i.e., we obtain the equation of the
feedback stochastic optimal control
∂
∂t
V (X, t, T) + min
u
φ(t, X, u)+
'
F (X, u, t)V (X, t, T )
=0, (7.65)
where we have replaced Y → X and τ → t. On the other hand, the optimal
control follows from the relation
u
∗
= arg min
u
φ(t, X, u)+
'
F (X, u, t)V (X, t, T )
. (7.66)
We remark that (7.65) is the extension of the Hamilton–Jacobi equation
to stochastic processes. In fact, if we identify V with the action S
and consider that the derivatives now concern the initial state and the initial
time instead of the final state and the final time, we obtain the mapping
V → S
∂V
∂t
→−
∂S
∂t
∂V
∂X
→−
∂S
∂X
(7.67)
and therefore with (7.56) considering D =0
∂S
∂t
+ max
u
F
∂S
∂X
− φ
=
∂S
∂t
+ max
u
H(X,
∂S
∂X
,u,t) = 0 (7.68)
with the Hamiltonian H given by (2.94).
The above-derived control equation (7.65) must be completed by the cor-
responding boundary conditions. Because of definition (7.48) we immediately
get
V (Y,T,T)=0. (7.69)
Further boundary conditions follow from the Fokker–Planck equation. For ex-
ample, absorbing boundary conditions for X = X
0
requires P (X, t | X
0
,τ)=0
and, therefore, V (X
0
,τ,T)=0.
Let us illustrate the treatment of the control equation by a simple ex-
ample. We consider the bounded diffusion of a particle under the drift force
(2.139))
7.3 Feedback Control 209
F = −(u
1
− 1)X − u
2
with u
2
≥ 0andu
1
∈ R, while the noise is defined by
u
1
σXdW . Thus, we have the Ito stochastic differential equation
dX =[(1−u
1
) X − u
2
]dt + u
1
σXdW (7.70)
with the two-component control (u
1
,u
2
). Thus we obtain the diffusion co-
efficient D =(u
1
σX)
2
. In principle, the model corresponds to an Ornstein-
Uhlenbeck process in a space-dependent temperature field
8
. Furthermore, we
require adsorbing boundary conditions at X = 0. The performance to be
minimized is the functional
J [Y, t,u, T]=
T
t
dt
dXu
γ
2
(t
)p
u
(X, t
| Y,t) (7.71)
with γ>1, i.e., we are interested in a maximum escape rate over the border
X = 0 at a minimum external force u
2
. Thus, we have the boundary conditions
V (0,t,T)=0 and V (Y,T,T)=0. (7.72)
The control equation is now given by
∂V
∂t
+ min
u
u
γ
2
+[(1− u
1
) X − u
2
]
∂V
∂X
+
1
2
(u
1
σX)
2
∂
2
V
∂X
2
. (7.73)
From here, we find the pre-optimized control functions
u
γ−1
2
= γ
−1
∂V
∂X
and u
1
=
1
σ
2
X
∂V
∂X
∂
2
V
∂X
2
−1
. (7.74)
We use the ansatz V (X, t, T )=g(t)X
γ
which satisfies the first boundary
condition and which yields u
2
= g(t)
1/(γ−1)
X and u
1
= σ
−2
/ (γ − 1), and
therefore
g
+(1− γ)g
γ/(γ−1)
+
γg
(γ − 1) σ
2
(γ − 1) σ
2
−
1
2
= 0 (7.75)
with the solution
g(t)=
C
1 − exp
:
γ
2(γ − 1) σ
2
− 1
t − T
2(γ − 1)
2
σ
2
;
1−γ
(7.76)
with
C =
2(γ − 1)
2
σ
2
γ [2 (γ − 1) σ
2
− 1]
(7.77)
also satisfying the second boundary condition. Thus, we have a constant con-
trol law for u
1
while the second control law is given by
8
Note that the Nerst–Einstein relation requires that the diffusion coefficient and
the temperature are proportional.
210 7 Optimal Control of Stochastic Processes
u
2
=
C
1 − exp
:
γ
2(γ − 1) σ
2
− 1
t − T
2(γ − 1)
2
σ
2
;
−1
X (7.78)
with singular behavior, u
2
∼ X(T − t)
−1
for t → T .
Finally, it should be remarked that functional (7.48), which represents the
Lagrange formulation of a given control problem, can be also extended to the
Bolza formulation with
J[Y, τ,u,T]=
T
τ
dt φ(t, X(t),u(t))
X(τ )=Y
+ Ψ [X(T )]
X(τ )=Y
(7.79)
or to the Meier formulation
J[Y, τ,u,T]=
Ψ [X(T )]
X(τ )=Y
. (7.80)
In both cases we have the boundary conditions
V (Y,T,T)=Ψ [Y ] . (7.81)
While in the case of (7.79) the stochastic control equation (7.65) is still valid,
the Meier case (7.80) requires the substitution of φ = 0 in the control equation.
7.3.2 Linear Quadratic Problems
Let us now consider a linear problem defined by the stochastic Ito equation
dX(t)=[A(t)X(t)+B(t)u(t)] dt +
R
k=1
d
k
(t)dW
k
(t) (7.82)
and the expected system performance
J[Y, τ,u,T]
=
1
2
T
τ
dt
X(t)Q(t)X(t)
X(τ )=Y
+ u(t)R(t)u(t)
X(τ )=Y
+
1
2
X(t)ΩX(t)
X(τ )=Y
(7.83)
with the symmetric (and usually positive definite) matrices Q(t)(typeN ×N)
R(t)(typen × n)andΩ (type N × N ). Then, the optimal control equation
becomes
0=
∂
∂t
V (X, t, T)
+ min
u
1
2
XQ(t)X +
1
2
uR(t)u +
'
F (X, u, t)V (X, t, T )
(7.84)
with
References 211
'
F (X, u, t)=
XA
T
(t)+uB
T
(t)
∂
∂X
+
1
2
∂
∂X
D(t)
∂
∂X
. (7.85)
Thus, the pre-optimized control is given by
u
(∗)
(t)=−R(t)
−1
B
T
(t)
∂V (X, t, T)
∂X
, (7.86)
and the control equation now reads
0=
∂V
∂t
+
1
2
XQ(t)X −
1
2
∂V
∂X
B(t)R(t)
−1
B
T
(t)
∂V
∂X
+ XA
T
(t)
∂V
∂X
+
1
2
∂
∂X
D(t)
∂
∂X
V. (7.87)
We use the ansatz
V (X, t, T)=
1
2
[XG(t)X + V
0
(t)] (7.88)
with the symmetric N ×N matrix G and obtain
0=X
˙
GX +
˙
V
0
+ XQX − XGBR
−1
B
T
GX
+ XA
T
GX + XGAX +trDG . (7.89)
All terms of order X
2
yield the Riccati equation
˙
G + A
T
G + GA − GBR
−1
B
T
G = −Q (7.90)
with the boundary condition G(T )=Ω. The remaining relation is
˙
V
0
= −trDK . (7.91)
The solution of the Ricatti equation (7.90) now allows us to formulate the
complete control law from (7.86)
u
∗
(t)=−R(t)
−1
B
T
(t)G(t)X
∗
(t) , (7.92)
while X
∗
(t) is a solution of the linear differential equation (7.82) considering
(7.92). Thus, the control law of the stochastic feedback control of a linear
quadratic problem is completely equivalent to the control low of the deter-
ministic control of linear quadratic problems. The effects of noise are only
considered in the function V
0
(t) while G(t) is not affected by D(t), neither is
u
∗
(t). The only difference is the minimum expected performance V (X, t, T)
which differs from the minimum performance of the deterministic model by
the term V
0
.
References
1. B.J. Ford: Biologist 39, 82 (1992) 193
2. A. Einstein, Annalen der Physik 17, 132 (1905) 193
3. C. Bender, S.A. Orszag: Advanced Mathematical Methods for Scientists and
Engineers (McGraw-Hill, New York, 1978) 201
212 7 Optimal Control of Stochastic Processes
4. J. Zinn-Justin: Quantum Field Theory and Critical Phenomena (Claredon Press,
Oxford, 1990) 197, 201
5. C. Grosche, F. Steiner: Handbook of Feynman Path Integrals (Springer, Berlin
Heidelberg New York, 1998) 201
6. H. Kleinert: Path Integrals in Quantum Mechanics, Statistics, Polymer Physics,
and Financial Markets (World Scientific Publishing, Singapore, 2004) 197, 201
7. C. Holland: ‘Small Noise Open Loop Control’, SIAM J. Control 12, 380 (1974). 204
8. C. Holland: ‘Gaussian Open Loop Control Problems’, SIAM J. Control 13,545
(1975). 204
9. V. Warfield: A stochastic maximum principle. PhD Thesis, Brown University,
Providence, RI (1971) 204
10. A. Ranfagni, P. Moretti, D. Mugnai: Trajectories and Rays: The Path-
Summation in Quantum Mechanics and Optics (World Scientific Publishing,
Singapore, 1991) 197
11. W.H. Fleming: Deterministic and Stochastic Optimal Control (Springer, Berlin
Heidelberg New York, 1975) 204
12. J.H. Davis: Foundations of Deterministic and Stochastic Control (Birkh¨auser,
Basel, 2002) 204
13. R. Gabasov, F.M. Kirillova, S.V. Prischepova: Optimal Feedback Control
(Springer, Berlin Heidelberg New York, 1995) 204
14. T. Chen, B. Francis: Optimal Sampled Data Control (Springer, Berlin Heidelberg
New York, 1995)
8
Filters and Predictors
8.1 Partial Uncertainty of Controlled Systems
Suppose we have a system under control, described by dynamical equations
of motion for the N-dimensional state vector X(t), and we have obtained an
optimal deterministic control curve trajectory X
∗
(t) and the corresponding
optimum control u
∗
(t) by the methods described in Chap. 7 by neglecting all
noise terms, then we may write the desired evolution equation:
˙
X
∗
(t)=F (X
∗
(t),u
∗
(t),t) . (8.1)
On the other hand, the real system considering the influence of the stochastic
evolution equations may be described by the Ito stochastic differential equa-
tion (7.23). The noise terms always generate deviations of the real trajectory
X(t) from the nominal behavior, Y (t)=X(t) −X
∗
(t), which require the con-
trol u(t) instead of the nominal control u
∗
(t) in order to keep the deviations
Y (t) small. Considering (7.23) and (8.1), we obtain the evolution equation:
dY =[F(X
∗
+ Y,u
∗
+ w, t) − F (X
∗
,u
∗
,t)] dt
+
R
k=1
d
k
(X
∗
+ Y,u
∗
+ w, t)dW
k
(t) . (8.2)
For small w and Y we may use the linearized stochastic evolution equation
dY (t)=[A(t)Y (t)+B(t)w(t)] dt +
R
k=1
d
k
(t)dW
k
(t) (8.3)
with
A(t)=
∂F(X
∗
,u
∗
,t)
∂X
∗
B(t)=
∂F(X
∗
,u
∗
,t)
∂u
∗
(8.4)
and
d
k
(t)=d
k
(X
∗
,u
∗
,t) . (8.5)
M. Schulz: Control Theory in Physics and other Fields of Science
STMP 215, 213–264 (2006)
c
Springer-Verlag Berlin Heidelberg 2006
214 8 Filters and Predictors
Thus, one obtains, together with the expansion of the performance up to the
second-order in Y and w, a stochastic linear quadratic problem which we
have discussed in Sect. 7.3.2. In particular, one obtains the control law (7.92)
presenting the classical linear feedback relation.
However, the application of this theory to real problems causes some new
problems. The first problem belongs to the stochastic sources which drive
the system under control. It is often impossible to determine the coupling
functions d
k
(t), which connect the system dynamics with the noise processes.
In addition, it cannot be guaranteed that a real system is described exclusively
at the Markov level by pure diffusion processes related to several realizations
of the Wiener process. In principle, the stochastic terms may also represent
various jump processes or combined diffusion-jump processes.
1
Since the majority of physical processes in complex systems consist of a
sufficiently large number of different external noise sources, the estimation of
the noise terms can be made in the framework of the limit distributions. This
will be done in the following parts of this chapter.
The second problem belongs to the observability of a system. It means
that we have the complete information about the stochastic dynamics of the
system, given by the sum of the noise terms and the matrices A(t)andB(t),
but we are not able to measure the state X(t) or equivalently the difference
Y (t)=X(t) − X
∗
(t). Instead of this, we have only a reduced piece of infor-
mation given by the observable output
Z(t)=C(t)X(t)+η(t) , (8.6)
where the output Z(t) is a vector of p components, usually with p<N,
C(t) is a matrix of type p × N,andη(t) represents the p-component noise
process modelling the observation error. The problem occurs if the reduced
information Z(t) and all previous observations Z(τ) with τ<tcanbeusedfor
the control of the system at the current time t. Such so-called filter problems
will be considered in the two subsequent chapters.
Finally, it may be possible that the dynamics of the system is unknown.
The only available information is the historical set of observations and control
functions while the system itself behaves like a black box. In this case it is
necessary to estimate the most probable evolution of the system under control.
1
We remark that the stochastic Ito differential equation is related to a Fokker–
Planck equation, which is only a special case of the differential Chapman–
Kolmogorov equation (6.91). This equation is valid for all Markov processes and
considers also jump processes. Thus we may reversely conclude that (8.3) can also
be generalized to combined diffusion-jump processes.
8.2 Gaussian Processes 215
8.2 Gaussian Processes
8.2.1 The Central Limit Theorem
Let us first analyze the properties of the stochastic contributions to a control
problem if a detailed characterization of the noise terms is no longer possible.
In other words, if we consider the N-component noise vector
dξ(t)=
R
j=1
d
j
(t)dW
j
(t)=
R
j=1
dξ
j
(t) (8.7)
or the discrete version
ξ(t)=
R
j=1
t+∆t
t
d
j
(t
)dW
j
(t
)=
R
j=1
ξ
j
(t) , (8.8)
then dξ(t)andξ(t), respectively, can be interpreted as a sum of R indepen-
dent random quantities dξ
j
(t)andξ
j
(t), respectively. In future we consider
the discrete representation (8.8). The extension to infinitesimal small changes
dξ
j
(t) is always possible. Formally, each ξ
j
represents an event of a stochastic
process realized with the probability distribution p
(j)
(ξ
j
). We remark that
the events ξ
j
must not be a weighted realization of the Wiener process. It is
also possible to extend dW
j
in (8.8) to arbitrary independent diffusion-jump
processes [1, 27, 28] corresponding to the differential Chapman–Kolmogorov
equation.
In order to characterize the stochastic processes driving the actual system,
we need only the probability distribution of the sum ξ(t) while the knowledge
of the distribution functions of the single events is a secondary information.
The number of these independent noise terms becomes very large for the
majority of complex systems. On the other hand, the often-observed lack of
distinguished features between the elementary noise processes and the ab-
solute equality of the single noise terms ξ
j
(t)in(8.7) and (8.8) gives rise to
the reasonable assumption that all the events ξ
j
(t) are realized with the same
probability distribution function, p
(j)
(ξ
j
)=p(ξ
j
). For the sake of simplicity,
we will use this physically motivated argument also for some of the following
concepts.
In other words, we have the typical situation that the external sources
produce a series of randomly distributed events
{ξ
1
,ξ
2
,...,ξ
R
} , (8.9)
but the system utilizes only the sum
ξ =
R
j=1
ξ
j
(8.10)
216 8 Filters and Predictors
for its further dynamical evolution. We assume for a moment that we know
the probability distribution function p (ξ
j
) for the single events. In the follow-
ing context we designate p (ξ
j
) also as an elementary probability distribution
function. Because of the statistical independence the joint probability for the
set (8.9) is given by
p (ξ
1
,ξ
2
,...,ξ
R
)=
R
j=1
p (ξ
j
) . (8.11)
Let us now determine the function p
R
(ξ) for the sum (8.10). We get
p
R
(ξ)=
R
j=1
dξ
j
δ
ξ −
R
j=1
ξ
j
R
j=1
p (ξ
j
) . (8.12)
The Markov property allows us to derive the complete functional structure
of the probability distribution function p
R
(ξ) from the sole knowledge of the
elementary probability density p (ξ
j
). It is convenient to use the characteristic
function (6.50), which is defined as the Fourier transform of the probability
density. Hence, we obtain
ˆp
R
(k)=
dξ exp {ikξ}p
R
(ξ)
=
R
j=1
dξ
j
exp
ik
R
j=1
ξ
j
L
j=1
p (ξ
j
)
=[ˆp (k)]
N
. (8.13)
What can we learn from this approach? To this aim we provide a naive scaling
procedure to (8.13). We start from the expansion of the characteristic function
in terms of cumulants. For the sake of simplicity we focus for a short moment
on single component event ξ
j
∈ R. In this case we may write
ˆp (k) = exp
∞
n=1
c
(n)
n!
(ik)
n
, (8.14)
and because of (8.13)
ˆp
R
(k) = exp
∞
n=1
Rc
(n)
n!
(ik)
n
, (8.15)
where k is now a simple scalar quantity instead a vector of a certain dimension.
Obviously, when R →∞, the quantity ξ goes to infinity with the central
tendency
ξ = Rc
(1)
the standard deviation σ =
Rc
(2)
1/2
. Since the drift
can be zero or can be put to zero by a suitable shift ξ → ξ −
ξ, we conclude
that the relevant scale is that of the fluctuations, namely the variance σ.The
corresponding range of k is simply its inverse, since ξ and k are conjugate