Jacques J., Raimondo M. Semi-markov risk models for finance, insurance and reliability
Подождите немного. Документ загружается.
Markov renewal processes
87
3) It is now clear that we can immediately consider an MRP defined by kernel
Q without speaking explicitly of the basic (J,X) process with the same kernel Q,
because the basic property (2.6) is equivalent to (5.1).
6 MARKOV RENEWAL FUNCTIONS
Let us consider an MRP of kernel Q and to avoid trivialities, we will assume that:
,
sup (0) 1,
ij
ij
Q
<
(6.1)
where the functions Q
ij
are defined by relation (2.2).
If the initial state J
0
is i, let us define the r.v.
(
)
n
Tii, 1≥n , as the times (possibly
infinite) of successive returns to state i, also called successive entrance times into
{}
i .
From the regenerative property of RRP, whenever the process enters into state i,
say at time t, the evolution of the process on
[
)
,t
∞
is probabilistically the same
as if we had started at time 0 in the same state i.
It follows that the process
(
)
()
,0
n
Tiin≥ with:
(
)
0
0Tii
=
(6.2)
is a renewal process that could be possibly defective.
From now on, the r.v.
()
n
Tii will be called the nth return time to state i.
More generally, let us also fix state j, different from state i already fixed; we can
also define the nth return or entrance time to state j, but starting from i as initial
state. This time, possibly infinite too, will be represented by
(
)
(
)
,0
n
Tjin≥ ,
using here too the convention that
(
)
0
0Tji
=
. (6.3)
Now, the sequence
(
)
()
,0
n
Tjin≥ is a delayed renewal process with values in
+
.
It is thus defined by two d.f.: G
ij
being that of
(
)
1
Tji and G
jj
that of
() ()
21
Tji Tji− , so that:
(
)
(
)
() ()
()
1
1
() ,
() , 2.
ij
jj n n
Gt PT ji t
Gt PT ji T ji t n
−
=≤
=−≤≥
(6.4)
Of course, the d.f. G
jj
suffices to define the renewal process
(
)
(
)
,0
n
Tjjn≥ .
88 Chapter 3
Remark 6.1 From the preceding definitions, we can also write that:
(
)
()
()
0
1
() () 0 ; , ,
1()
ij j
ij
Gt PNt J i ij I
PT ji G
=
>= ∈
=+∞ = − +∞
(6.5)
and for the mean of the
()
n
Tii
, 1≥n , possibly infinite, we get:
()
()
1
0
(),
ij ij
E
Tji tdGt
μ
∞
==
∫
(6.6)
with the usual convention that
0( )
⋅
+∞ = 0. (6.7)
The means , ,
ij
ij I
μ
∈ are called the first entrance or average return times.
Lemma 6.1 The functions ,,
ij
Gij I
∈
satisfy the following relationships:
1
() () (1 ) (),, , 0.
m
ij kj ik jj ij
k
Gt G Qt G Qtij It
=
=
•+−• ∈≥
∑
(6.8)
For each possibly delayed renewal process defined by the couple (G
ij
,G
jj
), i,j
belonging to I, we will represent by A
ij
and R
ij
the associated renewal functions
defined by relations 2(2.4) and 2(3.12) so that:
0
'
0
() ( () ),
() ( () )
ij j
ij j
At ENtJ i
R
tENtJi
=
=
=
=
(6.9)
and by relations (5.8):
0
() () ().
ij ij ij
R
tUtAt
δ
=
+ (6.10)
From relations 2(9.7), 2(3.9) and 2(3.14), we get:
()
0
() (), ,
() ().
n
jj jj
n
ij ij jj
R
tGtjI
Rt G R t
∞
=
=
∈
=•
∑
(6.11)
Or equivalently, we have:
()
0
0
() () (),, .
n
ij ij ij jj
n
R
tUtG GtijI
δ
∞
=
=+• ∈
∑
(6.12)
Proposition 6.1 Assumption
∞
<
m implies that:
(i) at least one of the renewal processes
(
)
(
)
,0,
n
Tjjn jI≥∈is not defective,
(ii) for all i belonging to I, there exists a state s such that
(
)
lim , . .,
n
n
Tsi as=+∞ (6.13)
(iii) for the r.v. T
n
defined by relation (1.4), given that J
0
=i whatever i is, we
have a.s. that
Markov renewal processes
89
lim .
n
n
T
=
+∞ (6.14)
The following relations will express the renewal functions , ,
ij
R
ij I∈ as a
function of the kernel
Q instead of the m
2
functions G
ij
.
Proposition 6.2 For every i and j of I, we have that:
()
0
() ().
n
ij ij
n
R
tQt
∞
=
=
∑
(6.15)
Using matrix notation with:
ij
R
⎡
⎤
=
⎣
⎦
R , (6.16)
relation (6.14) takes the form:
()
0
.
n
n
∞
=
=
∑
RQ (6.17)
Let us now introduce the L-S transform of matrices.
For any matrix of suitable functions A
ij
from
+
to represented by
ij
A
⎡
⎤
=
⎣
⎦
A (6.18)
we will represent its L-S transform by:
ij
A
⎡
⎤
=
⎣
⎦
A (6.19)
with
0
() ().
st
ij ij
As e dAt
∞
−
=
∫
(6.20)
Doing so for the matrix
R , we get the matrix form of relation (6.15),
()
0
() ()
n
n
s
s
∞
=
=
∑
RQ. (6.21)
From this last relation, a simple algebraic argument shows that, for any s>0,
relations
()(())(()()ss ss
−
=− =RIQ IQR I (6.22)
hold and so, we also have that:
1
() ( ()) .ss
−
=−RIQ (6.23)
We have thus proved the following proposition.
Proposition 6.3 The Markov renewal matrix R is given by
()
0
,
n
n
∞
=
=
∑
RQ (6.24)
the series being convergent in
+
.
Moreover, the L-S transform of the matrix R has the form:
90 Chapter 3
1
(),
−
=−RIQ (6.25)
the inverse existing for all positive s.
The knowledge of the Markov renewal matrix R or its L-S transform
R
leads to
useful expressions for d.f. of the first entrance times.
Proposition 6.4 For the L-S transforms of the first entrance time distributions,
we have:
1
1
()( ()) , ,
()
1( ()), .
ij jj
ij
jj
R
sR s i j
Gs
R
sij
−
−
⎧
≠
⎪
=
⎨
−=
⎪
⎩
(6.26)
Inversely, we have:
1
()
,,
1()
()
1( ()), .
ij
jj
ij
jj
Gs
ij
Gs
Rs
Gs i j
−
⎧
≠
⎪
−
=
⎨
⎪
−
=
⎩
(6.27)
7 CLASSIFICATION OF THE STATES OF AN MRP
To give the classification of the states here, we will proceed as we did in the case
of Markov chains: that is, by considering the embedded renewal processes or
delayed renewal processes of return times in the different states of I.
This gives the following definition.
Definition 7.1 The state j of I is said to be recurrent, transient, aperiodic or
periodic with period d. according to the associated embedded renewal processs
is recurrent, transient,r aperiodic or periodic with period d.
Moreover, j is positive (or non-null) recurrent iff
j
j
μ
is finite.
The next proposition establishes the interaction between classification of the
states of an MRP and that of the same states but for the imbedded MC
(, 0)
n
Jn≥ .
Proposition 7.1
(i) j is recurrent for the MRP iff j is recurrent, necessarily positive in the
imbedded MC,
(ii) if
,
sup
ij
ij
b <∞, then j recurrent for the MRP implies that it is positive
recurrent,
(iii) j transient for the MRP implies that j is also transient for the imbedded MC.
Markov renewal processes
91
Definition 7.2 State j of an MRP is accessible from state i, or j can be reached
from state i, if there exists a strictly positive t such that:
(
)
() (0) 0PZt jZ i
=
=>
. (7.1)
Remark 7.1 A state j is periodic for the MRP iff the d.f. G
jj
is arithmetic.
It is clear that the periodicity of a state j for the MRP bears no relation to the
periodicity of this state in the embedded MC.
For the periodicity in the MRP, Çinlar (1975b) had nevertheless proved the
following result:
If j can be reached from i and if i can be reached from j, then state i and j are both
aperiodic or both periodic and, in this latter case, have the same period.
Definition 7.3 An MRP will be called irreducible if every state can be reached
from any state.
From
Remark 7.1, it follows that in the irreducible case, all the states are both
periodic with the same period or both aperiodic.
From
Proposition 7.1, we can deduce the following result.
Proposition 7.2 An MRP is irreducible iff the imbedded MC is also irreducible.
Definition 7.4 An MRP will be called ergodic iff it is irreducible aperiodic, and
if the imbedded MC is also aperiodic.
The semi-Markov kernel
Q corresponding to an ergodic MRP is also called an
ergodic kernel.
From
Remark 7.1, the ergodicity of an MRP implies that of the imbedded MC,
but the ergodicity of the imbedded MC only implies the irreducibility of the
MRP.
8 THE MARKOV RENEWAL EQUATION
This paragraph will extend the basic results related to the renewal equation
developed in section 4 of Chapter 2 to the Markov renewal case.
Let us consider an MRP of kernel
Q.
From relation (6.15), we get:
()
()
0
1
0
() () ()
( ) ( ).
n
ij ij ij
n
ij
ij
R
tUt Qt
Ut QR t
δ
δ
∞
=
=+
=+•
∑
(8.1)
Using matrix notation with:
92 Chapter 3
0
() ()
ij
tUt
δ
⎡
⎤
=
⎣
⎦
I , (8.2)
relations (8.1) take the form:
() () ().tt t
=
+•RIQR (8.3)
This integral matrix equation is called the Markov renewal equation for
R.
To obtain the corresponding matrix integral equation for the matrix
,
ij
H
⎡
⎤
=
⎣
⎦
H (8.4)
we know, from relation (6.10) that
() () ().tt t
=
+RIH (8.5)
Inserting this expression of
R(t) in relation (8.3), one obtains:
() () ()tt t
=
+•HQQH (8.6)
which is the Markov renewal equation for
H.
Of course, for m=1, this last equation gives the classical renewal equation (4.1)
of Chapter 2.
In fact, the Markov renewal equation (8.3) is a particular case of the matrix
integral equation of the type:
,
=
+•fgQf (8.7)
called an integral equation of Markov renewal type (in short MRT), where
(
)
(
)
11
,..., ', ,..., '
mm
f
fgg==fg (8.8)
are two column vectors of functions having all their components in B, the set of
single-variable measurable functions, bounded on finite intervals or to B
+
if all
their components are non-negative.
Proposition 8.1 The Markov integral equation of MRT,
=
+•fgQf (8.9)
with
f,g belonging to B
+
, has the unique solution:
=
•fRg. (8.10)
9 ASYMPTOTIC BEHAVIOUR OF AN MRP
We will give asymptotic results, first for the Markov renewal functions and then
for solutions to integral equations of an MRT.
To finish, we will apply these results to transition probabilities of an SMP.
9.1 Asymptotic Behaviour Of Markov Renewal Functions
We know that the renewal function R
ij
, i,j belonging to I, is associated with the
delayed renewal process, possibly transient, characterized by the couple
(G
ij
,G
jj
).d.f. on
+
.
Let us recall that
ij
μ
represents the mean, possibly infinite, of the d.f. G
ij
.
Markov renewal processes
93
Proposition 9.1 For all i,j of I, we have:
(i)
()
1
lim ,
ij
t
j
j
Rt
t
μ
→∞
= (9.1)
(ii)
() ( )
lim
ij ij
t
j
j
Rt Rt
τ
τ
τ
μ
→∞
−
−
= , for every fixed
τ
. (9.2)
The next proposition, due to Barlow (1962), is a useful complement to the last
proposition as it gives a method for computing the values of the mean return
times ,
jj
jI
μ
∈ , in the ergodic case.
Proposition 9.2 For an ergodic MRP, the mean return times satisfy the following
linear system:
, 1,..., .
ij ik kj i
kj
p
im
μμη
≠
=+=
∑
(9.3)
In particular, for i=j, we have:
1
,1,...,,
jj k k
k
j
jm
μπη
π
==
∑
(9.4)
where the
,
i
iI
η
∈
are defined by relation (3.14), and where
(
)
1
,...,
m
π
π
=π is
the unique stationary distribution of the imbedded Markov chain.
Remark 9.1 In a similar manner, Barlow (1962) proved that if
(2)
,,
ij
ij I
μ
∈ is the
second order moment related to the d.f. G
ij
, then:
(2) (2) (2)
(2)
ij i ik ik ik kj
kj
pb
μ
ημμ
≠
=+ +
∑
(9.5)
and in particular for i=j:
(2) (2)
1
(2 )
j
jkk llkkkj
kkjl
j
pb
μ
πη π μ
π
≠
=+
∑∑∑
(9.6)
with
[
)
(2) 2
0,
(), ,
kk
x
dH x k I
η
∞
=
∈
∫
(9.7)
provided that these quantities are finite.
9.2 Asymptotic Behaviour Of Solutions Of Markov Renewal
Equations
Under the assumptions of Proposition 8.1, we know that the integral system
(8.9), that is
94 Chapter 3
[]
0,
() () ( ) ( ),
ii j ij
j
t
f
tgt ftsdQsiI=+ − ∈
∑
∫
, (9.8)
has the unique solution
[]
0,
() ( ) ( ), .
ijij
j
t
f
tgtsdRsiI
=
−∈
∑
∫
(9.9)
For the asymptotic behaviour of this solution for t tending toward
+
∞
, we have
the analogue of Proposition 4.2 of Chapter 2, i.e. the key renewal theorem.
Proposition 9.3 (Key Markov renewal theorem)
For any ergodic MRP, we have:
[]
0
0,
()
lim ( ) ( ) ,
jj
j
jij
t
j
jk
t
j
g
ydy
gt sdRs
π
πη
∞
→∞
−=
∑
∫
∑
∫
∑
(9.10)
provided that the functions g
i
, i belonging to I, are directly Riemann integrable.
10 ASYMPTOTIC BEHAVIOUR OF SMP
10.1 Irreducible Case
Let us consider the SMP (Z(t), t 0≥ ) associated with the MRP of kernel Q and
defined by relation (5.6).
Starting with (0)
Z
i= , it is important for the applications to know the probability
of being in state j at time t, that is:
(
)
() () (0) .
ij
tPZt jZ i
φ
=
== (10.1)
A simple probabilistic argument using the regenerative property of the MRP
gives the system satisfied by these probabilities as a function of the kernel
Q:
0
() (1 ()) ( ) ( ), , .
t
ij ij i kj ik
k
tHt tydQyijI
φδ φ
=− + − ∈
∑
∫
(10.2)
It is also possible to express the transition probabilities of the SMP with the aid
of the first passage time distributions , ,
ij
Gij I
∈
:
() () (1 ()), , .
ij jj ij ij i
tGt HtijI
φ
φδ
=• + − ∈ (10.3)
If we fix the value j in relations (10.2), we see that the m relations for i=1,…,m
form a Markov renewal type equation (in short MRE) of form (8.9).
Applying
Proposition 8.1, we immediately get the following proposition.
Proposition 10.1 The matrix of transition probabilities
Markov renewal processes
95
ij
φ
⎡
⎤
=
⎣
⎦
Φ (10.4)
is given by
()
=
•−Φ RIH (10.5)
with
.
ij i
H
δ
⎡
⎤
=
⎣
⎦
H (10.6)
So, instead of relation (10.3), we can now write:
[]
0,
() (1 ( )) ( ).
ij j ij
t
tHtydRy
φ
=−−
∫
(10.7)
Remark 10.1 Probabilistic interpretation of relation (10.7)
This interpretation is analogous to that of the renewal density given in Chapter 2,
Remark 10.2: the “infinitesimal” quantity dR
ij
(y) (=r
ij
(y)dy, if r
ij
(y) is the density
of the function R
ij
, if it exists) represents the probability that there is a Markov
renewal into state j in the time interval (y,y+dy), starting at time 0 in state i.
Of course, the factor (1
−
H
j
(t
−
y)) represents the probability of not leaving state j
before a time interval of length t
−
y.
The behaviour of transition probabilities of matrix (10.4) will be given in the next
proposition.
Proposition 10.2 Let
()
(), 0ZZtt=≥be the SMP associated with an ergodic
MRP of kernel
Q; then:
lim ( ) , , .
jj
ij
t
kk
k
tijI
π
η
φ
πη
→∞
=
∈
∑
(10.8)
Remark 10.3
(i) As the limit in relation (10.8) does not depend on i, Proposition 10.2
establishes an ergodic property saying that:
lim ( ) ,
.
ij j
t
jj
j
kk
k
t
φ
πη
π
η
→∞
=
Π
Π=
∑
(10.9)
(ii) As the number m of states is finite, it is clear that
(
)
,
j
jI
Π
∈ is a probability
distribution. Moreover, as 0
j
π
> for all j (see relation (9.89) of Chapter 2), we
also have
96 Chapter 3
0, .
j
jI
Π
>∈ (10.10)
So, asymptotically, every state is reachable with a strictly positive probability.
(iii) In general, we have:
()
lim lim ( )
n
ij ij
nt
p
t
φ
→∞ →∞
≠ (10.11)
since of course
,.
jj
jI
π
≠
Π∈ (10.12)
This shows that the limiting probabilities for the imbedded Markov chain are not,
in general, the same as taking limiting probabilities for the SMP.
From
Propositions 10.2 and 9.2, we immediately get the following corollary.
Corollary 10.1 For an ergodic MRP, we have:
.
j
j
j
j
π
μ
Π= (10.13)
This result says that the limiting probability of being in state j for the SMP is the
ratio of the mean sojourn time in state j to the mean return time of j.
This intuitive result also shows how the different return times and sojourn times
have a crucial role in explaining why we have relation (10.13) as, indeed, for the
imbedded MC, these times have no influence.
10.2 Non-Irreducible Case
It happens very often that stochastic models used for applications need non-
irreducible MRP, as for example, in presence of an absorbing state, i.e. a state j
such that
1.
jj
p
=
(10.14)
We will now see that the asymptotic behaviour is easily deduced from the
irreducible case studied above.
10.2.1 Uni-Reducible Case
As for Markov chains, this is simply the case in which the imbedded MC is uni-
reducible so that there exist l (l<m) transient states, and so that the other ml−
states form a recurrent class C.
We always suppose aperiodicity both for the imbedded MC and the considered
MRP.
Let T=
{
}
1,...,l be the set of transient states
(
)
TIC=− .From Proposition 9.5 of
Chapter 2, we know that: