Genshiro Kitagawa. Introduction to Time Series Modeling (Введение в моделирование временных рядов)

Подождите немного. Документ загружается.

INTERPOLATION OF MISSING OBSERVATIONS 149

Example (Interpolation of missing observations in BLSALLFOOD

data) Figure 9.3 shows the results of a numerical interpolation ex-

periment for the BLSALLFOOD data. Within 156 observations of data,

in total 50 observations are assumed to b e missing: y

,···, y

and

101

,···, y

120

. They are interpolated by AR models of orders 0, 1, 5, 10

and 15, respectively. It should be noted that order 15 is the AIC best

order. AR coefﬁcients are obtained by m aximizing the log-likelihood

deﬁned b y (9.31 ). Interpolation by the AR mode l of order 0 is equiva-

lent to replacing the missing observations with the mean of the whole

time series. Since the interpolations with the ﬁrst or the 5th order AR

model cannot capture the wave pattern of the time ser ie s well, inter pola-

tion with these models cannot reproduce the actual observations sh own

by the symbol ◦. On the other h and, by using the AR model of order 10,

the cyclic be havior of the data is reproduced well. Further more, it can be

seen from plot (e) that the AIC best model of order 15 can reproduce the

details of the data remarkably well.

Problems

1. Show that, using the transformation (9.7), w e can ob ta in a state-sp ace

model, which is equivalent to the one given in (9.1) and (9.2).

2. Assume that the state-sp a ce model x

= x

n−1

+ v

, y

= x

+ w

for

the 1-dimensional state x

is given, wher e v

∼N(0,

), w

∼N(0,1)

and x

∼ N(0,10

(1) Show the Kalman ﬁlter algorithm for this model.

(2) Show the re la tion be twe en V

n+1|n

and V

n|n−1

(3) If V

n|n−1

→V as n → ∞, show th at V satisﬁes the equ a tion V

−

V −

= 0.

(4) Consider the Kalma n ﬁlter algor ithm as n → ∞ (stationary

Kalman ﬁlter).

3. Let the solution of the equ a tion V

−

V −

= 0 be denoted by

V . Obtain the variance of the one -step-ahead predictor, ﬁlter and

smoother in the steady state. For

= 1,0.1,0 .01,0.00 1, evaluate

these variances.

150 ANALY SIS OF TIME SERIES WITH A STATE-SPACE MODEL

Figure 9.3 Interpolation of missing values (bold line: mean, thin line: ± (stan-

dard deviation) and ◦: observed value). Orders of the AR models are 0, 1, 5, 10

and 15.

Chapter 10

Estimation of the ARMA Model

In this chapter, a method for efﬁciently computing the log- likelihood of

the ARMA model is explained based on the state-space representation

and the Kalman ﬁlter technique. Applying the numerical optimization

method shown in Chapter 4, it is possible to maximize the log-likelihood.

By th is procedure, th e ma ximum likelihood estimates of the parameters

of the ARMA mo del can b e obtained.

10.1 State-Space Representation of the ARMA Model

Assume that a statio nary ARMA model (autoregressive moving average

model) of order (m,ℓ) (Box a nd Jenkins (1970), Brockwell and Davis

(1981))

∑

j=1

n−j

+ v

−

ℓ

∑

j=1

n−j

(10.1)

is given. Here, v

is a Gaussian white noise with mean zero and variance

and ˜y

n+i|n−1

is deﬁned as

˜y

n+i|n−1

∑

j=i+1

n+i−j

−

ℓ

∑

j=i

n+i−j

, (10.2)

where ˜y

n+i|n−1

is a part of y

n+i

that can be directly expressed by the

observations until time n −1, y

n−1

, y

n−2

, ···, and the noise inputs until

time n, v

, v

n−1

, ···. Then, the following relations hold:

= a

n−1

+ ˜y

n|n−2

+ v

˜y

n+i|n−1

= a

i+1

n−1

+ ˜y

n+i|n−2

−b

˜y

n+k−1|n−1

= a

n−1

−b

k−1

. (10.3)

Therefore, setting k = max(m,ℓ + 1) and deﬁnin g the k-d imensional

state vector x

= (y

, ˜y

n+1|n−1

,···, ˜y

n+k−1|n−1

)

, (10.4)

151

152 ESTIMATION OF THE ARMA MODEL

the ARMA mod el can be expressed in th e form of a state-space model:

= Fx

n−1

+ Gv

= Hx

. (10. 5)

Here the k × k matrix F and the k-dimensional vectors G and H are

deﬁned as

F =







. 1







, G =







−b

k−1







(10.6)

H = [

1 0 ··· 0

respectively, where a

= 0 for i > m and b

= 0 for i > ℓ.

In summary, an ARMA model can be expressed by a state-space

model wher e the coefﬁcient matrices F, G and H are time-invariant a nd

the observation noise is zero.

10.2 Initial State of an ARMA Model

In order to apply the Kalman ﬁlter to the state-space representation of the

ARMA mode l, it is necessary to specify the mean x

0|0

and the variance-

covariance matrix V

0|0

of the initial state. Since x

0|0

and V

0|0

express

the distribution of the ﬁlter withou t any observations of the time se-

ries y

, they can be evaluated by computin g the mean and the variance-

covariance matrix of x

in the stationary state. Firstly, since E(v

) = 0 , it

is obvio us that E(y

) = 0 and E( ˜y

n+i|n−1

) = 0. Since the expected value

of x

becomes 0, the initial state vector is deﬁned as x

0|0

= (0,···, 0)

Next, if the (i, j )- th element of the variance-covariance matrix V

0|0

denoted by V

i j

, it can be obtained as

= E(y

) = C

= V

= E



˜y

i−1|−1



= E





∑

j=i

i−1−j

−

ℓ

∑

j=i−1

i−1−j



∑

j=i

j+1−i

−

ℓ

∑

j=i−1

j+1−i

i j

= E



˜y

i−1|−1

˜y

j−1|−1



MAXIMUM LIKELIHOOD ESTIMATE OF AN ARMA MODE L 153

∑

p=i

∑

q= j

E(y

i−1−p

j−1−q

) −

∑

p=i

ℓ

∑

q= j−1

E(y

i−1−p

j−1−q

)

−

ℓ

∑

p=i−1

∑

q= j

E(v

i−1−p

j−1−q

) +

ℓ

∑

p=i−1

ℓ

∑

q= j−1

E(v

i−1−p

j−1−q

)

∑

p=i

∑

q= j

q−j−p+i

−

∑

p=i

ℓ

∑

q= j−1

q−j−p+i

−

ℓ

∑

p=i−1

∑

q= j

p−i−q+ j

ℓ

∑

p=i−1

p+ j−i

. (10.7)

Here, we note that C

and g

are th e autocovariance functio n and the

impulse response f unction, respectively, of the ARMA model, which we

can obtain by the method shown in Chapter 6.

10.3 Maximum Likelihood Estimate of an ARMA Model

Using the above results for the state-space representation of the ARMA

model, we can co mpute the log-likelihood for the ARMA model speci-

ﬁed by th e param eter

= (

,···, a

,···,b

ℓ

)

. Firstly, the in itial

state x

0|0

and its variance-covariance matrix V

0|0

are p rovided by the

method speciﬁed in the preceding section. Next, the one-step-ahead pre-

diction of the state x

n|n−1

and its variance-covariance matrix V

n|n−1

can

be computed by th e Kalman ﬁlter for n = 1, ···,N. Then, using (9.23 ),

the log-likelihood of the ARMA model can be obtained as

ℓ(

) = −

log2

−

∑

n=1

logd

n|n−1

−

∑

n=1

−y

n|n−1

)

n|n−1

, (10.8)

where d

n|n−1

= HV

n|n−1

and y

n|n−1

= Hx

n|n−1

(Jones (1980), Brock-

well and Davis (1981)).

The maximum likelihood estimate o f the parameter vector

can be

obtained by maximizing the log-likelihood ℓ(

) using a n umerical opti-

mization method. It is noted that the variance

can easily be ob ta ined

without applying a numerical optimization method by the following pro -

cedure, which is similar to the method shown in Section 9.6.

1. Apply the Kalman ﬁlter after setting

= 1 and obtain d

n|n−1

and y

n|n−1

for n = 1,···,N.

2. Obtain the maxim um likelihood estimate of

−1

∑

n=1

−y

n|n−1

)

n|n−1

154 ESTIMATION OF THE ARMA MODEL

3. Obtain the lo g-likelihood fun ction for the AR coefﬁcients

,···, a

, and the MA coefﬁcients b

,···, b

ℓ

by ℓ

′

,···, a

···,b

ℓ

) ≡ ℓ(

,···, a

,···, b

ℓ

The maximum likelihood estimate of

′

= (a

,···, a

,···, b

ℓ

)

can be obtained by applying a numerical optimization method to the

log-likelihood function obtained by the above-mentioned pro cedure. In

actual computation, however, to satisfy the stationar ity and invertibility

conditions, we often apply the following transformations of the parame-

ters.

For th e condition of stationarity for the AR coefﬁcients a

,···, a

PARCORs c

,···, c

should satisfy −1 < c

< 1 for all i = 1,···,m. It can

be seen that this condition is guaranteed, if the transformed coefﬁcients

deﬁned by

= log



1 + c

1 −c



, (10.9)

satisfy −∞ <

< ∞ for all i = 1 , ···,m.

Conversely, for arbitrary (

,···,

)

∈R

, if c

is deﬁned by

−1

+ 1

, (10.10)

then it always satisﬁes |c

| < 1 and the corresponding AR coefﬁcients

satisfy the stationar ity con dition.

On the other hand, to guarantee the invertibility condition for any

(

,···,

ℓ

)

∈ R

ℓ

, let d

be deﬁned as

−1

+ 1

, (10.11)

and formally obtain the corresp onding MA coefﬁcients b

,···, b

ℓ

by con-

sidering d

,···,d

ℓ

to be the PARCORs.

Then, for arbitrary

′′

= (

,···,

ℓ

)

∈R

m+ℓ

, the corre-

sponding ARMA model will always satisfy the stationarity and invert-

ibility conditions.

10.4 Initial Estimates of Paramete r s

The initial values of the AR and MA coefﬁcients are necessary for nu-

merical optimization procedures to obtain the maximum likelihood esti-

mates of the parameter s of the ARMA model. For AR mode ls obta ined

by setting ℓ = 0, the initial estimates of the AR coefﬁcients can be rather

INITIAL ESTIMATES OF PARAMETERS 1 55

Table 10.1: AIC values f or ARMA models with various orders.

ℓ

0 1 2 3 4 5

0 141.1 73.3 58.5 377.0 54.1

1 105.3 64.6 138.9 55.7 62.4 58.0

2 43.0 39.2 41.0 228.7 23.7 25.0

m 3 41.3 41.1 92.5 14.8 25.4 21.1

4 40.5 41.9 27.0 16.7 14.0 22.9

5 40.9 42.8 18.6 14.3 13.9 16.8

Table 10.2: Log-likelihood values for ARMA models with various order.

ℓ

0 1 2 3 4 5

0 −68.6 −33.7 −25.2 −183.5 −21.0

1 −50.7 −29.3 −65.4 −22.9 −22.2 −22.0

2 −18.5 −15.6 −15.5 −108.4 −4.8 −4.5

m 3 −16.6 −15.5 −40.2 −0.4 −4.7 −1.5

4 −15.2 −15.0 −6.5 −0.4 2.0 −1.5

5 −14.5 −14.5 −1.3 1.9 3.1 2.6

easily obtained by ﬁtting AR models by the Yule-Walker or least squares

method. However, for ℓ > 0, we should carefully select initial values,

since the log- likelihood may have several loca l maxima, which accord-

ingly yield different estimates, depending on the choice of initial values.

Therefore, it should be noted that we cannot always obtain the best esti-

mates using the default initial values, which are automa tica lly set by the

time series analysis software.

Example (Sunspot number data) Table 10.1 shows the AI C values

when ARMA mo dels are ﬁtted to the logar ithm of the sunspot number

data with in the range 0 ≤ m, ℓ ≤ 5. From Table 10.1, it can be seen

that the best model is the ARMA model with m = 5 and ℓ = 4 . On the

other hand, Table 10 .2 summarizes the values of th e log-likelihoods o f

the ﬁtted ARMA models ℓ(i, j), 0 ≤ i, j ≤ 5 . From Table 10.2, the log-

likelihood values for the ARMA models denoted in boldface, such as

ℓ(1,2), ℓ(3, 2) and ℓ(2,3), etc., are apparently too small compared with

156 ESTIMATION OF THE ARMA MODEL

Table 10.3: Estimation of ARMA models wit h modiﬁed initial values.

order log-likelihood AIC

(0,4) −22.5 55.0

(1,2) −23.8 55.5

(2,3) −12.2 36.4

(3,2) −15.4 42.8

(3,4) −0.4 16.3

(4,5) 2.0 16.0

(5,5) 3.7 14.5

the values for other surrounding models. Note that the maximum log-

likelihood of the ARMA (i, j) sho uld be larger than or equal to those

of AR (i −1, j) and AR (i, j −1), i.e., ℓ(i, j) ≥ ℓ(i −1, j) and ℓ(i, j) ≥

ℓ(i, j −1). Therefore, this means that the log-likelihood for these mod e ls

did not converge to the global maximum from the default initial values

of the parameters.

By assuming that a

= 0 in the ARMA (3,2) model, we can obtain

the ARMA (2,2) model. Therefore, it is expe cted that the maximum log-

likelihood value for the ARMA ( 3,2) model is larger than that of the

ARMA (2,2) model. Acc ordingly, the AIC values for the ARMA (i, j)

model in Table 10.1 do not increase by more than 2 compared with those

of the ARMA (i, j −1) and ARMA (i −1, j) models.

Therefore, if the log-likelihood value fo r the ARMA(i, j) violates

either ℓ(i, j) ≥ ℓ(i −1, j) o r ℓ(i, j) ≥ ℓ(i, j −1), then it indicates that we

could not obtain the global maximum of the log-likeliho od function in

estimating the param eters of ARMA(i, j). Such log-likelihood values are

highlighted in boldface in Table 10.2.

In these cases, a be tter model can be always o btained by using the

coefﬁcients of a model with a larger log-likelihood among the left and

above models in Table 10.2 as the initial values for numerical optimiza-

tion. Table 10.3 shows the log-likelihoods and the AICs of the models

obtained by using these modiﬁed initial values. It can be seen that those

initial values certainly satisfy the conditions that ℓ(i, j) ≥ ℓ(i −1, j) and

ℓ(i, j) ≥ ℓ(i, j −1) for all i and j.

In the maximum likelihood estimation of the parameters of the

ARMA models, it is certain to increase the AR and MA orders gradually

by using the estimators of the lower order models. However, it should

INITIAL ESTIMATES OF PARAMETERS 1 57

be also noted that even if the above-mentioned conditions are satisﬁed,

it is not guarantee d that the estimated parameters actually converge to

the global maxima. We often see that since the ARMA mod el is very

ﬂexible and versatile, if we ﬁt an ARMA model with large AR and MA

orders, it has a tendency to adjust itself to a small peak or trough of the

periodogram. Thus the log-likelihood has many local maxima. To avoid

using those inappropriate models in estimating ARMA models, it is rec-

ommended that the power spec trum of the estimated ARMA model be

drawn, to ensure that the estimated model c onverges to the global maxi-

mum.

Problems

1. Usin g the expression of equation (9.5), obtain the initial variance-

covariance matrix for the maximum likelihood estimates of the AR

model of order m .

2. Describe what we sho uld check to apply the transformation (10.9) to

parameter estimation for the ARMA model.