Texts in Statistical Science

Time Series Analysis

Henrik Madsen
Tech nical University of Den mark

CNprNrl 6l tWPICIIC I' .... ImpMt 01 the

l .ylDf" F,ancIsGroup." inform. bu..-om

Chapman & HallfCRC Taylor & Francis Group 6000 Broken Sound Parkway NW, Sui te 300 Soc" Raton, FL334872742 (12008 by Taylor & Francis Group, LLC Chapma n & Hall/CRC Is an imprint of Taylor & Fran(is Group, an Informa business No claim to original U.S. Government works Prin ted in the United States of America on add-free paper 109876S4321 Intemalion,,1 Standard Book Numbl':r-13: 978-1-42OO-S967-0 (Hardcover) This book contains mformation obtained from authentic and highly regarded 5Oureel. Reprinted materia l is q uoted with permission. and sources are indica ted. A wide variety of re(erenees arc tisted. Reasonab le efforlS have been made to publish rel iable da ta and inform ation, but th e author and the publisher cannot assume responSibility for the va li di ty of all mate rials or for the consequem:es of thei r use. Except as permitted under U.S. Copyright Law. no part of this book may be r~rmted, reproduced. transmu ted. or utililed In any form by any electronic. mechanical. or other means, now known or hereafter invented. mcludmg photocopying. microfilming. and recording. or in any information norage or retrieval system, without written permission from the publishers. For permission to photocopy or use material e lectron ica lly from th is work. please access www. copyright.com (http://www.copy right.coml)orcont ac t theCopyrigh t Clearance Center. Inc. (Ceq 222 Rosewood Drive. Danvers. MA 01923. 978-750-8400. CCC is II not for.profi t organi13t1on that provides licensrs and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC. a $Cpa rate system of payment has been arranged. T ...dema rk Notice: Product or corporate names may be trademarks or registered are used on ly for identificalion and uplanalion WIthout intent to infri nge. Library ofCongreu Cat a loglng-in-Publleation Oata Mad$Cn. Henrik, [9SS11mI' series analysis I Henrik Mad$Cn. p. em (Chapman &. Hall/CRC texts in statistica l Klenee series; ". 72) Include. bibliographical rderences and IIldcx. IS BN 9781 42OO-S9670 (hardback: alk. pa per) I. Time-$Crie$ analySi$.1. Title. [[. Senes. QA280,M322007 SI9.SS-dc22 Visit the Taylor " runcil Web site at http,l/w""w.taylonndfrancis.com and the CRC Prell Web ,Ite at http: //www.cr<:pleu.com
trademar~5.

Contents

Preface

x iii
xv 1

Nota tio n
1 lntroduc tion 1.1 Examples o f t iml' ~ r ics ..... 1.1. 1 Dollar to Euro exchange ratl..' . 1. 1.2 Numbt'r of mont hly airline pas..'>t.'ngt'rs 1.1.3 Ileat dynam ics of a buildi tlg 1.1 ..1 P redator.prcy relationship 1.2 A first crash course . . . . . 1.3 Contents and SC01 )C of the book

2
2 2

3
5

and

Multivariate random variables 2. 1 ,Jo im. a nd lIIarginnl densities 2.2 Conditional distributions .. 2.3 Expt'Ctations and moments . 2..1 ~ I olllents of multivariate random variables 2.5 Condit ional expect ation 2.6 T he mullivariatc normal distribut ion. 2.7 Distributions derived frOIll the lIorma l dist ribution 2.8 Li near projections 2.9 Problems . . . . .
f

13
13

14
15 17

20 22 23 24
29

2:z

2007036211

Rcgrcssion.based m e thods 3. 1 The regressio n modd . . . . . 3.2 The general linea r model (CLM) .. 3.2. 1 T ...('ast !;(Iuarcs (LS) est imates 3.2.2 ;\I ax.imum li kelihood (M L) N;limatcs . PT('(liction . . . . . . . . .. . . . . . . . . . 3.3 3.3. 1 Predictio n in t he gencral liuear model R('grcssioll and expont"nt ial smooth ing . . . . 3.1 :.1.1.1 Pr\ictio ns in the const a nt meltn modt'l

31
31 33 31

40 44 45 47 48

3.5

3.6 3.7

Locally constant mean model und simple ('xpollcntial smoothing 3.1.3 Prediction ill trcnd models 3. U Lo<-al trend and exponential smoothing TilllC' S(>ries with seasonal ''8.riations 3.5.1 TIl{' das...ical decomposition. 3.5.2 Holt-Winters procedure . . . Ciolmllllld lQ(~al trend model-an example ProblC'ms... . . . . .

3A.2

50 52 56 59 GO
61 62

65
69
70 73

Linear dy na mic system s 1.1 Linear systelilS in til(> lime domain 4.2 Liut'1lr :-;ystcms in the frequency domain 4.3 Sampling.... . .. 4.4 1'11{' z-trllnsform 1.5 F'r{"Qucntly used operators 4.6 The Laplacl' transform 1.7 A cOInparison betwccn transformations 4.8 Problems... Sto c h astic p rocesses 5.1 luLrodlloioll .. . .. 5.2 Stochastic proccsscs and their momenlls 5.2.1 Characteristics for stochast ic processes 5.2.2 C{)varianC<! and correlation functions. 5.3 Linear processes . . 5.3. 1 Processes in discrete time 5.3.2 Pr~'S in continuous timl' 5.4 Stationary processes in the frequency dOIlUlin 5.5 COllllllonly UI~..'(I linear prOCSS(.'S 5.5.1 The ~IA process 5.5.2 The AR process 5.5.3 T he Aru. IA process 5.6 NOIl-blatiouary models. 5.6.1 The A IU~ l A process. 5.6.2 Seasonul models 5.6.3 ~ I odels with covariatcs 5.(3A ~ I od(,[s with t ime-varying mean val ues . 5.6.5 Models wi th time-varying coefficients 5.7 Optimal prediction of stochastic processes . 5.7. 1 Predictioll in the ARIM A process 5.8 Problems . . . . . . . . . .

78 80 87
90

91 96
97 97 97

Ide ntifi catio n , est imutiou , and m o d e l c hecking (3.1 intro<illClion Estimation of COVl\rillll{"e Il.lld rorrdntion functions 6.2 (3.2.1 Autocovariance and autocorrelation fUllction...; . 6.2.2 Cros&-("()vRriflll{"(> fLnd cr~'H:orrclation functions 6.3 Idcntificlltioll 6.3.1 IdentifiriltiOl1 of tll(, degrcc of differencing 6.3.2 Identification of th(' AIU. IA part . . . 6.3.3 Coilltl'gfillion. . ... Estimation of param('t('rs ill stRlldard models 6.4 6.1.1 r-. lom{'lIt t'stimatC8 . . . .... (3.4.2 T he LS estimator for linear dynamic models 6.1.3 T ile prediction crror method .. 6AA T he ~ I L 1Ill'lllod for dynamic models 6.5 Selectioll of the modcl order 6.5.1 The 3utocort"l'lation [UII{"tions 6.5.2 Testing thc model 6.5.3 IrlfOrllllll iull ("I"it('ria 6.6 ~ l odcJ checking .. 6.6.1 Cross-vulid/lUoli 6.6.2 il.C!iiduru analysis. 6.7 Cnse study: Ele<:tricity consumption (3.8 Problems .. . .

145 115 116 146 150

152
153 15-1

156
157 157

159
163 166 170 171 171 174 174 175 175 179

182
187 187 189
190

on
103 107 107 III 113 117 117 119

125
130 130
132

131 13 1 135 135 137 110

7 S pectral a na lysis 7.1 The p<'riodogram . . . . . . ... . 7.1.1 lI armonic allal~sis. . . . . . 7.1.2 Properties of the pcriodogram 7.2 Consistent estimates of the spectrum 7.2.1 The trullcated periodogram . 7.2.2 Lag- and spectral windows 7.2.3 Approxillllitive distributions for spectral estimates 7.3 T ill' CfQ6S-Spectrutn . . . . . . . 7.3. 1 Tilt' (1)-sI>t'Ct ru lII and tile quadrature spectrum 7.3.2 Cross-amplitude spectrum, phase spect rum. t"ohcr,IICt' Spt.'CtrlUll, Wlin tipcctrUIll 7.1 Estimation of t he cross-spectrum 7.5 P roblems . . 8 Linear sy s t e m s a nd s tochas tic pl"Ocesses 8. 1 Iklationshi p bCl .....('('n in put and output processes . 8. 1.1 ,\Iolllc lil rela t ions . . . . . . ...

194
195

196
200

203

206 206
209

210
21 5

215
2 16

8.2 8.3

8. I 8.5

8.6

8.7 8.8 8.9 8. 10

8.1.2 Spectral rclutiom. .. Systems with measurement lIoise IlIpm-outpul llIodds .. 8.3.1 Transfer function models 8.3.2 Difrereu<-,<, {'(Illation models 8.3.3 Output error modeL<, .. IdentifiClltioll of tmuskr fUllction models f.. luitipic-input models. 8.5. 1 ~ I olllcnt relations .. 8.5.2 SpN:'trai T('lations .. 8.5.3 Identification of multiple-input models. I:::'timation... . . . 8.6.1 Moment estimates . 8.6.2 LS l'!!;tillllltl'S ... 8.6.3 Prroictioll error method. 8.6. 1 hit ('StillllllP;, 8.6.5 Output efror method hfod el ciu'('killg . P rediction in trrulsfN fUliction models 8.8.1 fo.lillilllum vnriancc controller Intervention Illodpls Problem... . . . . .

218

220
222

9.8 !J.!J

r-.hxld checking. ProbleIllS ..

271

278
283 281

222 223 223 223

226

10 State 10.1 10.2 10.:1

226 227
227 228

10.4

229 229 22!J 229 230 230 233

10.5

10.6 10.7

space m odels of dynamic system s The lillea.! stoChill:il ic stal e spru:e modd Trall.!>fer fumtioll and slatC spaCi' formulaliOlls L uterpolulioD, re<:ollstruetioll, alld prediction 10.3.1 Tile Kahllan filter . . . . 10.3.2 k-btcp prroictions ill !oolate !oopoce models 10.3.3 EllIpirintl BaycsiRIl description of the Killman filt{'r SOIllC COlllmon models ill slat{' S P I\CC form 10.1.1 Signal extraction .. . ... Ti.me series with missing o\.)Mf\'8tiolls 10.5.\ E~stiml\tioll of autocorrelation flll\(tions l\IL estimates of state spa'(> models Problems... . ..

286

288
289

296
2'J6 299 301
307
307

:\07
310

238
241

2\1
247

9 Multivariate time series 9.1 Statiolllu'y sl.ocha..,tic proccs.scs a nd their moments 9.2 Linear processes . . . . .
9.3 Till' IIIllhimriate ARMA process 9.3.1 Theoretical covarian('(' matrix fUllc tions 9.3.2 Partial correlatiOll matrix. . .... 9.3.3 q-conditioncd partial correlation matrix 9.3.4 VAll reprcscntation .. :-\on-stationary models . . . . . . . 9.1.1 The multiwt.riatc ARI~IA process 9.4.2 The multivmiate seasonal model 9.1.3 Time-varyillg mode!.s Prediction. . . . . . 9.5.1 ~lis."illg value:s for some signals Identification of multivariate lIlockb 9.6.1 ld{'mificatiOIl using pr('-whitcning Estimation of parRIIlI'ten; . . . . . 9.7.1 Least 8quRr('S esti mlltiOIl 9.7.2 An I'xteuded LS mC'thod fo r multi\1lCiklt> ARMAX models (t h{' Spliid Illt'lhod) 9.7.3 f.. IL estimates . . . . . . . .

21\9 251
251

11 Rec urs ive estimatio n 11 . 1 Recursive LS 11.l.l Re<:ursive LS with forgetting 11.2 Recursive pseudo-linear regression (RPLR ) 11.3 Recursive prroiction error met.hods ( HPEM) 11.1 t-.lodel-ba.'>Cd ad"ptivc C!>tima.tion . . . 11 .5 t-.lodels with till1('-varying parallietcrs 11.5.1 The regr{'S,>;ioll model with time-varying panullctcfS 11.5.2 Dynamic model.; with tillie-varying parameters

313
313

316 3 1!.l 32 1 32 1 325 325 326

255
259

260
260

9,4

261 261
26 1

9.5 9.6

9.7

262 262 265 267 269 269

270

12 Renl life ins pire d proble m s 12 .1 Predietion of wind power production. 12.2 Predietio n of the eonblllllptioo of medicine 12.3 Effect of chcwing gum . . 12.1 PrPdiction of stock prict.'S .. 12.5 \\'astewlltcr trcatment: Using root zolle plallL'i lUi Scheduling systelTI for oil d<."Jivery 12.7 Warning system ror slippery roads 12.S Statistieal qUlllit\' {'omrol 12.9 \\'a.o.;t<."wutcr treatment: ~ I odeli ng and control 12. 10 Sales llUlllbcrs .... 12.11 :\Iodelillg and pl'<'<iiC'tion of stock prices 12.12 Adapti\'{' mooeling of ililerest rat{'::;
Appendix A The solution to differe nce equations

331
333 334 336

338
310

3-11
31 1 345

3.\7
350 352 353

355
357

271 271

Appendix B Partial autocorrelations

The aim of this book is to give an iutroduction to timt' st.'rie:3 aJlIlly~is. The emphasis is on methods for modeling of linear :;tochu:.tic systcms. Doth time domain and frequency domain description:; will lX' given; however. {'mphnsis is on the time domain description. Due to the highly different mathematical approachf:'S 1ll"l'(lcd for Iincar and non-linear systems, it is instructiv(' to deal with them in 5etx'fate textbooks, which is why nOIl-lillear time series anniysis is not a topic ill this book im;tead the reader is referred to ~ I adsell, Holst, Iwd Lindstrom (2007). Theon.'ms fire HSt'd to empiLtl.';ize the most import.nnt results. Proofs are given ouly when they clarify the r('!;ults. Small prohll'llIs ar(' included ilL the (lnd of mOSt chapters, and 8 ~parate chupter with real-life problems is included IL>; the final chapter of the book. This also sen.'eo 1\,>; a delllonstrat ion of the mall)' possible applications of tillll' l;{'rie:; analy~is in areas such as physics, cngineering, IIlld I.'(:ouollletrics. During the se<luence of cJlIl.ptt'rs, lTlore advUllt'{.'(i stochastic models arc p;radualiy illtroduced; with this npproach, the family of linear time S('ries models and methods is pnt into a clear relationship. Following un initial dlHpter ('Owring stlltic model>; nud methods such as the use of tht' gcneral linear model for time series data, the r('St of the book is devoted to stocluu.tic dYllamic 1Il0dl'ls which are mostly forllluillted as ditrereIlcc equations, as in the famous ABMA or vector AR~IA processes. It. will be obvious to LII(' feadt'r of this hook that l'ven knowing how to solve difft](,uct' t.'(juations becomes important for understanding the bd18\'ior of important. aspects such as the autocovarian('(' functions and the nature of the optimal pre<lictions. The importlUlt concept. of tillll'-varyillg systems is dealt with using a state space approach and the Kalman filter. lIo\\-'t'Vl'r, the strength of also using Rdnpliv(' estimation IlItthods for on-lint' for('('/istillg and control is often not adequiltcly recognized. For instance, in finance the classical methods for forecasting /ITt' ofu'll not very lIS1'ful, but, by IIsillg adoptive tcchlliquC8. interesting re.ults nre often obtained. TIl{> last dutptpr of this book is devoted lO pro1>I(,lIIs inspired by r('a1 life. Solutiollh to the problf'llli'l ar(l found at http://wwv.imm.dtu.dk/-hm/ time. series. analysis. This hOUl(' page a!:;o coulnins additional exer<:i!';('S, called assignments, intendE'd for ht'ing solved lI!'ing n {otHpUlt'r with dcdieat('(1

software for tillLt, St'ri,,!; Hllalysis. I lUll grall'ful to all who hnv(' contributed with USC'ful comments and suggestions for improwlll<'lll. Esp('("ially, I would like to thallk my ("OII('agu{'S Ja.n 1I0lht, H(,llrik Spliid, Leif ~Iejlbro, Nie l,., Kjolstad Poulsen, and IIcnrik Aalborg Nielsen for t hdr vahlllbl(' ('Ommems and suggestions. FUrthermore. J would like to thank forlllN studC'nts Mortell Boicr O lsell, R llSlllUS Tillllstorf, and Jail ;\'ygaard Nielsen for thdr greul ('frort in proofreading and improving the first IllfUl lIS('ript ill DlluiJ.;h. For this 2007 edition in English. J would like to thank i)(>von Yates. Stig l\IortClbell, and Faunar Om Thonlfl.rson for proofreading alld tlwir wry uS('ful suggC'Stions. III particular, I run grateful to Anna Ht'lgfl J6nsd6ltir for her assist(UlCC with figures and examples. F'inally, I would like to thank 1\Iortell HIlghohn for both proofreading and for prop06ing ami ('r(,!lti ng a new layout in LNfEX.

Notation

Lyngby. J)1'tlIunrk

H('1Uik

fIf(UI.~(,11

All vectors are L"OiulIIll \'('('lQl':'j. Vettors Ilud matrices arc emphru;ized using il bold fOIlt. Lowercase iC'ttcrs ilr(' used for v('('tor!) and uppercase letters are used for mll.tric('!:i. Tr!l..Il~I){)bi llg is d(,lIot('d with the upper index or. RWldom variables are always writtell usillg uppt'r('IL'>C letters. Thus, it i~ no t possible to dist inguish bctW('('B n Illultivariate random variable (random vector) and a matrix. Ilow('\'cr, ralldom variablC!'i Hn:' assigued to letters from thC' last part of tit, IIlplml)('t (X, Y, Z, U, V, ... ), while determinist ic term!) arc assigned to letters from the first part of t ll(' Itlpllllhtt (a, h, c, d, ... ). Thus, it should be poosiblt 1.0 distingu ish hclw(.'(..'ll Il mutrix a nd a random \"ect.or.

CHAPTER I

Introduction

Time series analysis deals with statistical methods for analyzing and IIIcxit'lillg an o rdered ~('(lllt' lL et' of observations. This modeling rcsults ill a stoch(lStic process modcl for the system which gCllerated th(' dala. The or(\l'ring of observations is mO/'lt often, but not always. through time, particularly in terms of t"qually spaced time intcr\"uI8. In some applied litenHurt', time seri<'S are often called signals. In more theoretical literature a time serieR is ju;;t till ob~rvcd or lIIc8..<;urcd rcalizatioll of II stocha...,tic process. This book on time' series analysis focuses 011 modeling using linear model!!. During the s<'<jUC'IlCC of chapters lIIorE' and more advanced models for dynamic ~rstcms afE' introduced; by this approach the family of li near time series models and metbods are placed in 1\ struct ured relationship. In a subse<went book, lion-linear time S(>ries models will be considered. At t he sluue time t he book intends to provide t he reader with nil understanding of the mathematical aud stat ist ical bo.ckgroulld for time series analysis allel mod('ling. Tn general the theory in th is book is kept in a second order theory framework. focu88ing Oil the second ordt'r cliarRcteristiCll of the pcrsistl'nc(' ill ti me as measured by the alltocovariancc and autocorrelation fUllctio ns. The separation of linear and nOli-linear time series analysis into two books facilitates a clear demonstration of the highly different mathematical air preaches that arc needed in eneh of these two cl\ses. [n linear time s('ries analysis some of the most important approaches arc linked to the fnet that sUI>erposition is valid, and that clns.<;ieal frequellcy domain approaches are ciirt'Ctly u;.;abl(". For nOli-linear time series supcrpooitioll is not valid and frl'qucncy domain approaches are in general not very lL~ru l. The book call be seen as a text for graduates ill ('ngi neeri ng or ;;cience d{'partments, but aL'iO for statisticians who want to understand the link betW('('n lHoci('ls IlIld methods for linear dynamical ~ys t('ms and linear stochastic processes. The intention of the approach taken in this book is to bridge t he gap ixotwE't'n ~ci('lItists or ellgio(!o('rl:!. who often 11IIn' a good understanding of IIlt'thods for dcscribing dynamical syst('ms, and statisticians, who have a good understanding of slali;;tieal thoory liliCh as likelihood-ba.'>("(\ approa.chC'S. tn classical statistical analYMis lhe correlation of data in time is oflen dbregarde<i. For instance ill fl'gff'Kt,ioll analysis til(' f\.<;sumption about SC'rial

I NTRO[)! I('TION

l. J

EXAMPLES OF TIMF, SEIlIES

tUicorrclatcd residuals is often violated in practice. In this book it will b' demonstrated that it is nucial to take this autocorrelation into ac<:oulll in til' modeling procedure. Also for applicfllioll.!; such as simulations And fOT(,(,lLSling. we will most ofl('1\ \)(> able to provide mllch morc rCf1.!>Ollablc ami rcnJistie results by taking the autocorrdntion into accollnt. On the other iUUld ad('quate lIlethods and models for lime series IInfll\'sis can oft(>11 be seen as a simple cxtcllhion of linear regres.<;ion analysis wl~cre previous ollM'rmtiolls of til{' dependent variable are induci{'d ns {'xplawllory \'ariables in a simple linear regrcs.,,>ioll t,rlle of model. This facilitates a mther C1L'>Y approoch for IIl1d(>rstnnding tuany method'! for time series anHlvsis. as demonstrated in various dmptcrs of this hook. Theft' arf' a lIIullber of reru;ons for studying time series. Th{'S(' indtlri(' n charact('rizution of time S('ri(>s (or signals), understanding and lUodcliu!1; tlLt' datn gellerlltillg syst(>nJ, forecl\.>;ting of future valucs, aud optimal cOlilrol of n system. In th(' rt'St of this chapter we will first consider sollle typintl lillie S('rics and briefly mention the rl.'usons for studying them and the methods to mit' in ('I\('h eas('. TlwlI sOllie of the important Illcthodologit!s lind models art' illtroduced with the help of 1\11 eXllIllpk' when> we wish to predict the 1II0llthly wllellt pri('{'S. Finlllly the contents of the book is outlined while focusing on the Illodel structures and their ba.':Ik rclfllions.

<..>
00
00

"

00

"

Q3

Q4

QI

200l

Q2 Q:I 2005

QI

QI

Q2 Q3 2006

QI

QI Q2 2007

Figure 1.1: Daily US dollar' to Sum iutcrooflk CLcitaJlge mtl'.

~
~

~
~
~

1.1

Examples of time series

In this M'Ct iOIl we will show examp!t.-s of lillie serie;. and nt the same time indicate possiblt, appliclltions of time series analysis. The examples COlilitill both lypical ('xamplcs frolll ecollomic :;tudics and more technical applications.

1995

1996

1997

199b

1996

2000

2001

2002

1.1.1

Dollar to Euro exchange rate

Figure 1.2: !\umber of mOllthly fUn be ,'jet'n in the serif's.

airlill~ IHUUf'fl!Jl"rs

in thl" US. A cleor annual t'oriotion

The finot ('xllrnple is the dnily US dollar to Euro interbank exchange rnte shown in Figure 1.1. This is a typical economic time series where tilllt' S(>rie!ot III1a1ysis could Ix> used to fonnulat(' II model fo r forecasting future vnlues of the exchange mle. The analYSis of such II problem relates to the models and methods d<'S('ribcd in Chapters 3, 5, Illid 6.

1.1.3

Heat dynamics of a building

~ ow let us consider a mort' tCi:'lllli('lll t'xHlllple. Figure 1.3 on the following

1.1 .2

Number of monthly a irline passengers

Next WI' consider the IIUll1\)('r of 1II0litbly I\irliue passengel'!' in tht' US shown in Figure 1.2. Fa!' this Sf'rips 1\ elCll( al1nual variation is S('(' II . Again it might be II;-,('ful to c(mSl ruet a model fo r making forPC'a."!t.'; of t he futu re lIuIIII)(>r of airlinf' pa...SC"ngefS. ~lodl'ls and lIIeth(xls for analyzing tillle S('ries wit h M'a.'iOnni variation an' dl'M'ri hc'(l ill ChapH'rs 3. 5. and (i.

pa w' shows meilliurcments from an unoccupied test bui lding. The data 011 t he lowcr plot show the iudoo!' !lir t('lIIpl'mtufe, while on the upper plo~ the a mbit'lit air telllpefaLIIf(', t h(' h('l\t supply, and th(' solar radiat ion a re shown. For this exnmple it might b(' illteresting to characterize the thermal behavior of the buildiug. As It. part of thaI tht' so-('allf'<1 resistance against heat flux from iu:-;ide to outside can 1)(' estimated. T he rcsistanc(' chnrocterizcs the insulation of the building. It might nbo IX' IIst'ful to ('Slahl ish It. dynam ic model for lht' building and to f'Stinu\I(' thl' l illlf' constants. I\I1Owlc'(\g(' of t he time oonstanL'l ('an be used for designing optimal oolllrollel'S for the heat supply.

'I

I NTR.ODUCTION

J.2

A FIRST CflASH ('OURSE

Input variables
~

E
~

" S! ";

5"" -

~-

f"
~

r..luskral

~o :e~oo

"'" :t
0

'"

~~~~~~~~~
On. I [ 1983 Oct 12 1983
Oct 13 1983

1 850

lSOO

18iO

Oct 11 1983

Output vllriable

Figure 1.4; Armuallll troded skinA of mu.;kmt and lIIi,tk by the Hudson's Bay Company afler' logarithmic irn'l$for~lIati(m . It is 'lOt lx>ssible to classify olle of the series as input and the other 8rne., as OUil)Ut.

''''''

18!Xl

1!J()()

I!)JO

" '2
"

~~
tN

g)

during the 62 yellr period 1850 19J L 111 fad t he population of muskrats depends on t he population of mink, and the population of mink depends on t he number of muskrats. I n sud I Cfl..';t'S both series must be included in a multivariate time scriCii. This series has been considered in many texts on time wries analysis, and the purpose is to dC'i;Cribe in gell('ral the relation ootWf'(' 1l populations of muskrat and lIIiuk. l\kthods for ullulyzing such multivariate series are considered in Chapter 9.
0:00 12:00 Oct 11 1983 0:00 12:00 Oct 12 1983 0:00 12:00 0:00 Oct 13 1983 Oct II 19S3

1.2

A first crash course

Figure 1.3: Mt'.QStcIYmM.I" from an tJrloccupied te.~t building_ The input t'ariable.8 all"

(JJ solar rodialion, (2) ambient air tempernlure, alld (3) heat input. Tile output va"'ablt' as thf' indoor air temperature.

Let IL.., introduce some of th(' mO!iI important concepts of time series a.nalysis by considering an example whcn' w(, look for simple modcls for prcciicting the

For this case methods for transfer function modeling os descril){'(1 in Chapter 8 call 0(' 1lS('(J, wll('re the input (explanatory) variables arc the solar

Inonthly prices of wheat. In the follow ing, IN PI dellote tht' pri('{' of wheat at time (month) t. The first naive guess would be to say that the price next month is the same as in this mouth. H(,llce, the predictor is

radiation, heat input, find outdoor air temperature, while the output (d'l>('n. dent) variablr is till' indoor air temperature. For the methods in Chapter 8 it is crucial that all the signals can 1)(' classified as either input o r out pu t. :-;cries related to til(' sysH'rn considered.

(1.1 )
This predictor is called tile naive predictor or the pen~istent pn:dicior. The syntax used is short for a prediction (or (,!!limAtP) of PHI given the observations Ph PI I, .... ~cxt month, i.e., ut tilllC I + I , tll(, 1\('lnal price is P1+ 1. This means that the prediction error or innot'afion ma.... be computed as

1.1.4

Predator~ prey

relationship
11

This f'xllmple ilIustrntcs

typical multivariate time S(>ri(':;, sinc(' it iJ:i not

p~sibl(' to clru..o;i(v OIl(' of thl' M'ti('s as input and the other Sf"ri'S lL'l output. Figur<' 1..-1 shows 1\ wi(\f'l.v studied pn.'(l.at,or~prey CIi.SC, ntllnd.v til(" M' rif'S of

tullluully Imcl('d skillf.; of muskrat and mi nk by til(' HmL-,()IIf.; Buy C()Iupnny

( 1.2)

i NT IlO1)UC" ' ON

1.3

CONTENTS AND SCOP E OF T IlE BOOK

By mmbini ug &:jUntiolls (1 I ) Illld (1.2) we obtain tile stoch(L.'1tir lIIothl for t iJ(' wlll~at pricf'
( 1.3)

the modeh; can be written in a more (''Ompnct fo rm. The AIl( l ) model can be written as (i-r.pB)P, and !lieS('iL'>Olllli nl(x\C'1 in (1.7 ) as

=E,.

random variables (wMle Floise), the process (1.3) is ('ailed a mndom walk. The random walk lIIodd is vcry often seell in fimul(;c nud {'("{)lIolll('lrirs. For this model the optimal predictor is t he naive predktor (1.1). The random walk can be rewritten as
h;
Z(,TO melUl

If {ed

n s<'qU{' IIC(' of unt"Orrc!ntt'C!

(1.9) If \\-e furtherlllore iUlroduce tile dijJrrY'ncf' operator

'V

(1- 13)

( 1.10)

(1. I) which shows that the random walk is nn integration of tlu::' lIoise, auel that tll(' varian('(' of PI is unbounded; therefore, no stat ionary distribution exists. This jJj illl example of n nOIl-stutionaql ]JTocess. IIow('y('(, it is obviolls to try to {'Ollsidcr the mOTC general model
( 1.5)

callN:\ till' A R( I ) Tn()(lr/ (the fllltoregressive firs t order model). For tl lis proc"("S.-; H stat iollary dbtribution Cx iJils for 11;'1 < I. Notice that the random walk is obtailll'(l for r.p J. Another ('andidate for fi model for wheat prices is
( 1.0)

\vhicil assUiliClI that the pri<.-e this mouth is explained by the price in tile samf' month I&;t yE'fLr. Thb. S('('ms to be a reasonable guCSl; for a simple modC'!, ~in("(' it is well kno ..... n t hat Wh('At pri('(' {'xhihiu; a seasonal variatiorl.. (The n oi~ pl"O<'eS.>;CS in (1.5) and ( 1.6) are, despite the notation used. orcoun;(>. IIOt tht, ::<8 llle). For wlwllt priC<'S it is obvious that both the actual price and the prict' in t he sallie month ill t hf' previous year m ight bE> used in II description of tilt' ex pected pric(' next month. Such a model is obtained if w(' fL'>-'i\tIll(' thnt tilt' innovation Et ill model ( 1.5) shows an allllual variation, i.e., the cotn bill('(! JIlodel is

thell tht'random walk canlK- written \1P, = E/ uliing a very compact notation. In this book thCS(' ki nds of notlltions wilt be widely used in order to obtain compact equutions. Given a time SfrifS of o llM'rvro monthly wheat prices, Ph P 2 ,, PN, thl' model slructmY! CUll be identified, aud, for n. given model, the time series clln be used for l)(.1mmeler eMi7llalion. The model idcnhjicatiu1t i:; moot often bo..:;cd 0 11 the estimated autocorrelation function, s ince, liS it will be shown ill ChaptN 6, t he autocorrdation fUllction fulfils the sUllie difference equat ion as t he lUodC'1. T he autocorrelation function shows how the pric(, is corl'('lal('(\ 10 pr{'violls prices; more specifically the autocorrelation in lag k, called p(k), is simply the correlation between P, and P, k for slalioullry pron'S.-;t'S. For tile lIIonthly values of the wheat price we might expect a dominant allllual vnriatioll and , hence, that the autocorrelatioll in lag 12. i.e" p( 12) is high. The models above will, of COUI"S(', be generalized in the book. It is important to notice that Ihe:sc prm'Ci>.'i{'$ n il ht'long to the more general elM'] of linear prON.'SSeS. which again is strongly rciat('() to the theory of lineM systems as demonstrated in the book.

1.3

Contents and scope of the book

(1.7)
l\lodt'ls s l1("h fL'l (1.0) and (1. 7) aI'' ca ll'd se.aBQnai models, and they I\r(' USN) vcry oftel1 ill l'('ollomNries. Notic(', thllt for~' = 0 w(' obtain the AR(I) model ( 1.5). whilt' for I{) 0 th(' most simpl(' !t('{\sonal 1110<1('] in (1.6) is obtained . Dy introdudug th(' backward shifl opautor B by
( I. H)

A~ m ~mtiolled previou~ l y, this book will concentrate on analyzing a nd modeling dynamical sy~ t emJ-; lL~itlg statist ical methods. The a pproR.ch tR.kell will foclls on the formulation of appropriate models, their theoreticul characteristics. and on links betw('('u tlie lIIellll)('T'S of the class of stochastic dynamic models cOIll;idered. 111 gelleraL the models eonsid ('re<i arc all linear a nd formulated ill discrete time. Howev{'r, sonl(' r('silits r('\al('(1 to continuous t illle models arc proVided. This SC'Ction d('S{'ril)('s liJ(' eOlllen!s of the subsequent chapters. In order to illus trate lht' r('latian betw('Cn various models, SOIlJ(' fUlldalllf'ntal eXlllnpl('S of the consider(.'ti lIIod('l'l art' olltli nf'd in tli(' following section. lJowever, for more rigorous dt'S('riptions of til{' d('ti\ils r('latro to the models we refer to til(' following chapters. 10 Chuptf'r 2 1he {"OIlCt'pt of IIIIlIt imriat' rR.ndom vR.riablt'!> is inlrodu('('(1. Thi~ dlaplt'r also introdu("(", U(-(j .... o;ary fUlldamt'lllai {'on('('pts such 8.', th('

lNTnootJe'Tl0N

1.3

CONTENTS AND SCOPE OF TIlE nOOK

conditional meall and the linear projection. In generaL the chapter provides the formulas Illld methods for adapting a second order approach for characterising random variables. The second order approach limits the attcmioll to fir"t and ;;('('Oud orcll'f ('('lIlm] 1Il0nWllts of the density related to the random vnriablc. 'This approach link:; closd.r to the very important SE'C:Ond order charact'risatiOIl of stochastic prO<'t'SS('S by the autocQvariancc function in suixseqllcllt. chaptNli. Although lillie series arc realizations of dynamical phenomena. non. dynamkal methods arc often used. Chapter 3 is devoted to df.'S<'rihing .~tatj(' "ux/fL9 applied for tillie S('ries analysis. lIowen'T, in the rest. of the book dynamical models will be considered. The mcth(}(l<; introduced in Chapter 3 ArC' all linked to the class of regression models, of which the general linear model is lhe moot important nlember. A hrit'f dCM'ription of the general linear model follows Iwre. III the following, let Yi dellote thl'dependent variable and x, = (XIt,X2j, .. xpI)T n known vector of p explanalory (or indcpendem) varillbl{'S index(>{1 by the !illll' t. The yf"tll~lTIllinear' model (GLM) is a linear relation between the variables which can be written

SystclII Input Figure 1.5: Schnnatic

n>pn'..If' ntafion

Output
of II ijnror "II.dem.

input

Xt

and the corre.pondillg OlltPlltYf b the c01lVo/ution

y, =

L
~

Ilk;]",

l'

(1.12)

,
Yt = L XktOk +,
k=1

The sequence {h k } is cailed thc impl1lse response jundion for the linear dynamic system. For physicul s.\'stellLs wilcf(' till' output dO(.'8 uot depend 011 future w,lu('S of the input. the sum in ( 1.1 2) is from k = O. lJased on the impulse rf'!o;pon!ie fU llctioll we will obtaiu th(' /1"Cfluency response function by u Fourier transformation, and the tm1l.'ljeT" f!lndian hy lIsing the z transfonnatiOll. A very important model belonging to the model class described by (1. 12) is the linear ditTel'f'1Ice efllL(J./ion ( 1.13) Chapter S considers stochast ic proc(>S.."t'S, and t he foclls is on the lineflf stocllru,:tic process {V,} which is defined by the <."ouvolutioll
}'t

(1.11 )

where j is 1\ !,NO IIlt'an nmdolll variable, and 0 = (0 1 , O 2 , . . , Op)"I' is a Vl.'Ctor of the l' parameters of the model. Notice that t he mod('1 (1.1 I) is a stntic model sinCl' nil the variJ:lblcs refer to the same point in time. On-line and rccursive lIl('thods arc very important for time series analysis. These methods provide us with the possibility of always ul>illg the IIlCbt recent data, e.g., for oil-ii Ill' predictions. Furthermore, changes in time of the considered phenomena calls for adapth'c models, where the parameters typically are allowed LO vary slowly ill tilll'. for on-line predictions and conlrol, adnpth'c estimation of paramcters ill relativcly simple model,; is often to be preferred, since til(' alternative is a rather complicated model with explicit lilll(,--vl\fying parametcrs. Adaptive lI1ethods for t'Stim3ting parameters in thc gelleral linenr model are considered ill Chapter 3. This approach introduces exponelLtial smoothil19, the Holt-Winter procedure, and trend modcl8 as importunt specil!l

L
k,'O

1;-'kf" l - k

(l. L4)

'''''''''. The remaining chapters of the book consider linear systems lind appropriRtt'
rf'iat{'(1 dynamicaI1lux.lelli. A lincar system converts an input series to an Output series as illustrated in Fig ure loS, III (,llIl.pt('r 4 we int roduce linear dynamic deterministic ~yM('ms. In this chapter. one should note that for raudom variables capital INters nrc \I!'i('d wh('rcN! for d('V'flllinbtic variahlt'S WI' use lower case il'ttf'f'l"i. Al; a backgrollnd for Chaptn .j, one !>bould be aware that for lin(ftf Hml tim('-im-ariant tlYlilclII>; the funduflleUlal rdation belwt"{'n tbe dewnnillbtic

where {el} is the so-called whitc noiS(' process, i.e., a sequcnce of mutually ullcorrelated identically distribut('(\ Z('fO mean random variables. Equation (1.1.1) defines a zero Illcan process, ho .....e\'cr, if the mcan is not zero the mcan Ill' is just added on the right hand side of (1.14). Notice the simi larity between (1.14) and (1.12). This implies that a transfer function can be defint'<\ for t he linear prOC.....tiS as for tbe deterministic linear systems. Stochastic procCS6CS with a rational transfer function arc the AJU..tA (p,q) proCf':S.'>, t lw ARlr-.-JA(I', d, f/) pro('es... , and the llluitiplir,ntive 8'&;Ona.l procesSt'8. These important processes arc considered in detaiL As an example the proccs..'l {Yt } g iveil by

(1.15)
where {eel is white nOise, is known Nl the A RMA (I', q) process. Notice the similarity betwet'n (1. 14 ) and ( 1.15). Such a process is useful for describing the data related to the dollar to Euro exdm.ng(, rate ill Section 1.1.1 , and the 8ea8oDai prooess models are US('ful for modeling the monthly number of airline pe.ssengers in Section 1.1.2.

10

I NTRODUCTION

1.3

CONTENTS AND SCOPE OF T i lE BOOK

II

G ivell a time series of observations Y\, Y2,"" YN , Chapter 6 deals with identification, estimation, and model checking for finding an appropriate model for tile underlying stochastic process. This chapter focuses on time domain methods where the autocorrelation fUllct ioll is the key to an identification. Frequency domain methods are typically lin ked to the spectral analysis which is the subject of Chapter 7. T he iiO--calle<l transfer /tmction models are considered in Chapter 8. This class of models describes the relation between a stochastic input process {XI} and the outpnt process {Ve}. Basically the models can be writtcn
Y! =

on the input-output relations. but also on the internal state of the system, is the class of stall; SWtCl; model..; iutroduced in Chapter 10. A state space model in discrete time is formulated using a first order (multivariat.e) differt'llct' C<llIilt ion dt'scrihing the dynamics of t.he stute ved01', which we shall denote X t, and a stat.ic relation betwccn the state vector and t he (multivmiatt') obS('rvation Yr. . MorC' spcdfically the linear stu/.f' .~l)aCC modd cousists of the sy..;tem equation
(1.18)

L h"X
k=O

t_ k

+ Nt

(1.16)

and t he ObSe11!ation cqlmtio71

(1.19)

whe re {Nd is a correlated noise procC5.'>, e.g., an AllMA(p,q) process. This gives rise to the so-called Box-Jenkins transfer flLnclion model, which can be seell as a combination of (1.12) and (1.14) on the previous pagc. It is relatheJy straightforward to include a number of input processes hy adding the corre.pondiug lIumber of extra convolutiolL" 011 t he right hand side of (1.16). An important assumption related to the Box-,Jenkins tran~ fer function models is that the output process does Hot influence the input process. lienee for the heat dynamics of a building example in Section 1.1.3, a transfer fUlietion model for the relation betwccn the outdoor air temperature and the indoor air tem perature Clln be form ulated. This model can be extended to also include the solar radiation a nd the heat ~upply (provid(,>(i that 110 feedback exists from the indoor air temperflture to the heat supply). In the case of multiple processes with no obvious split in input and output proces.'ieli, the llluitivariate approach must be considered. In Chapter 9 the multivariate linea' process is inl roduc(:d as a.n tn-dimensional stochastic process { Yi } defined by the multivariate convol ution

Yi.

tPkEt_k

(1.17 )

b<O

where tP is a coefficient matri.x, and {Et} the Illultivariate white uoi<;e prOCt'S.<;. T his form ulation is used in Chapter!) as the background for formulati ng the multivariate ARMA(p,q) P'YJCCss (also calk-d t he Vector-A RMA(p,q) process) and ot her related models. As Illentioned previously, the muskrat-mink Cil...,*, reprcscnts a problem which must be formulated as a multivariate process, simply be<;ause the popula tion of minks influences the population of musk rats and vice versa. Until now all t he models can be considered as input-output models. The p urp~ of t he modeling procedure is simply to find an a ppropriatf" model which rela tes t he output to the input process, which in lUany cases is simply 1h(' white noise process. An important class of modeb which not only focllses

where X I is the m.-dimensional, latent (not directly observable), random statc 1Jector. Furt hermore u / is a deterministic input vector, Yi is a vector of observable (Illeasurable) stochastic output, and A , B , ami C are known matr ices of suitable dimensions. Finally, {eu} and {e2,d are vector white noise processes. For linear state space models the Kalman filter is us(."(i to estimate the latent state vcctor and for provid ing predictions. The Kalman smoothe, can be IIsed to estimate the values of t be latent state vector, given all N values of t he t ime series, for l'! . To ill uslrat(' all example of application of the stat(' spac(' Illodt'l, eonsider II.gain t he heat dynamics of the test building in Section 1.1.3. ~ I adsen and Ilolst ( 1905) shows that a second order systcm is 1l()C(led to de;cribe the dynamics. Fu rt her more it is suggest.ed to dcfine the two elements of the s tate vector as the indoor air temperat.ure and the tem perature of t ile heat. accullluillting concrete floor. The in put vect or U t com.,ists of the ambient ai l' tem perature, the solar radiution, and the heat input. O nly t he indoor air t(>llIperature is observed, a nd hence, Y t is the measured indoor air temperature. Using the state space approach gives us a possibility of estimllti llg the temperature of heat a(TlIllllllati ng in the concrete floor using the so-called Kalman filter techn ique. In general, the paramet ers of t he models arc as.':!umed constant in lime. However, in practice. it is ofte n obscn'cd t hat the dynamical characteristics change with time. In Chapter J I recwsive and UdUl}tivc methods are int roduced. BII.<.; ica Hy the adaptive sche mes iutro<i llC'e II. 1ime window related to t he data, SITch t ha t newel" datil ob ll'liw; more influence than older data. Tbis leads to met hods fo r adapti ve for(''Custing a nd control.

CII APTER 2

Multivariate random variables

An rt-dimensiorwi mn(iom lmri(Lble (1TUldom "ector) is a vector of random variables. The random vector is written

II

scalar

X ~

X,

()(n

X ')

(2.1 )

Random vectors wililliso be clCllottxi multivariate nmdom variables.

2.1

Joint and marginal densities

En>ry random variable hf\."i 1\ distribution fU!lclion. The II-dimensional random variable X has the joint distribution jurtction

(2.2) If X is defined on a continuous sample space, the joint (probability) density junction is defined as
I ( x., .... x" ) _ 8"F(x" ... x .. )
-!:lo J::I

vII "

-vI"

(2.3)

and the relation bctwC('1l tiu' distrilmtioll and density fu nction is

F(Xl"",X'I) =

l.z;~

... i:f(t), ...

'tn)dtl ... dtn

(2.'1 )

A random variable, X . is cnlll'd discrete if it. takes values 011 a discrete (countable) sample s pacf'. III this ('sse the joint dcm;ity funct ion (or mass Junction) is defined as
/(XI, .. ,7.. ) P{X\ =x\ .... ,X.. = x .. }

(2.5)

The joint distribution and mass fUllc l ions are n-'lo.tl>(1 by

F(I\, ... ,J',,)=

L ... L
1,<", In<r .

f{th ... ,t,,)

(2.6)

14

i\]ULTIVAJUAU: RANDOM VAIIIABI. ..

.s

2.3

EXP~CTATION S

ANI) MOME:'OTS

15

For Il sub-....ector (X\, ... ,XIt)T (k < marginal density function is

fS(Xt, . ..

Il)

of the random v('{'tor X th<-

,.:rio) =

I: I:

f(x\, . .. ,In) dXk+l ... dIn

(2.7)

Th(' conditional distrib1ltion/ut/ction is then fOllnd by integration or s umm a~ tion as previously dCl;('ribcd ill (2. 1) uud (2.5) for t he continliOUS and dis{Tele cases, respectin'iy. It follows from (2. 11 ) lhnl

in the contiuuous cru;c, and

(2.8)

Ix.dr.y)

h x,.'(y)Ix(x)

(2.12)

and by interchanging X and }' on the right hand side of (2.12) .....e get Bayes' rule:

if X iii dbcTctC, III hoth

('a.'!(.'S

th(> marginal distribution junctioll is

.. () _ Ix ,. ,(x)!>(y) I )1\ .,.1/ h(x)

(2.9)

(2. 13)

PS(Xj .. . " II;,) = F(I\, .. .. :C,b~ .. .. ,x)

\\.{' now d efine indel)('ud('nl'l':

D EFINITION 2.2 (INDEPENIJENC":)

P]CIl.'j(' note that w(' will writ' Ix(x) Rnd Fx(x) instead of f(x) and F(.r), fcspccti\'cly, whenever i~ is ncccssaf}' LO emphasize to which rando m variubl(' the fll11('tlon bdoup;s.

X 100d Y a.n.' indqJt,ltdent if

2.2

Conditional distributions
which corresponds to

I x.> (r, y)

Ix(')Iy(")

(2.14)

In liul(' SNit'S Imalysis ('omlitional distriuutions plRY all important role. especially in relation to prroiction and filteri ng. For insta nce, ill prl,'(lkl ioll, where we wanl [0 gi\:(' 1\ slllt{'/ll<."nl about future valu(':oi of lh(' liml' series gi n'n pa...,' ohM'rVRtions, then the conditional distribution contains all available information about thc futurc valu('. u't A and B d('lIot(' sollie t't'''nL,. If P(B) > 0 thell tlip conditional probability of A occurring givcn that B occurs is

Fxy(x,y) = Fx (x) Fdy)

U X and Y arc iudcj>('nd('nt, it is d('urly seen that

(2. 15)

IYI' ,.(y)

Iy(,)

(2. 16)

P(A 18) ~ p (AnO) 1'(0)

We intcrpr('t P(A
SIlPPO~;('

(2.10)

Bear in mind that 11 two nmdom variables are independent. then they a," uL,o uncorreiatcd, while unoorrclatcd variables arc not. necessarily independent.

I il) Il.'l '"the probahilit.y of A given 8."' that til(' continuous random variables X and Y havc joint. d('nsity. /x,~. If we wish to liS(' (2.10) to d(,t('rll1ill(' the conditional distribution of }' givl'll that X tak{'S lll(' vallie 7, we have the problem that the probability P(Y :5 y I X = x) is ullddincd us wc lIlay only condition 011 ('V('lIts which lu\V(' a strictly positiVl' probability, and P(X = x) = O. lIowl.'vl.'r, /dx) b positive for somc ;r; Vallll'$. Therefore, for both discrete and COHtillllOIiS random vrinble; we II.W the fo llowing definition.
T he cOlldilio1lui
DEFINITION 2.1 (CONOI IION/\L m:ssl'ry) cl('n.~ity (flmdion) of Y given X = .r is

2.3

Expectations and moments

For a discrete variable X , til(' ('xp('('tA.tion is

ElXI

L XP{X ~ x),

i.{'., an average of tIl(' po~)sible valuei'! of X, e<tch value being weighted by its probability. For continuous variables, ('xpt,<.tatiolll'l Uf(' defined as integrals.
DEFI NITION 2.3 (Ex PEerATl oN) Tht' expectation (or mCan value) of a continuous variable X with density (unction / x is

I)

.\

_n(f,Y) r (Y ) !x(:r) .

(Ix (x) > 0)

(2. 11 )
find}' ili a.\"

..... h('rp / \) is t il(' joint d('ll!;it)' (unction of X and Y. Bolh X nlllh imrint(' random mrinhlC'l-;.

1)(,

EIXI

I:

xIx (x) d.<

(2.17)

16
this imcgml exists, I.e., if

MULTtVAIUATE RANDO~l VAnlAIlLES

2. I

]\ IOIlI ENTS OF MUI.TIVAnlA'n~ RANDOIII VAlUABI.ES

17

WhCIl{'VCf

:"-Jow let X be an
(2.18)

II-dilll(' Il ~iOl\nl

rnlll\Otll mriablc allo pnt Y = g(X). where

9 is a function. Ah in (2.20) we have

Ellxll < 00
~ot(' t hat

EIYI ~ EI9 (X)J

usually

w('

rulo,"" the existence of J g(r)d;r only if JIg{x)ld.r

<

Xl.

~ J X ... J X g(.rl,"" .r,,)J.-dx] .. .. ,.r n ) (ix i ... dX n

. ""'" - :x;

(2.21)

Remark 2.1 The e.xpectation opcmtor E can more generally be defined by a StieUjes
i.(' ..

In particular. setting g(Xt.X2) = aXt

illtcg''fl~

+ bI2 implies

that
(2.25)

E[a X,

+bX,1 ~ aE[X,J +bEIX,1

EIXI

~[

: xdFx(x),

(2.19)

Thus, the expectation opcmlol' i.~ a /inca I" OJH'futor. The {'Iemenlos X . and X J in lite random vector X (2 .21 ) can be uSf'd for
(kfining mixed. moments

where F, i:; t.iJ{' dil'it rilmtion (unction for X. T his d(>fi nitiOIl covers both discrete a nd cont inuous variables. .. TIlt' exp('('t lltioll is called the first. moment, becausc it is the first moment o f Ix with rCtipcct to the line x = O. If X is 1\ rnlidollL variable find 9 is a function , then Y = g(X) is also a mndom variable. To calculate the expectation of }' , we could find h lind tiS(' (2. 17). Ilowewf, th<' pro('('&'! of finding f\ can be complicated; ins tead, we

E [X~X;J),

(2.26)

where Q and j3 are integers alld similarly for t he corresponding cenlral molll('lIts. Of special intcrest is the covariance of X, 11ll(1 X)

CovIX" X,1

~ ~

EI(X, - EIX,I)(X, - E[X,1l1 EIX,X,I - EIX,I EIX,I

(2.27)

can exploit that

EII'1

EI9(X)1

J :

Thp covariance gives information about t he s imultaneous varia t ion of two variables and is u~f\ll for finding illterdl'Ilt'lldcncies.

g(x)!x (x)dJ.

(2.20)

By !l1>plying (2.27) and the linearity of the cKpectation operator, the following important calculation rule fm' the coranullce: Cov[aXI

we

obtain

+ bX2,CXa + dX,) =

This provides a method for calculating the "moments" of a di:;triblltioll.

DEFINITION 2. 1

+ Q(lCov [X\. X .. ] + bcCoV[X2 ' Xa] + bdCov(X2 X"J

a.cCOV(Xh Xa)

(2.28)

(h lom:NTS)

where X I, .... X I are random vllrillbl{'S and a, ... ,d are constants.

The Il 'th mOf/lC1lt of X i:;

EIXnl ~
and the n 'tli cc.nt7YJi moment is

EI(X

EIXJ)"I

I: I:

2 .4
(2.2 1)

Moments of multivariate random variables

,n !x(x) <ix,

We now consider IIlultivariate random variables. In time series analysis v.'C oftell usc on ly the first llIomcm (the mC1I1I valuc) and the second central moment (the variance). Therefore, it is helpful to have the following d efinit ions:
(2.22)
DEFINITION 2.5 (EXPECTATION OF TilE HANDOM VECT OR )

(x - EIXJ)"fx(x)dx.

The expectation (or the me.an v(llue) of the rondom ,'cctor X

is

The second cen1ml moment is also called the variance, and till' variancc of

X is givcn by

(2.29)

V"'IXI = E[(X

E[XJ)'I ~ EIX'I

(EIXJ)'.

(2.23)

18

i\IULTtVARIATE RANDOM VAlUABLES

2.4

t>.IOMENTS OF r-.nJLTIVAHIATE IUNDOM VMIIAllLES

ID

DEF1'1tTION 2.6 (COVARIANCe)

The col'uria1lct> (matri.c) of X is

.. Remark 2.2 U, for inMall('f', E is positivc dcfinitc, we ofum write 1:

DElm~n ION

>

O.

Ex = Vllr!Xj
E[(X - I' )(X _ /, )T[

2.8

V",[X,]

Cov[X\.X 2 ]

Cov[X],X,,]

Iand a
(2.30)

,Thf' covaria11ct: mabil: of 1\ mndom \'('('wr X with dilll{'llsion p and mean J.J.. random \"('("tor Y with dimensiOll q and mean v is

Cov[Xl Xtl
Co"[X,,. X,J

V.,[X,[
Cov[X". X,] ...

eo,,[X,. X,,]
V",[X,,[

Exy ~ C[X . Y ] ~ E I( X

I' )(Y

v )TJ

(2.3 1)

Ex is called the oovurian('(' matrix of X.

SomC'timcs
W('

shall

U!iC

the notation

It can be clearly Sft'n that C{X . XI

VarlXI.

Tn EormM 2.2 (CALCU LATION RULES Fon TilE COVARIANCE)

(2.3 1)

Lt't X and Y be dejifu;(l (Ul in /)('jinilion JUi, and let A and B be 11 x p and m x q real mahices. Let U and V II(' IJ- ami q-dillun.~iollal mndom Ilccior8. Thrn
C[ A (X

For the variance', u;, we sometimes usc the notation u". Th(' rorrdatiOIl of two random variables. X, and X j is R normaliUltiOIl of the covariance and can be written

+ U), B (Y + V )]
Uf1"

~ A

q x, y [a T + A C[ X. V ]B T

+ A C[U , Y ]B T + A C[U , V ]B T
Important sp<:cial casc.'

(2.35)

Cov[X;. Xjl = p'} = JVlU'[X,J VarlXjJ

U'j
U,Uj

(2.32)

V.,[A X I ~ A V",[X]A T C[X+U, Y [ qX. Y I+CU. Y

(2.30) (2.37)

Equation!; (2.30) and (2.32) ll'ad to the definition

DI:.:FtNITlON 2.7 (CORHELATION MATRIX)

The conrlalioll matriJ for X is

I
R~ p~
p"

Proof FoUow!; dir('('tly from lh{' ddlnition of tht' {"()....arianct' and the linearity of the expectatiOIl Ol){'rator..

p"
I

p,,,
I

:
(

P'' )

Compare til{' rules abo ....{' with t he rules for !;cillar random variables gi ....en in (2.28). (2.33)

P'I]
TltEOREM 2.1

Example 2.1 (Linear trnnsfonnations of randml"l variables) Let X he au lI-dillll'IISiOtllll nllldOll1 variahl<- with mean JLx and (."()variallc('

Ex, :\ow \lit' introduC'(' a n{'w linear transfoflluuioll

ntudOIll vnriHhll' Y = (Y] ...... , Yd T by till'

The covariU1lN' matrix E (J1Id tht corrclatiml matrix R arc (a) .llYlllmctrir 1l1ld (b) positivt' semi.definite.

a -t BX

(2."ti)
flU:!

Proof (a) TIl(' ~~'lllInf'lry is ob"iolL'. (1)) USP Var(zTXJ:2:: 0 for all \'11111('1; of

when- a hi a (k x I) vt'Ctor A.nd B i!; a (~. x 11) IJIlltrix . B~' Il~illg tllf' the expt'ctation opc.'ro.tor hi IiIK'Rl" Wt' fiud
py ~ E[Y] ~ E[a

thClt

+ BX]

= a

+ BE[X[

a + B/,x

(2.39)

20

!\ I Ul.'llVAIUAT E HANDOM VAllIAB LS

2 .5

CONDITIONAL Ex p~~C'rNnor-;

2[

Rnd

b~ \L'iillP;

tile !-.tate rule; (2.3!j)

Mild

(2.37) for cakulati ng the oomria nC('

Equation (2,18) sho\\'S that the conditional expect.ation operator is li near,

DEFI NITION 2.10 (CONDI TI ONAl- "A IHANCE)

Ey = Var'a t BXJ = VllrlBXj

BVar[X ]nT

BExBT

(2. 10)

The conditiowd vm'irlnce of Y Kivell X is Picas(' note that. sometimes COyi', -I i!-. used also in the mult ivariate cu.<;e. H..ud the conditional covari(Htce I)('tw('('n Y a nd Z giv('n X is

(2.49)

2.5

Conditional expectation

C[Y , Z IX [ ~ E[(Y - E[ Y [X I)(Z - E[ Z [X [( X l

From R{'umrk 2,3 it is clearly matrices of random mriablcs,
SC'CII

(2.50)

The conditiOIll\1 f'xppctation is t he expectation of 1\ random variable, ~iv('n wl.lues of another random vuriabl(. Late r it will b(' shown that t he optimal predictor (in tl'rlllS of minimum varinnce) is the CQuciitiollui meal], Conditional Ul('ans are a lso used in filtering.
DEFINITION 2.9 (CONDlTI OI'AI. EXPEerATIOI\)

that Var[Y IX [ and C[Y , Z IX j are

TIlEOREM 2, I (TilE VAJUAN('E SEPARATION T IIEORE\I)

[..{'i

X , Y awl Z be rundo,n I'ariablf.'s, Then

TIl(' o:mditioMI t":rpeciation (or c01ldiliollal menll) of tile random \1\riablc }' given X = I is

E[YIX
.. Remark 2.3

~ r[

!:

V.,[YI ~ E[V"' !Y [X [[ + V",[ E[Y IX II Cry , Zl ~ E[C[Y . ZIX[[ + C[E[Y IX I, 8[Z [X II

Proof Qlllit.l{'d
S<'e.

(2.51) (2.52)

"Iy xoz(y)dy

(2.4 [)

for instance, Jazwinski

( 1970),

If ~'C know tilt valu(' X = .(1 and lh t' condit ional dCllsity function, thell w(, HrC able to t'akulat(' th" conditional In('all. For anolht'r vah1(' X = Xl we will get

A~"'IlIll('

Example 2.2 (Linear model ) }" is dl'finro by the lin{'\J' Illodel

a different value for lhe L'Ondil iOli al mean. lienee. the conditional exp{'<:tutioll of Y givt>n X x is written '4-'(x) = E[l'I X = .'). Il(>(,AUSC t he conditional expectatiol1 ocp('nds on the vHlup J' taken by X, we cnn abo think of lit{' rollditional (>x('('tRtion as a function I,:'(X) of X iL~Ir. ..
THEORE~I 2.3 ( PnoPE IITIES OF' ('O:-;DlTIONAL ~ IEA ""S)

v=

XO+ f

wher(' X and art' lIIutually inci('p('lldl'llt random variables wilh 1II('llIl P.\ alld I', 0, and \'arianc(' 0'\ nud f('SJl(.'<'tiwly" \\"(, a."mnl<' that (J is known_ B.\' u:-;illK the Ih('Qr('lIIs fil)()v(" it follows that

(1;.

ETIX' ~
Let X, Y IIwl Z be mndom variables (with joint
rml numbers, and 9 is a Teal fUII('tion. Then
dUI.~it!J

qxo + 'IX [

SO

/x.}',z) . a ll1ul b alY'

Var[Y IX l

Var[XO + fi X ] -

0';

HrUt", WI' M'f' Ih,.j for a give-It X J', we haw thl' "pn'(lktion" ElFlX xl-= x8, and til(' c-orr{'spolldillg ull('l'Ttainty is given hy tIl(' variance Var!}'I X ~

E[YlX] = E[l') ,

if X lInd l~ a1'(' i1ule,>endent

(2. 12) (2.13) (2.11 )

x[ ""

O'~ .

E[Y[ ~ E!E!Y[XII Elq(X) 'I X[ ~ g(X) E[Y[XI P.19(X)Y[ ~ E[g(X) E[YIXII E!"IXI ~ a
E[9(X)IX~ = g(X)

Tilt' lUiUgillill nwan of ). i:-;

(2.15)
(2.16 )

and by IUliug (ViI) Wf' gt't th(' marginal vnriam'('

V"'I)I

q\'",[)'IX!J + \ ",[E!) X I

(2. [7 )
(2 '")

u; -+ (J:lui

E/rX
Proof Qmitll'il

+ dZYI

- c E[X, )" . dE[ZP "

(2.11)

but follows frolll

on pagt' 11 nlld (2..11).

Hence, the variance eeparation theorem }ieidlo tbat tilt' IImrginal \lU'bl.ll('(' of Y .. COIDpOIJl!d of the variance of plU8 8 8CaIed contribution from t.he .-....ofX.

22

~ I UI:rlVARIATE RANI)OM VAIlI,\Il!.ES

2 .7

DISTRIBU TIONS DERIVED ~llOM TilE: NOIHIIAL OISTIOI3L;TION

23

2.6

The multivariate normal distribution

2.7

Distributions derived from the normal distribution

In time S(>ri('S analysis th(' normal distribution and the distributions derived from t he uormal distribution lITe of major interest. For eXfllUplt, tilt' lIIul ti\'flriate normal dbtributiOIl is t he fundamental tool used for formulating the likelihood fUllction in later chapters. \\'p assume thllt XL. Xl , ... . X" fire inde pendent random varifl bl~ with 2 1 2 II'C wnte . X ,E."( ') . Illpans It]. Jl2. .. , II ... , a n d v8n8Uct'S ~ It" (1; 1 , (71.- ... On' :\ow, defint' tilt, mndolll vtoclor X = (XI.X! , .... X .. )T. Bet'auS(' til(' mlldOIll

~ Iost of the tc:st qUlllititics 11.';('(1 ill tilllt:' series Ilnalysis Ilre imsed on the normal dbtributioll or o n 011(> of the dislributions d('rivoo from the normal d istributiOIl. Any lincur combinalion of 1IaTilt/dly distribuird random t'ariables is normal. If. for iruitanc{', X E ;:>';(1-'. E ), the n the linear trallBformation Y = a + BX defines!l norll1811y dbtributcd random variable I\S

varillhl('S

flrC'

indC'penciC'nt, it follows from (2. 14 ) on page 15 that

(2.56)
Compare with Example 2. 1 011 I>agc 19. Let Z = (ZI, ... , Z,,)T 1)(' a ,"('('lor of independcllt N(O, I ) random mriables. The (central) \,:1 dist1"ibulion with It dCY1r.:CS 0/ f1'Ccdom is obtailU.'d IL'! t h{' SQuared SlIlll of II ind pw'ndellt N(O, I ) random variables, i.e.,

I.dx\ ..... r,,) = lx, (;cd'" 1... ..(;1',,)

" - -1e x p [(Z, ~ '")'] II , IO,,f'ii 2a~

m;:., ".') (2,)"/' exp [~;

t. ["' .,"']'j
(2.53)

X2 =

" 2: Z; = Z",Z E \,2(n)

, .. I

(2.57)

By introducing l ilt' mpan J.L = (1'1, ... , Il"rr Il.lld the covariance Ex = VarIX ] ding(a? ... . a~), this is writte n

From t his it b dear tllllt if 1'1, ... , Yn Il.l"t' il lde pt'ndent N(/l" a;) random variables, then

h(x)
A gell('ralizatioll to the Ipll.eI..; to til(> fo llowi ng:
(I\SC:'

(2.58)
since Z, = (Y; It.)/a, is N(O, I) distribut('(1. For Y E t\n(l-' , E ) (E > 0), wc ha\"C

wht'J"(' the covariance matrix is n fun matrix

DHT.... ITION 2.1 1 (Till:; MULTI\ARIATE NOIU1A I. D1STRlIll'TIO~) The joi nt d{'lIsity fUll ction for thp n-di mensional random \.. riable X with mean 11. and covariance E is

(2.59)
This follows by using Theorem 2.5 a nd (2.57). The 1W'l-tenlrol,\ 2 di,~tnbulioll with '1 dt'gt'et's of freedom and non-centrality parameter'\ appears whcn consid('ring the sum of squared normally distributed variables when the nU'aliS art' not ll('('('SS8rily zero. Il ence,

j,(r)

1 ::--:-c;;;....r.=~exp [ '( x I-'f E '(x (211")"/2 Jdf't E -"2-

1 1. )]

(2.5 1)

where E > O. We write X E N(I-' , E ). If X E N(O, 1) we say t hul X is slandardil.t'(l normally dbtributed.
TIIEOREM 2.5

(2.60)
A ny ll-dimflMirJUlIl nonwdly disI1'ibll/(:d nmdom Ilori(lb/t' wilh m((w I' am[

rou(JI"iallcc E ran be written as

I. + T e

(2.55)

where'\ = I I-'. COIll PIl1"(, (2.59) I-Uld (2.60). Let X?, .. . , X~. denott' independent ,\ :I(n" '\,) distribu ted HlIldom variables. Then the rcpnxiuctiolt InY)IHT/Y of t h{' X2 dbtribution is

4/,"'E

Proof 011\' to liJ( l"oyllllllt'lry of E , there alwll.Ys c'xisi.... 1\ f('nlllliltrix so lIull TTT. TIII'Il ih(' f('l-;ult follows from (2.1O) on pAgt' 20 auel (2.:m) Oil E

f: X? E \' [f: f: ",j. '0

u ..
1 ,. I . I

(2.61)

If E is singular with milk k < II, thm y7:E Y is \ 2 distribut{'(i wit h k degrees of rreedom and nOll-N'ntmlit.\ pa.rauwt('r'\ 4I-'T E - p.. wherf' E -

24

l\IULT IVARIATE RANDOM VARlt\lILES

2.8

LI NEAR PROJECTIONS

25

denote:,; a gPllefalize<i inverse (called g-inverse) for ~ (SC'C, for instance. Rao (1973) ). The (stU/inti) t di~lriblltion with 11 degrees oj fn:cdom is obtained fl.... Z T = (X2/n)1/'1 E t (n),
(2.62)

Y - E[Y IX [

where Z E X(O, I). X 2 E :\. zen), and Z and X 2 nr(' indepeudcllt. The nOHcentral t distribution is obtained from (2.62) if Z E N(IL I). and we wl'i!p T E 1(11./'), The F' distribtJtion wilh (IL. III) drg/'r,n of jrT't'dom appenrs as the followinA ratio
p ~
where

x
E[Y IX [

X -'/ 2 m.

xUn

E F(n, m)

(2.63)
Figure 2. 1: The projection E[Y IX ] ol Y on X .

X; E \ 'len), xi E "Z(m), IUld X; Ilnd xi are incicp<'udent.

It is cleml.\"

seen from (2.62) that T2 E F (l, 11). The non-reflt.ml F disi1'ibution Wil h (n. m) (l('gret',~ of freedom Rnd nouCt'ntrality pammctn' >. is obtnilwd frOI11 (2.63) if X~ E "Z(n, ..\), E \ 2(W), and nud xi Ilr(' ind('pendcnt. The non-central F distribution is writtcn F E F(n, m; '\).

X;

xi

Proof From T hrorf'1ll 2.4 011

pl\gt'

21:

C[ Y , X [ ~ E[C[Y, X IX II + C[E[Y IX [, E[X IX II

~
~

E[O [ +C[a+ BX . X [
B Vru-[X[

(2.68)

2.8

linear projections
E[YI
which l(>nds to
~

This seclion t'ontllin:s the fundalllental theorems used ill, e.g., linear regr('s;;ion, where the iudpp('nd(,llt variables arc stochast ic, 1:IS wt'll as in linear pre<iictiou:s and the Kalman filler.
TIIEOREM 2.6 ( LINEAR PROJECTION)

E[E[Y IX [[
a
~

E[a

+ BX[ ~ alB E[X [,

E[Y [ - B E[ X [
(2.69)

Let Y = (YI , . , },,")T and X = (X ., ... , X", )T be nmdo"l vectors, and let the (m + n)-dirlll"f1llional vector (Y . X )" have tlie mam
and cotJal'iance Define the lin('ar projcction of Y on X
E[ Y [X [ ~

i.(>..

[{Iuation (2.65) is now ohtain<'<i by using the vaim.'S for a and B ill (2.61).
~ow

E;Vru-[YX[[

",,[Y - E[YIX [[
Var[Y

,,+ BX
=

BXJ

Then tlte TJr'Ojcrtio71

and the variance 0/ the projcctiO'l e'1m' M giuf'1I by

= E yy

+ BExXBT a BX . X ,

BExy - EyxBT

(2.70)

E yy C[Y

(2.G5)
E[VarIY IX jJ f""inally, the p,'Ojc.l'lion error, Y
C[Y

I: y x Ex~ Exy

CIY

E[YIX[. X '

Eyy - EyxEx~E;:x (2.66) E[Y IX ], and X al'l' unrorrelatcd. i.e.,

E[Y [X [, X [ - 0

= E yx

BExx = 0

(2.71)

(2.67)

OrlhogOlHl[

Rcf('rring to (2.67), w(> !'fay thnt the error, (Y E[Y iX '), fIIHI X ar(' (I.'" illustratro ill Figure 2.1. TIl(' liu(>l\r projt,(tioll.c; a boVl' 9i1'" the

26

11uLTIVARIATE RANDOM VA IlIAHLES

2.8

LINEA II PIlOJE('TI ONS

27

minimum variance amo1lg aU iiI/tar projections. This is s hown in the

Ilext

The variancc of tll(' prt'(linion crror is

thoor('lIl. Since we (lCC moot1r inlcr(':;l('<1 ill tlw ''<lrianre, we !\SSumc for the moment that. J.Ly = E [Y j = 0 and J'x -= EIXj = O.
T ,mORE\f 2.7 ( PROJECTION T Ii EOHEM)
I~et

Vflr[}'l- p:~}'I] ~CO\' [}2

= a1( I

Ihe mwlom vcctors Y ami X hlUl{' zero mean. Theft tlte lincar
E[Y[X[ - BX

projectiOl~

(2. 72 )

T he resu lts can also b(' ~'n dir('(~tly by lIl>ing Th(>orellt 2.8 bt'low. Sitl('(' (}/i. y:.!)"r is tlorllllllly dist ributed. the optimal litH'/\!' prroidor is the optillllli predictor.

which gives the minimu m variant(, among all linear proj('('tions is found by
8011';'19

III general E[ Y IX j is not necessarily lin{'ftr in X . This is illustrated by til(' fol lowing example: Exa mple 2.4 (Non - line ar E [Y IX ]) COIl!olid<'r th(' deTlsity

C[ Y

BX , X [- O
fIlly

(2. 73)
other linear

P roo f Assume that BX sat isfi es (2. 73). Let {3X denote projcction. Then

Ix ,} (J'. y) =

~ (J'

+ uk

-(.0:+11)

(r. Y) E (R~)

(2.7 1)

Va,[Y - {3X [ - Va< [Y

~

BX

+ (B

- (3)X [ ~ (d .. " to (2.73)) (3)T

'The marginal dt'lisity for X is

- V",[Y - BX[ + (B - (3) Va,[X[(D

Va,[Y

DX[

(2.75)

T ht' conditional deu.sit.\' for }'IX

=;r

is

The theorems noon' art' ha<;('(1 all a se<'Ond order representation for til(' "('Ctor (y T, XT )T, and the projcctions nrc r~tricted to lX'ing lin('ar projections.

. () _ h,,.(r, y) f ~.\;rY - I\(:r)

Now the con<iil i{)lIIt l (>x p <'<'tfl. t iOIl
lx'COIllt'S

- ,+./

_ r

+ Y, ,

(2.70)

Example 2 .3 (L inear pred ic t o r ) Let (l'\, )'2)"1' UP normally dbtrihutf'd with Zl'rO mean ali(I covariance

q,,) , .
q,

E[I 'IX

rl

1""
()
,

;r +!I -lid y--e y

~.r

TIl(' corr('latioll
pag(' 18.

ix-tWC'{'1I

Y I and l'2 is thcn p =

012/0102,

d. (2.32) on

(1

I +.r

00

i('-IIdy +.r

1""

Jj('- I/(ly)

(2. (7)

As.... utn(' that 1'1 is obscr\'('(1. The projection lhl..'Or('m ubove can 110W b(' us('(\ to find til{' optimalliu(,llr prt'(lictor f2 = (!}'I by solvinp;

I 2 +.r --(2 >r)--I +.r I +.r

HenC'e, EYIX j - ~:~, which cl{'arly i~ nOIl-lillt'ar ill X.

it; linear.

whidl I('nds to

For Ilol"l llully dhtrihuted random varia hies, however, the conditional mC'an Wt' hnw til(' following tllt'orcm.

n\( o ptimnl line.ar pmlictM L .. tllUJi

TH EOHI::~' 2.H (PROJEeno:-.;s FOR :'>IOIlMA I .I.Y I)ISTlllUtTED VAIUABI.ES) Let (y T. X T)T be a normally di.<;iribu/{d rYJlIdom l'"';(Jblt' milk ml'aTl
and COt'w"iaFl(,("

28

fo, l Ul.TIVAR I ATE RANDOM VAn I AB I. ~:S

2,9

PIlOIH.I~MS

2D

Then Y IX is 110tilwlly llistributcd with mean

(2.78)

The solUlion h", is known ItS tllf' Wiener' jilttr ill d~ret.(' time fLnd the WiellerKmm.l9Qrow filter in cOlltili tLOtLS till\('. S('(" also Wieller (1949). The vari811('(' of the r('('OU.'itrn('tion error b var[x,H

arid t'ariallct

VariY IX ] =
Fiuthrnllore, the rfTOr', (Y

Ey y -

:EyxEx~E;:x

(2.79)

E[YIXj). and X are independent.

,. "

tht., )~]

=CO'o'[XI

k-

f::h l.,Yi.XIH]
,"'I,

(2.82)

Proor Omitted

S('(',

for instance, JazWiIL"ki (1970).

E[Y IXJ) and X

~:h('re the right halld side rollows

b,v lISillj.\ (2,SI). In collununicatiOIl IltC'Ory a rn>qupntly used mood is

~ Remark 2.4 Without the 8.';Sumption about normality, the error (Y

Yi

X,

+ (',

(V~3)

were 1I1l('orrt'lfttt'(i; hut if we consider nOflllal diSlribulioll, til('Y Itn' also iudt"I)('ndcnt. <I
~ Remark 2.5 A morc mathematical treatmcnt of t.he projection thcorem involves inner prodm'! spaCt'S; S('(' ~l f\.{ lsetL (1992) or Brockwell ami Davis ( 1987). In til(' formulation uS('(i above, til{' illlll'f product is defined by the covuriuJlcc, i.e.,

wh('n.' ) ~ is th(' 1II('&ur('(\ l'ignal. X, is th(' true signal, and et is !loiSt', Tlt'llel', thp \ \'iener filter givNi nil estilllat(' of til(' trll(' Signal. Another ve ry important ('xampl(' is the til>!.' of tlte li near projection theorem for deriving the KnillliUl liller, whidl is dt'al t with ill Chapter 10,

(X , Y ) ~

qx, Y ).

2.9

Pro ble ms

An applkalioll of lilf' proj('('tioll theorem is illustral{'(1 ill t il(' following

importam t'xamplC',

Exercise 2.1 lA>t the two random "Ilriablcs X alld y, and their corresponding sC('Qlld order
moment repres<'lltation , be givell by

optimal linear reconstruction) COllsidt'r two lOut ually ('orN'intt'li Mocha,-'.;tic pn:>c'-'~ {X,} and {) i }, bot h with nll'IlU \'lIhll' 1:('[0, ;\S,'iUIIlI' that the aut<,K-'O\(lriaIlC(' and tr~""('{)\'lIriIlJl(""
fllncliom; an' known (t h('S!' fUllctiolls, which arC' introdm'(..-l f()r1IlRl\y in ChaptC'r 5, F;hl' a IOtlu S('('olld lIlonll'nt representation for {X,} and P,}), WI' ('()Il~idl'r tilt' probll'lIl of ('!o;tillll'l.ting thC' \1l.hw of X, tI< h~' using 01>""[\'('(1 vaiu~ of ) ', at lilll(' f" fl t 1, . -', /2, I\ote, that no rl':;trietion 011 tht, vniuC' of I.: i~ gi\'t'li 111 pmftil't', how(''o'('r, WC' m()to;t oftf'n hR'o'e I.: 0 or k' ":). 0, Tht, opliUlIlJ liJwar proj{'(:tioll

Example 2.5 (Wiener Hlter

EIX ) ~5.

Var[X I = )2

ElY)

10.

Var [YI = 22 ,

p(X , ),.) =

Two ncw random variables 1-1 a nd L ar(' now d('fin('(l by

J-/ = 2X +31'"

L = ,X + 2Y
('akulatp the S('('t)lld or<i('r mOIll(,lIt n'prCht'litation of Exercise 2.2

(/J L)T,

~\;,," =

" Y; Lh"i , "

(2.KII)

COlisider the linear Ulexi('l

is, dup to Iht' Pl'Oj('('lioll Ihf'orf'IIl, giVPII tt.'i the soitllioll In

(I+{iX +t

" "' )~ 1-0, L""')

"

:0;.

i,. ' '

"t~

where (I and fJ IU(' C'OllstRnts, X and, nr<' lIIutually t1llcorreiatcd ralldom variabi(>S with ,m ean vnlucs E [X J I'X And E[eJ = 0, variances VarIX) = O'~ Bud VRr[e] = find ('(Jrrt'inliolll'l p(X, )') p,

0';,

Note that in (2.Ml) the numlM"r of equatiolUl is equal to unknown alUM of he.

the number of

Q uestion 1 Filld E ;n,\"~ IIlId VIU'WI,\' Question 2 Find

EYI ami

Var. l 'j.

30

'\\U LTIVAJUATE RANDOM VAlUABLES

. 3 "lol,lHl moment NlUmatcs Question .:.:; <, .7 n set of ('(Illations to determine the . ') 'ill'rc pIX Y) nppcars. (liint: t-.lolllcnt csllmat('S nrc found by o f ((I. ,/ \.\ , . 'I, .... moments of the random variables as a function of the pamlllcLcrs ('xprt .......;lI1v, , and 1)\('11 solving for til(' SCL of unknown parameters. )

C II APTER 3

Exercise 2.3 I'" '\ ~""'~ I.('1 \ ' l" or,

ddilll,(\ as

It

.,..

~"'(Iu('ncc of random vnriablcs. A new random vurmbl('

j'

.
IS

Regression-based methods

wiwn' X is

random variabl(' with the

&'l.DlpJe

space ON

:=

{I. 2, ... }

lUl(l

E[NJ
It is a\:.o given that

20,

VarIN] = 22

In spite of the fact that A 1illl(' se r i~ i!i IlII outcome of a stochastic proct.'S:> And, thus, an obscn'ation of a dynamical phenomena, methods, which are usually rdated to lile a nalY!iis and modding of htatic phenomena, arc oftellllpplicd. A class of such methods is closely related to the standard regression analysis.

E[X, [N

jJ

E[X,J

2
r oF s

V"'[X;[N ~ jJ ~ V"c[X,J ~ (k)' CavlA'r. X. IN ")1 = Cav[X", X~l = O.

Question 1 [kt('rllliu{' EW] and VnrlYj.
Question 2 D{'Il'rllliu(' til(' rovariane('

3.1

The regression model

Cov[Y,ZJ
wil('re

The dnssical rcgn-s... ioll lIIodel is 1ls<...Q. to dehcribe a s tatic relation b('twccll Il dependent variable Yt and p independent variables X It, .. , X pt ' In lime serie; analysis, the obscrmtiOIiS occur successively in time and most frequently with an equidistant time distance. Therefof(', all index t is introoueed to denOle the variahle Ilt time origin t for other applications t denotes an arbitrary index. In its most general forlll the rcg,.r.!f,~iolt model is writtell

(3, I)
Z = XI

+ X2+ .+X.. ,\,.

a EN

where I( X I t; 0) is Ii known mmhcmatical function of the p + I independenL variables X l"'" (Xu, ... ,Xpd T und t, bUl with unknown parameters 0 = (0 1 , ,0m)T. The independent variable t is introduced to indicate thal tlIP model dflSS de.crilx'<l by (3. 1) COlllnins lIIodels where I is a function of l, c.g., l(t,O) = Osin(wt), and CI is n random variable with E[cd -:- 0 and Var[cl] = FUrthennof(', it is 8,.,sullu'<.l llll11 CoV[CI"CI)] = O""l E 'j, i.e., O"i::: q"l E ... Finally, Ct and X t arc as!iullwd to be independent. For tile regre;sion IHo<id (3.1) for Yi given X t = Xt, it holds:

u;.

ElY, IX,

x,J

f (x"t, O)
O"~

(3.2) (3.3)

Vllr[Y"X t =

xtl =

('ov[Yi" l~JI Xt

xtl = 0"2 E'J

(3.4)

where 0";, = O""l~". The lotnl variAl ion for Yl dcpelld301l the variation of X l; see Theorem 2..J 011 p!l ~e 2 1. In this ,haptC'r WC' ;;111\11 r('Strkt OUl" attt'11liOIl to models with fix('d and nOIl-stochn...,U, inciC'pendf'llt \1\rinhl('S. 1.('., X t X t. 1I0 ...."('v('r, most of the results arc nl'iO mlid for indcp<'nd('nt mndom variables .

REGRESSIO N-IJASED METIIODS

33

Example 3 .1 ( Dlrcct. radialiol1 in clear skies, part 1) In St-ctiolll'i:U throup;h 3.3 .....(' ..... mlx> following a ca.<;e study to show how the mpthucb, un' Ilpp1i('d to soivt' H rt'ai problem. The pllrpOM' b to C'Staillish
a stochftloltk rllodd for dir('('t radiation in clear skie, as a function of till' solllr ek-vahull. TIl(' radiatioll i~ kliOWIl to d('I)(,lId ou other vnriabl('f> as .....ell, ... uch 88 waiN \1\1'01" IIlld (l('rOliQl.s. but thhe dependf'nrie.llrt' iguon'fl. How('H'r. hy nm!;idNilig 1\ stodu..... ti( Ilppronch, til(' modd is able to give all illdir<'Ct drsrription of ('freels dm' to variations of the ignor(>(i \"ariahl(~. In this I'xnlllpj('. only solar radiation in clear skies is conbid('rNI. '1'111' ()IItIt'r\"tstion~ Hr' hourh mill( .... from Fl'hruary I, 1006. t hrongh D('("(1II1)('r 31, 197:l, ..... ll('r(' tl\(' total num\x>r of hourly observations was 69,3&1. Only 1.183 ul~'f\ations of til<' totE,1 "l't currl'Spond to clear sky during da\"timc. Figure 3.1 ... hu" ... 111(.... (" Obl>(T\l1tiolls as a fUllction of the solar eleVRtion. FUrttwnnor(', tll(' figur' ... how" twO dav ... with cl('ar ~ky during til(' entire day. The direct radiation is actually 1I0t 1lU'3Sur('(1, but is caJculatPfI using obs<,t"ved. hourly 1Il1'(\..'illn'III('lIts of Aloha! O 'I,~t"b) Imd difflL'i(! P 't,d!r) radiation , along with the wlnr ('l('vntion, hy isolAting the direct radiatioll in til(' runellHlI('ntnl n'lal iulL'..llip

.' ....

~ .

.' . . .' ,.- ..~~~ . . '!... : . .....: .. :- :. ...

'.~' ~.'"

'

,"

, .. . '. ; .,

"- 2/7 1968

]()

20

30

40

50

Sohlr elevat ion, deg. Fig ure 3. 1 : Now/y ob.~en.'('(i riilY'ct mriialiOfI ill d ew JJI.:iCfJ
a.'J

a fWlction of fJOlrlr
filial

(3.0) wlwn' lt.,I;t is t ilt, Imj\''iufl'(l din'(l radiation at lilll(' I, denoted ~ }'i in thl' fullowillJP,. A dt'l(,fmilli:-;til' IIItxll'1, dl,:,>(Tibinp; thl' relationship h('tw('('n dir('('t radiation alld solar ('J('vatioll. h".'i 1)(,('11 ~IIAAt'St('d by Paitri<igt' and Plall ( 1976):

elevation. Two dayfJ wiUI denr s},"y for Ih e enti,.,. day are shown, alOl!9 with UIC mooel.

3.2

The general linear model (GLM)

The gencral linear model is H v('ry irnporttl.nt special case of tbe rcgression model (3. 1) wh{'r{' the funct ion f is lillC'ar in the parameters 9.
D EFI f'tTlON 3.1 ( CENEltAL LINEAH ~ I ODEL)

wl1('r(' h, i ... tilt' ...olar p\('vation at timE' I, f(hd is thl' ('orn"'pondillg din'(t radinli()lI, ami Ol ond O"l Uf(' paramctcn. of th(' model. suggestoo to be IOOO\\:/ml nud O.06dt'g-l, n>spE'ctiwiy. As lIleUlioJl('(illhov(>, til(' "'(JIM (I('mtiun Illom' b lIot abl(> to d('S('ri\)(' th(' total variation. Thus. th(' 010(\('1 is expan(h>d to the 'ilotlla.'itk lIlodel
(3. 7)

The genemllincar' modd (eL M) is

model :-.tructure
...

l\

ft'gressioll model with the following

YoI

-X, TO + E"t

(3.9)

where XI = (:rll,.' . is a k nown vector and 8 = (Ol , ... , 0p )T are the wlknown paraul{'tcl'l). E"t is tI. nl.lldOIll variable with mean E{c,J = 0 and co....arian("(' Cov(Et"E"'J] = (J"2 I: f).
Let us consider some eX6mpl('i;

,xp,f

wilerI.' tl\(' d'via\iol1 from the dl'tNministic model (3.6) is des('ribl'd hy tilt, random variabl(' :t. ('onsirkr now titt' variall{'t' :-.tnlclure fur ft . By isolatinv, Iill' din'Ct mdintioll from (:Hi ) airel l\.'isUllling that the \'ariance of tlU' 1lU'a..'illrt'IIWllts of }~.gtol. and YI.tI,f are (ollstaBl, a l"ea.<;(.lIlahll;' rt'iatium.hip for the m.rinnc(' i ...

or a gCllcmi lillear model.

(3. 10)
(3. 11 )

a) Constan t m can mode l

y, = Otl +E",
xI I,

Var!)",I ' Vn./"!etl =. ~ -- .

Mil

0 =00

(h,)

. b ) Simple linear regression model

}j
XI

This hudtucy i8 n'<-ogn.iu-d in Fi,;uJ"P 3.1, where it is Meen that the variallC(> tlf tht' difC("t radiation ~ 88 ~ 80Iar elevation incl ! _

(I, .::d/ ,

00 -+0 , ::,+0:1
(J -

(3.12) (3.13)

(Ou.Od'/"

31

3.2

THE c.:EN EHAL L INEAR MODEL ( CLi\I)

35

c) Quadratic mode l
YI

An estilllate for till' variarll'I' of

0 is

00

+ B\Zt + 82z; + C{
8 = (OO.OI ,02)T

(3.11)

XI = (l,;:j,z;)T.

(3.15)

v"'liJi
where (j2 = 5(O)/(N _ pl.

2,' [;>(8 ) 1

L,

(3.25)

d)

~Iodel

wit.h two independent variablos

YI
XI

00 + O\Zll

+ 02Z2t + Et
O =(OO,O\ .(}2)T

(3. 16) (3.17)

.. Remark 3.1 TIl(' term uruL'f'ightcd is UM'{I sin('(' 110 <'O rrsid{'ration is taken of re;idunls y,,.hich may ha\'c a larg{'r varillllc{' or r('!o;iduals which may be correlated. lI{'nc{', ullweight{'(l least squar(':; c:.tillmt<'S aT(' only reasonable if the mrianc{' of th{' residuals is COII!>tlllit (i.<'., :E" = i) and til(> residual,<; are mutually independeJlt (i.e .. :E ij = 0 for / ::f j). Lllt<'r ill this S{'(tioll \V('ighted least squar<'S est imates will be cOllsidert:,'<1. In ordN to dblinguis ll b('tw{'<'11 t he two types of 11'IL>;~ :.quares estimates. ulI\vdghtl'd It'l.:;t ~qll aT{'S ar{' often called OLS f'.~'im(lt('s (Ordinary LclL-;t Sqllll.n'S) and the weighted ~tillll\t('s are called IVLS csli1llat('.~ (Weighted Least Squares). .. Sill(,(> it is not possible ill gl'll('ml to filld an explicit expression for tll(' LS estimator. IIlIlJl('Tical I/l{'thods have to be 1IS('(1. A cla.ssic-hi reference to nOIl-lillear parameter ('!o;lilllfltioll is Bflfd (197-1). 3.2.1.1 A simple numerical procedure

( I ,Z]t.Z2,)T,

e) A mixed mode l

y,
~I =

00

+ 0 1 sin( ...../) + 02f.1 + OJz\t + 04Z21 -+- 95Z11::21 + C/

l

(3. 18) (3. 19) (3.20) (3.21)

Cl'IIlIl ot

o {1

I ~ 200

< 200

(step variable)
Z2t, ::\ tZ,u)

x, = (I, Sill( ..... '), .,. Zit,

"

0 = (00 ,0\, 02 , 03,OI,O(,)T

The mO<)f'1 l 'j 00 +OI(sill(wl 9~ +E:, is not a linear mode] sin('(> it be written on the form Y1 xlo + Ct-

3.2.1

Least squares (LS) estimates

L<-t us IL";UIll(' thnt th(' jltlrfllll('lers in t he regression model (3. 1) 011 pag' 31 or the general linear moor! (3.9) on tilt' preceding page arc unknown. Then lhc~' mny lx' ('Sli llll.ltcd from the following N observutions of the dep,lIdcnt and independcnt variahl(>!;
(3.22)
thes<' observations Wi' want an e>timatc 8 of 0 such thal !(x,; 8) dcscriix's thc OOOcrvutions as well as po&:>ible, as defined by some mCfI.'mre of clOM"Ilffi'S. Ir til(' SllIll of Mluar('(1 ('rrors (SSE) Ely,. - !(X,; O)J2 is chOS('n lI'{'1l Based
011
w{'

It is well knowlJ that for ullconstrained optimization , the value O' which millimi7.e. S. will S3lbfy the {'(lUlllioll

",,8(0')

(3.26)

A Taylor expallsion of 5(0 ) in the neighbourhood of 8 = 8 0 is

have

S(O) - 8(0 0) + (8

8 0)""',S(00) 80 )

( LS eSTIMATES) Thc Least Sqltat'Cs (unwei,qhtcd) estimates arc found (rom

DEFINITION 3.2

+ ~(8 - oof H (8 0 )(8 2

(3.23)
11 (00)

+ ..

(3.27)

where H (6o) is lilt' so-cnllC'd l1essian mutl'iJ, given a.,>

(j = arg min S(O),

wh('r{'

0' S(O)

8(8 )- 2::1", ,,

!(x" 8)J' ~

2:: , ,,i(8 )

80, eo) 9 "", 9~

(3.28)

(3.21)

i.{'., (J is thi' 6 thnt millimi7('S til(' s\IIn of squarl'd re;iduuls.

T he 1IC$.... iaJl mat rix is /I ~yrulllf'tr ic matrix with dime nsion equal to lire lIumbcr of PllfIUII('lt1'!i ill 6 . It is ru;:;uIIll'd t hat thl;' partial derivnt iv{':-j 1If(' continuous ill 6 0

36

I1EGRESS10N ~BASE I)

METHODS

3.2

TIlE GENER,\I, 1 . I NEMt 1>101)1::1.

(CLi-. I )

37

An approximation for S{O) in 00 is

5.",(0) = 5(00 ) + (0 ~ 00l' 'V ,5(00)

Proof
5(0 ) = ( Y ~ xO ),(Y ~ xO ) 'V,5(0) =2xor (Y ~ xO) = 0 i.e., O' is found from x T zO = zry '" (3.3-1) and (3.35) follow dirC'Ctly.
THEOREM 3.2 ( PHOPt:;HTII::S 01 TilE

+ ~(O ~ ool' H (Oo)(O ~ 00)

(:J.29)

The minimum of the quadratic approximation is found by putting V' ,S..pp(D) = 0, tllHt is
'V,5(00)
with th('

+H

(O o)(O ~ 00) = 0

(3.30)

The LS estimator

gipl'n

LS ESTIMATOR) by (3.35) has the following I11Vperties.

By putting the old wUlle 8 0 = 8, fmc! denoting the new vuiu(' 0, +1, \\'{' ('ud lip NelJ!tOll.Raph.~on method
(3.31 )

i) It is a liFlmr function of the obsnvalions Y.

ii) Jt is 1mbiaaed, i.t' .. E(O] = O.

iii) Var[91= E[(l1-0)(O-O)7] =a 2 (x T x) -I,

which is a ll it('mlivc method for finding 0 , which is based 011 suC((~..,i\'{' qlladrfl.lic a pproxiTwltiol1s. Otlwr numerical procedures, C.g., the Gau.u-Newtol/. method, t1Sl' the fac t that the optimiznlion is a lcru.t squares problcUl.
iv) jj is IJLUE (8e.~t UW'1tr UYlbiased Esti71latm), '/lIhieh means thai. it has the .91l1allest variance a1ll011g all f'slimltiors which u; a lincar function of the obscnJali01l..~ (111 1 ' soc(lilcd Gauss-Mm'kov theorem).

3.2.1.2

lS estimates for the general linear model

For the gCIl{'mi lincar model it is possible to find an explicit expression for the LS ('!>liml1lor. C'oll:;id('rillg th( linear model (3.9) Oil p<'\g(' 3:J, tii' obs('rvatiOTls
can be writlen

Proof
i) From (3.35) we have lhat (j = LY , where L = (x T X )- I X 7' ii)

(3.32)
0'

EO! =

E [txT xl 'xT y )

y = ::z:8+
As RS1;uUI('(1 ill the ddiuit ion of the linear model (3.9) ......e ha\'{' Var'j E[T] q2:r:, where E = [:E,j].

(3.33)

E [(xTxl 'xT(xO + E)) = (x 1'x )' '(x Tx)8 = 8

iii) \Ve have

(3.36)

Ele]

0 aud

(j - 8 = (x T x)-'x 1' y = (x T x)' 'x T E'

which leads to

(J

= (x7'x ) IXT(XO + E') _ 8

3.2.1.3

Unweighted lS estimates

For the case of no weights we have E = I .

THEOREM 3.1 (LS EST1MAn;S) Least SfIUU1Y!S f'l!li"wt('.~ fur 8 in lilt? lineAl" model (3. 33) art fOlmd by .901rling

Vnr[6) = E [exT x) 'x 1ee7'x{x T X) -I]

(x 1'x) IX'/' E[uT) x(x T X)-I
(12{X T X) -I

1M nonnal equation
(:1.:11)
iv) GlllittN\ (3.35)

d. Krndall and StU/lfl (J9S3).

36

I1EGRESS10N ~BASE I)

METHODS

3.2

TIlE GENER,\I, 1 . I NEMt 1>101)1::1.

(CLi-. I )

37

An approximation for S{O) in 00 is

5.",(0) = 5(00 ) + (0 ~ 00l' 'V ,5(00)

Proof
5(0 ) = ( Y ~ xO ),(Y ~ xO ) 'V,5(0) =2xor (Y ~ xO) = 0 i.e., O' is found from x T zO = zry '" (3.3-1) and (3.35) follow dirC'Ctly.
THEOREM 3.2 ( PHOPt:;HTII::S 01 TilE

+ ~(O ~ ool' H (Oo)(O ~ 00)

(:J.29)

The minimum of the quadratic approximation is found by putting V' ,S..pp(D) = 0, tllHt is
'V,5(00)
with th('

+H

(O o)(O ~ 00) = 0

(3.30)

The LS estimator

gipl'n

LS ESTIMATOR) by (3.35) has the following I11Vperties.

By putting the old wUlle 8 0 = 8, fmc! denoting the new vuiu(' 0, +1, \\'{' ('ud lip NelJ!tOll.Raph.~on method
(3.31 )

i) It is a liFlmr function of the obsnvalions Y.

ii) Jt is 1mbiaaed, i.t' .. E(O] = O.

iii) Var[91= E[(l1-0)(O-O)7] =a 2 (x T x) -I,

which is a ll it('mlivc method for finding 0 , which is based 011 suC((~..,i\'{' qlladrfl.lic a pproxiTwltiol1s. Otlwr numerical procedures, C.g., the Gau.u-Newtol/. method, t1Sl' the fac t that the optimiznlion is a lcru.t squares problcUl.
iv) jj is IJLUE (8e.~t UW'1tr UYlbiased Esti71latm), '/lIhieh means thai. it has the .91l1allest variance a1ll011g all f'slimltiors which u; a lincar function of the obscnJali01l..~ (111 1 ' soc(lilcd Gauss-Mm'kov theorem).

3.2.1.2

lS estimates for the general linear model

For the gCIl{'mi lincar model it is possible to find an explicit expression for the LS ('!>liml1lor. C'oll:;id('rillg th( linear model (3.9) Oil p<'\g(' 3:J, tii' obs('rvatiOTls
can be writlen

Proof
i) From (3.35) we have lhat (j = LY , where L = (x T X )- I X 7' ii)

(3.32)
0'

EO! =

E [txT xl 'xT y )

y = ::z:8+
As RS1;uUI('(1 ill the ddiuit ion of the linear model (3.9) ......e ha\'{' Var'j E[T] q2:r:, where E = [:E,j].

(3.33)

E [(xTxl 'xT(xO + E)) = (x 1'x )' '(x Tx)8 = 8

iii) \Ve have

(3.36)

Ele]

0 aud

(j - 8 = (x T x)-'x 1' y = (x T x)' 'x T E'

which leads to

(J

= (x7'x ) IXT(XO + E') _ 8

3.2.1.3

Unweighted lS estimates

For the case of no weights we have E = I .

THEOREM 3.1 (LS EST1MAn;S) Least SfIUU1Y!S f'l!li"wt('.~ fur 8 in lilt? lineAl" model (3. 33) art fOlmd by .901rling

Vnr[6) = E [exT x) 'x 1ee7'x{x T X) -I]

(x 1'x) IX'/' E[uT) x(x T X)-I
(12{X T X) -I

1M nonnal equation
(:1.:11)
iv) GlllittN\ (3.35)

d. Krndall and StU/lfl (J9S3).

Example 3.2 (Simple linear regression model) For til(' stodlll.'ni(' pr()('l'~/r8Ildom '~rtriable Yi defined by
}'j
(t

P roof Same as for Theorem 3.1 Oil pugc 3G. However. when calculating thc derivatives olle should apply, for any sYlllllletric N x N-matrix. A , it holds:
OT

+ (iI, + ,

A{:J = oT (fjT A Tf
~

the ol);..Crmliom; ( FJ ' .rd ..... (Y",.;r.v) are available . ....or nil t ht'S(, oi)M'rvati()l1~ till' 111\)&1 i~ writtt'li

Wr" A''j,,)7'
fjT Au .

( l" ) ~ (: 7 ) (~)+ ; )
Y"
I
X\ EN

when a and (3 arC' N-dimensional vceton;. i.e. ,

S(8 ) ~ ( Y - x8 )TI; ' (Y - x8 )

(3.41)

According to (3.:J5) IIII.' LS

~tilllntor

yield<;

ii
that is.

V,8(8 )
From Iwre the resl follows.

(3.42)

(3." 3)

T he variance of the estimator (3.10) is

TIl(' \'llril\ll{'(' is

Apart from t his, the (':0;1 iUlalor has III(' propcrties Illelltioucd in ThoorclIl 3.2 on page 37. It i8 also possible !o find 1111 t,~tillll\tor fo r the variance, (12.
THEOREM 3. I (ESTIMATOIt Fon TilE VARIANCE)

An unbiased estimator' for' (12 is

3.2.1.4
Let
liS

Weighted lS estimates
introdu('p .....e ig hted least squares estimutes for 8 in til(' lill('llr lIl<xlpl

_2

~---

S(O)

(Y

xO)'I' I; '(Y - xii)

(3.41 )

Y = x8+

(3.37)

IJ.!h(;f~ .Y is the number' of ob,'f'n.'(tii01L' lUlfll) is the nlJmber of parameters.

Using E [E] 0 and Var[E] = E[u T ] (12 E , ",here E i:. as:sunwd knowu, the weighted least sqtwres ('stimate.s for 8 art' found by minimizing

P roor Omitted

M,{,

K<'Ildall and Stuart (1983).

S ~ (Y - X8)TI; ' (Y - xO)

TIIE:OBE~I 3.3 ( WE IGI ITED

(3.38)

LS

ESTIMATES)

The. 1L'eigli.ted least squares estimate for 8 in the linear model. is found by solving the uOrHllli ('(jlLll! ion
(xT E -1 x)O=xT E -1 y If x'l'E
I x IW,.'I

.. R emark 3.2 A weight{>(1 lea.';t square; problf'1II can b(' expressed as an Ullwcighted lellSt !!quares probicill by 1\ suitable transformation of the variables. The tran:.formation i~ givell by T heorem 2.[; on page 22, which shows that we can alway!; Slid I) linear tranSfOl"llllltioll determined by E = TT7', such that e in (3.37) Cflll be writ tell as E: = T e. T hllt is,

'1"

(3.39)

Y = xO+Te
",Iu're Vaf[e] (12 T. 13y IlItiitiplyillg both !"iides of the above equation by T - l , th(' weighted le8.';t squlln'S problem is brought into lIll lInweigltt('(1 1('Il.<;t squares

ftJll milk

U!t

91"1 (xY E- IX)" I xTE -1 y

ii =

(3.-10)

problelll.

..

40

R ECHESSIONBASEO ~I I~T IIODS

3.2

TH E G ENEnAL UNEAll MODEl. (CLt-. I)

41

3.2.2

Maximum likelihood (ML) estimates

Proof Frolll (3. 16) it follows

\\"(, shall now (xmsidcr maximum likelihood est imators of tht' parnlllPt('l"s 9 and q2 in the linear mudd Y = :to + ~. T he i\I L estillllU~ IIr(' ha..,ed on nn assumption of normality in E and thus also in Y .
THEOREM 3.5 (~ IL E:STI~IA.TES)

DlogJ...(0,u 2; y ) Du2 =
= -

l\' 1 1 q 2+2u4(Y
[u
2 -

-7' x9) E

(y - xO )

~ ~'I

~r(Y - xO)TE~I(y - %0)]

Consider the N-dimensional rYmdom toariable Y rtd let Y E :-.'(x8 ,021: ) tdU'T'f> I: is assumed known. Then the maximulll likelihood estimator f01' 8 i.~ equit.aient to tile It:(1.~t IJqUlUl: estimator
(3 .45)

Solvi ng for q, it can be 5('('11 that the estimatc (3.47) impiie; that the partial derivath'c is O. A comparison bt,twt't'n (3. 11) and (3.17) shows that the estimator given by (3.47) is biased. T HEOHEM 3.7 ( PnOPERTIES OF TII~~ ML ESTIMATOR) The AfL e.~ti"wlor given by (3.45) JutS the /ollowi'l9 properties

Proof The density for Y is

i) i t is a Unear function 0/ the OU.WT1.1/ltions w/tir/t now implies that it is

" (y )
Sinc(' rhf' lik('lihood function is equal to the simultanf'OllS density for oil the Oi)M'rVilliolls, the logarithm of t he likelihood function, L, is
"21og(21f)

normully disl,fiullt.ed.

ii) It i.~ 1Lnbia..~r,tl, i,r" E[O ] ':" O. iii) It has variance

V..

2 log(dct I:} - "'2 10g (n2 )

(3. 16)
x 8 )+(',
Wl'

I8] ~ EI(O-

0)(0 - On

~ (x T I; -' x) -' u'.

ivY It is an effiril'rlt ntimatm.

Proof il. ii ) and iii)- see the proof for Theorem 3,2 on page 37 and usc that Y is now normally distributed. iv) omiued.

2u'

_1(.

xO )TE-'(y _ xO)

l\' 2 1 1" "2 log(u)- 2(12 ( y - x8 ) 1: 1(11

where c is a constant. By differentiation with respect to 9

obtain

a.<.:;umed that E is known. If E is unJ.:nown. one possibility is to use the reia.xatio'l algorithm (Goodwin and Payne 1977. p. 49) , which has the following steps.
bt'ell

Until now it has

Relaxation a lgorithm Setting 'Ve1og L = 0 implies that the maximum lik{'liiJood estimator (MLE) (j is found by solving the normal (>(]uut iou a ) Select a value fo r I: (e.g., I: = f ).

b) Pind the estimates for this vailic o f E , e.g., by solving the normal
Ifx7'I; IX has full mnk, (3.45) is now obtained..
THEOll!:;!.!

equations.
c) Consider th(' r<'Siduals tid and calculate the correlation and variance structure of the residuals. Then select Ii new value for I; which rcflects that correlatio n and \Ill.riall('t' structure.

3.6 (t-. IL F ...<;TIMAT OR FOR

THE: VAR I ANCE)

Thf maximum likrlihood ('.~timator for u"l is .qitlcn by

dl Stop if convcrgcnc<,; otherwise go to b).

(:1.17)

Example 3.3 (Direct. radiation in clear skies, part II)

Wc contillu(' our analysis from Example 3.1 on page 32 with til(' slIggcsti mo(]('\ which iududN; two paranj(>t('~, 9 1 Illld 92 Wilh ohs<'rmtioll!'o for tll(' din'(t mdiltl ion and th(' bOlar ('I('\'nlion, these parameters can \)(' ~timat('(l , along with t\l(' \ariftlKt' (12, by u... illg the maximum \ikdihood lU<'lhod. Thi!; iUlplil':O; I hnt the 1<*.. function S(9), IlS indicated in Definition 3.2 on pagt 31. b minill1il,f'(1 with rt'!o.p{,(t to 8 , !:iuch thal
O ~.,gm;n( Y -f( O ))
T

Table 3.1: The maximum likelihood ('lIijlllO/flB of the pammctcrs in th e model/or direct rudintion ill clear ISI.:ie4 tdlll diJJcrwl colJo,.iaw:e strlJ('/Uf'f.

COVilrianCC' given by

0.

(IV /m'l
798.6 822.6 827.8 812.3

O deg 'I 2 {
0.0798 0.0706 0,055 1 0.0614

aN
34631.8 3917.6 25180.1 2302.1

-,

E=I
Equalion (3.52) E<luHtion (3.53) Equation (3.51 )

I;

(Y - j (O))

(3 18)

Wilf'rI' for,\' ollM'rv('(1 Wl.hll~, til{' n'('tor~ of e.tim8t('(1 and ollM'r\{'d rcspolI:-;('

nrl' d<'SCrilX'd by f (8 ) = (h . ... ,J:d T and Y = (Y1 , .. , YN)T, I"CSIX'cth'('ly. Til(' w'('tOI" (} (0 1 O'l f is tht, unknown paramet('rs. Since thl' d ell'nnilli~tk mo<i('\ is non-linear. there is no explicit solution to the minimil,atiOJl problem. Thf'rt'forl' an il{'ratiw pr(x't'(lure il> npplied and hef(> the relnxntion algorithm on th(' pr('C('(ling page, along with the Newton-Rapilsoll method on pagt' 36. i:-. 11~'(llo find till' ('1;\illlllh'" ii. The Illl.\ximumlikcliilood cstinmlor for tli{' mrian('(> is similar to til(' linear CN;e ill; indicated in (3.H), bill for Ilonlitwar {'tlst' it is

"

.E..

>:. ( I)

.2 :g
~

"
~

(2) ,' . (:I)

(3.49)
To gt'\ r(>lL<.;(lttahlt' t'l>l imall'S, Ilw {'o\'ariam:{' Jtmtrix E ha.<; to h{' dl'filled appropriat<'iy Iknce, both the variance and th(' corl"<'lntion structur(> of 0', nlU~1 lw cit-';<ribl,(1. B~' in~p(-'(:tillg t hp traj{'('tories of the dil'('("t radiation in Figure 3.1 on pugc 33. it is S{'('11 that time coIL'i('('uti\'e ohservation:; ftPIX'ar to ht' dCI)('lI(i{IIt. This indicates that the noise M'{ll1f'IlC(> {!t} i:; strongly correlat('(i ill time. It is assumed that the OOIT('lation strllcturl' is an (>xl)()lI{'ntial dt'Cflyinp; function of th(' time dil;tau('(' between two Ol'lli('f\11tiOIl~, i.e.,
Co, [, I,. eI J J - pll. tJI
(3.50)

" is ~
" " ~
~

10

20

30

40

50

GO

Sotnr dC'vation, dl'g.

Fiaure 3.2 : Direct mdiatOll irl dear .d.: i c". Pjrloi modd (curve 2) . modclwithnllt
ronsitkmliOrl of tI(lriOrlCt' olld cOTTV'lalion sin.tr h.lrt'll (cunoe 3) arld the modd based on the OO(ffidt'tlt.~ suggf.'.dNi by Paitn'Jgc arlit Platt (J976) ill Example 3. 1.

where p is the hOllr-to-hour corr('lation. Furthennore, til(' residuals an' a...... umed to bt' Gaussian distributed with mean ....ero and ('t)\'arinllt:e matrix 02E, Whl'H' tlJ(' n x ll-dimelL';ional matrix E, as sugg(';;tcd by (3.8) and (3,50). i~ giv(,11 by

CIOrn>latioll slrlletnr{' is CQII:o;ici{,rt'{l ill thl' lIIod('l. til{' matrix becolllcs

(3.53)

(E.,)

sin(h r,) si n (ht)) .

(3.5 1)

:iad hE>nce. tilt' varialll'f' of Ih(> ci('viatioll~ from III{> model is a.......;umed to be
JiadepenctE'1lt of the !tOlar 1']('mt.iOIl
hilt. wil h Hrong correlation betw(,(>11 h01lrly . If nu t'on:-.idt'rHtiOIl i.~ takcn of l'itlwr tht' varianc{' 01' tlw correlation tun'. the comrilUl('t' sl ruC'1111'1' ii"i E [ . The ML t'Stilllfl!('S ii and iT 1 for the four difft'rent C'ovariance matrices 1\1'(' in Tahl(' 3.1. Fur tilt' 1I10d('1 {'ollsidering both WlrianC(' al!d cOl'relation ute, tht t'tOtilllall'd hOllr,to-bollr ('um'I!l.1 iOIl i.~ ji =- 0,859. which indicnt{'S

If ouly Ihl' \'Miall('e structure is considered, the matrix is giVf'u by

{l:" }

~;2(h ). . '.

(3.52)

whidl. illtli{'ah'N Ihnl d('\iatioIl.N from I ar(' M.,mmed to lw nIutually indt>pt'D,1('lIt. hUI tht' vtu'iaUN' il1c.......8f!('t>i with d('CI'('8Hi..ug !tOlar elevation. If only t1M'

hourly ('()rrelaliou in thl' d('vit&tioll frolll Ih{' d('t('rlllill~t i c l1Iodd.

In Figuno 3.2 thf" modt>1 in t>quatioll (3.7). with two diffl'rmt Cm'8rifUlIT

2 ami :1). i~ ('ompan>d wit h til(' tllodpl with tlw sIIAAt'st(>(\ frotn EXlUllplt, 3. 1 (curve 1). A large diff(,fCIlC(, is dct(~l('d bt'tw('('u ,Il(' rnod('lh, I\hou1. 150 W 1m 2 , sinet' the estimate. for til{' models ill Tabl(' 3.1 11ft wry difft'n'nl from t il(' sUIU!;~t('d ('()('fti cic-nls. Thc two lIIodels, {)II(' with the covnril.lll('c structur(' as th(' idcntity matrix aud Ihe other om' as Eq11<uioll (:U) I ) implit'loo. an' \"i'ry similar alt hough till' larg('~1 diffen'm'C i~ "hout ,J() Wur l O({"uring ncar solar eiemlioll of 12.
~tru(' tnr('S ((' urV('~ ('o("flki~'nl,~

3.3.1

Prediction in the general linear model

3.3.1.1

Known parameters
}.'!QW1l

Consider the linear lIlod('1 with

param.ete,s.
(3.55)

3.3

Prediction

where E[gl l = 0 and Vllr[E"d (12 (i.e., constant). Tbe pn.'<iiction for a future value Y,+l given the independent \'lIriabl(' X LH = Xr+ f is (3.56)

Fundamentally, prOOiction is based on the following important theorelll.

TIIEORHt 3.8

Var[ }~+1

Y; +fj

Vfll"[I+ll =

q2

(3.57)

Ll't}' be a mndom l'(Jriablc with mean E[Y]. tlmt the minimum oJE{(Y is oblained Jor a E[Y].
Proof

0)2]

Compare with EXIlUlplt 2.2.

3.3.1.2

Unknown para meters

E[(Y

0)'] ~ E [(Y - E[Y]

+ E[Y] of]

It is often the ease that t he p!lrlunl'lers arf' unknown but there may exist :.ome I.'Stimalcs of (). If tire eslillllltC'i nrc found by the LS I.'Sti!llfitor
(3.58) thl'1l t he vlIl"iancc of the prcdictiOIl ('rror can be stated..

E [(Y - E[Y])'] + (E[Y]- 0)' + 2 E [Y - E[Y]] (E[Y]- 0) ~ Va,[Y] + (E[Y ] - 0)' ~ Va< [Y]
TIl{' equal ...ign holds for a

EiY] find the result follows.

This It'luL,,

{O

an ('xprcs,<.;ioll for the optimal prediction.

.
(0.51)

TIIEOR,"~M 3 10 (PREDlC'TIOf'l; IN rm: Gt:::-IEtlAL LINEAR MODEL) A.'ttume thai the Imk-nown parameter., 8 ill the linear model are estimated by ~;ng Ihe least square, m('{hod (3.58), then the minimum Varil.UK"C prediction

Y,+, =

E[Y,+lIXt+l

x .... t] = xi+lii
I"H

(3.59)

Tm:Otl..f.M 3.9 ( OPTIMAL PRf.DICTIO:-':) It holds thai

min E

The variance of tht' prf'<iicliOIl t'fror

= YI+l ~

Y; +t

becomes

[(l'

g(X))'IX ~ , ] ~ E[(Y - g(x))'IX ~ ,]

~

(3.60)
Proof (3.59) follows inllllt'(liatt>ly siucc tbe mean of tht' prediction error is O.

wh"" g(x)

E[YIX

x].
Var[Yt+t

}'iHj

Var[xT.tO

Proof Follows immc<iiately fTom Th('()n'f11 3.8.

+ tH' %;+,ii) Var[x:~,(O ii) + Ct+tJ

+ (12 + 2 Cov[xi+l(1J - ii). CHf]
.f.

xi-, Var[ii]xtH
Thrort'm :i.n is twry important in I inl(> SNips analy:-;is. The Ihrorelll says lhat if tht' criterion is to minimize the expected value of thc SQuared prC'<liclion error, tlWIl thl' optimal prl'llidiort i~ ('.lflJ(U to tlu' wnriitimw.l f.I'pn-/atifm.
- q2

+ x:+t Vl\r[O)xt


"

.
~.'

:.

'"::: 3: ~
~

~ . .-

. ..
:

:.' ,

" ~ ~

,"

'.

,.
"

. (2)

'i

- :;: ."
. :;:
~

.'
"

"I.

. ,

~
0
10

.'

" is
40

"

g
0

.,.',.'

.. ........ ,"
~'~'. ~:'
.~

, .:..,..,:. .. to'. ..' 0:.> '. ....~ ~:' ~. I ,~ ...-:.! .. ,i.:"" .

:'~.

'~'.:, . .; ',j;'..~/:~'..~;.::~' ... .~.;:!.;: _. , ~ ':.: '~".:"~":'. " ~/: L'

... :: ..... ~~ .
-~

:.~.

...... '

; ... ,. . :.... . .... -.-.. ..

.' ........ ,
:~
"'~,"f:'~"

:.

:~.:"" ~-.".''''';;'''''." ~" ,' ,r ... O:':-.... f" .~l'):

':

.i

":'~

::~"'

..

'

_(I)

......

:~ ""0', '

lit

.' . (2) ..... -t--;o

,'.

"

.:.

"
(3)

, .; . :

.:. . ... . '/. .:, < : . : ..... . .. .

., ,'.. ... .,'. .

.,'

. '.
'

t.

.~: .. ,'.

,.

""

.-

20

30

50

()()

10

20

40
801M ('\CV(ltiOIl, deg.

50

Solar eleva t io n, deg. Figure 3.3: 8inJ.lj/a1i0l1 of di",.ct mdilltiml ill clear sJ.:if's. Cun'(' 1 lias hour-Io-hom" CQn l'/a/io11 equal to zcrv: CUrt't' lias IlOurto-/lOul' con'f'lation {'({w illo 0.859 .

Fi&ure 3.4 : Forecasting oj dirert radiation in df'Ar IIbf''' . Curuf' I ha., no r.lJI)Uria nCf' ~trurtllre ; curue only vafimu:e ~ b"Uct ure; and (:UI1.'(: 3 both va riance alld corT1!lation
.'ltructUft'.

If we uS(' a n ~ti ltl a t(' for fI'l (sec Throrem 3.4 on page 39 applying E li l(,u f\ 100( 1 H)% ('QlIjilil'nCf'- 'ilttcT1-,ol/or' the future Y1H is Kivell IL'i
YHl

I)

t,,/2(N
=

I))
0

JVarlct+t1

Yr+t 1

/2(/1.' - p)(jJr l-+ -X ~'{~+t(-x ~T~x ~)--~'x-,+-t

(3.6 1)

wh('r(' t n /2(N -I') in t il(' 0/2 quant ile ill t he t distribution wit h (N -1') dl'gr{'('S o f r('('(IOIII and N is l li(' number o f obscr vutiOlls. A ('Ou fi <i!'I\("(' intE'r vnJ fo r a future value is also called a 1nY'(Uction in/ITt.ai.

Example 3.4 (Direct radiation in clear skies, part III) Now it is iUuJ;tmlt'(\ that also for llpplicatioll!S, stich 3.."; ~imulaliOIl ami pn'fliction, it ii> also cru<'iai to tak(> into a ccount hoth thl' ntrinl1('f' nud till' corn' In! ion ~ tru("t un'. Figure' :1.3 illIL.,1 r R t~ sillllilatoo din'{'t radial ion for t .....o diffj'l"f'llt ('ovarinllcc !o;tructures wh('t(> the solid. long curve indicates the dC'tcrJllinistk parI of the llIodel. ('urV!' 1 i~ 1m.,;('(1 Oil the mvarinll(,(, givp]J ill (3.52), i.t~., witb 110 cou!o;irl(>mliou of tht' tim(' l'orrC'latioJl, and it i~ S<,,(,II to be nlldliRliliA fll"Ouud t!i<' df't('nnilli~ti(" Pllrt. Tbi.~ i.~ dearly illl'Ollfliel with III{' two slx'dlk days ill Figurl' 3.1 on pngr 33. By considcring til(' esliIllIlH'(\ corn'la! ion ill E gin'1I h.' (:1.:)1 ). till' iIIolit'l is ilion' ablC' to 1"('produce till' !';('quellc(> of hU('CI'S.... jno ol!:-<t'rvAlioll,"" B.." ('\In''I' 2 indinlt('S. ,.\ nollwr applkllliull showing import I\IIC(> of til(' ('O\'llrian('(' sl rmt IIrI' b pn'flictiOlI. III Fij(lIrt' a.l. tLR~lill tlu' long cun.(, i..... til(' dct(,f1uiniHtit- pari of

F
3.4

:the

the Ktochastic model and the short ('urv(> illdic;at('S observed direct radiation up to II = 42. corn'Spoudinp; to 11 a.lII. Now a foreca.st of the direct radiation is wamed at II 11.0\0. at 12 noon . CUf\'es I, 2. and 3 are the unit oo\'fU"iallC1' matrix and lhl' Slructllr('S in Equations (3.52) and (3 . 51 ), .JeIIptlC1.iwly. Doth curyl'S, rxcluding the oorrc\ation structure, give forecasts equal to till' point 011 til<' dl'll'rlllini.... tie part ('()rrN'<pondillg to h = 48, while &be ot"('("ast considering the hour-to-hour corr(>ialion is fOlUld below the .determini..,tic Clln'C', EL... ('XI)('('h'(i , ThC' fora.... t... without consideration o f the correlation structur(> are obviou.sly about 108 \\'/ m 2 too large . In Figure 3..t the oonditional di.o;lribution for thl' future valucs are df'pictoo. corresponding to t!6timatcd \"8.fian<'e; in Table 3.1 011 PAg(' 13, Comparing til(' curvcs shows impurtlUlC"t' of th(' varianN' And correlation structure in forecasting. k ~ case stud.v, of din'("t rl\lliatioll in clE'ar ski~ depending only on solar 'hdi.atlOn.lw! fewalf'(i the importan('{' of various m(>thods given to this point .3. Th. wi... ,,. ond """'Inlion "",('1m", ~. ,hown to he veey t III ord('r to g~,t corr('('t values when Wl' arc making simulations predictiolL'l.

Regression and exponential smoothing

In Set-tion 3.2

w(' consid('roo regression models relating the dependent variable Y to a set of independent va.rillbles X I,., . , Xp. Such a model call be used

for predit'tiJJg tiL(' d(>pt'lldl:'nt varible Y if fut tll'e vailles for tli" indl'Ilt'lld<"llt \'Uriabl('8 are !:ipccified. Another possibility, which we shall consider in this M'('liOIl, is t llllt only ol:v.-;ervations of t he dependent v3rillhl(' Hrt" ,nHilahl'. In this situation bot h th(' modeling and t he prediction have to be bllMXl 011 Oi)M'f\1l.lions of }' on ly. T iwrerore, we <-'O lisidef model'! of tiLe' form

predictioll , i.e., f'V+21f',+I, whit-II CUll b(> found by using Theorelll :UO. In many cases, however, it if.; lIluch <'(lsil'r to liSP til(' updating equations givell in lh(' following tiJmf('1I1.
TIIEORE\I 3.11 (UP I}'\TI~C: ~ OHE(,ASTS) In the constant mean model (3.63). 1M one-step prcdidiorl can be I1pdat('d a..~
= IV + I }',\'+I

Yi.
LA.'t

= f(t; 8 ) +e,.

(3.62)

IL'! fir.-,t introdu('(' thf" relation between 8n exponential weighting tDg(>ther with the I('nst sqU8I'{'S method and simple exponential smootliing by (''Ollside-ring a ...cry .simple- modd.

N -

(3.67)

(3.68)

3.4.1
1I1e(JIt

Predictions in the constant mean model

Proof Using (3.6-J) lind (3.6S) at. t1w time origiu N
mOtJt>I,

A very simple model belonging to the class of models ill (3.62) is lilt' ro7ls/(mt
(3.63)

+ I;
- (3.67)

where (t is th(, ('OllstanL and {ed is !l scqUI:'U('C of ind('pendC'lIt, identically dist.rihutNI (Li.d) random variablcs wi th E[etl = 0 and Varlet] = (12. A process ted with tilt'S(: chllmcterhitics is cHlied white noise or t ile innOU(lfio71 1'7'O('t'5S. Gi\'('11 till' ObM-'rVl.ll ioIiS l'l, ... , l'N the least squarcs cstimator for 0 is
(3.61)

1 Y,\'+2IN+I = N + ] (l'N+I 1

+ Nl'N+IIN)
_

= N+ , (YN+I- Y'N+IN+(N+ I )YN+ 11N )

= l'N+I]N + N

+ 1 (YN +I

}~N+II"')

"" (3.68)

We now illlroduce the step prediction Y NH1N as the prcdirtion of l'N I given lhe ob.~rT'l!(ltiotlS lit limp origin N. Using Theorem 3.10 o n page 15 the prediction is
(3.65)

Remark 3.3 From (3.6S) it is S('('n t hat t il(' pnodiclor }~f',+tIN is equal to the estimate at time origin N, say Os, of thf" IIll'an . This implies that (3.67) can be interpreted M the rocl1nrivt' estimation
(3.69)

ThC' \'8riIUK"e of th(> prediction error is

(3.66)

of th(' parameter fr ill the constant. mean model. Note t.hat t he influence of Dew observations decreases as lil(' Ilumber of observations increases. .. The equations above incli<'ut(> that the one-step prediction at t ime origin

which is also found by using Theorem 3.10. ThIlS, the variant(' of the pn'(lktioll error depends on the number of observations used for estimating 0', and for N -+ 00 the variance approaches
Var[YN+t-YN+tIN]
-+(12 .

~hl:' one-step prediclioll at 1I1(> time origi n lllnd the prediction error (l'N+I _
Most. frequently only the one-step prediction, i.e. , YN +1IN, is of interest. Given t il<' new ob:;corvntiOIl YN + 1 , we are illteret;ted in tile lIew one-st<'p

N + 1 call be cxpr('s,<;("(1 I\S a lin('ar eombinutioll of the predict ion frolll ol'igill N and the most f('('cm obscrvatioll Y N + 1 (3.67) or as a linear combination of

YN-tII N) (3.68). For the compul/llion of the new fo recast it is important. that only th(' last obscrvatioll Hnd t hc Illoot rccent prediction error have to be Since the m(,8n 0 in the mod('] is assumed constant, each ob&>rvRtiOIl eontnbutes equally to the predictions.

1IkIre.:!.

3.4.2

Locally constant mean model and simple exponentia l smoothing

800

In practi('(' lilt' asslllllPtion of a global OOlUj t a n t TIlf'all, s\'i in tIl{' pr('violls M'Ction, is often too restrictive. It might be obvious to allow for fi slow variation in limp of tilt' m('an. Heuristically, in such cases it would he more r'S\.'iOliahle to give more weight to the most recent obscrmtioll and less weight to the ObM'rwltiolis in the pS\.<;L Let us consider .....eighL'l which dttrea.'iC gconletri<:ally with the age of the observations. Then the prediction of }'N+t at. the time origin tV is
N

700
600

YN+tIN = ('

500

.,vYN _) =

c{Y,.

+ >'YN - 1 + .. , + >.N - IYd

(3.70)

100
300

) ", Q

The constant >. (1 )'1< I) is called the forgetting fadm' or tlt(' di.~co1Jnt cot'fficirnt. In Brown (1963) it is lllPlitioned that a value bet.....een 0.7 and 0.05 will most often be reasonable, The normalizing conslrult c is choscn such that tIl{' SUIII of the weights is I. Since
(3.7 1) the normalizatio ll constant is
1-.\ c = 1 ~ >.N'

200

100

" " "

~ I Olllh

30

'"

."

48

"

00

Fisure 3.5: Predictf'd

(3.72)

lHlle.~ Ji91HY'.~ 11.~illg fliml)ir e.;qx)rIetltiai smoothing

(>. = 0.9).

If N is ifu'g(' thell C ~ (1 - >.) a nd we obtain an exponential weighting o f the past oiJscrvations, i.e.,

Dr:FI"IITION 3,3 (SIMPLE EXPQr..-EI\TIAL SMOOTHING)

The sequcnce SA' dcfilled as (3.76) is called .'fiml/ie f'.xponenliai smoothing or first order exponential smoothing.

YN + l 1N

= ( I - A)

L A)}~"' -J'
j2

(3.73)

Succ.essiue one-step predictions are calculated by using

YN +2IN+l

= ( 1- A)

LA
)20

YN + 1- J

In the timc .series lileratur(' !~ .<Jlfloothing ('QrUJlmil 0 = ( I - A) is oft('11 considf'rNI instead of A.

J

= ( 1 - ).)YN + 1 +A(I - A) L).)YN )2:0

Bxample 3.5 (Simple exponential s moothing)

(3.71)

IL is clearly sccn t hat these results can be gencralized to successive I-step

predictions, i.e., (3.75) The equation for thc s Uc'SSive one-step prediction leads to th(' following definition.

IJIeriod lJI.\
3.4.2.1

SW{'tii~h company ha.. ... tls('(1 ex:polll'lIlilll stlloothing for pr('(Iic'tillg III(' -.onthly hales IigHT(':'. for llipir pr(Xlucl. Th(' sail'S figures for a fivc-yellr and, I he prooictcd vuluC'S arc shuwn ill Figure 3 ..5. A forgetti llg factor = 0.9 IS WI('<i.

Choice or initial value

AB all

initial ('>Itilllfll(' for S'u in (3.76), the sim ple arithmetic a\'('rag(' of SOliit' h!celll historical data or the fif!;! part of tlw ron~id('red !;('rics-----ean be US('(1.

TIl(' influence of So will decllY mpidly sillce

(3.77)
Alternatively thl' first ob:;ervatioll can be used as thc initial value. This initial value will be mObt reasonable if a level changes rapidly: Sft' r-.lakri(h\ki~ IIml Wheelwright (1978).

where the transition l7IalriJ: L , is a fixed p x 1} nonsingular mat rix. For example, t'xpo rJ('lltiflb. polYllomials. harmollic functions, ami lilll'ar combinations of these functions so.lisfy (3.81). Remark 3.4 Equation (3.80) is a lillPar lrlodf'l in thr da..'iS of the regression moopls given by (3.62) all page 48, which contain fUllctiolis of tillle as independent variables . .;\o tC' that the fore<:a.st functions are defined relative to time origin N. This parameterization turns out to IX' marl' com'cnient for updating and fOT making predictions. .. Let us imrodll('('
~1l\(,

3.4.2.2

Choice of s moothing constant

Thl' Ml100thing ('OIL',umt 0' = (I - A) determinCti to which extent the pn..,t observations influence the prediction. A small vallie for 0 r('Sults in fl slow r~ponS(' to dlll.llg<'S in thC' m('all \-alllC of thc pr()(:e;S. A high value results in a rapid response to changes in the level, but a lso a high S<'llsitivity agflinst rH.Ildom irrf'gllilu ob&rvations. Let liS consider how the smoothing constant 0' Cfln he chO&'It. Gi\,<'11 th<' ol)M'r ....nt iOlls Y I , . , Y", Imd a wllliC Ct, we Cllll calculate the olle-step prediction error, ('do:) = Yt Ytlt-l(a)
(3.7$)

important examples of trend models.

a) Cons tant m ean m o d e l

(3.82)
hi writt<'11 in t il(' forlll (3.80) by introducing f (j)

= I,

yielding L

=/

(0)

I.

and lh('

SHIH

of til{' S<llI!U'{,{\ one-step pred ict ion errors (SSE),

b) Linear tre nd model

(3.83) (3.79)

5(n) ~

I: e1(n)
(=1

The smoothing constant which minimizes S(o') is then used. For lhis minimiZAtion a Illllilerkui procedun. such as the ~ewton-Rn.phson procedure, sec (3.31) on page 36, can be used.

Fo r this lIlod('l, t i l(> form ill (3.80) is obtained by illtroducing f (j) = (i,j)T. The t nUlsitiOIl matrix and the initial value of f are
L (:

~),

/(0)

G).
(3.84)

3.4.3

Prediction in trend models

c) Quadratic tre nd m odel

Y,Y+J

In this section the dass of tTend models is introduc('(1. A tr('lld model is fI rt'b'l"{'8;ioll model in which certain fWiction,; of time are taken as indcpelldC'1lt variables. A IIY'1ul Hl(xirl is Jl. T<'gr<'Ssion model of the form
(3.80)

-= 00

+ 0]) + 012 + EN+j

j'

Here. t hE' fo rm in (3.bO) is obtnined if f (j) = (1 ,j, j2/2)T.

where f (j) (fl(j), ... , f,,(j))T i~ f1 ....ector of known forecast fllnction.~ IU ld () = (0 1 , , Op)"1' is a vector of parameters. The noiS!.' C OlllpOt\("nt , {{I' I = 0, I, ... }, is H. s(,(]Ul'II{'(' of mutually independent. identically distribu\('d random variables with E[cd 0 And Var[cIJ = (12 (whit<' tlOist. F'lIrt h('rmon" it is ns.sulUed lhat f (j) satisfi es the differ(,llC'P (,(]luttiOll
-0

o I
I

0 )
0
I

/(0)

m.
(3.85)

The facto r

P /2 instead of j"l .Yklds It !li mpl('r difference equation.

d) Ic'th order polynomial tre nd

t U + I)

Lf (j) .

[s.tart vaitl(, f (O)}

Il rr(', we have f (j) = ( I , j.

I
L ~

" ~ T "2, ... , tr)

and the matrix L becollles 0 0 0

I

3.4 .3.2

Prediction

0
1/ 2

' (0) ~

0 0 0

Using Theorem 3.10 on page 45, the prediction Y ,v+tIN, i.e., the prediction of ) \ +( given the obS<'rvatiolis at time oriJ:,tln N. 1x><'o1llC'!;
YNHIN =
-

1'

(l) ON

....

(3.00)

I l k!

I /( k

1)!

ruld the variance of the prediction error, ('N (f) = Y,v+l-

9:. . +I IN

is (3.9 1)

L is thus a lower triangular matrix, having {L,]} - 1/( i - i)l for i ;:::: j.
0) Harmonic model wit h the p e riod p
YN +j= O O+ OlSIIl

. (2rr) p .

J+O'lCOS

(2rr) p

J+EN+j

(3.86)

Gi,,-cll fill (:stinllltt' '(j2 of (72 , w hich can be obtllined by using (3.44 ) on pAge 39, a 100( 1 - 0 )% prediction interval for til(' futur{' value YN +( is

i.t'., f(j) =

( l, sine;)j.ro(~{~)j)T and

YN+1iN t o /2(N -

p)JVar(eN()J
(3.92)

L ~ (~o
("OS

-sineplf) COS(2;)
('OS s ill

' ' (';)

= Y"' +'IN t n /2( N - p)o Jr l -+-,-r=-(-(-) F -N --:,1:,-(-)

,;n(';))'
j

3.4.3.3

Updating

The matrix L is derived from

When the next ol)l;t'Tvation YN + l is avaihtblc. the e,l.i1l1fttc can be updated by Wiing the following theorem.

2rr) ().+ (P

I) = -

2rr) (2rr) (2rr) P + (2rr) p p (p sin (2rr) sin (2rr) p + cos (2rr) p cos (2rr) p
sin
< :01;

TII EOREM 3.12 (UPDATING OF PARAMETERS IN TIlEND MODELS)

Git'Nl the next ob,9crvation YN +I, the parllmetcr estimate fiN+! is un'itten
j

8 N + 1 = F N + 1"

-I

N+l

(3.93)

3.4 .3.1

Estimation

WIhere the 1IIJdating fornmlas for F and h

ar'

Civ('n tilt' ObS(>Tvatiolls YI , .. , } 'N. the unknown paranwtcl'l'l () (,fll1 be e~limat{'d by ! hI' LS method. All N ObS('TVIILiolis art' written
}'2 ) YI

(3.94)
Gnd

(
0<

+ )) (F(-N +I ('I) . 0+.

f r(-N
2)
2

. .

. .

(3.87)

Proof (3 .91) is

0.

rc:>ult of t he defi nition of F , and (3.95) follows [rom

N

YN

I T(O)
y =
'XN8

EN

ItN~
(3.8l\)

= LJ( -i)YN + I
",.0

-1

+ f:

U~i n g

ThoorCIII :U 011 page

~16,

the LS estimate is
( T ) - 1'XN T Y 'XN'XN

~ , (O)YN+I

+L

N-I

f (-j - I )YN_,
N
1

- /Ii 0

)=0

'"

f (O)YN+ 1 + L (3.89)
= f (O) Y N+ 1 + L

f ( - j)YN- j

].00

wl1('T(' Fr.X~XN and h 'V 'X( Y . The ind ex J\ is illtrodu('('(llo strt'ss t ha t th(' (~timatc is based on the .\ ol)S('rvmions. !l.vailahl(' lit lim(' , "v',

I IiN
1IS('(1.

1rben- the diff('f(,llct, ('Quation f (j + I) = Lf(j) is

3 .4.4

Local trend and expone ntial smoothing

and tbe variancl' of the prcdiction error, 1 ,v(f) = YN+l -

YN+ tl " , is
(3.102)

In the previous section gloool trend models wen' oolll'iidt'rt,<i, i.(>., models where tile' parnmC'ter 8 i" con"laul in time, and each observation (rcccilt or PMt ) had the same weight. Ilowcvcr, it is useful. liS ill Sft'tioll :t.4.2. to allow for SOlUl' tim{' variation of til' pllrl!.lIlt'l('r 8. This means that observatioll':; ill the past should be given 1t'SS \n'ight than recent obl;('f\'lltiolis in tilC' lC'ast squar('S criterion. In this SC('tion tilt' ('Oll("('pt of local trend models is introduced by rollsidering a jmyf't/i1lg /(I('lor or a discofmt factor ill the lelL"t sq arf'S criterion. In the present derivation tiJi:o; is dUlIt' simply by considering WLS (nnd an ILpproprint(' weight malrix E ) instead of LS (or OLS). In adaptive least squa,~s (or di.~colt1lt('d lettst squart's) , the parameter cstimo.t('S at timc origin N, 1l.T('

whet{' F N is giVl'U hy (3. 100). ~ote t hat ther(' is no difference bclw('('u the prediction equal ions ror the ~Iobal and local tr{'nd 1Il0dd. Tbe differellce lies in the estimation procedure.
TIIEO B E~I 3.13 (UPDATING IN LOCAL THEND MODELS)
",71f1l

the next observation Y N +1 is al'ailable, the /xnYl.f/ltler ('stillwt(' (j N+ I is

8 "-+1 = F",+, h "'"+1

-,

(3.103)

UJlit'n the nX;IJrsil'f IJwJalill9 of F and h is

ON :: IU'gminS(8;N)

(3.00)

where

S(O;N) ~
ThC' fo rg{'tting foctor A

",

and
(3.97)
Proo f SUrIl('
liS

,\'IYN , - /'" (-j) OI'

(3.104)

J 00

(1).1 < 1) determines the expollt'ulial discount of pnst

for Theorem 3.12.

oi)S(>f\'8.tions. Drown (HJ63) !>uggt>l;ts ch(>O/'ii llg A stich that 0.70 <).P < 0.95, where p is the numbE'r of paramcters in the model. Howew-'r, it is natural to choose a forgetting faclor ucoordiug to samf' criterion. e.g., the value which gives lhl' minimnm variance of the prediction error. Thl' estimation problC'1Il ill (3.00) can be formulated as aWLS probl(,lll, sinC(' the model for all N obS(-'rvations is wrilt{'u a.o;
(3.98)

For N -+ DC, AN / (_N) / T(_N) -. 0 for nCl\ri)' all functions (e.g., polynomial:; a nd harmonics). This implies that the following stationary or steady attUe value of F exists
(3.105)

In t he steady stat( CA.,;(, the procedure for updatillg the parameters is givell by the following t h('OrCIH.
THf:OREt.!

wher(' Y = (YI , . ... YN )T, X = (fT( - N + 1), .... / 7'(01' I\nd VfU{] = al E , and where E = diug[l /).N I .... , I/ ).,I]. This \\'LS problem is soh,'cd by using Theorem 3.3 Oil pa.ge 38. The !;Olution to (3.96) is

3.1 1 (UPDATIl\G IN STEADY STATE)

- = F N. 8_ hN
v

Ita 8tl'ad y state thc upd(Jiing oj the pammeters ill tlte locally constant trend
mOOd
i.~

(3.99)

8 .. I

L 8 ,v

T~

+ F - I /(O)jYIV + 1 -

YN+l 1N ]

(3. 100)

where FN = x];E-I x;v and h ,., = x];E -l y . Using (3.98) it is seen that

W&ere F

i~ given in (3.105).

Proof

j / (-

S('(' Ahrahnm and Lcdoltcr ( 1983. p. 103).

FN ~

N-'
j"--O

L '\' f (-j) f T(

j),

hN =

:L A
j"'O

j)YN-J'

(3.100 )

In EXlllll pl(' 3.6 o n til(' fo llow ing page it is illu~tn\tt'(J how t he steady state

\'aIuc of F is cakulllte<l for a linear t relld IIIO<lt'1.

In Sectioll 3. 1.2 the locally cOllstant mean model wa.'l IIMXI to defi llf' simple eq,oO('lItial silloot hing. SilllilRriy, t he' much wid(' r cla...." of locally cons taut mooe ll! ('1\11 h(' 11;,('(1 to define gem'ral cxponcutia.1 s moot hhlg.

Using Theorem 3.10 til(' prediction of y,.,. +l giwlI all observa.tions at time

N is
(3.lDl )

58

R EGHESSJON-UASED MI~TIIOJ)S

3.5

TI ME SEHIES WITH S~:MiONi\I. VAlUATIONS

59

DEFINITIO:>i 3.4 (GF;NERA I. EXPONENTI AL SMOOTHING)

T he gCIl<'ral rxponclilial ~mootlling is related to a trend lIl()d('1 with pflrRIIlClers 8 = (OI .... ,Op). For a given smootbing constant Cl: = 1 A the I) ('x]lOlIential smoothed values arc defined as

the sl<"ady stalc valuc of F is

F - ('" A
&ml the paramet(,f

~)

0" = F - h .\
Or. sinct' Os

.
(I

~tilllllt('!o,

un'
I -,\'

( ( I - ,\)'

whrn' A(A) iJol describing a mapping between the parameters of the slUoothoo values. This is illustrated in Example 3.6. ili,N is obtained from (3.100). Using a k'th order polynomial trend modcllcoos to a defillitioll of lile k'tli O1yi('r t'JPO,wntiai smoo/hing:

~}..",. = ( I

iii .l\'

(3.107)

wiwre S~'I is lh(' k'th order exponential smoothing, which clearly is a s imple if is givcn by Definition 3.1, Illid for a k'th order polynomial trend, Ol.N is the local lewl, 02./\' is til(' 10('fi1 slope, 8 3 ,N the IO{'ili ('urvature, ele.
CXPOllClllitii Sllloollling of S~- Il . The relAtion to

This gi\'~ tile ex])oll(,Tltial w<'ighl('(i cstilllfit<'S of the local level, 00 , and the local :-;10])(', 01 , lRt u s now illUJ,tmle the rplatioll to tht.' douhlt.' (2nd order) eXpOllt'UI iul emoot hing, As c\('l;("rilwd by (3. 107) III(' 1st lI.l1d 2nd order ('xponPlltial -.oot lll'd vaiu{'S afe given as

S,~I
Sl21
'"
>OJ

(1

(I

+ '\SI~1 _ A)SII~ + ,\S:2: N.V

'\)Y",
,,11' [2j

Example 3.6 (Double exponential smoothing)

This ('xnmplp iUu...lral(.'s lh\' rdation between 10<'<1.1 linear lreml And dOli hi\"'
('xpOll('llliA.1 smoothing. Consid('r the IincaT trpnd model

00 ..... = 2S,\" - S,V - .., _ -,\-(S,) i ' \ III - (8,) '21 0

('warl)' i... an exampl(' of til(' relation in Definition 3.1. ill til(" ltK'al tr('nd lI10dei on' obtwned b,\"

Thc stcady state value of F is

rJiI,I~ F" +I
Sil1('(' fOf

~"V (~J (I

01),.\

+ 81..'011

1>'1 <

I it Ilolds

(2+ 1 >. '\/) S:~ _ (I + 1 ,\ AI) S.~I

~predlkt;on pron><lurt' is

called (ioll/Jlt' t'J"IK)fIential smoothing_

J' 0

1-,\

Time series with season al vari ations

tn-nd modt'ls ronsiderro in 5<><-tioll 3.1.3 can IX' uSN! to handlt' tim' with seasonal variatious (('.g., nn IUllllml variation). In this S('Ctioll SOI1}, mf'thod~ ba.'>('(1 011 t he n'Kn~~ioli or t'XPOl1t'lltil1i Mlloothing principles oonsid('f('(!. In lat('r dl'II){{'Th d~'llalllkallllod('ls for d('S(Tibiug tiUl(, S('ri~ 8('a:.;cmal variations are introdu('('(1.

60

REGRr,sSJON-BASED MgTlIODS

3.5

TIME SERIES WITII SEASONAL VAIUA'II ONS

vi

3.5.1

The classical decomposition

Altf'rnal ivciy trigonomctric functions can be nS('(i

In til(' tr{'nd modC'] the vl'lriation of the obscrvutions is dC('OIllPOM,(] into a trend and II random error, i.e., VI = T t + lOt. where T/ is the tTend and c, j!o, the rAndom error (typically whit(> noise). If the time series also shows a seasonal variation. the classical dCCOUlp<)l>ition Illay be used. In the cla.ssical decomposition til(' variation i!J oolllmoniy decomposed in a trend T,. a sooMnai or cyclic effect SI. und a random cnvl'" Ct- The tfend is mCk:>t frequently
de>c:"ribt'(] by low order polynomials, and the se8M)naJ effect by trigonometric functions or seasonal indicators (w hich will be introduced later). The rtUidOIll (>fraT is most frt'(lu('ntiy cOIUiidcrcd to be white noise. If the effccts aTC ndditive, then the additive decomposition can be used (3.108)
This mod('1 might be appropriate if the seasonal effect is independent of the

SI=~A)sm
m .

(j2.

st+"PJ

(3. 112)

where the parameters are A. and ~i, which arc the amplitude and the pha,<,(', respectively, or the harmonic C'ompoUl'nt at the frequency...J} = j2;./s. The c'OInponent (3,1 12) is not linear in the parameters; but using the identity ~in{r + Y) = sill{r)cos(y) + cos(r)sin(y). an alternative parameterization , which is li!waT in the parameters, is obtuined. The paramf>t('rs of lhe aoovl' lIlodds can be c:;timated by using, for illst811('(>, the I(~.ast Mluares method. The esti mation procedure, f\S ....'Cll n.s the prediction formul as follow from wllllt. we hnvc:;Cen previously in Sect.ion 3.2. S('('. ('.g.. Abraham and Ledolter (1983) or l!nrv('y ( 1981) for a further discussion of the classical decomposition.

mean. Uow('v('r, if tht' siz<' of the seasonal effe<:t appears to increasc with th(' mean, it might be more rellsouuble to consider lhe multiplicative dccmlllJOsilimt
(3.100)
By lIsing a logarithmiC' transformation the multiplicati\'(' model is brought into a.n additivC' mod('l. ~ot.e, ho\\'Cver, that litis transformation will stll.bilil(' the \'arian("(' rorr{'('tly only if the error term is also thought to be multiplicative. Anotht'r pos.';ibilily is to uS(' mixed addilive-multiplicalil'e m(Hleb. Traditionally the trf'nd c.omponent, T I , is described by polynomials in til(> time l

3.5.2

Holt -Winters procedure

For the locally constant trend models in Section 3.4.3, a si ngle pammeter (the smoothing conl)lant 11' = 1 - ..\) is lL';ed to describe the variation in time of th(' p<uamclcrs. liowever, the seas()IIal compOIlCllt lIIay be more pcrsistent than the trend component., and thus it is obvious to usc different smoot hing oonstams. Holt (lOS7) rormulnted n doublc exponential smoothing with one smoothing cOflstanl for the lpvpl Ilnd 11IIo11l('r slIIoothing COUSlallt for the slope. Winters (1960) extended this procedure of using different 1>lIloothing constants to the additive modd (3.113) wbcr(' till' trpnd is linear, i.e. , TN+J = I'N+) = J.tN + O,d, and where the 8e&Onal component is described by seasonal indicators. i.e.,

(3. 110)

USllnlly we select k = 0, I, or 2. A seasonal component SI with a period of 8easonal indicator,,'! (or dummy variables)

samples may be r\eM'rilwd by

S, = L::S.6"

.. ,

(3. 11 .1)

St =

J=l

L 0) at) , L 0) = 0
)=1

(3. 111 )

wh('1'(' S, = S;'.~ S.+2. . .. for j 1, .... 8 and L::=1 S, = 0. The procedure is u1>ually referred to us Holt- Winters procedure. The J)I'occdure npdlltf'S thr('(' 1>Tnoothed stati1>tic.s fL, nnd S recursively ill order &0 prov ide adaptivc Ctitilnates of the seasonal and trend com ponents.

ii,

where the !;pecinl Krollc<:ker's delta = I if t corresponds to til(' S('fL'iOlllll tin)(' point i, Ilnd ot h('rwiM' = O. For illstanc(' for monthly data wit h All anllual S('1\SOnnl I)('riod, if t C'orrC'Sponds to :\Iarch (i 3), tl\('n only 81:1 i:; Ollt' and tile n'ht b .lero. Unfortunately. for larg(' values of .~, til(' formulation of n lUod!'] with H ~'IL'i()!1lI1 efr{'('t r('(luir~ a huge Illlmb<>r of panull('tt'!"S which might conflict with til(' gl'!l<'rai id('a of parsimoll~ in 1I10del building.

at.

ati

3.5.2.1

Estima tio n

An ffitinlll.lp of th(' lfl,fl (or mfall) of tht' timt' series at tinl(' origin N
band by the r('("un;iw equation

+ I is

(3. 115)

62

REGRESSION-UASED Mt:;T I10I)S

3.6

G I,QIlAL AND LOCA L. THEN!>

MOD~:L

AN

t~XAMPLE

63

It is St:lCn that the new estimate is a weighted 8\I'Cr agc of the new ol:>b<>rvation adju:;I(,(! for tht M'a.'tOllal componem 8N +1 _ 11 8!!.d the previous ~timutes for the level and the slope, JiN and iJN' Thus. (JiN + i3N) is the estimate of the 1(' ....<'1 I\t tilll(' origi ll N + I hlL.'>('(i on the observations available at time ol'igi ll
Y,." . I

Global model
Locl\l mod('l

-- --

N.

C'orr(':opondingly. the l'!>timatc of the slope at time origin .'\1+ 1 i!; fOllud as (3.116)

i.e., as a v,."t'ighU'd aW'ragt' 1)('~\\1'('11 the new observation of tlJ(' :.\opt' (;t". + 1 -fiN) and the previous ~limat(' PN. Fin/Illy. tile M'H.o;QIlII! eDmpOIll'lIt is updated lL<;i ng (3.117)

------- -o
2

which is n w('igilt bplw('(>n the new observation of the seasonal compOJlent (YN+l ilN+.) alld the previous e;tiwalc 8N+ 1_". 3.5 .2.2 Prediction
ft

Fiaure 3.6: GIOtN11 and locallincar In.'mt.

Using til(> Cllrrt'llt ('!;titllflt(>;; of til(' kvel, slope, and seasol1111 ('ompOII/'IIt., prediction of l't. H, givcn tllc observations at t ime origin N, is

I..et WI a.....'iumc

YN+/IN
('tf' .

ji.N+{J,.J+SAH-,.,

(=1,2" ..

,8

(J.1I8)

JCxample 3 .7 (Globa l and loca l trend model) tha t w (' (1)11';Jd(' r l:I Iil1eaf trend model and assullIe tha t III(' obsen'l:ll iou!-i !-ihnwu ill F ip;lII'(, :;.6 ar(' available. LC't us firs t co nside r t he model:

1''''-+/IA :;;;ji..'V+IJ...-t+Sf.;f-t _2",

t=8+1, .. 28
)"'\'} 01) +OTJ+ :\1 }
is ..... hit< noise ..... it h a>ro ll1('an

3.5.2.3

Choice of smoothing constants

:;;; o. I. . . }
z.o
2.5 3.5

a nd mriall{,(' u 2

Several ~ibl(' optimality criteria for choosing the set of smoothing conlltant!-. (0'1.02 and 0':1) exist. AbrAllRIIl And Lrooit{'r (1983) sugge.l tlUtll h(' smoothing constants nrc chosen so that the sum of squared one-step prt'dictiQn f'rror.~ i~
lIIinilllizro. i.('.,

Using the' a bo\"' mod ('1 fO f nU !;ix oh!-.{'("vatiQIL'. gi\'(~

" L)Yr - N

., .,
~z

" <,
(~:) +
':1

}'jll-d ,

(3. 119)

3.0
4.0

<,
~~, ~(;

..

Y = X 6 B 4-E

1
(]

with respcct to (01,02,03)' T he minimization is carried out by u!-.itLp;, P.g., the Ncwton-Raphwll pco("('(illl'(' (3.31) frolll page 36.

3.5

3.6

Global and local trend model-an exa mple

100

G ( - 15
.~

r,;;

If))
0.1-129) ( 1:-1,;)) (J,U571 - UI.5

This S<'Ctioli ("'ont/tim' !liT ('xiU nple fl'ganlillg a g lobal amI lo('al lill('nr t r(,1I(1 mod('1. Funll(' rlllor('. l h(' ('xalll pif' ilh l!-.t ra tf'S in delai! all t il(' ('ukuillt ions involved, lind th(' 1Hllllh('r of o bservat io ns (.'\' = 6) i;; kept low to f' nMlff' tlm t ~'ou 1"1111 follow III{' ('!IiT-ulations withont R compu t er. The available ol)h('r\"lllio n ~ IUl' !-.linwlI ill F igu l"(' 3.6.

= Z(o,. 11 =

( Ig., ) - -IIUi

.. = F.- 1he = (O.52.1X 6 U.1419

(3.!lO3) 0.331

01

REGRESSION-BAS ED ~'!ITI I ODS

3.7

PRODl. EMS

65

U:-;lllg t 11('';(' ('"liIllHt{'1> lh(' sol id line in Fib"ure 3.6 is obtaill("(l. UII,.'>{x]

011

lllf'

8.\'uilabl(' OlNT\'ntions (N r 6) thl' pr{'(iiction of Y6+f is

l";;"16 - f

.,.

(f) 96 =3.903+0.3311

,-.

b) Local linear trend llIodel Wp choose the forgetting factor A = 0.9. The mod('1 is unchanged: but duE' to th(' diH'Ount of lhe pa:;t ol>M'fValiOlls, v..t' g't
F.
6

All IIlIl>i<l..">I'(I,,,,1 imah' for (72 i"

(y

~~ "/(_)/'(_') ~ ( 1.68r.O ~ ) ) 1O.2H-1

J~

%ool"(v

X(0),'(6 - 2)

(-O.2-IN)1 + 0.0791 O.5!Xf"

+ (-0.2-11)2 + 0.428',1 . (
j

0.-103)2

, "'~ L"I(
J--O

10.284) 35.U61

11.902 ) ( , 28.5~
0.5732 ( U.1630

U.(630) ( 14.902 ) 0.0717 -21:L'lSO

=;:

(3.858) 0.308

"\OW is

lin

f'Stilllalt' of til('

vllrifU1Cl'

of th(' prediction error (',,(/) = }iHI

~;ttlll

'\;;h(l)]

ii' (I

+ F(/)F, '1(1))
0.1129 0.0571

siopt'.

0.153' (I + (I f) (0.5238 0.,129) (I)) t

n, ]5:i (1.5231:! + O.28G8f
2

0.0,)71(1)

Colllpar('(1 with 'ht' glohally ('Oll~lant ![('lId 1110<1('1. we.' dearly M'e a i>mallcr This lcndcncy for 1\ :o;mnll('f slope for til(> mort' rec('nt o\)s(>rvatiOlls is recoglli~('(1 ill Figure 3.G. U pdating the parameters Again \11\' nt'xl obSf'I'vatioll iii Y7 = 3.5. With the adaptiVE' scilPlllc the updating equatiolls fin'

A" an ('xarnpl('

F,

F,+"/(_6)1r(_6)

~
Thl' 9()~ ]m'(lkliOJI inh'n'al for }7. givcn the obscrmlions t G, is
}716
As.-;ulll(' IllIlt III("
nl tilll('

1.6~5G

origin (

1O.2R4

10.21:11) 35.%1

+.

00' ( I ) (1 6

-6)

10 (J!;(6
1/t-J1

2)JVar[C6(1)] = ".23-1 1.320

3.5.

5.2170 13.173
I

13.173) 55.0U3

II/1M rnll;on ; .., ItT

h7 =).L

Ito + J (O)Y7

Updating the parameters

F,

0.9

C ~) '( 11002) + (I)

2~.~O
16.912 )
J9.1~

0 3,i)

F,
(_ 6

I ( 6)1F(O)
15
55
91

I,,) + (I )(1
-6

-6)

7 ( -2 1

21)

II,

L ] h6

+ f (O)Y7

O.!>202 ( 0.1272

0.12(2) ( 16.912) (3J:i20) O.O1f)3 39.13,1 == 0.222

., (18.5) + (1)3"
40.5 0

it iM noted that the ncinptiH' mllun' of thi;; lIlethod illJplies a lower

for Klohally t'St imat('C1 paralll('H'rs, which is in accordance with .de,n", S('('u in Figure :l.u.

.V

8,
A

0. Ili 1:1 ( O.lOil

0.10(1) ( 22 ) ~ 0.0357 - 59

(3JW6) 0.250

Problems
I'

' ~=~~.3.1 on th(' following page shows ~JJJe obS{'rvations of a stochastic process
and all indepeud(,llt vnriabl('.

U>lwrllr)' for a

dl'CfC&I{"

in the 810pe is clearly set'h.

00
Ta ble 3.2: ObsC1"lIotio'fl8.

REGIlESSIQN-IlASED METI IODS

PROBLEMS

07

Question 1 State how (p, o. J) can be detcrmiul'd u...,ing the lea.'lt MIUlU{'!j lIIC'thod aud !!pedfy the (.'orre;..ponding matrices. 2

3
2.0

5
2.0 4.0

7 4.0 1.5

y,
I,

1.0

4.0

0.5 1.0

3 .0 '
2.0

3.5

3.0 2.5

Question 2 Take W W; t~t iUll\tor for (II, (l",/J). Hint:

= = '1fJ

(i E Z) and (i :5 q) and lind the lea...t squares

.V \'

L
Question 1 Forlllulate a model for the variations in Y, IUld rllilleters.
~timale

COS

(w;t) =

12

the pa.N

Question 2 Kllowing that Xs = 0.5, find a 90% coutirle-n('e interw'Il for l's. Question 3 AssulIle thaI.. we haw oil:;('rveci only the time series {y,}. Using this assumption, calculate a 00% confidellcc interval for YM

L , , cos(w, I) sin(w,l) = 0
Question 3 Tilt, part of Yi's \'UrintiOllS which coruponent WI is given by
N

Exercise 3.2
COIl!-.idpr til(' rt.'gression model
('1\U

be

d~CTihcd

by lilt' periodic

YI
wilf'rf' E[{EI}[

j1:r t

+ ,

O. Suppose that N observations Rff' gin'n.

Question 1 AR.. UIIIC that Var[cd (12 /:r~ but that thc clements of til(' M'qU'llC'e {Ed ar' mutually ullcorrelatcd. Consider tlir Ullw{'ighted least squarl'8 ('!jlilllalor (i'J.).
Is tllP estimator unbiased'! Calculut<> t ile variance of tilr E'$timator.
Question 2 Calculate lhe varial1("(> of the weighted least S(lut\r~ ~tilUator (;i). Qoestion 3 Compare tluo' variances of (;J*) and

[ (w,)

L (0 C06(Wlt) + ;isin(w, t)2

1=1

Sbow, this oontributiOLl can be written

a.'>

t (w,)

~ (ii' + iJ')~
W,.

l (w.l is callt'(l the intensity (of the variations) ill

(Ii).
In M airport
Il('(lr Copenhagen the, wind speed is measured every hour using CUp-aIlt'1Il01ll('t('r. Supp()!';c tl1l\t now (t = 8) the following obtiervntiolls of the wind Spet>J lire givell [m/lll:

Question 4 As:.UUlC now that VarlEt] = (12. but that th(' ('\('ll1cnts of tile S('(luelice {Ed ure mutually correlatt'd. \\'e assllme that

p[Et,E!
C.oll-;ider the llllweighted least

p. 0,

for 1.: = 1 for k ~ 2 4.4. 3.4. 3.3. 2.5, 7.3. 4.9. 1.8. ,1.1 ;."'..... til(' fifth measurcment wa." performNI. the anemometer was moved thl' nor1l1111 uU'lIsllring IDem ion (2 m abovc ground level) t.o the roof of a

P["t.Et-.d
MIUJU'{'

e.timator.

Is the estimator unbia..'-;('(I? Cakullltt, til(' variance of the estimator. Exer cise 3.3 Suppose that N mudd
2(14- I o~r\'ulions 1',,1'2 .... l~", are givcn. Considrr t ht'

:1Iui!d;nK.
~1IS1""n 1 Forllluinte 1\ suilabll' model describing the vtlI'it\lions ill the obs('f\'('(1
spet'(l.

1!Iu",;'n 2 Estimat(' the parallJ('\('rs of the model and C8pt'Ciaily /.L>;st":io." (model) differt'n{'(' ill wind speed at the two mpa.-;urelll{'nt location.". 3 P r('(lkt thc wiud !-.ptwl nt tht' old lIlt'IIMlring IOClition 11.1 tllr next

wiwrl' {Ed is 1\ sequencc of lIlutwdiy ullC'urrf'illt('(i random variablel-i with "1<,1 o und Vur[t tI (12.

(I = 9).

68
Exer cise 3.5 Consider a model with local constant mean.

REGRESS10N-I3ASEI) METHODS

Question 1 COlIsid('r tht' formulas rdaled to the local trend model. Show that the prooiction of Y..v +/, given the obsen'ations YI, Y2,"" YN , is giV('u by

CHAPTER 4

Linear dynamic systems

Question 2 Show that for>' red u('ed to
--0

I (and fixed N) the prediction equation is

1 ~ N L.. ; =0
=

}'N+I IN =

YN ' J

In tlj(> following w(' will consider siguals (Of pro<.,'cs;;cs) ill discrete time {XI;t = O . I, ... } and in continuous time {x(l); -Xl ~ t ~ oo}. In practical situation!;
di.'lCfel(> signals a rc oflcll
It

r('!iult of COlll inuous signals being sn.mplcd, and this

Compuf(' with t he result in Section 3.4.1.

Question 3 Now the ClISC (N --0 (0) is considered (). fixed). Using TIH."Or(,11l 3. 11 Oil pl\g(' 49, show t hat IlpdH.ting of the prediction is
U1U... stCp

will be di"us.<;e<.i later in t his chnptcL A sy.~t('m COllvcrts an ifl1m! x(L) (possibly multidimensional) to a (possibly mult idimens ional) 011tpUt y(t):

x(t)
SystC'1II Illput

.(t)
Out put

and compare wi l h tile r('hUll ill Section 3.4.2.

Exercise 3.6 Consider the Ilumbers from Exercise 3.1, and suppose that only {YI} can IX' ob:.erved. On th(' basis of a large datil. set, it. is found H.{'ceptabl(' to tiM' a local lincar trend with a forgetting factor>. = 0.9,
Question 1 BI\S('(\ on the abo ....e Ul('ntioued facts, calculate a 00% oonfidell(,(, interval for Ys . Question 2 Assume that the next observation is given as yg = 5.0. Vp<lall' th(' panunctN C!ltimatC'S and calculate a 90% confidencc imerval for Yg.

10 mat hematical terms the system is described by an operator .r--oftcll called a filt IT. W ithin this franl('work we will limit oursel\'(~ to the following.
DE. II'''ITl O~ .1.1 (LI~EAR SYSTEMS) A systelll lli said to bp linear' if

(4. 1) The dass of linear systems call further be divided into, e.g., time--invariant BOd tim('--varying sy:.teUls. DEFI" ITl O~ 1. 2 (T IMF':- II\VAIUA:'>lT SYSTEMS) Byst(>111 is said to be tiuW-i1UI(J1-i<Ult if

y(l)

.1'[,'(/ )["* y(/- T)

.1'[x(, - T)[

(4.2)

1. 3 (S TABLE

system is said to bt' fitl1bk if ally constrai ned input implies a const rained

Let us consider SOliI{' ('xtll1l pl('!i .

70
Example 4.1 (Non-linear system) \\'(, <'Ollsili('r I Ill' :;Yht('1lI {y,} .1'[{.cdJ (discrete till1C') dC'finro by
y, = (xd
This sp,t('m is lIon-lilll'llr sinN' p.,XI.l + A2.1'2,,)' I- Al (.cU)' + A2(.cV)'. TIl<' syst>m is not linlt'-imllnrult since y, -= (xd' and y, _., == (x / .,.)'-" -F (rt _.. )' Fillnll~-, lll(' s),stt'lil is Hot stnble since. e.g., Xt = constant> 1 i lllpli~ Ihllt
Yl_xforl_~.

4.1

LINE AR SYSTEIIIS IN TilE T IME OOMAlt'ol

71

In discrete t ime the sequcncc {hI;} is also call<'<l the impulse response fundioll or just t he impulse rcsponS('. siu('(' tilt' out put is hI.; when the input is Kronecker's dfita sequence (somet imcs referred to t\!; the impulse function) and is dcfiued

by
15k =

I {

rod' ~ O
for k = 1. 2, ...

(4.G)

CaM'S

Im;tead of (-1. 3) /Iud (.fA) t he colwoitt/ion operator and the o utput is sim ply written as y = h * J:.

* is often used in both

Examp le 4.2 (Linear system)

The systt'lIl lit'fiIlt'd by is lill'nr sim'!'

Tllfo:ORE~1 1.2 ( PnOPEm'IES OF TilE COI\VOLUTION OPERATOR)

The r()1!t'oiution operator haa the following P1TJIH'riics:

aj II *9
1+A2X2., !J=.\dxu+Ox\.t])

9 *" (SYUUlIPt7';Cj.

(>qIl,t+..\2I2.tl+O{AIJ'I"

+ A2(Y'.!,I + 0.r,2./
It is ('m.;il.v
M'('U t

bj (h. g). j = II

I)

* (9. f)

(associative).

hnt tilt' syst('111 is both time-invariant aud stable.

r} "

*6 =

It , where 6 is tlw impl jlM> ftmrtioll.

' 4.1

Linear systems in the time domain

Proof Ll'ft for til(' f('nder.

For li n('ur and lilll'-i nVllrinnt sYl>tcms we have the following fUlld a lll{'1l1nl [(sull.
TIIF.ORE\I 1.1 ( hWU.SE Ht-:SPONSE F UNCTION) For' any iillcur and time-invariant system theJ-e exists a

functiol1 h so that th('

Olltput is obtaiTlf'il by applying the ('Ollvolutioll integral

(1.3)

.. Remark 4.1 For 8 giWIl (pnntlll('terL!:cd) sy:.tem the impulse response function is oftell 'nmel most ('Ou\"{'ui{,lltly by !oirnply puttillg :r = 6 and thell calculating the tespollhC. y = II; d. Throfcm 1.1. This is illustrated in 8xample 1.3 on tht' ilUuwinJ!; pnge. ..
D EtT\H IO:-;1. 1 (CAL"SAL SYSTD1S)

ill COIlU/UI01L.9

timf

(md (1.9

the convolution sum

Yt =

L
1;=
00

A syMCIll." is said to Iw phy.~i("(JUy fCMibll- Of cawai if the output at time t doe; not depcnd on futurC' vnlues of thc input, i.e.,

hk J

( I. I)

ill di,~c1'('te

time. S01ltftimes toe wite y = h *;r. wher('

* 1.9

Ihe eonvolutioll

11(1,) III

0,

for

<0

(4.7) (4.8)

0,

for k<O

opcrator.
Pmof Omitt('(1.

in continuous and db(Tt'te tillit'. respt'(:tiv('ly.

-"'---------

Having introduc<'<l the impulse respousc fuuction we introduce the following.

THEORE\tI.3 (STAfllI.lTY FOR

Th(' wdght fUllction It(u) is callC'd the impulse respmue fUllction 1)('tltlLM' t ile ompul is h(lI) wlwll the inpnt is the Dimc ddta-junctioll J{t ). 6(t) is also called th(' impul.'le j1mctiorl And is d('fiJl("<:1 by

['1~EAn

SYSTEMS)

J:

aufficitnt rondilio1t f07' a lincar .~y.~lcm iJeil'g stable is that

(1.9)

[(/)5{t

I, ) dl

[(/0)

(1. 5)

72

LINEAR DYNAM IC SYSTEMS

4.2

LI NEA It SYSTEMS IN TIlE FREQlE'ICY DOMA IN

70

L
k
-0<)

4.2
,h"i<OO

linear systems in the frequency domain

in continuous and discrdf' time, respectivciy.

Proof Omiued.
Example 4.3 (Calculation of hie) Considl'" l h(' lincar, tillll'-invarialit systC'1Il
YI 0.81.11
1

Fur ~Ill(' 3naly!ois it is more cOilYcnient to dC'M""ril){' It system in the frcqul'm'y domain . c.g .. if the systcill is generating signals with pcriodicitiC!-. For a linear timf'--iuvRriant system we hllve M'('II that t he relatiou betw{'('11 input IUld output for n sy:;tem in continUOIlS time is given by the cOII\'olution integral

y(I) =

I:

h(ll):r(t - u)dll

( U5)

Sillt'(' the output in the frequency dOlllain is obtained by the FOrJ11C1'

2It

:rt

( til)
~.

tmll4arm
l'(w) =
"'''''1"('''; is till' frrIJIlt'tu'y

n.,' put t iug .r a.

"

W(' S('('

that Uk

h" = 0 for k < O. For

= 0 we g('t
( U2)

1 :

y(t)l'

'-'I

dl

(Ll6)

Yo

O.Sy 1 + 21\0

D-. O.8xO+2xl-O .. 2

Lt'., 110

2. Continuing,

Wt'

g<'t

Yew)

1 :I:
1.:

Irad/S]' we get by sllU:.titulioll of (4. 15):

h{ll)X(t - u)(' ..,1 d1ldl

.III
Y2

n.Ryu + 26. - ISo _O.8x2+2xO_l=O.6

O.R,,] 0.18

==

I:

"(IL)c .., ..

[1:

h(u)c' ''''Udu

I:

x(1

u)c .... (1

.. )

dt] dll
(Ll7)

;r(v){'

''''V dl'

where t' = I
.!Ik - O.S.l: 10.6 (I.:
111'11('(', thc impul.se f0iI)OIlS(' fUlll'lion is

u. It is S<'Cu that the last integral is the Fourier transform X(uJ)

> 0)

{ L
hy
S(l)
in continuous t ime a nd

ork<O fork=O fork >O

of tilt input signal ;c(/); d. (4 .16). The first integral is the Fourier transform of &be impulse rl~I)()IlS(' function, and is railed the !rrquellcy response function (or lOIn('tim(':o; 1h(' lnuujo' JUI/('lion), and is delloted 'H(w). Thus, it is determined by

1t(w) ==

'0.6

!:

h(u)e-'''''' Ju,

-00

< W < 00

(4. 18)

From (U7), th(, following is seen.

which clearly rep rc:;(:ut.s fl ('Ililsal syst('IJI: l)(>finition I I Furt/wrmorf'. thi' s.v~l(m b stahlI' since L~ IlIk 2 + 0.6(1 0.8 O.b 2 )=5<x Based on the impulS(' r~pon1'j(' rUllrl ion, we define tht' step
f'!'.~po1ISe /tmdio71

cr.

,n ...,.,,,,,, 1.1

( FRr:Qt E'\'CY IlESPONSE FUNCTION)

a linear. tirne-inva,iant system the output in the jn'.l[uency domain is by mUltiplying lhl:' input in tlte frequency domain with the jrequency ..."",,,,, junction, i. 1:'.,

~ /~ h(u)du
Follow~ dir{'('tly.

Y (w)

H (w)X(w)

(4.10)

w('

( I. 11 )
in discn'lc time. Th(' fUII("tion is ('all"d the stcp r('SpouM' function bccallliC it C<lual:. thl' Olltput whl'll till' input is It ~t{'p.

For 8. sy:;t('111 ill di.'crete time

l '(w) =
t,

apply thE' discrete Fou rier transform

L
-0.;,

y,e-i...JI.

'11":5 ""

< 11"

(4.20)

74

LINEAR DYNAMIC SVSTE:MS

4.2

LINEAR

SYSTE~IS

IN TilE t

IlEQlJE~("Y

DOIo.!A IN

75

From ca1cull1l ions C<luivrucnt to (4. 17) it is seen that. Theorem 4.1 is also valid for SYht{'IIlS ill distTctl' time where the frC<luclicy r('!o;pOIl,'i<' fUll c tion is determined by

y(t) =

I: I: I:

Y(w)c~'dw

( 1. 28)

'H(w) =

00 L

in continuous lime and simi lnrly in discrete time when changing the integnttion
,,-ith :;Ullllllfltioli and dmuging the limit:; according to (4.27) and (4.28). The pair (l.18) and (4.22) is most frequently used for frequency response fUllctions, wht'nas the pair (.1.27) Ilnd (4.28) is tII~t fm:lu('ntiy used in statistics (because

h"c-, ..;k..

-11"

<

1T

(4.2 1)

B8.<,('(1 Oil fI. gi\'l'1l [r<'<lucDcy response fUliction 'H(w), the ililpuiS(' rCSI>OII!;(' function can be found by the inverse Fourier transform

tbe area under the spectra ('<juals the total variancc).

Furthermore, an alternative pair of Fouri('r tran."forms (Jenkins and Watts 196X) is
S(v) s(l)e- i2 ". ... t dt S(v)e,2,."t (lv

h(l) = - '1~ 'H(w)c""'dw, 211" -.xIlk

(continuous time) (discrete time)

(<.22) (,1.23)

. = - I ; ' 1t(w)eo..l" dw,

211"

...

set ) =

(1.30)

\Ve observe, lhftt for systems ill discrete time, we considN w only in the interval [-11",11-]. This is due t.o aliasing of harmonic fun ct.ions applied to <'fjuidistant observations. The problem wi!! be di:;cussed in Srction 4.3 on
page

78.

'H(w) is ill K('ul'rnl complex sinee the impulse res ponse fUllction is not usually an ('\,en function. TIllis, it is possible to split 'H(w) into El r(>al and a complex purt
(1.2 1)

whefC v is called the frequency ([II] = S I = liz). Since w = 27TV, we get the relation S(w)dw = S(v)dv that. S(w)2~ = S(v). u>l liS consider t he output that is obtuinoo when the input is a single hormo71ic signal, i.e., the input is givell by

.,.(t)

At"""

A CON(wt)

+ iA sin(wl)

(4.3 1)

Then the output

h(,('OIII'S

where G(w) is the aml,Utude (amplitude function) or gain and (w) is the phase (phase function). Let us com;ider two signals ill the time domain w(t) and x( t), respectively, as v..-ell as the corrt>SI>onding Fourier transforms. II"(w ) and X (w). The con\"Olution in the fru:IUCIiCY domain is givell by
1 Y(w) ~ 2rr

y(t)

1 : J :

"(.)x(t
h(u)Ae'.... {t

u)d.
- u)

; 00 IV(v) X (w _~

1 v)dv ~ -2 (11'. X )(w)

~

(4.25)

At',.,t

I:

dn

II( u)e ....;t. du

(4.32)

= '1t(w)A(',,.,t = G(W)ACi( ..H~~))

And by taking all inverse Fourier transrorm we get this is S('('n till' following.

y(t)

w(t)x(t)

(1.26)
i'In""lEM 4.1) (S[r\G LE I HEQl"E1'>CY I{ELATION)

Remark 4.2 In general it. holds t.hat convolutioll ill the time domain corresponds to lJIultiplication in the frl:'quellcy domain. This is clear from (4. 15) and (4.19) on the preceding page as well as (rom (4.25) and (4.26). ..
It should h(' not('(1 tim! different placement of the constant 1/(211") for the pairs of Fourier tml1s/orms is ofLen uS<'<! in the literature. Alt('rnatively, th(' pairs of Fourier transforms may be defined
Y(w) ( 1.27 )

ainglt' harmonic iT/puL 10 (t iinf'ar, time-invariant sy.~tem will gil1e (Hi olL/lml the SIUJl(' frpquC'ncy w. Tlw aml,litJui!- o/lhe output signal equals the ~lit,.,j, 0/ the input 8';glla[ tll1Jltiplied by C(w). The change i1l phase /mm to output i.~ ,pew).

4.4 (AD RC e lement)

'J'O,,",de. tht' f1t'c-1riral Hy!'iu'm ill thl' fiF;urt ht'low . Tht, illPIII \()It~('
~. "~It)

and th(' output \'OhllRt 11(') .

76

LI N~~AIt DYNAMIC SYSTEMS

4.2

LINEAR SYSTEMS IN TilE FHEQUENCY DOMAIN

77

~----;: . =--A\~-~:;:-~--<o

1
0'

G(w)
yet)

(t)

ReI

0+-'='----- . /2

(I(w)

o -+-----=:,~-- w
fIIure
By ('ollsiilpring tht' circuit, it is S('('n that
cdy(l)

4.1 : Am/lIi/ude (yaill) and pll~e for all nC-elrment.

df
(Iy

1 n(I(I) - y(I))

[n'<III('I1(,} rrsponsc function for lll{'

SYSll'lIl

Iwconlcs

RCdl

+ y ~ x(I).

(1.33)

Th(' solution is
(L)'I)

w. . ~('(> dirt'Ctly.

by compuring tht, {'ollvoiution iutegral in Theorem --1.1 on pnge 70 lhat thl' impuiM' f(~p()n<,(' function becolll('S
'I.'P

1/(w)

1 + I..JRC'

(.1.36)

{'H.kui;\l(' til{' runpliludc.

( 1.37)

h{lI)

_1_1;' u/RC RC

for u > 0
( 1.35)
fOf[l<O

The tilllf- ('OfMtlmt is T nco Th(' exponential d('('ay and th(' time com;tant nf<' ilI\1strated in tile figure 1I('low.

(w)

arg{'Jt(w)}

=arctan( flew).

( J.3H)

h(u)

lIpIiI,.d. awl till' pha.'i(' for the ne' f'if'lUt'lll is ~hown in Figur<' 1.1.
filwr (as dl'MTibNI pr('viously) where tile slow vlU'iatiOIlS (small values of undisluriwd. while the fa-'ll "ariations Me damped (filtcrt-d). is called -.::,;;::.,:, A filtt'c. where the fast varintions un' k('j>t intact, while the .. Itr(' fiitecro. is called a highpa!I.9 filter.

1fi"

,,

,,

, ,
T

4.5 (A t b n .. delay)
IlOOlIIicK' 8 "}'htC'1lI ""Iwrt' til' input l'iignlll is dc'ln.ycd
T lilll(' units.

i.(> ..

, u

y(t)

.r(t

7),

(T

(~Ollstllllt )

I1!SPOIUI(" bCCODW!;

If tllf' input L'I a pubtt 6(') d('fiuitioJ) of 11(11).

w\'

ohtain the MipmlSC h(,,) (,,(1rrt'Spondiug 11) the

h(u) = 6(u - T)

(4.10)

78

LINEAH DYNAMIC SYSTEMS

SAMPLING

79

G(w)

o --J..o--'---'-------./2

'I'(w)

The sampled sigllfll is tILt' signal WI' obtain when the continuous signal is Dlultiplied with a series of 6 junctions defined by

j(l ) =
"

L
' -0:;

5(/

"T)

( 1.14 )

Figure 4.2: Amplitude (gain) aFld phase oj a tiTlle delay.

Sinn' j(t) is a periodic function, it cnn he ('xpanded in Fourier series; i.e., it

can he written as

j(t) =
and th(' fr('(lu(,Il(,Y
r<~ponl;('

F.. e"',"",1

(4. 15)

function becomes where .....o = 21r / T is the SUl7I1Jli"9 !''C(IUCIICY, lind

(1.11 )
Tlit, UUlplitud(' of 111(' ~ys'{'m is G(w) = land tile phase is 41(.... ) =- wT. It iJ; S('('II frolll Fig-ufl;" 1.2 that all fre<lu{'ncies p&S tllldisturb('d, hut Ihm
til(' phlL'-<'

F.. = -I T
I "'" _

JTI'
-T/2 T/2

j(t)c-..,,,,,ot dt
;n"",1 ([/

4>(w)

-')t.

for w oc.

/ T12 6(t)e
T
I

1 = _

Example 4.6 (D!sign of low-pass filte rs)

All idt,1l1[ow-pa....... filte r is a Hilt'f. which cuts off frl'qut'lIl'ies higher thaH tht thr('Shold fn'<j ll('llc,Y Wo whik frf'quencies lower than Wo pa......" undbturh(od. WI' will now inw:o;tij.\!lt(' how to :-.('I('(-t th(> weight fUllction in tlu' lillll' dOlUain so that w(' l\("hi('v(' such an ideal loU'-pass filter

the Fourier expansion of n series of 6 fUllctiolls becollll..'S

j(t) =
Fourier transformed becomes

L
" ' -00

e"""ot

(4.46 )

'''''I S"'-'l)
!..;I > ..Jo
W(' find

( 1.l2)

It is lIot l)(liisib[(' to construct this fiher (not ('au.~al) "i!l('t' 11,(11) is ddilwd [or !lll II. E\"('1l for "off-line" calculfltion~ \H' can only oblaiu nn approxim8tioD to 1i1(W), sillc(' II/(u) cannot b(' Applied to the input l'ignltl in II. nUll-finite tilllt' inh'rvtl,1. An irlwllH",d-1X/..S,'1 jiUl~f' b a filter when' the !'iigulli pMtIlII ulldbtmlwd in the interval [,.}I.W2), while it is cut off outsidE' this intl'rva!.

.
I

[cos(;',)U)d...J

"

(U3)
=

2.

L
-~

5(w- nwo)

(4.47)

again a series of 0 functions. Let us consider lhe sampled signal

I.(t) = x(l)j(t)

(4.48)

~ r(t) be givt'll ill tht:' frequency domain by X(w). Since multiplicatio n

4.3

Sampling

t im(' domain corr('spollds to convolution in the frequency domain we get

I X.(w) = 21r

When a continuous signal is regi~t('f('(1 or !l.('quirt:'d at diM'rt'tt' 1illl(' illstaJlt1'" \\'t'say that \\l' an' sampli ng til(' signal. It is obviolLs that iuforllli\tioll i:. 1 0Kt wl\('11 II sigll1l1 i:. hlImp1('(1. L<-t u.s consider the C(l.'>(' where 1\ ('()ntilllloll:O; :o;ignaJ .r(f) is :o;amplcd (or l\('quin'(l) nt Ifll'i(Jj,\I(ml time ill.~t(Jnc(!. T (.~(lflipliJi!1 Ijme).

/ 00

-00

J(v)X(w - v) dv 5(v ,,,,,,,)X(w

1 /~ ~ -T L

v)dv

-00"'.<:10

ou
From this it is seen that

LINEAR DYNAMIC SYSTEMS

-~~-~ . . "

z- TRA NS FOltM

81

(4 .49)

The sampled signal ill tbt, fr('qllt'licy domain is thus It p<'riod ic function with till' p('riod wo0 Furthermore, it is sccn t hat if X (w) comili us values outside the interval [-wo/2, wo/2J = [. 7I"/T,7I"jT] these values cannot be disti nguished
frolll values inside till:' int('rval. This phenomenon is knowu il.'i (Lli/J..~i'lg. Values of X(w) outside the illtervai [ 7r/T.7r/T] will thus be added with valuC8 iru;id{' the intl'fval. The frequency w = rrjT is called the Nyqui~t frrqumcy. If X (w) contains values aboY(' tht Nyqubt fn'qlle ncy, wc will encounter problems du{' to flIiR."iing. Figure 1.3 shows a spectrum which we willl'OllsidcT later. Howewr. if X (w) is os showl! in the figure, thl'lI X, (w) will be as shown. Til(' pre\'ious discu~ioll can be used in the problt'lJI of choosi ng a fo, nitable sampling time T, si n{"(' tbe' Slull pli ng frequency 2rr/T should be at least twin' as lnrgp as the limit frequ('llcy for X(w). More prccbdy, W(' llllvt' th" fo llowing.
TIIEORE~1 ".6 (SIIA .... NON'S TI I EOREM)

qw)

-,
x,.(w)

lA't there be given a 81gMI in continuous time x(t) lIIhrre X(w) = 0 for Iwl > Wr (i.f., the ,~i9nal i.~ limiird to w,,). This signal is completely described by thf rorre"pon(iing sampled signal XI if the sampling flr.qtlcncy i.~ (It lmst tulic(' a.~ larye a.. We. i.e .. 27r/ T > 2..J.. ~ T < 7r/wc.
Proof follows dir{'("tly.

Altt'rtlatively, Shannon's thror('m tan be forlllulated: Any continuous signal.

x(t ), call be reproduced from th/;' corresponding sampled sig-nul, x.(l ), if, al1l\ onl)' if, it does not contain frcqucntiC'S above the Nyquist frt'<jU('llCY. Alin....,illg can result in fa ul ty conclusions since a penk ill X(w) over tl1l' ~yq llist frC<jucncy might be strong e nough to ap pear 1\..., a lX'ak at another
frequpncy for t he sampled signal. A practical solution to this problem i!; 10 choose a shorter sampling tilll/;' tiUUl what was init ia lly thought of and then ch()(W' It. fil te r which cuts off the variations approximately outside the de.in'<i sam pling interval. The mOIst !;impl(', hut no t always tht, lII()f; t id('al, method is to apply subsampling in terms of the meau of k !;uccessive values. This fi lter is of course lIot Ilde<lllate whell it is important to r('Stoff' the high frt.'<lllellcy vnriation.

'(w)

.,
Aliasing. forom
cofitillUOUS

o
to sampled signal.

4.4

The z -transform

TIl(' .:-t ran~form is a 1I.<;('ful way to dc:scribe dynamical systems in discrete lilll/;'.

82

LINEAR DVNA/lllC SYSTeMs

4.4

TIlE z-T RANSFOIlM

83

DEFINITION 4.5 (TIlE: Z-TRANS FORlo.l) For n M'qu('ll('(' {.Td the z-tnUisform of {xd is defined as the complex fUliction

" _ Ipic 4.8 is !:It'('11 inulledinlcly thaI the

z-tran~rorlll

of the impulse flUiclion 6" is

Z({x!}) = X(z) =

L
I=-oc

00

:rIZ-

(4.50)

ZI(6.))
jpmlple 4.9 (Solution of a difference equation)

(4.54)

The z-transform is defined for tlJ(> complex variables z for which the Laur(,llt :series (4.50) is collvergent.

til(' Iincar timl'-iuvarinllt sYl'ill'1U dl'fin('(1 by tilt' difference ('(Iuation

Example 4.1 (z-transform of a seq uence)

For tilt !';('(lU<'u('t'

YI I

O.OYI

6/ .

Yo

{.rd

ddin('(]

(l.';

thl' z-trruu.form We' obtai n

for t <0
fort~O

0.0
I

o.o~

l'

the z-t ran.. 'iform

i)('('OlILfti
~

thih it i!-. !-.\'f'1l ihal the solution lO thl' difrerelJcc equation is

ZI(r,)) ~ XI') ~

L2-','
toO

L (2z) ,
1=0

00

Yt = o.()'

1,2 ..

(wit h til(' regioll of ('ollverg('II{'(, giv(,11 as

1=1 > 0.5).

In the

('XUlllp](' wt'

exploit that
00

Tht, adwull ag(' of ll!'ing tllC' z-tmllsfOl"ln for lincar lilllt.'-illvariam systems is from the following.
",,ORE" 4.7 (CONVOLUTION
]:>{

_ 1 _ = I+x +x2+ ... = ' xk I x Lk=O

TilE TIME DOMAIN CORRESPONDS TO

IUI.T"""(".,""",, Ir\l HI'. z OO~IAIr\)

::;;:~:,,~the .~cquCW.Y."9 {11A:} (md (xd. and a 8cqUcnce {YI} defined by the
~

and that this geomNrie series is convergent. for The z-transform is a linear operator. i.e..

Ixl <

l.

Yt =
( 1.52)
"

L
-<:X.

"~;XI

"

( 1.55)

which is readily

S('CII [rom the definition (4.50). Of special importance is the backward :;;hift operator.

l '(,)

II (,)X(,)

(4.56)

DEFINITION 4.6 ( BACKWARD SIIII;"'f oPBRATon)

The backward 8hift opemtor z

is defined by z

-, = Xt.1Z
t00

,
t= -oo

l :!;(' D{'fillition 1.5.

ill (I.5u) is called lllt' t1Yu/8f(,. fW/dion of the sy~Lem and it e<lll be
( 1.53)

'X(,) ~ z' ZI{x.})

Similarly, we hnv{' Z({Xt Z({.rI~A:}) = z"X(z).

11(,)

n,)
XI')

(1.57)

h t is til(' impul!'C' n'sp0I1SC' flllletioll

81

LINEAR DYNAMIC' SYSTEMS

TilE z-TBANSFOBM

85

THEOREM 1.8 ( THE UNE A Il DIFFERENCE EQUATIO:';)

Ttlt:OIlE~1 4.9 (STABI LIT Y)

The difference equation

71v
l.

,~stem
,\

... .. ,

de!Jenbcd by the difference equation (<1.58) is stable 11 allilic poles lie v:ithin the unit cirrle in the ('01ll1,le:r planf'.
sec, howc\'cr, Appendix A, "The solution to difference

represents a linear time-invariant system with iTlput {:rd (Uld output {Yt}. i.~ a .mutant positil1t inte.yer-valrwd time delay /7Y)m illTJUt to Qutlmt.

THEORE ~I 1.10 ( FII EQUEN CY HES POr-;SE AND TBANS FEH FUNCTIONS)

Proof The system is linear b(,'(~auS(' the difference equation is linear. Further_ more, the system is timl'-illvariant because the coefficicnt s and the time delay are constant.
Using the z-traru;form all both sides of (4.58) ,yields

For a linear time-invariant and stable system. the j1-cf(U/''1lcy

IUP01lSF

" fou nd /)y Ctaluatillg the tramjer junction OIl the IInit circle {z E C I.e., by putting z = ('lW. Hence,

Ilzl =

junction I},

(1.61 )

- _ . , Follows hy

fl

com parisoll between (4.21) on page 74 and (4.57)

011

R
"rom (4.57) it is fieen thal the transfer function is

4.10 (A simple system)

tht> Iilwar till\('-invarianl sYhtem defint'<\

I/ (z) = (bo + b1z- 1 + ... + b'lz -q )z-'I" (I + al z I + ... + apt p)

Besides the time delay op('ralor z
polynomial i n
''1"

(.1.59)

8.'1

y,

:rt - O.9I/ ]

the IlUlllerntor !'Onta ins th(' following

z
B(z) = (60
=

the

z-t rltll~(orlll,

we find the transfer function

+ biZ 1+ ... + bqz- q) z--q(boz q + bt Z 01 - 1 + ... + bq)

1l2Z-1) ...

.t-tntu.,,(orm, the frequency rehpOIls(' funclioll is oht aincd

lIqZ

= z-q(z - nd(z - n2)'" (z - 7l q)bo = ( I - Il IZ t)(J _

(I

t)bo

'H(w )

H(c'''') -

n.Dc ,...

where we have uS(..>(\ t he fact that any polynomial can be fadoril.('(1. The root~ 11,. 112 . 1 n q which lIlay hI:' ('Oll lplex. are caller! the zeros of the system 1md 1m' found as t he solution to zqU(z) = O. For the denominator we introduce

1111"')1
Ute ptm.<;('
0("')

/1I(w)7t(",) .

)1..,

LH,,,,(,,,)

A (z) = (I
=(1

+ a,z-l + ... + apz- V)

)qz I}( I -A2Z 1) ... (1

(4.6 1)

A" Z I)

(4.62)

) .;: arctall [ arg ( 1(w) 1

O.!l<in(w)

O. ('osW

()

wher(' til" (,'ompi('x IIl1mb(r.; ,\10,\2 . .. ,'\" are call"d tit" polN of the sy!-,tCIU. Using the introduc('d polynomials the transfN fUll ctioli for the systt' lII defined by the diffcn'ucc ('(Illation (4.58) can be written

-:;:::~.1.11

(SVSTB1!-l 1'< SERIES A"IO 1"1 PAIIAU,E I.)

in s<'ri('!o;
.\

(Lf'l )

" 1 11, (,) f-

86
tN'

LINEAlt I)YNAM IC SYSTEMS

F'H.EQUENTLY USED OPERATORS

87

hat'e the lHJ1l.s/cr /wlction

from X to Y.
(4.61;)

4.5

Frequently used operators

and /01' .,y.9tems in parallel

11, (,)

Ja the previous sectioll the backward !>bift operator was introduced in the .t domain. In this section we will introduce Ii number of o perators which directly applr to the time M'rics {It} or t he stochastic proces.'i {XI} in the time domain. Without ]os.'i of generality the sampling time is assumed to be 1. In the foll owi llg we ~hllll lise extensively the blLrkwam shift ope711lor B
defined by
(4.68) If B is used J timcs ....'{' writl'

8(8(- (BI I ) " ' ) ) = WI,

= II_)

(4.69)

1M transfer function from X to Y

i.~

(1.66)

Thp control t heory lilerftture most frt..'qut'lltiy uses q-I instead of B. By aIIDg q' ] a more dir('('t correlation with a z-transrorm is obtained since it is to jtllil ('XChllllgt' :: I with lJ- 1 , t her{'by chllnging X(z) to XI_ Within ,.."""met,;,, lhe backwa.rd shift operator 0 is most frequcn tly dcnolcd L (for

Proof &-en immediately.

TIIEOREM 4.12 (SYSTEM \'>"11'11 A FEEDBACK LOor)

U"".pon,,i;"'gl, lhe fonuard shift opemtor F is defined by

FXt=Xt+1

( 1.70)

For lhe feedback sy.,trm l'(z)

FJ Xt =

Xt+J

(4.7 1)

.. (,Iea.rly :;('(.'n !hl\! F is the inverse operntor for Band vicc \'c!"SU, i.c. ,

H,(z)

B.
Tbt' (backwa.rd ) di1TnY'ltce o1)Crator \7 is dc-fined as
(4. 72)

tlte tmn.~fer function jlYJm X to }' is

tht unit Olx'rAtor I is defined by is SCCn from (1.72) that.

lXI

'1.

'1= 1 - B

('.73)

summation opcmtor S is gi\'Cn by

and hence E(z) --' X(z)- 1I 2{z) Y (z). FUrthcrmort'. we hAW Y(z) = l1\( z)E(z), tlml tlwn V(z) = 1I 1(z)X(z) 1I J{z) 1I2 (z )'(:). Oy isolat ing Y(z) j X(z), ( 1.67) fo llow~.
= H 2(z)Y(z),

Proof 5'(z)

S Xt = (1 + B + ... + B' + ...

)Xt

('.74)

v'

_ 1_=

L+ B + ... + B'+ ... =S

('.75)

For a morc COlllplt'x syst('1II I hI' lotal tram.fer fU1wlion ('fill hl' fOUild hy Il.... ing (-1.65), (4.66). a nd (1.67) rl'lxaledl.\". A ... an fllt<'rIlaliw', Ma.wn.~ rolf' {flll h(' ftpplit'<1. as d escrih('(1 in J('IIS1.'II. H" holdt, ami :'\idM'1I ( I9fJ2).

that Sand \7 ar(' the invcrse olX'rntors of each olher. thl' ......u(uon diffn-eTlcc opemlor \7. is dl'l1l1(>(1 by
\7.Jf =
It - It

_II

( I - O)JI

(-I 76)

88
All the dE'fined operators arc linear, i.e.,

LI NI~An DYNAMIC SYSTEMS

FRf.Q IJENTLY USED OPERATORS

89

(4.77)

fA,j' {",j
11"0 = AO-'O 11") - A (1/10

(<,J.

Ii? 0)

( 1.83)

Wll{'f(' 11 is auy of the introduced operators. Furthermore the operators are comlllutlltivc, i.e. ,

(4.78)
For ench of the above dt'fiucd operators, it is possible to dcHuc new combined
ol}('mtoT8. For the power series

+ AOW) + A; _ I'I/.'I + ... + AOI;:.'.

1r;

A,11-'0

0(:) =
It

L o,z'
.",0

(4.79)
lise th(' .....ell-known rciatioIL" fo r the series A(:), 1I{:) and 11"(:).

m'w operator II is defined by

a( lI ) ~

L n, H'
i _0

(4.80)
Ull

11K'ort'lIl 4.13 ('(III b(' uiK'<l , e.g., to fiud th(' i nvNse combined operators. "'low',"'A 1(13) faT A(8) is found by putting (O) = A I(D) and 11"( 0 ) = 1.
4.11
(Box-Jcnki'L~

If 0, = 0 for aU i higher tilan SOme wLlue, thell obtained. For example we have

o1x;mt01' lKJiynomial is

transfer function)

'-.....d"' tlw diffI'T(,ll('(' equation

(4.81)
It

q-ordcr polynomial in B.

bac-kward shifl operator B, (-1.84) b wriwn

( 1.85)

... Remark 4.3 By the definition (4.80), an isomorphi~m between the class of combined operators Ilnd the class of power series is obtained such that t he calculation rules wh ich arc used for power series also are valid for the combiucd operators. III particular we operate with the combined operator polynominls exact ly a.:; we opNate with ordinary polynomil\ls, Using the ahoY(> definition a vcry strong tool for description of linear time-invariant systems ill di*r('l(' lime i~ obtained without considering the z-tmllsforlll.
The si milarity wit h po.....er series is illustrated in the following.
TIlEonEM 4.13

odtlcl"o til(' o!wrator

..p(Il)

polynomials

+ '0', I3 + ... + 'Pp B") ..;(0) = (WO + WI B + ... + """ 13")

(I

( 1.86)

(a7)

(1.88)

F01' lite opemtor H the jollowing opemlors are gillen

,,-"(8)"'(1!)8" r, - h(8)I,

~ [~h' B'] r,

(1.89)

to Box and Jmkins (1970/1976), Iht'

mobl and h(B)
=

such that
A( II)",( II )
~

II)

(4.82)

1II(}(I('1 ( 1.&1) is calloo n ;p_. I (D)w( B) B' ill ('8.11<'<1 th(' tru"..'fjl r the system which dearly i.~ baH! on the impulse rc:-POIIltt' Com""", (4.89) witb (4.57).

90

LINEAl! DYNAMIC ::iYS'1 EMS

TH E LAP I. ACE TltANSFOIlM

91

4.6

The laplace transform

The Laplocf' o]><'rator is a linCaT" opcmtor, i.e., ( 1.92) wbkh iln nwdiately follows fronl tile> d('finition. Of special importance is the following.
THEORt:~1 -1.1-] ( DtFF"ERE"'-TIATION)

The LapllK'e lmm,[orm is a useful t ool for description of dynamical syslt'llis ill continuous time.
DEFINITION 1.7 ( THt~ LAPI.An; TRANSFORM )

Por a given function in continuous time, x{t), the Laplace transform is dcfiu('(i

as
C(x('))
provided the intcgral exists.

~ X(,) ~

1 :

c-"x(t)dt

(4.90)

,.,. til l bi/ateml Laplace trons/orm

dx(') } ~ ,c {x(' )) _ c{ -

d'

. X( . )

( 1.93)

Exa mple 4 .12 ( La place transfo rm o f a function ) For x(t) drfiJl('d hy

fort<O

fort 2:0
til(' Lap]II.{'(' lWlIsforl)l is

c. {d~~') } ~ ,X(,) -

I(O)

C{r(t))

foX) :r(t )e

'tllt

[7(t)C
-I(O)

.01]: +}j + . X( . )

l "C ;r(tJ<'

Mdt

for R(s)

> -ln2
~2

E OREM -1.15 (TIME OI::.J.AY)

a time delay we have'

whtrc R(.,) d'lIott'S lilt, real part of s. Compare with Exrullpl{ 1.7011 pagt for thl' z-trnl\ljfOrlll /tlso for til(' n>gioll of conW'rgt'IIC(' .
.. R e m ark 4.4 Thl' Lapll\cf' tfl\nsform is oftt'll used 011 causal systems where /(1) t < OJ therefore, o.n alternative definition is

c {x(, - Til ~ X (s)cC{r(t - T)}

( 1.95)

0 fo r

I: I:
eJT

x{1 - T)C' -" elt

x(t
T)e-,(I-r j el(t - T}
eO' X (,).

~ ,

.. C {r('))

This is ('alit-xl titC' lmila/crol Lalilace transform, and (1.90) is Laplace transformation. From the exnmpiC' ahoV(' it is
;;(,('1 1

,1111(,<1

til(' bil{lt(llJl ..

t ransform, combined with tllhiN; for the illverse Lapll\ee ~on'''' is I\. ~lrollg tool fo r solving differenliul equations.

that marc generally

.... 13 (Re cleme nt)

( I !Ji )

the simph' ~' lt'('t.rieal ~p;tl' lII fmlll r. :(;\lII pl~ 1.1 Oil !Jag' 75 given R(
,d y(l)

1\.'>

forR(,,) >

lJ.

T h(' inckx

It d('Il()t~'s

tI\(' ullilatf'rru Laplue(' tra usfom L

dt .. yet ) -

x (t),

fI(t)

0,

for t :O:; O

92
By wtting t he input
vo l tu~('

Ll l'n:AR DYNAMIC' SYSTEMS

T ilE LAPLACE TRAN SI-"O RM

93

equal to t he Dirl\Cdclts.-function 6(1) {\ud

using the LnplnC<' I ransform (:-;in('('

C{ 6(t)} - I),

___r T he system is linear because the differcntiul ('(Illation is lint'at and tbPt"-ill""8riant btx'ause the coefiki('nts and Ihe lime delay are constant. _ Using the Laplace transform on both sides of (4.99) gives

(RC) . \ '(,) - y(O)

0'

+ Y (s) =

Y(s)

= I -t

(l1e).,

=,

11I1C
I I RC

From CUH) it is seen that llit, inverse L.1.p\ll("(' transform gives

The troJI!t/cr /tmclion for the system described hy difrerential ('(Illation

y(t) - _I_f' tIne IIC

whieh obviousl~ is t h(' impuJSt' pit' t.J.
THEOREM
r('Spon~' [UTlc tiOIl

( 1.101)
found previously in Exam-

Iloo .,w"".to< contains a

polynomial

I. 16 (CONVOLUTION I N TilE TIME DOM A IN CO RRESPONDS TO

O(s)=bosq+b1s q l++bq
= (s - nil(s - 112)" (s - l1q)bo
ftl ' "2 .. . .. Tlq IUC' the zeros of tilt, system. Similarly til(' denominator contains the polynomial

(1.102)
('1. 103)

MULTIPLICATION I N TilE S-DOMA I N)

Consider tltt' junctions h (t ) arlli ..z:{t) cmd a junction y(t) defined by lIu' tOln'C!luUon intcgml

y(t)
Then (4.96) implie.9 that

I:

h(u).r(t - u)ciu

( 1.00)

A (s) =

sP

+ Ul,9P

+ ... + Up
A'I)

Y(s)
who", Y (s)
~

1f(,) X (,)

( 1.97)
}'1 , A2 ...

= (.9

Ad{s - A2 )' " (s

(4.lD' ) (4. 105)

(y(t)} , fI(, ) = (h (l)} and X(s ) = (r(')}.

P roof As prC'viousiy for the Fourkr tram;form, see Throrclll 4..1 on page 73_

. ,A,. are called the poles of the SY8tCIJ) . t il(' introduced poiynominis, the transfer function for the continuous 101I)'>" n is wriUC'n
H (,1 )
1. 18 (STABILITY)

I/ (s) is the trunsjer function for til(' continuous time system, and it is dC'1U' that

_ 8(s) , - u - A(s) e

(4. 106)

Y (.,) H (s) = X(s) ~

00

' 00

10('). "d'

(.1.9<)

by the differential equatioll (4,99) is stable prul.'itird "',. ""'/',,0 lir jrlllit left part oj the eOmlJir-x s-plane, i.e., !R{A,} < 0 jor all i .

.~y,~lfm d.e,~cribed

where It(L) is t he impulse response fuuction.

THEOREM 4, 17 ( TilE LINEAR OIHEHE"IT1AL EQUATION)

linear timf'-illlPfLriallt and siab{t .~ystem, described by lhe lmnsjer jUl/ction tJu;. ft"f'qltl'1u'Y rY'lipQnse- jlmction is jound a,'I the tmnsjer ftmclion on 1M imaginary an.", i.('., by putting s = iw. lienee,
4 . 1!J ( F REQ IP.NC Y RESPONSE AND TRANSFER FUNCTIONS )

The differelltial equation

fi"m,' time-im'ar'ianl r( t) to tit(- Oil/put y( t).)

(J

represeTlt.~

,~yshm.

(r

i.~ rl

lim! (Ielay jrom the

'It(..-)

I/ (iw)

(1.1 07)

LINEAR DYNAMIC SY~TJ.;MS

COl\I PARlSON BETW EEN I'HANSI-'ORMAT IONS

Proof Followli immed iat.ely by a ('om parison betweell (4.18) on page 73 and (-1.98) on page 92. &'(:tioll L i on pngt' 80 d('l;('riiws how t he total transfer function is obtnill('(1 for systems ill series or parallel and for feedback systems. Exaclly the sanll' cnlclilation rll l{'S eau 1)(> uS('(1 for ronti nuous time systems.

,I

o
3

20
'I

'!I(s)

4.7

A comparison between transformations

Assulllt' .;r(t) is 0. fUllction which is sampled at the time instant'> {... , -'1', O. T. 2T.... }. III St>ction1.3 on page 7~, it was shown that the snmpled signal J .. (t) can be written

0
Z(,~) = e~T

I.(t) =

2: 2:
00

x(t);(t - nT)

(1.1 08) 0 4

..l(z)

13y using .c(t)&(I) = J(O)&(t), we have

.,.(1) =

x(nT);(' - nT)

2
( 1.1 09)

'!I(, )

n=

-')0

T hen using the L aplace tr(lnsform gives

1 0
(.1. 1101
4 .4: Prom :'1-plcme to z-plane (a SamlJled conLimU)us signal). :R denotes fUJI and J th,. imagina11l part of lilt compla number s or z.

x.(s) =

2:

x(nT)! {;(t - nT))

Since C{&(t)} = 1, usi ng Theorcm 4.16 on page 92, we hll.\'(

1;(' - nT)) = e-.. T !{;(t))

t hus, (-1.110) ('an be written

= c ,oT

X.( ..,.) Finally, using the substitut ion

r(nT)t'-&nT

(.1. 111 1

t hat the ztransform OOfft'Sponds to the La place transform signal. l 'si ng t ill' su bstitution (4. 11 2) it is seen that when the functio n from .;r to y is written IlS Il mtional funet ion in the Laplace {oJ (i.e. , (,1.101 ) witho ut t imC' d{'lay), t he n the transfer fu nctio n fo r . lIlp'Iro syshm is writtm as a rutiOlllll fU llction in e-~T (see (4.59) o n
M.'ell

is ewarl.\"

p .1 12)
(4. 111) is wriLt{'1I

X.( . )I.
w h N('

.,

2:

r(nT ), "= X l')

0, 1, ... }.

S (:)

; ~ t il(' Z'-trn ll~ form

of {J( 71 T ), It

cho ire of sam pling t ime T imposes the Ill!lpping ddint.'(1 by Z{.i) = e 5T , is a IIlnppiug of t he ('olll pl('x .~-plallt> onto t he com plex z-plane as ill Figure 1.4. It maps t he imaginary axis, :R(s) = 0, onto t he 1 :1 = I, in thl' COIll Pll'X z-plarw. F\lI"thC'rmore, t he left hulf of t hc s-plan(', ( n (.~) < 0), fllll.~ in<;ide t ht' unit ci rcle i u the <:omplex :plane. lIniquell(,ss, only ,.,. plllile VIl]m's in lilt" n'gion: -1f < :l(s) :5 7T a rt' sil1e(' t h(' Ilmppi llg will pul a ll t ht' \'alm'S s 2p1f , Jl E N into tht' ill t he : planE'. mapping is part k ulnrl.... u!>dul for It transformatio n of til(' location of Z('r(1II; from diM'r('t{' to r ont; lIl1o U S t ime Ilnd viee vcrsa.

LrN~:AR DYNAMIC' SYSTE~fS

If we choose s = tw, so that s is (we aL~ M't T = I )

011

the imaginary axis, lliC'n we obtain X,

X, (i...,) =
"

L
~ oc

CHAPTER 5
x(n)e ......
011

(1.111)

which obviolLSly is the Fourier transform (see (4.2 1)

page 71 ).

Stochastic processes

4.8

Problems Introduction
tum .w'ries {XI, t = 0, 1, ... } is a "mlu(JtioTi of a stochastic TJ1ucess {XI, l = .. } and an illlj>ortam npplication of modern time series analysis is to e.g., a physical plwnomcnon as a. stochastic procc;s. This book provides only a brief introduction to stochastic procl's-'>(.,..,. Its purpoS<' is to provide only tile most essent ial notation and definitions to introduce a class of 1I~'f111 stochastic pro(('s'-;t.'S. Por a rather detailed "",,,,,IUll k,,, to stochastic processes, we refer to Crimmit and Stirzaker (1992), (19S3), Yaglom (1 962), Pnr"',.('n ( 1%2), and Cox and Miller ( 1968). ~ I ore oriented introductions 3re found in, e.g .. Pilpoul is (1 983) and Davis Villter (1985).

Exercise 4,1 A systcm in discrete' l illie can be described by the following difference equation:

Question 1 Determine the polt~ !tlld zer06 of the system. Is the systcm Mabl{,'? Question 2 Dctermine the impube rCSl>onsc function (to Ilnd incl uding k = i). Question 3 Determine the frequency response fUIJ{; tioll aud skPlch thl' amplitude function. COlll pllrt' lilt, amplitude function with the pole placement cornmellt! Exer'cisc 4.2 Cousidt'r a lSy~tem which can be described by the following diffC'nmtial equation:

d'y(t)
dt"l

+ 5 dy(t)
dt

+4 (t)
Y

~ x(t)

Stochastic processes and their moments

".-h<>s',ic proceJJS is defincti as a family 0/ random variables {X(t)} or {XI}, t belongs to flU illde.I set. III th(' first CA.W W{' are considering a proccs..'i time while in the latter a process in discrete time. In t his only procl':'i,-;('S in continllollS lillie will b(' described due to the nota tion, the 'Suits ~md definitions can easily be formulated in discrete time. :A"'~I'''''',;; cprocess is a fUllction ha.ving two IlrgUIIlC'lIts, {X(tw) ,w E O}, n is the sample .~pa('(', or the ensemble, of the process. This implies that tbt>!!('t of all the possible time series that can be generated by t he process. a fixt'd t we say that X(t ,) is f1 random t'ariabie, and fo r fi xed wEn we a N'ruization of the process, i.e., a. time series, X (',w); d. Figure 5.1 on
P Rg{'

Question 1 DctenniuC' the trllllsfer function for the system. Is the systc'lIl stable? Question 2 Determine the frequency response function for the systl'Ul. Question 3 Suppose we want to lSillllplc the signals. Spe<'ify time T. Question 4 Dt'tt'rlllilw the poles for the sampled system.
A.

suitabll' sllmplilig

often only onl' realization of the I)rocess is available, aud in such it iii Ilec{'"sary to assume scveml properties for t he process in orciN to distributional charact('ristics about the process. a stochast ic process is n family of random variables, its distribuSIH'CificRtions are lIlor{' or IpSs t'qui vnlt'nt to what we have seen for I mndOIll va.rinhll's. function , h' (t, l, ..\' (1.,)( 7], . ,x n ), is called the n-dimensional probdistribution function for tlJ(' proc~'i {X(t)}. The family of a ll lil('S('

98

STOCHASTIC I'IlOCSSES

STOCIIA STIC PJlOCESS ES AN]) lIIEJR MOMENTS

99

All important concept is thl' mL/ocovan:Wtcc f1J1lcl.ion:

1.\ X{fl. tz) = ')"(t), tz)

CO" [X

(td . X(tz )J

~ E[( X (/,) -,,(1,))(-'(/,) - ,,(I,))] .

(5.3)

we haw' qZ(t) = '"I( t. t ). The MT..oFlfi moment is given by function . The mC'1I1! vailII:' and the autocovariwICC'" functioll th(>~erond order momfflt 1TfJ1l'.~entation of th<.' process. In a similnr manne'r w(' call define' 1U001l('uts of higher order. For exam pit"
8

special

C'IL.'>C',

K(tI .tZ .... ,t n )

E [(X(I , ) -"U,))(X,) - "(I,)) (XU.)

- ,'(1.))1

(5.4)

the momenl fWlction of n 'tit ol"rie,'. For more detailed iliforlllll.tion 011 order 1ll0Illt'uts, S(*:, Brilli ngcr ( 1!J81) or Papoulis ( 1983). &><tiol1 5.2.2 \ve will consider lIlC' autoeovarillnce functions mor(' closely.
>

lillie
Figure 5.1: RcalizalionJJ of a st()('.htJ.f!tupr0ce3S.

Characteristics for stochastic processes

!tf>(tion \v(' will introduce a number of useful characteristics for stochastic Thb giv('l; us a possibility to limit our attention to stationary and prut"t's..w s.

probabilil.\' distribution fUllctions. for n = L 2 .... and for all vnlu0> of t, forms the family of /irtik-dimt.nsional probability dislrib'Jtion fUlidions for till' proc<'SS {X(t)}. The stochastic process is comp\{'teiy charactC'rizoo hy 111<' family of finite-dinl('llsionai probability distribution functions. Just. as it i.'! nd'-'<Iuatc to characterize random \~driabl{'i) by thei r mOlll('lIt~. it is suitable to characterize stoch&tic pr~ by their moment !undio'!., The simp\to:;t m OUlI' lIt is determined by the mearl t'aiw; (or the 1IU'm1 "(liut' junction):

'T

is said to be slmllgly stationary if alJ finite-dimensio nal lriloutio",,,

M('

im'flrililit for changes in timp. i.e., for e\"ery n, any set

.. t n ) and for any II it holds

/X (to),

, .\ (I~)(.cl'"

,.r,,) = I'(t,+h), . .. .\""{In+hj(Xl ... ,X n )

(5.5)

(5. II
EAK STATIONAH ITY)

which is It function of /. The ('xpretoisioll (5. 1) is used in ca."iI' th' slImp\!' spatc for X(t) is <"OntillllOIlS, while the corresponding surnmlltioll is uS('(\ W\1('lI t ht' sample spa('e is discr('t('. TIl(' rtll'(!,U vllluC' iJ:; al.so referred to a.'l t11(' Jir.~t "LOmelit of t hI' prOCi'S..... I n ll. similnr fAshion w('" d<'fi llt> the variance (or the t'lUilUUY fundion) of tlU' pro(('S.>;
"'(I)

is said to be IJPf'(lkly .~t(l tionary of OlYit:I" k if all the first J.: . a~c ilivaril\llt to ciJaugl's in time. A W t'llkly statioll{U")' process of

2 Is sllllpiy ca.ll('(l w('akiy slatiolHuy.

stationary proct'.~.~ i.~ rhnmcinucd by til( fact that both the meall and the.l!ariaun aFY' ("(mMant, lJ.'hiif tlu' (UJlorot'ari(!nN' fundio'l depend.s tilt' tnRl' tiii!erf"lI("(" i.(., by (II t z).

\',rr[X('II" E [(X( / )-,,(.))'].

100

STOCI IASTIC P ROCESSES

STOCHA STIC PROCESSES AND THEIR

" I O~1E~TS

101

Proof Pollows directly from Definition 5.3.

Markov processes

In tht> remainder of the book v..-e shall use t he term stationary to den01(' n
weakly stationary PI'OCe8.<;, DEFINITION 5.1 ( ERGODlCITV) PROCESS)

prot"t-'S.....

is called a. Markov process, if for tl

<

t2

< ... <

i ...

A stationary proccs.., is said to be ergodic if its ensemble a\'cmges equal approprillte time averages. By this definition we meanlhat any &tatislic of X (L) call be determined from 1\ single rculizatiOIl :c{t). This is a general requirement. In most applicatioll!> we consider only specific stuti::;lics. 1L'i, e.g., the mean value. A stationary pro('('SS is said to be mcan-ergodic if

distrihution of X (l .. ) given (X(tJ), .... X (tn _tl) is the same as the IItrib,,,;,,,' of X (t.I) given X(tn - l). Th is implies that

P(X (I .. ) OS zIX (t._,) ..... X(t,)) ~ P(X(I.) OS xIX(t._,)}

(5. 10)

E IX(I)I

rx(t,w)/(w)(W in

= lim 2T l
T-o;;

r x(t,w)dt, Lr
T

(5.6)

A Markov pr~., is thus characterized by the fact that all information X(tn ) from past obscrmtions of {X (t)} is contaiucd in t he previous ....,;;"'" X(t .. r). Ow' to the fact that only til(' rnOl;t r(;'Cent observation is the process is also called a fiTl:lt order Markov process. 5.1 (AR(I) PI'"OCCSS, part J) be a lI(.'(IUI'I!t"t' of UllcOITt'ialoo normally distributed variables with oand Varh : - q2, i.e., {:I} is white noiS('. Let {tt} 1)(' til(' input to
syslc.1U (iIfined by the diff<'rcnre equlltion
}'t - } ',
I

i.e., if the mean oj lhe ensemble eqtu!.L'i the mea1! over lime (see also Figun' 5.1 on pnge 98). 5.2.1.2 Normal processes

DEFINITIO:>l 5.5 (NORMAL PROCESS) A proa:~s {X(t)} is said to lx' H normal

+ E""
I 2, ...

(5. 11 )

process (or II Gaussian process) if nil t he n-dimensional distribution fu nctioHl; /;tt,) ., "X( I ~ )(x I, . , J',,) for Imy 11 are (muh idi mell,sional) normal (Jj,stributions.

:,.,lIn.. R Il('W stociJa.-;tif' proccss P'}.

fYt-1 + ,
II \ ' -1

<>l'I

:j

-r

l3y slIcc(>ssivt'ly substitutiuj{ on thl' right hand .side of

is _'n that }i can he ....Tittell

(5. 12)

A !lormal process is com pletely specified by itl; mean value function

I,(t.) ~ EIX(I, )I
and a utocovnriancc function
(5. 1)

(5.13)
(5.S)

>(1,. I,) ~ CovIX(I,), X(I,)I.

Introducing the \"('('tor . p. = (jl(t\), J.l.(l2), . .. ,11(l,,))1' and the variance mlltrix E = b(t"t))}, the joint distribution for X = (X(lJ), X(i:.!), ... , X(tn)T 18 givcn by

(5.11)

fx(x)

2rrn/2 <1('t:r:

J:tt . -

CXP (

2 (x - /J.f E '(x

- 1-')

that [<pI < I. For lol ::::: 1 tilt' varilu\('(' is unbouud('(l. we haw the co\1ltiilJICI' fumHOll for (tl > l2)

where E is t\SSumro to be regular . For fl stationary 1l0rllllli proc('~" tht' IIw8ll value is constwlt lIucllhe IIlItocovariance fUllction only dcpends on (l , - ij) .. Weak s tationarity And strong stationarity ar(' equivalent for 1I0rlllai proC<.'l>SCS, siu('(' a normal procl""", is oompil'tely chnrftCteril.('(1 by t h(' fir.;t tWO IIlOlllcnts.

Cov[Y" , lt31 = Cov ktl +9I, 1+ .

.,,-1 2 (1 +<i+o'"f 3 .'I - f U ?

t-(II"

1(12+ .

'!3+,pE"'2

1+" . 1

".)0 2

102
sint(' COy[e-f,,!I,

STOCHASTIC PROCESSES

S"OCIIASTIC rnOCESSES AND THEIH

~10MEN' I S

103

+ tl~-l + ... J =
,..(1), l'l)

0 for all I,

> tl. Similarly for

t\

< [2.

Purely stochastic processes

l,-/,U}.
11 -

\\", ui>M'rw, that ,),(/ 1 .12) d('P<'nds only on the time dHf(,rt'lll't'

/2. i.e.,

5.8
proct'SS is
~d

STOCHASTIC

to lx> (purely) stocha.o;tic, if it ('an be wrinen ill the fOflll

XI =t,+VII-1 +1,''2, 2+'"

(5.15)
Sill('l' til{' Ill('an value imel 1Il(' variancc Ilrc (:oll!-itant for

191 <

aulo('ovnrillllcc functiOIl d('p('nds only on the time diffcfClI('(', {Yd is ,~lfltj(lIIfl1'Y proces.<; for Ij < L

1, and tht' 8. ulf'flklll

""I~~' of llll{"orrclated stoduL'itic v[lrillhl('S, where [11 =- 0,

IP~ <

00.

It is w('11 knowu thnt Il Slllll of lIormall,v distributed rnndom \'Ilriable; is Ilonllllily dist dbuted. lI('n('(' {l i} is a normal proc(,,~_~ sinc"(' e, i!o. normally distributed. The procl'S."! is t hili> .~tlY'my{y stationary for I4'J < I. hom (5.1 1) it is 1;(-,('11 thA.{
(5.16)

An

illlportCUlt theorem for stationary proces.'iCS follows.

PORE" 5.2 (WOI.O'S DEC'OMPOSITIOl\") ~iQnury l}roce.!~S iT! discrf'te time {XL} can ~ written in the fornl

and hereby that

P~d

is a \iarkov

prOC<'SS,
1'1'0('(':-;,<;

of first onl('r (i n short AIl(I)). TIll' !lallle illdkat<'S llwt IIl'W VUIIlI'S of Y art' obtained by rcgn'S..,>ioll on old mlu('s of Land tht' ord('r is given hy the order of til(' dim'n'm't' cquatioll.

{Yi} is culled all AutoH.cgrcs.'Si\'('

is

(I

lntll'ly .~t(}('ha8tic IHucess awl {Dd i~

(t

IJlt1Y'/Y determi1listic

Covariance and correlation functions

stocha.-.tic
pr(){'('~'i,

5.2.1.4

Deterministic processes

DIFI~ITIO~ 5.7 (DI:;TERMINISTIC PROCESSES)

{X(t)}, wc havc the autocQI'm;ance funelion

A process is said to be dctE'rministic if it can be predicted without wlCt'riuinty from past observations.

Example 5.2
A
pro('(~", {}~,}

defined by

thf' variance for till' process is 0"2 (t) 'Y x.\' (t., t). thp aut()(,O'''Il.'latiol1 function (ACF) is definl'd liS
p\ '(

. (/ /)_ (t t ) I, 2 -

P ..

:01

yO"

J "2(

1xx(t " t,)

tl t1
) '(

l2

(5.20)

wh('ff' AI and At are random vnriabit':>, b

Ii

dctt'rlllini!o.til" pl"f)('\~"',

Example 5.3 Til(' pr()(t'!o..... {ll} dpfinp<1 by

the pr()('~'i is stntiollll.r,)". the-n (5.20) is only 8. function of the tim(' il Dplloting the time difrercnce by T, we have for statiolla7'Y th(' autrX'(w(lfiun('c function

(5.18)
wlwl"(' tPl lind (/)"2 }1ft' ("Oll.... lants, i...; a clf'lf'rminlstic an' random \"ariabl('~.
Tht' proccss in Examplr 5.3 i",
IlIl

1xx(r)

Cov[X(t),X(t

+ r)['

(5.21)

Pf()('('S. .... Hpn' Al

and A3

aulocondation function..
p.\\ T

t'xNnplp of a sullM,t of thr dctcrminj"tic proc('S.-.;('!o; where all tht, \"IlIU(~ of tIlt' pr(){'C;;s can b(' ('xpr(~M'(1 IL'> n linear cOllibiulltiou of a finit(' II\lInl)('r of variahl('S.

. .( ) _ 1.\.\:(T) = "1"XX(T) " "'Ix'>:: (0) O"x

(5.22)

i... I h('

mriall('('

of the

pr~;..

PI('a.'o('

IIOt(' lhat P.\

.dO)

1.

101

STOCHASTIC PROCESS ES

STOCHA ~TIC

I'HOCESS ES I\NO n l EIR MOMENT!:i

105

Let us consider the stochastic proct'SSe8, {X (t)} and {}' (I)}. Th{' covariauC(> bctW('('1I these two stochru;lic processes is describftl by the cross-covarianCe function

Sine<' Varl>'IX (t)

+ A2X(f + r) 2': 0 it
(A~

follows that
T) ;::: 0,

+ A~h(O) + 2>'1 >'2"(

Oxdl.,.t,) ~ Cov[X(/,), \'(',)J ~ E [(X(t,) - ~x(t,))(Y(t,) -1,,.(t, ))1

(5.231 (5.211

sine<' {X(l)} is slnlionary, and hereby Var[X(f) = Var [X(t + r)J = "( 0 ).

wi",. ,'x(t) ~ E[X(')J Rnd I'Y(I) ~ E[Y(')J.

Similarly to (5.20), we defi ue the cross-correlation flmrtion (CCF)
(5.25)

If we put Al = A2 = I we obtain -yeT) ;::: -,,(0), and if we put >'1 = 1, and >'2 = - 1, we obtaill-Y(T) S -yeO). In conclusion, 1,(r)l::s -YeO).
For any choice of z, 11, Ilnd time ill it holds that

wh('r{' u~(t) = Var[X(t)J and u~(t) = VarIY(t)J. Thus, the CToss-covarillllN,1 and cfoss-correiation functions for stationary processes become

vw-[i>'X(t,)] ,,0
.",1

''''(T)
pxdr) =

Cov[X(,), Y (t

+ T)J,

" " L L "',1(1" I,) ,,0.

,~

I j",1

-y,\'),(r) = "(,\'),(r) v"(xx(Ohyy(O) aX u )

It

Thc autocovariance function for propt'rties.

stationary process has the following

h(T + h)

TIIEOIlEM 5.3 (PROPEIITIES OF TIlE ,\UTOCOVARJAN('E: fT~CT I 01\') Let {X(t)} be a stationary process having the autocot'ariance ftmdion -yeT) . It then holds that

o(T)1 ~ ICov IX (, + T + h) - X(t + T),X(t)JI ~ ICov [Y(t ), X (tJII wh.,c Y(t) ~ X(t + T + h) - X(t + T) " VCov [Y(r), \'(t)J Cov [X (' ), X(')J ~ V2 (,(0) ,(h)h(O) _ 0, fo, h _ 0

i) ,(T) ~ '(-T).
it! (~ontinuous in

O.

ii) I,(T)[ ",(0).

iii) The quadrotic form of z gil'en L~I

[:;=,

Z;Zj"(t,

tj }. i, for any:, n
Cl'ff'//,J'/Urt'.

and limes t non-7lcgatil'e definite. iv) If"f(r) is contirlUOUS for

T

= 0 then it is CQntinuQl!"~

Proof

,(T)

Cov[X(,), X(I

+ T)J

from Tlu,'Or('m 5.3 and (5.22) on page 103 that the autocorrelation stationary stochastic proct'SS is an C\fCIl function. This implies =- p(r). It :;hould be noted that from ii) it follows that [p(r)[ :s 1. thE- autocovariancc and the autocorrelation funct ions Me symmetric the lag k = 0, lh('S(' functions are often piow'(l on ly for nOll-negative 'Wi...." '" in Figure 5.2 on the next pagc. The plOL of the autocorrelation 18 8Om('tim('S cal1t'd a cOJTClogrom. we have the followin g properties.
for
8

-"y,

CovlX(t + T), X(t)J Cov[X(t), X(t - T)J "}(. r).

5..1 (PROPEIl:TIES OF TilE CROSS-COVARIAN CE FUNCTION)

'"" "(yxC T).

1 ActuaJJv. til\" bivariate p~ (.\ (t), },(t))T IlIW;t be ~tationarl.

106

STOCHASTIC pnOCESSES

LI:>; EAJI 1'11OCESSES

107

p(k)
1.0 1.0

p(k)
,

.= 0.5

,, ,

0.5

0.5

0.0

5
Lag k

0.0
10
= -0.9

Thl' cros.vcorrelation fUIlCliOlI is not symmetric, find tilerefore it should poticro that lhe cross-correlatioll flltldiom, is a mea.<;urc of not only the ....""'hofa relation het ....'{'{'u to stocha.<;tic process bllt !llso ils dif('("tion. To full de-;criptioll of the rclaliom,hip it is thl'll important to examin(' the ",lKon.,I.,iiool function for bOlh positiv(' (k > 0) and negati\'c (k < 0) lags. of the cro:;:;.-correlatioll functiOIl IlS /I function of the lag I.: is sometimes the cro.~.<;-CO"Y'logTl1l11. ~ u.-; iJlustratc the previous fact by CXllIllp!l'. This example concerns the ...' "">ch,,,;,ic proct'S.'1 &'> ill Exltlllpl' 5.1 on page 101.

5 Lag k

10

1.0
0.5

p(k)

1.0 0.5 0.0

- 0.5 - 1.0

p(k)

- 0.5

<

5.4 (AR(l) process, part 11) the SIUIW jlf(){"('Ko; as in (5.11) on page 101. The pr()("('j"g b stalionary 1. We find for (I.: > 0)
O(k)

O( k)

Coyl", Y,-<i
= C'OVI1'1 1 + f'" l't
= C'ov[ii I,l't-.d

d
= ...

0.0
0.5

M(k
')'(k-)

I)

.',(k - 2)

- 1.0

4>.\;")'(0). Sinc(' ')'(k) is 1111 CVI'Il fUllction it

0
Figure 5.2;
E;J:alllpfe.'

5
Lag k

10

5
Lag k

III

O(k)

"0(0).

of autocorrelation junctiQ7Is lor AR(J) proccsSf!&.

. .Io<",,,II,"'iool fUlictioll h(,(,OIlt(~

(5.28)

Proof
'X,.(T) = CovIX(t), Y (t + T)I = CoylY(t + T), X(t)1
= C"vIY(t), X(t - T)]
= "/YX(-T).

ii) Since Var[A\X(t) +..\2Y(t + r)J;? 0, we obtain

k is often rd('rn'd to a.., t 11(' lag, i.e., timc dib(IUlc(>. ,. .offiriirn< (ktf'rmillcs the IlIl'mory of t he pr{)('(>S.<;. For (j) cl()f,;(> to alonK lIU'mory, while tIl(' 1II1'11I0r)' is short for slIIall values af 9. will ~cillute for (/J < O. Thi." is ill1lstrall"(1 in Figure 5.2. be noh'(i that far (j) < 0 tlwn' pxists no e mbedded first ardt'r pro('{'!o.... in ('()utililiOIlS tim('. This ('(Jrre-pollds to the fact that 1 z = c mllIlO! be soiv('{i with rf'sl)('!'t t o 8 in rfLW z is Ilegative.

>.hxx(O)
Taking Al

+ >'hyy(O) + 2A\A:Z'YXy(T);::: o.
=

I/J'Yxx(O), >'2

1/J1'YY(O), and l'mhl)('(lIl('lltiy .AI

l /J1'xx(O) , A2 = - l/J1'Y)'(O) the result follows .

It follows from Theorem 5.-1 o n the preceding page mId (5.27) all PflKC' I(M that. for tile cf()t;s--('orrC'iatioll fUllction, it holds that pxdr) - p~:d or). tU}(! that IPn(T)1 $ I. rillruly, it should be 110100, that whii(' for the Hon it holds thllt p,dO) I, it holds for the cr()S.';--oorrciation fUllction thai

-"'].

is COHl'{'rn('(1 with lincar stocho.'stic prol1'SS('S in th{' time domain. in Stttioll 5. I stachru;tk pro('('S.'>t's ill t he frequency dOlllain will

Ip,,(O)1 < I.

II lilll'lIf pro('(~:-, {l~} ran Ix> intt-rpr('t('(i fL'> tht' output from a iill('ar wlwn the input is "",hitl' HoiM'.

108

STOCll ASTtC PROCESSES

LINEAR PROCESSES

109

e,
White noice

.-(B)
Linear filter (5.34)
ll.J

Figure 5.3: J/lustroli(m of a linear l)roct'.9S white ,wise as in]rot.

the. output from a linear filter ilati1l9

b referred to as the trr:msfCl'ltm('ti01~ of lhe proce..... exists lUI illveI'SC o perator J'T( D) so thllt
(5.35)
pr~

DEFINITION 5.9 ( WIII T!:: NOISE)

A proc'('S.'> {Ed is slIid to lx, (.I romlJietely m1ldom proceslI or tl'ltite noise, if {Ed is a seque nce of mutually uncorrclatcd identically distributed random variable; with mean vaitlc 0 rule! constant variance This implies that

(5.3-1) mil be written ill the form

(5.36)

(7;.

111 = E[EII = 0, = VarIed = u~. '}'.,(k) = CoYlE!> EHkJ = 0, for k #- O.

Por white noiS(' we obtain the autocorrelation fUll ctioll

0-;

00

~( B) ~ , + ~ ~. B . ,=1

(5.37)

P'(k}~ {' k~O o k = I ,2....

The name white noise is due to til(' fact that t il(' ~ pt'("lral d('nsity for {EI} ill (.'OII:-;t8nt, i.e .. f(w) = O';/2rr. In other words the variations of {Ed can he d('S('ribNi by a uniform proportion of all frequcncies. This will 1)(' discu:s:.cd in Sc<:lioll 5..1 011 page 113. A~ discllssed in Chapter 4, the output from a lincur, till1(>..invarianl !:i~"St('m cnn be written as a cOllvolutiolL of the in put by a weight fUllction (t he impulse r~pOllse fuuction).
ITIQN I LINEAR

7I"(D) ('xi~ts we call d etermine (5.37) lIsing II Tnylor serle> expansion and 1h(" Sf'(11Il'IlCe {11"I,;} is then referred to as the 1r weight.~ of the Equation (5.36) is called t he invCTse form. ~ function for the linear process is given by

''''ooco""i,"".

'"In -(J.)

Co"fYt Y,. ... k]

COY
O'~

[f i~l-i'
, ,,,,0

f,1f', eIH
, '0

;l

(5.38)

'"
,0

00

W "\-'iV.+k -

(gem'raJ) linear process {Yt } is a process that can be written in t.he- form
~

cru;c til(' ....arianc(' brtomcs

I',

I'

'"' .,. ', W . ....',' C.I

pr( ~ " ~ (~ s:_'._ _ _~

o'~,

'hYCO) =

0'; L V:.
,,,,0

(5.39)

. { ~t} is white

lloi~e

lind /I. is the me-an vahlt' of tilt'

~"'''i',",. and
/unrtiort

ill particular the variance obviously only exist when

TIl(' S('(Jllcllce {v'I,;} is oft('n ref('rrcd to as the V' l1'eighis, ane! (5.32) r('ferred to as the random shock fonll. PICllfJe. 7wliC(' thaI without lo.~.~ gNU"rnlity we willll..~.mf/le that It = 0 iu the following. I t is common to scal(' t ed so that \'(j 1. By inlroducing the OP<'rator

til{' right hand :-;id(' ill (5 .38) alld (5.39) exist. the 8utOComrianc(' fUllction is dctcrmilll'd IIsing the auiocovaria1lce
oc

(5. 'O)
.t-lr&f1.'iformntion of the alltoco\"ariRlJC{' fmlction .

v(B)

L
'01

r,:,.B',

110

STOCHAST IC PROCESSf.S

LI~EAR I'ROCESSES

III

By substitution of (5.38) we obtain (since

00
~

~',

0 for i < 0)

5.11

'lVElrrmll.ITY FOR LINEAl(

liuC'Rf pr()C('l>S 7I"( I3 )Yt =~, is said to be invcrtible if

r (z) =

a; L L ~',\-',+kZ
k
' -<Xi'

Ie

-0
~

1I"(z)

L 1I";z - '
;=0

(0.43)

17; L:t,:'d L
1=0
J _0

VJz

'J

fOf

!zl 2::

1.

-= a~1,:.'(z -lh'(z).

F'urthcrmore, s ince we have 1I"(z)t,:'J(z) = I, we get

(5. " )
P lease lIotc thlll one efin get frO Il1 1,"(z) = L:>t',z ' to 'P( B ) = L ~r, J3' h,\' rcpladng z-' with 13' (z 1 corresponds to B). From (5.40) it is st'('u tlllli r(e ..... ) is the F'OuriN transrorm of the Ilutocovariance functi on, which impli('S, I\S we shall SC'C, that (5.41 ) becomes very 1I~f1l1 ill dctNmining the sl)('Ct rum of a linear pnx.'('s/i. The COllvcrgence of tht' IiCries in (5.39) ensurcs that the vuriRtI{(, of till' prOCe&! is finit('. 1n Sect io n 5.2.2 we han>shown impo rta nt propcrt i~ for lit(> autocowl.riance function for a stationary proces..... Por linear proc('SSCS this rAll be summariz('{1.
- ) THEOREM 5 .. 5 (S IATIONAij[TY mil I IN.l::AB

Iealmrk 5.2 prt'viou.s definition ensures that the influence frOIll past observations of going :mffidently fa.<;t to 0 for invcnible prO<."eSSeS. F'urthcnnore, as we Bet' later, the claim of the prOCI:i;S bei ng invertible ('nsures identifiability ''''',un liu{'M proccs.'iCS (moving average pr~ which we will inlrodu('('

Processes in continuous time

ruwe
dOIl(>

with discrcte processes in the previous parab'Taph , we will

a linear proce..'i in continuous time n..'! the output from a liuellr system

rBOCI-'S~ t:S)

The linear process Vi = tJ( 13)'::t is stationary if

~

"('outiuuous white noise" II..... input.. Therefore, w (' will first. introduc(' '''(""()lItinuo us white noise .... many practical s ituations we are working with u process {X(t)} fOI" valu("S X(t) at given times tl and t2 are independent if It I - l21 > { , ~ u. small Illid 1f'SS thall th(' smalle~t time interval of physical interest. bt, ! he Cru;(! whcft, t he process describes the forces t hat work on in a 13rownian motion. A mathematical description of this illotio n to a definition of stochfl.Stic processes with indepcndent incr('ments.
PROCESS WITH INDEPENDE:-IT

,(z) ~ L>" ~'

, .0

convcrge.~

for

Izi ?

I.

tlw tim('s II < t 2 < ... < t". A pro<.'C'SS {X( l )} is said t o be a !.lith lwlependmt (or Ort hogOllal) irlN'ements if th(' random variables

Proof Omittro. Silll'C stationary prOC('!-,:-;('S 8rt' rhAfact(>rizro b..' the variUIl('(' iN finite, olle can interprC'! tIl(> r('Sult by colllparisoll with (fi.:19).
~ Remark 5.1 The daim i ll Theor('lll 5.5 ('llsun'!. that th(' infl ll (' ll("(' from past ()I)to,('f\(ilioIL~ of the lloiS(' \"8.rinble e:, gtlf'S sufficiently fas! to O. <II

"""""y illdepClldent. If the variables only are mutually uncorrclated

tht> proc('S..<; {X (t)} is said to be a 7}T"(K"e.9s u:ith mutually
ull001'"t"e/ated

mOtit important proces.... in the class of proCffi.<;CS with iudepC'uclent is the Wiener proct'SS.

Similarly, it is ('OllvcuiPnt to daim that th(' n1t('l"llIl! i\'(' formulation of tllf' Pi"O('('SS. 1I"(B)Y" Thlls. \\-"(' d('fill(' tlIP following.

11"

F.t,

wcip,hts b('longing to go su(fi<"i('uliy f/l..'it to

is said to be a Wi('uer 1)T"(K"es.9 (or Brownian

JJ2

srATI O~ARY

PROCESSES IN TilE I'nEQUENCY DO:\IAIN

113

i) P{X(O) ~ 0) ~ J. ii) The increments X (tl) X {to). X(t;! ) - X (ld. . X(t .. ) - X(t .. d 1U'l.' mutually illdcl>'ndent for arbi trary tillll' points 0 ~ to < t. < '" < t". iii) For arbitrary f n!ld " > 0 t he increment X(t + h) X(t) is normally distrihuted EIX(I + h) - X (/ )] ~ 0 (5. 11)
Vn,IX(1 where
q2

{dO} i... (;Ont iuliOliS white lloist,. Here P~(t) - It} is the OlLtpllL .......pond'n.LO the input {E(t}} from a linear sy:;tem having t he impuL-;<> function 1,.:.'(T).
Tbt' linear proccs..; {>'(t)} defined by (5.48) Im.o; the
1II('all

val uc (5 .49)

+ h) -

X(/) I ~ a'h
aut<K'O\"ariancc function
'"'rll

EIY (/)I ~ I'

is ('alled the variance.

(r)

Cov[)'(t), Y (t +7 )1 =
Ix,(,OUl~

Furthermore, it can be shown that flll t he finite dimensional dis tributions for a Wiener process arc multidimensional normal d istributions (cf., C'.g .. SoIX:l"k (1985)). Another important proct'ss belonging to the family of P IO(('i;.'>!'S wi lh indepelldC'nt inCfelllenCllls, but defined 0 11 a discrete sample space (No). btlif' Poisson process, ft." dl'SCribed in Crinunit and Stirlaker (1992). Por a Poi:-i!'OD proccss the probability of n events in the time interval t is

(1; fo<X. 1p(u)\J(r + u}(lu.

(5.50)

aspt'("ial ca:;c, the "arillllCt:

11~'

= ')'\ y(O) =

11; 1'XJ ~J2(,) d,.

(5.51)

P{n,t) = (.At)" eAt (5..16) Il! where>. is the illt<'I1Si ly of the Poisson prQCe$S. In Ulnl!J'ltlatical terms it is more difficult. to define continuous white i1oi:lt!l.S oPPOM'<I to discrete white noise; sec also the discllssion in 0kM'ndal (J!)9.j). The definition of white noise is baM><1 0 11 the ossumpt iOn that t he ~ pcctral dens ity is cOllstant, Le., f(w) = c, -00 < W < 00. But since lhe varianl'e of the process ccillais t he integral of J(w) (cf. next SCC't ion), t he variallt'e becomes infinite. Formally, t hese difficulties Olll be avoided by introdudng continuous white lIoise as a gene rali ~ed stochastic proCt'SS frOIll a g(,lIeralizOO time derivative of the Wiener process.
DEFINITION 5.}'1 (CONTINUOUS WIIlTE NOISE)

CQII('t' pt of a {'()JItiuuolls white noise pr()C('S.., as a gelleralized process t he lHltoeo\'ariance function in Dcfinition 5. 14 is nu approuch often In signal proc<'SSing H/:"t' Papoulis (1983). lIowe\'t'r, it turns that a sati..'..fal"lory illlro<iuction is obtained by lIsing a proc('SS in tilc dn.5.';. of wit II independent. illcrclllcllts. A classical rcferellCI' to t his discussion Ibmdal (1005) ...... hkh also contains nn excellent introduction to stocim..<;tic f'(llIl\ tious. 'lore rcct'ntly thc u.<;(> of stochastic differential equations .......Ii". hAA becolll{' vcry po pular----scc. e.g., MadM'n, Holst, a nd Lindstrom for ~'\'('nu nppliC"ations, Nielsen, Vestergaard , and 1 1 ud~ 1I (2000) and pt al. (2OQ.1) for thc lI.S(' in finance, Kristen&'n, ~ l ad.';('Il, and J orgenscn for IIlQd'ling ch('mical systems, Andcr&'n, ~Iad.sen, and liansen (2000) i t lw heat d ynami<'s of a building, J6nsd6L tir, Jacobsen, and (2{M)I ) for d{'1;(' ribing t he oxygen conte nts in a slllall creek, and Via (2006) for the II~' in astrono my.

Contin liollS white noise is formally defi ned as a generolized pro<..'<.':>s {~(t )} with alltocovaria ncc function

Stationary processes in the frequency domain

' ..cti,,., we will rollsider (wlakly) stationary processes, which in the ti me an' d('notro P~} (discrete time) or {Y(t)} (continuolls time). Such a I", chanU'lerized by til(' mean value JI and autacovariancc fU llction thiJ.; ~'('tioll an rulcrnati\'c way of charactcrizing til(' l)rOC~ will be

1'(') = U;O(T),
whcre 0(,) is the Oirac de lta function (S<'e Seclion 4. 1). The definition of white noise introduces a n a pplication, which has 1llaP1 si milarities to the impulse response fu nction for linear sy!:!tems.
DEFI NI] IOt\ 5.15 ( Lt NEAn PROCESSES IN CONT INUOUS T IME ) A linear process in continuous time (Y (t)} is a process that can be in the form

~T;,lle

, 1I1l11ll'ly. by illtrodul'i ng the Fourier tTl).lIsfunn of t II(' 8uto(.'ovari811cc of I h' procl'S.-;. 8hlnvll (ThroTl'1Il 5.3 on pag(' 104) that the aut()('Q\'8TianC(' funct ion dcfinit(>, Followillg fI. theorem of Bodmer such a nOIl-lIcgulivc huh' li".. call Ix> written as a Stit'itje; int('gral

O(T)

I:

.~' dF(w)

(5.52)

STOC'H,\STIC' PROCESSES

5. I

STATIONAHV PROCESSES Ir>. 'J lib FREQUENCY DOMAIN

115

for a

J)t'O('CS.,>

in (:ontilluoUS time, or
(5.53)

rcw)

few)
f7~ /2 f7~ / 2

u',

for l.I proc('SS ill discrete tlIlIC, wherc the properties of F(w) correspond to th,' properties of a distribution function. it holds that:

F(w) is nOIHlf'Crea.sing.
F{ ' OC/-7r) = 0 for pr(){"(>S.-;e5 in ("Onlinnousjdiscrelc lilllC, respectively.
F(oo/7r) = )'(0) for proccsscs in continuous/discrete time, rcsp<.'Ctively. The existence of (5.52) is given by Wierler-Khinlchine's theon:.m, wbile the ('Xistcllce of (5.5..1) is givell by ~J~ld's theorem, d. Prie!>lley (1981). F(w) is rekrrro to as the spcctml distribution of the prQC('SS. \\'e have sccn that any stationary process CIIll he formulated us Il slim of "purely ~tochllstk process 8nd a pllrcly detf'rillinistk process (TheorclIl 5.2). Similarly, tllf' sp<'Clral distribution call be writtl'll
Figure 5.4: Spt:dral di.,triblltioll alld demity lor the 11Ull' drtenllint.,tk process glue" by (5.61). This also iill'Strates a IWry simple lille 81H!Ctf1,m .

T II EORF.M 5.6 (RELATIONSJIIP IlETWEEN THE SPEC'TIWM AND Tm; Al1TOC'oVAIlIANCE)

.~pt'drum

fin' a ,'W"('ly .9t()('haslic l'roces8 lIlt' have the following ,-elations betwecn UU' and the auiOC011anance function f(w) = - I

wht.'re Fs{w) is an evell continuous fUllction and Fv(w) is a step function. Both (unction.s arc non-de<::reasing. For a purl' d,terministic proc('SS
Yt =

OCT) =

J: "'.,
2"
- 00

1~ C- '..!T...,(-r)(!T
(continuous lime)
(5.58)

f(W)dw o(k)C- ,k
(discrete time)
(5.59)

few) = -

2,

L
k= . ""

LA, COI'i(w,t + ,),

,=:1

(5.55)

Fs will becolllc 0, and thus F(w) will become a step functio n with steps at the frequencies w., i = 1 ... , k. In this CfISC F C8n be wriLtcn as
pew) = PD(w) =

Proo f Follows from til(' previolls Ie;ults.

few,)

(5.56)

.. Remark 5 ,3 Tht' Ill)()\'e rC1iults can 1><' exploited with (5.40) to cakulate the variance of a proccss giv{,1l its tmnsf('r functioll. Accordiug to (5.1O) it holds that

nnd {J(w,): i = 1, ... ,k} is often called the line spectrum. An eXlIlIlple of II linc spretrUill is shown in Figure 5.4. Of grenter interest is the ('fISC wl\{'re the process is pllrt'ly stochastic. In this case F(w) will be a ("OlltiuuOIlS function and if we disregard ca.ses of purl'iy theoretical inten~t, F(w) will also bl' differcntiable. Hereby we llrc nblc to introduC"(> the spectrum (or the S1H'ctml den..rity) for tile stochastic process as

few)
Suhstitut ing t his ('xpressioll illlo
(12

(5.00)
(5.59)

we

ha\'{'

...,.(0) -

/"11: e'o .... f(w)dw

few)

(5.57)

SinCl' W{' a......'HIl Il (' l hat Ff)(w) is zpro, WI' can suhstitut(' (IF(w) = f(w)dw into (5.52) and (5.53), thf'reby obtai ning t he following Fourier tmll..,,{onnations.

llij

COW,IO~LY l'SED LINEAR PROCESSES

117

when' the variance of II p rocess givell hy a trans fer fUllc tion 11' (sec (5.34) on page 109) Cll U \.)(> obtained by rl'Sidue calculations. <II

tilt' spectrum can most easily be fou nd using the transfer fun ct ion

&be Iincar process. Here we apply the following.

BIll" "" 5.7 (THE SPECTRl'M DETERM INED FROM A TRANSHR FUNCTION )

If. we allow f(w) .to be completely or partiy described by generalizing functions (delta functions), then (5.57) on page 114 and (5.58) 011 the prcvioll~ page can be lIi>C(i for FD O. This is ill ustrat{'(i ill the following example.
Example 5 .5 (T he spectrum for a p ure determ inistic process) Con;.id('r the pure drt('rministic proc('S.'!

tlu' linear IJrocetJS \', = 1p(B)Cf or 1T(B)yt = c, where {cd is white Then tltt" spectrum for the process is given by (1.1.1 E [-1T.11"])

V {t ) = A l COIi(Wot)

+ A2sin(Wut)

(5.62)

(,.61)
Follows dir('("tiy frOlll (5,41) 011 page 110 and the definition ;"'""i."" generating function (5. 10) on page 109.
011

where AI and 1\2 arc lIIutllaUy UllcOIT('!ated randoll! vlU"iabl~ ..... ith IIIt'lUI Val llf.., 0 and same varianC(' (1~. TIl(' alltocovarianC(' function h('Com~

tilt'
!iCE'

...,{ T) -

a; (coo{wot) ("Ob(WO(t + T)) + sin(.....ot) ~ill(WO(t + T)

= a~ cos(waT)

II 8f'('11 thllt thl' integral of f(w) equals the \CU"iallcc of the process, and (5.G!.I) on page 11 5.

Applying (5.5H) ..... P g<,t the ;jl)('('lrum

Commonly used linear processes

U8f'ful das.s of linear processes is the proce:sst'S where t,:J(z) can be lUI 1\ rational functiOIl : I,)(z) O(z )/1J(z), where O(z) fUld (z) an'

The MA process
"'):F" dT . 211"6(1.1.1 '""0). wher(' the ('(1111.\) sign follows from sl){'('tral di~tr ihlltion a lld dem;ity willtilell 1'(>("()1l1(' a.s sk{,tcilf'(l in Fig-urt' 5on thl' pr(>("('(lillF,: pag('. Plet\..<;(' Ilot(' that the slx'("(rlllll in this ("11..<;(' is a /->)X'("trum.

r: (' . .(. . .,

(5.63)

Example 5.6 (The spectrum for white noise) Sinc(' tht, autoc(lvurinIlC(' fUllttioli for discrete while Il()i~p

{(,}

is

is whih' noise, i!l called II M oving At'cfUf/f' IJr"QCR..JM of order q. In is d,'noted nn MA (q) process.

{ {)
w{'

f1 ;

for k

o
.t:1, 2,.

,IolA(,, ) proc('~.. is

for k

~("I

lilt" sJ}('(trnl (ku:.ity for whitC' lIui,,('

characterized by the fact that the 1>' weighu; &om a giW'1I pOint. Uaillg lhe backward shift operalor B, the r.. IA (q) a1H<) hf' writtcn a....
,llIIs

I ',
i.f'., un iformly tiistrilllltixi ~1~x1 ral d~'n"ily,

"""'''''M''' O{B) is thus giv('n by (5. 61). The eorrl':;pondiug tmnsf('r

110

C O\lMOr-LY l"SED I.INEA Il PROCESSES

119

TIIEOIlE\1 5.8 (t-.IA PRO('ESS: STATIONARITY AND INVERTISII.!TY)

An MA{q) process

i.~

i) Alway., .,ta/ionary.
ii) hwotible if the 'l)Q t.~ i,~ O(z I) = 0, Il'ith respect to z, are all within th" unit circle.
Proof

k ~ O

,,(1-)

~ 1~~; ~
1

{ 0

0,/(1 + O~)

Ikl Ikl

~ 1 ~

(5.70)

2.3" ..

autooo\1U'iancc function ')(k) til(' mOIllC'nt I,"l;timatcs for O. and

f1?

,~"..."n;"'<1 gi\'t'll thai ')'(0)

(} and 'Y( 1) =;J nrc kllown. This gi\'cs

i) Follows from Thoorf'1II 5.5 011 page 110.

ii) Follows from D('finition 5.11 011 page 111, where the cocffici('llts of the Tuylor t'xpansion of 9 I( B).
7r

\wights nr(' thl'

0. ~
a~
is chOI-l("J}
a:, 01 11 '

;J )C:), -I
plOI
~ I A(I)

(5.71 )

From (5.38) we find the auiocol'ariance /'Ilnclion JOT the MA (f/} pronH

Ikl Ikl
In particlliar thl' variance

~ ~

0, I ..... q
q + l , q +2 ....

of t lU' two solutions t hat giv~ 1011 < I in order to ~l o n' about thi:; is:sm' in Chapter 6.
process bccolll('S

tlK- !I1)('( 'tnull for til t'

(5.72)

Please l10te that lb{' lIutocovarillnce function (md, tims, the uutocorrl'illtioa function are l('ro for lags greater than q. FllrtIJ('rmore, it s hould I)(' not('(i thal ...,(q) = (7;0" will alwa.vs lX' differe nt from zero; othcrwiS<', it would at lHo!'j1 be
all

b- tht>

MA (q - 1) pr<x't'i's. Frolll (5.02) it is!;('('1l lhf\t the spectrum for the MA (q) proce.'1S is

1(",) ~ 2 ~ 0(.
whue W E [-11". "11"1. If Ct E N(O,a;), thcII (l~l"'" Yt ,,)'l' follows an 7l-dimen:-;i()uai , ""m.~ di!.tribution and tlms {Y, } is a normal process. 5.17

a'

5 .5 Oil till' following page' t h(' aUlooorreiation functio n and llJ(' ~tA ( I ) pr()('t's.;; /In' "howil for 01 = 0.8 and 0] = -0.8. MA( 1) PW("("""i it holds ill g(,lwral that when 0 1 is pooitive, p( 1) and till' sp('cirum is dOlllinat('(1 by low frcqu t'llcies. Whell 01 the p(l ) i!; n('gl\ t i\"(~ and the !;pC<"trum b dominated by high

~~

A R(p) PROCESS)

Example 5. 7 (The MA ( I ) process)

CUlIsid('r the MA(I) pro('('S.';

.....-

(} 't} given by
(5. 73)

~inc('

I + 0 1z 1 -=- 0 ror .;: T ht' a llt<x'ovarillllcc is

iii whit(' lIoist, is ('alled all autoregN'ssivc proccs.~ of ordu p (or

o(A)

fl
0

-0 . the pnx't""" i" inwrti hl(, for 0 1 < 1.

tII;I"'

A
ik

IJ j (J J

"

AR(p) process is l hus a linear procC'SS where lil{' 11' weights are z('ro p t- I. Al ternativt'iy, all AR(p) process call h(' writtt'll

,k , 2,:1.

(1+IP 1B +,. ' +op BP) Yi =1

(5. 71 )

120
p(k)

STOCIIAST IC PROCESS ~: S

CO.'.IMONLY lfSEn L I NEAil PROCESSES

12 1
. ApZ I). By dccompooitioll

e ~ 0.8

,
0
I

,
.. Lag k
0

I(w)

Wp ('ml write (z 1) = (1 of {z I) ',wcgct

>'1 Z-I) . ( I

(5.77)

3
O ~

,,(k)

0.8

1(",)

w
Ac('Ording t o T h {'Or('1I1 5.5 on page 110, t ile process is stationary if t/J, conn'rgcs fo r jzl ~ I. This im plies that in order to Qutlliu !o;tationarity Wt" n('('(1 1'\, < 1 for i = I, ... , p.
~"p...,.

,
0

,
2

wi th t h(' dlli lll for !>tabililY for lincnr lSystems. The equation

= 0 is called th(' characteristic cqtIldiott for tl1(' process. &>e, furtlwr-

.. Lag k
0

-!

A"l"'<Hlix A. "The solution 10 difference C(IU8tioIlS," thf' remaining part of thi:> section ollly stationary processes will be ill t h is (,fI..'i(' thc' following holds.
5. 10 ( A nO{.'OvA IU ANCE F'UNCTION FOR An PROCESSES) ""]00><",",,,,,, function /01' an A R(P) pT"OCtSS sati4ies the linear difference

Figure 5.5: The aut(J{;fJm>iatioll iU.flCtiol1 (l("/t) and spcdnllli (right) for the MA {I ) process Yt = ~I + 00., J (u; 1).

0'

(5.78)
(B)j = ~ t.
C brulition

when' (B) is a p'th ord('r 1 >olYIlomial in B. The 11"H.Il:;fer operator is (D) - I and the transfer fUlletioll is l/(z 1). The name autof"cg/"CSSiVl' is due to tIl(' fact t hat the value of }~ cnn be ~ as n regression Oil past values of 1he pro<.:ess, si ne('

o(O)+.o( 1)+ .. + pO( - p) ~ o;

0(' k).

(5.79)

(5.80) (AR An A R(p) process is

THEOREM 5.9

multipl)-'i ng (5.73) by
PROCESS: STATI ONARITY AM) I NVEHTIHILlTY)

}'t-k

aud finding expected values we get

i) Always inverlible.
ii) Slatiollary If the Tools o/(z 1) tmit circle.
Pro or

O. with T"f'spcct to z. all lie ujthi"

E[)j d 't

+ 01 )j

Ft _ 1

O(k) . n(k

+ ... T IPp l 't k)j I) + ... + .pO(k

p[ p)

i) I:OUows fTom t he d('fi nition of iU\,{'rlibility.

122 If wc write' out (5. 78) for k

Yuie- lVaikfT Nlucltions:

STOCHASTIC' PBOC ES~ ES

('O\lM ONLY USED LINEAR PHOC'ESSES

123

:=

1,2, ... , p, and divide by ')'(0),

W('

get thl'

p(k)

P(I)) ( p(l) I p(2) (

p(~)

p( I)
1
.. .. ' )) p(p - 2)
I

P("~')

.
p(p - 2)

(-')
>1>2

-p

! ,

and agaill by divj:-;ioll by ')'{O) in (5.78), it is seen that Hlso p(k) snti.!ofi(~ t ht.

difference equation (5.78) [or k = 1,2 ..... The difft'renl.,(, e<IUUliou ("un be solved explicitly (St"t' Appcndix A). If all tilE' roots ,\ I, ... ,'\p ill the characterist ic equation, (z' 1) = O. lire diff('f('lIt. then the complete solution call be writtell
(5.H2)

o ~J----L~~= 2 3 5
Tlif Ilulororrriation /lmction for th e A n (a) +0_1'1"", 3 E,.

__

Lag k

where the roots arc eithc!' rcal or complex. Especially, if it is

A.'!SllHll'd

that we

0.2, and 411 0.1. "'""ndlug to the two fin-I Yule-Walkf'r '<ruations we have

haY{' PI rt'al and p()i;itiv(' roots, 1>2 real and negative rOOLS, and 2P3 ('OIHp\('X

conjugated roots, then the roots cun be written

all

j= l ",,PI

n) b)

p( I) + d .,p(1 ) + .3P(2) 0 p(2)+ , p(1 ) +.;, ,."p( I)=O .

p(2)

-13l

j= 1, .... 1>2
)=1, ... ,1.1:1,

!,cf,f/J,

->1>2

p(1 )I, +0,).

where CtJ ' ;J), and ~J all hclong to tile interval 1 0, II. Now till' l'Olnpl('w solution can be written

pi I)

+ :j92 = r + OJ-01 - 'XI ~~ 3)'

,(k) -

,.,

L AJo; + , L, BA-PJ).i; + L Cj~; cos(OJk + Qj)

PI

1':1

P3

applit'l:l value!'! Wf.' obtniu p(l) = 0.3'1 aud p(2) = 0.27. For the i"nol'"~ur k. WI' UN.' Iht, f('('un.ivf.' formula P(k) o,plk - I)
p(~)

for k = p. p + I , .... It is Sft'n that positi\'e real roots imply All eXl>OlIl'lll iaUy dl'('Tf'8sing incr(>lU(>nt and negat ive rcal roots imply a shift ing IX)!>il in' BDd. ncgatiw eXI>Oll('util\lly d('('reasing increment. while two compll'x OOlljll~Hled roots gh'c nn increment in terms of a _dnmped wan'. It is usually quiu' l('diuUl to find Illl ('xplidt solul iOIl. An alternative (and often casier ) procedure is to calclliatf' tilt' val lit' ')'(0), ... ,,),(1) 1) IUld lhl'n dt'LI'fllli l]I' -y(p).')'(p+ I), .. . by us ing (5. 7~) on preceding page recursively. The same holds for the nlltocorrf'lal iou which is illw;tmU'd in tli(' following eXlimple.
Example 5.8 (Autocorrt'lations calculated. rccursiv('ly) 1.('1 liS dl'll'fluim' tht ulllocorrl'latioll fuuctiou. for til(' An(;-S) prvc18

>1>2p(k

2)

,plk

3).

==

no!).

0.03, p(fi)

- 0.01 , f.'te. The autocorrelation

",-,ibld,,,,1

in Figurf'

5.{).

tht> autO<'orrdnl iOIl fU1J(" iou consi);t); of an exponelltial dean additi\'{' t'xpollC'ntiully dl'<'rmsing hnrmonic wuvc. Th(' dUlr!\("lu; 011(' rC'al root ulld t W() ('Olllpkx ('onjugated roots.
(5.62) it is SC<'n thllt IIw Iilx:d'''Wll for an AR(p) process is givcn by

flw ) u, 2 -;C(,,:-:-;: ,"")".-;(,- :"'_)

I

(5.85) (5.86)

J 24

STOCHASTIC PROCESSI::.~

COW,IONLY USED I,INEAH pnOCESSES

125

5.5.2.1

The partial autocorrelation function

V" ili found by replacing the k'th (i.e. , the hL'>t) < :ohunn in P" with p,,:
Ap!>,lldix B. W(' hereby get
(/lLL

DEFINITION 5.18 (TilE PARTIAL AVI'OCORRELATION 1iIu-) The partial autocorrelation kk (PACF) is defined as

= PI

<h
i.e., 111.:" i5 the correlation cocfficknt between Y, Rild YH ill-between observations.
.\:

givclI an tllf'

_ I;, p'l '-I ~r

p,
J

P2~/12 ~ P~
I - f)~

p, p,
J
J.J=

p,
{>,

It is shown ill Appendix B that when the abo,,{' definition is used. !hit is the k'th coefficient (with opposite sign) in an AR{k) proC('h.<.;, which giH'I'I u.<;('fui formulas for dcteqnining u. Let us consider tlIP AR(k) process

'"
p,

p, p,
J

P3

wherf' the partial autocorrchltion thus is the coefficicnt for Yt - k. Using thia we CRn write the Yule-Walker equations

Appendix 13 n recursive formula for determination of ill u~'flll for IlIrge values of k.

'"

p,

p, '"
J

rPkk

is given. This

The ARM A process

;;;;rn"'55.19 (TilE AR~IA(p,q) prore...... {ytl gi V('1l by
where short
PROCESS)

<PH

is the partial autocorrelation at lag k. 8<11I3tioll (5.89) is written ill

is white LLobe is ('alied an ARMA(p,q) protess.

(5.92)

THEOREM 5.11 (PARTIAL AUTOCORR.ELATION FOR

A R PRon;SSt:s)

For on AR(p) process it hold.,

introdu("ing poiYllolllillls thing the shift operator B, the AJL\IA(p, q) be writtl'll

rPH::FO,
(/lkk

0,

k$p k>p

(B)Y, = O(B)".
and 0(8) ilr(> polynomials of order p und q, Te>1 >eCtiveiy. trans/f"r f1trJ("tion frOIll F: to Y becomes

(5.93)

Proof Follows from the previous.

On thl' other ha.nd, the partial nutocorrclatioll fUllelioll for Illi f..IA(q) will take vruu('S different from 0 for arhitrary iargc k. The partial autocorrelation function is obtaiued by ~lvillg (5.89) ""ea""''' with rl'Sl'f'ct to H for k 1,2,3, .... H pllc. Ollf' ('an d('tNlllin(' (/)u 1\.";
d('t Vk (/JH = det P '

p""".

(5.9 1)

5.12 (AH\1A PROCESS: STAfLO/l.'AIUTY A/I.'I) If"VEIULBILITY ) ""Ill". q) Ilmn'lIlI iH stationary if /II(' root.~ of q')( Z I) = 0 li(~ wi/liin th( int'o'iible if tht' roots of 0(.:;' I) lit' within the lInit cirdf'

126

STO('II,\STI(, PHOCESS/'!i

COMMO~LY USED I.INEAll PROCESSES

127

Proof Omitted. There is a close relation bctw(>('1l Thool'{'tn 5.12 and till' corresponding lil('Or('IIlS for All and r-.IA processes.

O(p)+O,O(P-I)++"O(O)
111 tile> rf'lIlaining part of this section all procC5tiCS arc ILSiilllllt,<! stationary. I3<>low \o\'{' show a gell(>rrullll'thod to determine ~h(' autocot'o,ianl'.t' /tmctioll = Op...,~y(O)
1(V + I )

+ ... + Oq"r,d(1

p)

+ ".o(P) + ... + ..,1(1)

1,-1)
(5.10 1)

lor an A IlMA (p, q) lJrQCCS.'.

Sill('(' }~
('1\11

=8p+,"rldO)++Oq')c)(q

lw ..... rittC'1l in the form

)', =
it is obvious that

L t..:'.et-i
0=0

thai only from lag k = q + I P will the IlUlooovluiancc lind, hence, m"m,,,,,,,'I.t;on fUllction consist of dnmped exponential and harmonic . TlIP codficil'llts ill tilt' complete hOlulioll arc determined using (5.99)

,I { ::/:.0

=o

fork<D

fork ;?:O

l3y multiplying ([;.92) 011 tIll' p['('{'('(ling page by et-k on both sides and t"kinK ('xp('('t(~l valuC's, WI' obtain lhe diffcrcnce equatioll

5.9 (The autocorrelation for all ARMA(l,l) process} tlw ARI\IA(1. I) PI'O('('Ss;

"Y.dk)+IPt1'E) (k- l )+"'+lPp"Yrdk-1J)=Ok(1;,

0,1, ...
'),)-(k) (t1~illg ([;.97)):

Fin-;t WI' clN('rmill(' 'Yl!'dk) using (5.96) and (5.97). For t'xRmplE',
1d(O)

".

,
(5. 102)
(5. 103)

"),r( 1)

')",). (k)
Prom this it fol1o ....'S that

'"tId!)

(0 1 - 9100)(1;
k

or~o

.. ) (using.
~,O(I)
+

([;. 101) :
""'I. dO)

Subs<:>qucntiy by lIluitil>lying (5.92) by}~ \lllue!., we get that ,(k) satisfies

on both sides and taking ('XI)("(tf'd

0(0) , 1(1)

+ 01")().. (1)
(5. 101) I), for k?: 2

, 1(0)

0 1""'1,) (0)

O(k) +~'O(k - I) + ... +,O(k - p)

oCk)
for k 0, L ...
(0) aud ), l (1)

-011'0'
",I'

(5.105)

= OA:"'r!l'(O) + ... + Oql"rdQ - k),

If J1

obtain

> q we obtain
")(0) 'Y(k)+rpl'Y(k

+ ~"I ')"(1)
(5.106)

I)+ -tp")(k
IIlld,

IJ) = O,

p,p+ I,.

0(1) , $,0(0)

In tbis

ClI..'i('

tbe

lIutOl'OV!It"iILlH'(,

bf'I1(,l', til(' autO('orr('la!iol1

- 91")(0)) (I
t

will (om,is! of dattl l")('(i ('xpo!l('ntial and sine function:; (d. Appeudix A UII
solution

of diff('f('I I(""(, ('(Illat ions). III(' ('oeflki('nt" of tllf' (olllpl('t(' :-.olutiOIl fOllnd 1I"ing(5.99) fork .. O.l, .,p 1.

0 1 (0 1

!pd)fT;

1(1) - -dl'\1(O)

+ 81

rr:

(5. Hl7)

128

STOCHASTIC PHOGI!:SSES

('OMMO:>lI.Y L'SED LINEA H PROCESSES

129

p(k)

Follows from the fact that the AIUIA(p, q) process can be wriltCllo.<; F'(IBI,,(B)V, = ~h i.e. , a.~ nn infillitt, order autoregressive' process. _

0.8 0.6
0.1
0.2

'J1K> );1){'Ctrum for an AIUIA (p, q) process is givcll by

1(",)

~ 2~ (c '")(c'") a; II + OIl' ,... + ... + Oqe-qf

= 211" Il +r C"""+"'+pC
ipo..ol2

q; 0('-.... )0('... )

0.0

(5 .112)

0
prrJCXSS

Figure 5.7: The outocorn:lalion fQr tile

Yi - O.7Y'_1 =

t:,

+0.5,.

j.

The inverse autocorrelation function

0'

1 +of

1(0)
'")'(1)

(1

29101

the statiOllfiry ARI\IA (p, 'I) proc('S8, {Yi}, given by

""
d
2

OI(i-tPi}-IP+Or- 2q) l OI} 2 ( 1~) .~

( O)l\

0(0 )"

(5.1 13)

t t ,} is white noise. Ifthc process, {Y,}, is invertible, then the process

O ( B)Z, ~ 4>( Il)"

.....
1(k)

(I - 01 4>11(0, -

(I - ?)

(J~,

(5.114 )

,1(- - 1),

k" 2

""'ionrue,. ARl\IA(q,p) process. This process is called the dtuu P'f'OCeSS.

5.20 . J~VEJt!;E AUTQCOI1.HELATJQ;./

Th(> 8l1tocorr<,lfHiOll fun('tion becomes

p(O)

p(l) ~ (1 p(k)

9,~,)(9, - ~,) 1+0 1 24> l O,

k~2

autocorrelation function (IACF) for the process, ,givcn by is found fl..., the autocorrelalion function for the dual process. The &DtQ('orrelatiOIl al lag k is denoted pi(k). note that if the prOC<'SS is UII AR(p) process, then the inverse _reh'tk,n corr<'l;pouds to the autocorrelation function for an r..1A(P) ThlL';, we ohtl\in the following.
5.11 ( I NVEIt.(iF. AUTOCORl\ELATION FUNCTION FOR A R PROCESSES)

(-,)' 'p(ll.

cou,!;ider for ('xumpi(' an AIUL\(1, 1) pro('e;1"i when.' 1 6, = 0.5, we g('t p(O) = I, p(l) .. O.R308, ,,(2) = O..')8J(l, 1,(3),. Tllf' I\ujocorrelalion function is skdched in Figure 5.7. As lhf' figur(' inc\icutN ti1(>fe i~ an expollential d('<'rcfI.'i<' froUl p( I ). starting vnlu<, d<'l><'uds 011 both J a.nd 0\.
wt'

If

AR(p) p7"Q('('.~.~ it hQid3 that

Remark 5.4 It is seen from (5. 100) that for sufficient ly larg<' vallll.'s of k, tll(' nu'LOCO..... ance fUllction fulfills a homogencolls difference equation detcrmill('(l by Rlltor<'gres.<;iv(' part of tlIP modc\.
TIIEOIU;!'.I 5.13

pi(k) " 0,
pi(k)~O.

k '5 p,

k > 1'.

(5.115)

,l'IJ&.".

from the pr<'viou.'I ]ll\rIIgrnph.

The I)(Hti(!l (w/(){"orrY'ia/iou f1l11("tion for an A RAIA(p, q) diff(',nli from 0 f07 arbitmril.1l1myf' ,,(,lues of k.

l'mN.~.~

nm iak('

"""''''l1utooorr('lntion can b<' us<'d as an alU'rnative to the partial autoin th( i(kntification of pur(' All PI'O<.X'SSl..':;. Compare Theorem 5.11 5.11 011 pllg<' 124.

130

STOC'I I AST le PROCES!:i E!oI

NO:-<_STA ri ONARY MODELS

13 1

Siuc(' till' illvl'r.;(' IHltQcorl'ehlt.ion funct ion can be found ~illl ply 6.'1 t h{' autocorrelation function for the dual process, the inverse autocorrd!llion for
ARMA
p roCl'S..">C!I

I'll l'Iuflkif' lltiy large lag-value. con:,ists of damped t'xPOII('lltinl

a nd harmonic fu nctions.

notatioll "integrated" ill DPiinitiou 5.21 is due to the fact t hat ~d } 'f if, and Duly if, YI = SdWj This !>hows thl'll Y -L-i~ obtaim'd I of tbe stationary nuc! invcrtible ARMA(p,q) procE$S W" 8 !>pedal cusc o( ti~(' A 111. \ 1A process, \vc ha\', the IMA (d,q) pmcess ~Iovin g A\'crage) , which is f\. proCl'SS in llle form

5.6

Non-stationary models
(5. 120)
AR/(p.d) process ( Intcgrnted Auto n cgrvssive) which is a pro<:ess in

In practical applications, one often has to deal with datu which caunot 11(> d~ril.)('d by a stationary pr~, e.g., due to long-term trends, periodic (r(,lllis. or a more general tilllC'-varying behavior.

(5.121)

5.6.1

The ARIMA process

Ix- :-.howll that the k'th orciN t'x ponentin! smoothing (sec Chapter 3)
pn'(lietioll if the underlyi ng model follows an l ~ IA {k,k)
(5. 122)

If a time series has a long-term trend, then this can be rcmonx\ hy 1\ suitahle difftrt'lIdllg of 1I1(' sNies. The differen\d series will then oftel1 be ad<'<llLuh'ly descri bed by a. stationary A IU.IA (p, q) process. lI ereby, a simple mod('l for 1l011-stutiollru'Y ph('110111ellft (,lin he achieved. an
D EI'I NITION I

AlU ~IA(p,d,q)

Pl'ocpss (d;:: 1), it holds that the sum of the

7T

Thl' pro('e>s.<; {Yd is ('ulled an integrated (or suulInarized ) a1J.to"('91't.~,~il't' mot>ing aV'Tay('. o r in short an ARIMA(p,d,q) 7n'OCc.~s, if it. ('lin 1)(> wriW'1l in the> form ( B)'J' \, ~ O (B)", (d EN). where {f"d is whiw noise. (z I) is a polynomial OfOrdN I), and 0(: polynomial of o rder q, both having all roots insidc the unit circle. It follows frOIl1 the definition t hat the proce5!;

.(1) ~

I>, ~ 1 + I >, ~ 0, , ,
, _0

oc

(5.123)

,..Alm,IA.I". d,q ) pr()('('ss ('ftll 1 )(' wriU('n o n the inverse form

is u !;tatiol1ury Ilnd il1V1'rtihl(' ARMA(p.q) process. If we intr()(luc(' a new polynomial cp( B) by
~( B ) ~

(5. 124)

(5. 123) that Vt. for d

I,

CIIII

lx' writl('u as
(5. 125 )

( B)'J"

( B)( I

0)"

the tra1li/fer junction from tiT(> white noise input proccs.'l to the Olltput {l'i} fo r the ARIr..IA process call be writtcn
/I (z)

P"''''''

\ ',

0(, ') ",,(Z-l)

0(, ')

lI1ean from !la.'ll obs<'natiOlls. It should he stressed. that the non-stationary process defined by

'V,}", - ",
wh('r(' it follow" Ihftt til(' IraJlsft.r flll1('tion ha.<; d I)()I('!-; in z inside til(' lin it cird('.
lUui ')

(5.126)

132

SToel l ASTle pnocESS ~:s

Ol"-S r,\'nO~AnY MODEL..'i

133

5.6.2

Seasonal models

1U1,lyiin.' (5.131) by Y,

(k:::: 12) und calc1Llating E'XIW<'ted values

.}'t

Box and J(>nkins ( 1970/1976), who han' been the main contributors to making the ARH IA illodds I>opular, have propo:;cd a clnss of models for de;tri biug a lime series with periodical behavior. For a time series wi th the seasonal period .', the idea is to let YI depend on Y,_" Y,-28' ctc.

E[l',)j.k[- <fJE[y, n )', .d- E[E",Yt kJ+BE[CI

O(k) = <I>,(k - 12), (k ~ 12) .

kJ

(p, d, q)x ( P, D , Q), SEASONA L MODEL) The prO<'('Ss {Yi.} is said to follow a TIIuitiplicntit'e (p, d, q) x (P, D , Q), smMrlai
DEFINITION 5.22 (MULTIPLICA1'lV E

..,.n'". (5. 1:11) in MA form

_.8 12 ),1(1 +Ollk, + .BL:! + ~ 2 0 21 + .. ) + (O D + 04lB 13 + O~2 n2tl + '"

model if (B) <1> (B')'V''V~ l', = 9(8)9(B')o,

cr.

wlJ(>re {r} is white lIoise, f/J and 0 arc polynomials of order p and q, r(>SP<'<'lively, lind $ and 9 are polynomial!; of order P and Q, which ha(' a llthf' roots inside the uuit circle. The seasonal difference op{'rator is denoted V " and 4) a nd 9 are pol.HX)o mials in B",

CavIl",. F, ~JrJ

.,eo-."[[(e t

t lEt

12+ $2.of24

+ }

+ (lh:,. I + O l.o, 13 + 0$2':(_ 2') + .. . )],

[(.~t~A

+ 41"1

12+A

+ 1I 2 '1._2 IH +... )

4'(8' ) = 1 + 4>1 0'

9(B') =

+ ... + lp B'P, 1 + 9 1 0' + ... + 9 0 B"'O.

+(0", 1.,, + 04>-'_I3+k+ 0$ 2 .o (_25 k+}l].

When describing tim' series wilh a season of one yenr and memthly' ol"e"cmti",,", ......e havl' " = 12. From Definition 5.22, WE' get that. Vd. cont.ributes with d poles in z :::; 1 whereas V!' = (1 - 8,)0 contributes with D x s poles o n the Ullil cirde are determined by

... ) + (02 + 02 $2 + 02 (1 )4 + ... )}O"; =

1 + 02 ~O";

,., +84> , +..

+,.3 + Ol'~ +,
- 12). (k
~

"l 0",

0 2 1"4>i"00t

"=0(10)=0

Zk

~C;k2ff/",

k-O - , I , ... , ,

I,

}0"2 _

~q2 1 ... ",2 (

since they lire the solution to I - z' = O. The 1)Q]es are, thu:;, unifono" distributed on t he lin it circlt' with the distance in angll' of 21r:/s. For d = D = 0 thl' procl'SS given by (5.127) is stationary, a nd it is ('asy that the autocorrelation fUllction for a pure seasonalmodd (0.0,0) x (P, 0, follows th(' srune pattern as the al ltocorrdatiOIl fun ction for the AIUt-.IA(P. mooel ,i fthelagsillbt,twtX'Il] ,2, ... ,s 1,8+ 1,.,., 28 1,28+ 1,. ,., ('tc., removed. For a (mixed) scasonaimodt'l (p, 0, q) x (P, 0, Q)" the a~I~;'~~~~~= function at lags thal M(' not a multiple of 8 will be determined by th(' AR IMA pflrt. This is illustrated ill t il(' following examplt' .
Example 5.10 (Autocorrelation runction ror a seasonal ",,>dol) Thb (,xl"ullple illllslT!l.tffi how tht' Rlltororr('iRtiOIl un('tioll for a llIodel b ddt'rmillf,(1. W(' will ('QllHid('r a Ulultil)linltin> (0.0, 1) x
!'>t'l\SOnal mode]

12).
bf'('OIll'S

,,"'ion

rwlctioll

0 . 10

P(k)

9/( 1+0' ) 0
94>/( 1 + 9')

1=1
k = 2,3 ..
k = 11

4>p(k

12)

I'

12, 13,1.1 .. "

_ (J = 0.5, we get lht' allt.orolTf'lation fun('lion ~h()"'n in next page, apooentlal df'Cfl'.8lW' in the rorrt'lation p"ttern fTOOI an exponential ~.aae hi dtl(' to the lint order autoregrt'& illl!pGDlDt, AR(I)" .

134

STOCIIASTJC l'flO(,ESSFS

O PTI MAL PIlEDlC-rION OF Sl'O(, IIASTIC PHOCESS~;S

IJ5

I,(k)
1.0

b )ll,,,,,,<[iv,ly. the followillJl; model CW I be used

( B)Y,

B, h (t ) + ... + ,1,h(t) + 9(B)".

(5. 134)

0.8
O.G
0. 1

modf'1 all the panullf'ters nrc ('stimMed s imultaneously. ilowever, it lw nOI('(1 thnt (5.133) And (5. 131) aft' not l'quinl.Il'lIt, becauS(' then 1)..... /,(t - p) should be included in (5. 13'1).

0.2

I
Models with time varying co effi cie nts
many phenomena it is obl:;cr\'ed that the dynamical charocteristics change tn ~udu;.IlSCS---it-lIlight ~ rcasoU',h1c 10 model such a Proccss using

0.0 ;':'--:----'':'::'---:-::_llL--,.,-----.l~. o 6 12 18 24 30 36
Figure 5.8: TIle aulOCO't"rf'iation fundion/or the process (1 - 4>B ''l) l't ( I +OB}I',

~~~~.~ "II;9i~.'c! ~\'hcTl' tile ('O('flicicnts 9j and 0, arc tilllc--varying. ~ a model in the form
(I

5 .6.3

Models with covariat es

+ (t) B)( Y,

I, (t)) ~"

(5. 135)

Some useful v(,n; iOIl of tile models with time-varying meallS ar(' ruo<if'is wlwre, e.g., llll' gf'IlC'l'Ili lincar model from Chapter 3 is uS('d to cxplnin n st.atic influence of soll1e {'oYarint('s or explanatory variables. The modt'i stru('ttlr(' it

( B)(Y, - X;(3 ) ~ 0(8),

WhNt' XI is Ii v('("tor of p ('xplaulltory variables and {3 is the paramc'r V(,\'tor of the gCllcrallill('ar lIlod(,] VI XTf3 + WI> where {1 1'd is 811 ant()("orrdlltt1i zero lIl('un procpss.

5.6.4

Models with time-varying mea n values

olle will ('x l)('('t thnt WI = W2) ('1111 be (,'Stimat<:d by using an it('rativc l'. Io; .. ill maximizing the corresponding likelihood function or minilllil, litllll of sqnar('(l r~idllals . 1 IO.....lv('r. n problem concerning this method ,\>,'OVido mel hods for thc identificatioll of til(' model strueture. \ Ve will tbis problclI l later ""lwl! IIdUPlivc fOf(.'<:lI..'iting models are discussed in II. Examples of identification of models with timt.... varying coefficients "und in ).lndSC'n (1985) I\lId Por:.hoh (1989).

SoIUt'tim('8 it is desirable to dcscrilK' a possible l'arialion of the m(lln IL o;iDlo e.g., linear dNCrininbti(' fUllctions !I(l) .... _It(L). This ('lUI hI' nch i('Hoo:1 by ~timating a mode l ill the form

Illid !-iUI>st,<}llf'lltly modeling the \'nrialiOIlS of (!I'd using all A R)' I A(p.q) cess, iP(D)Wt = O(B)",. Thl' total model becomes

'''''''mo

8P("lio li w(' consicier how to predict future values of the stochastic {l-i} h8.-';('d 011 pa..,t ol)S('n'IHiOlis frOIll the MUll!' proccss. I n a. later Wf' colIsic!t'r tilt' case w]II.'r(' we wish to predict the stochastic proccs''i Oil o ll!;('n'aliOIiS of" ('orn'lilt('(l proCt'Ss tXt}. l lillt we' h a.,(' obS('f\~\tions from tile prOCft>S up to time t, i.e., f}. W{, IlOW wanl to prC(litl tilt value of l~+.I;, and distinguish bclwccu
makc a forcclIs\ uf It fUlUl'{' ob:scrvaliollj ill other words,

(B)[Y, -

,3,1, (I) _

... - B,I,(t)]

O( O)E,.

(5.

In the e:.tinmtion of dj, .... ih till' covariance st ructure of 1\ '/ ha... In 1IM'(1. This can h(' don(' using a weightro lew,t ::Klllllf('t-; method, Il.H d""',i." in C'hapt('r 3, wil('re E is ehost'll so lhnt it is proportional to I ll(' :~~:.:~ !>lructUf(' of II ',. For applkllliolls t his can be done tLsing th(> al~ori\llIll ill &'('Iion :1.1.2 011 page 10. A proclical t'xampll' i:-. giVl'n in !\ lmls..u I"hyr!'god (I9i\S).

wll(,l'e 1'/. A' in time arc among the observations (a time wit h lIIi~sing obl;crvatioll:; ill otil('r words. k < O.
"'~i'",

will ('onsider oilly lite is.'ilIc of prroiction a nd we assulile an' 110 mis"illJl; nl hll'~, i.t'" WI' 1111\'(' til{' following obs('rvnt i oll~
- I .. ).

. . .{.

136
BfL';('(l on t he observations l j. and thws introduce the predictor

STOC' II AS'I' IC p nOCE;SSES

OPTIMAL PREOIC'T ION OF ST OCHAST IC PIIOCESSES

137

} ',-I, .. "

we St'{'k to pr(>{lict YI+A: (k

> 0),

Predict ion in t he AR IMA process

COf1."ider
all

AH II\IA (p.d,f/) proces,<" {Yd , dl;'fill('d by

(B)'l'Y, ~ O(D)",

9,
As
8

1t.1 = g( l',. Yt-l ,"')'

(5. 136)

(5.1 13)

criterion for the opt imlll predictor, we choose to minimize the eXpt.ded g(llt, VI_I, ... ) is chosen SO that it lIlinimiu'!'l tile criterion funct ion
S(ltla,'Cd erTO,.. This implies t hat

{Ed i:-; while lIoise wilh variance, a'l, process cun be formulated in ~'I A-forlll

(5. 1 11) (5.137)

w('

hav(>

Prom Theorem 3.9 on page H , we get that (5.137) is minimized for

(5. 115) (5. 131!)

writing til('
pr~" ill

g(l't, Vi -I, ... ) =

goP'" Yi- L.... ) =

E[Y'+k IYi . Vi I, .J.

inveT$C form (or All-form),

The optimal predictor (or fort.'Cust) for Yt+k becomes

(5. 146)

(5. 139)
i.r., lhe optimal predictor is the conditional mean. If we introduce the predictioll error

thevalUl'S fe, ,8 = l,t - 1, ... } are know n when { }'"s = t,t - i, ... } From (5.11 1) it iJi seen that t he opposite a lso holds. Thus, jf {f"d

fork '5 0

fork>D

(5.147)

optimal prroiction is t he conditionalm('sUl , .....e get from (5. 145) the

.....{' see from (5. 137) and (5. 138) t hat the optimal pr'CdictoJ" mirli7ltiz(.~ tM vmiance of the prediction cn"Or. The dn!.w back of solving (5. 139) is thaI the conditio nal menu is difficult to calculate because it is II C('(.SSfUy to kilO ..... the mu lt idimensional distributions for the process. it is c/lSier.
THEOREM 5. 15

(5. 148)
, .uh""'ti"g (S.I 1 8) from (5.145), we ObUlill thc pr'Cdiction cn'Or,

(5.149)

For a nonnalllJ"OC1!ss, {Yj}, the optimal li'leor p,'Cdicior i. linwr in vations. i.e.,

tilt!

ob .er(5.150)
that {cd is normally dis t ributed, WE' gCt thc following 1'n/f'rval for the VUl1.1l' of YtH

9t +1<11
or

" L /3) Y,.

)=0

(I - 0)(5. 1.,1)

ill tht> ft /'l. qUllmilf' in the standard normal di..,tribution. UEHlally

Proof Follows froUl (5. 139) nnd It p;('Il('ralization of Thoor('1ll 2.8011 pag(' 27.

:::o:r:;t:;h::"::ARI\IA model will not h(' known f'xactly. hut estimatoo TIl(' prediction error wit! th('n be larg{'r and (5.151) i!-.
("olifidf'lIe{' int erval.

138
~

STOCIIASTIC pnOCF.sS ~:~

OPTL\l,.\1, PREDK'TION OF STOClI\ST I(' PI{O(,ESSES

13D

Remark 5.5

In praclice k is 1;lIlfLll, and hellce, only a small number of t he l,;' weights flr(> needed. The needed V weights lire often most el.L'>ily ('alculated by "sending a unit pulse t hro ugh the system," 0..') illustrated in Example 4.3 011 pagr 72. ..

\ \'e now assume that olle timE' unit has pass<.'(l alld we get a Dew observation Yt + l We waUL to ('xamim.' how the prediction can be updated (&lId illlpro\"('(l). From (5. 145) and (5.147) we ha\'e

.....------+---+----+--+---+----+--+--_+_~ time
l+l

t+2 t+3 1+4 t+S t+6

1)f'{)(:e,~s.

91+1.:11+1

= l/ik If"t+1 + 'l/Jkf"t +

~'''+IEI

1 + ... = -Wk If"t+l

+ ~+k",

Tht f!'aluation of tile prY'riicli(JlI!I of all lMA(;t)

This formuln is used to ulxi.att' Ihe p,"Cdictio1l, (It i1; seen t ha.t whell ~'II becomes avnilable, then f"t+1 can be calculated.) In pmcti('(' the prediction (for('Ca.5t) is most eo..')ily obtai ned by applying lh(' conditional menn 011 til(' process written 0.... a difference equation

Yr tlll

Ell 'd l ',.l! I .. I E!et+d Y,.}i 1, . 1

J

Y, + ~ I Y/ 1 + ... + ~p+d }~t ,.

= C/ + Olf"t_l

+ ... +

Oqf"t_q

where the coefneiellls ~I, ... ,~p+d arc gi\'en from the ideHlity: ~(B) = ,,( D)V OII , sec (5.118) 011 page 130. If we wanl to calculate the k-stcp prediction, we write
Yi+k+~I}j+k
1

OlEIc! }~.l; 1, ... I+OlEIf"I_dY,, }~ " .. . ] + 0;1 Elcl <li l t, }, .[ .... ] )', +O+OICI +Ol!t 1 +03f"I_Z

+"'+~p+dY,+k p d

= E'+k
The forecast <'qulltion becomcs

+ O,ct+k_1 + ... + Oqf"t+k_q

Y ltH =
Y/+:il

f,t'lt +OZE:, +O:l.ot __ 1

l~'k lt = YI~311'
P
...

l'it21t+03Ct
k 2': 4.

Y;+kII=-Ir'IE{l'i+k-dYt.l'i ], ... ] - ... !/Ip+d E[Yi+k-p_dIYt, Y,"

I. ..

J
Yf

~:::.~o;:r,,:th(' ()1U'-M('p ]In'(iktion.'i is

+ E[f"I+kil'i, Yi.- I, ... j + 0 1 E{Et+k_ dY" + Oq E{f"t+k_qIY" i-" .. . J.

In the calculat ion of (5. 155)
w('

1,1 +

o? = 0;2. The Varill ll('{' of Cllll e&iiy lit' calculated nflN writing the pr()C('!o;s in + ( I +0 1 +O:l)Et
l

l;

~t (I +Odol ]

('xpioit that
j=0,I,2, ...

+ (1

+ 0, + O:l + (3 )f"t-3
+0 1 +02 +03le" __1 + ...

+ (1
)=0, 1,2, .. ,

E[Yr-jlYt,Yi-l, ... ]=l'i-j

E[Yt+jlYt'}~'-I,I=Yr+j:' j= 1.2 ...

E{f"t_jIYi,Yt
], ...

+(1 tOI+0:l+(3)Ef_i+""

j=E:t_j

E[c'+jIYt,l'i " ... ) - 0

j = 1.2, ...

which follows from the fact that {t" 3 = t, t - J, . } is known when t.t-I, ... } isknowll.

p ', .

Example 5,11 (Prediction in an IMA(l,3) process) W(' haH' giW'UIUI l ~ fA ( I,:I) pron'SS: Y, . }-; I
= , -+ 8t!,_1 + 8 2 (1 2

cr: = ( 1 .. (I +Od'J + ( I t OI +O:d'J " t ( k 3)(1 f-fJ O + 03 ),)(1;

1 .. 2

a:

a~ - (I
= (I

+ (1 + 0 1 )Z)(1; + ( I + OJ rot + ( J +0, +(2)2)0;

for
~. ~

:1.

variann;' of til(' jrt'(lif"iiulIl'rror gO!'!> to infinity wllt'll k go....

+ 83Ct-3.

140

STO('IIASTI(' PROCESSES

PROBLEMS

141

Previously we have 8(.'('11 thnt the 8utocovariallce fUllction fulfills II. hOmog<'ll00llS difference equation determined by the autogrcssivc part of th' model sec (5 .99) 011 page 126. Exactly the same holds for the predictor
Yr+kll'

basis of the process III Qut'Stioll I.

1-'1 = VX t

Ii

n('w process {V,} is

tbat {Yd is an MA( I) proccs.... ls {Y,} ~tationary'!

THEOREM 5.16 (DIHt:;RENCE EQUAT ION FOR OPTI/IIAL PHED[C'rOH)

3 Find the autocorrelation fUllction for {Y,}.

It.

F01' the A RJ AlA (p, d, q) lJ1'OCeSS, the 01)ii7llal predictor fulfills the homogencola
liifJerrnce equation
(5.156)

for k > q.
P roof Assume that. we hav(' obti('rvations ulltil t.ime I, writC' the A RJ MA process for YiH: and take the expectations conditional 011 obscrvlltions untU time t. Doing this the l\1A pllrt. of the modc\ will vanish for k > q. This is illustrated in the following example: E x ample 5 .12 (Predic tio n in the ARlMA (O, d, q) process) th' AHU IA(O, (l, q) proc{'!..'l

5.2 two-state ~ l lIrkov chailllllodel for a stochastic process {Y(t)} where art' denoted 1 and I. The number of transitions is determined by pr()("es.<; with Ii mpan event rate ~.

Show that the process is stationary with the ftutocorreiatioll fllIlC-

p(r) = c- 2 .\1 .. '

2 Show that. the spC<:tral densit.y is

1 4A f(w) = 21T 4,\lI + wll

aud P'i} be IlIl1tlllt.lly independent alld stationary stochastic prOCCSS(!!; 8pt'('tra l dC'nsitics fz(w) and fll(w),
1 Show that thc prOCf"S.'> {Zt} givclI by

('()II~idcr

For k > q wc obtai n the hOIllQg(noou. ... d iffl'rcllct' ('(Illation fur tilt' pn-<ii,""

Z, =Xt+ Yj
to&J"ru...y pro('(":'<S with the spC<:tral density

/,(w) = t.(w) + !,(w)

T Ill' rharactC'cistic equation for the diff('fcllcC equatioll hm; in 0 11('. Thi.<; IIlea llS that till' gt'llt'ral solution is
I d- 1'"
I . d- I
1\

Iidou ble

2 Consider now til' process {Zt} given by

Z, = X t
PCO<'I'!o.~

+ Yt

wht"fe til(' supersdpt t on till' ("O('flid('Jlls indicat{"S thai 1It(' I>"'CI i""I,,, "olld! i<; found using th(' information Mt Yt , } i .1 ... al time I. T hat is. t I,,> p,red" is /I polynomial of dt'gr('(' d I.

1+ 1\"1. (lal < I ). {Yd and {Wd are mutually independent eoch with 111('311 and varianet' Oll.

3 Show thal tlw sp<'Ctral densi ty of {Z,} is giwn by

f~(w) = 0'22 + a

211" I + a 2

2(\ ("os(w) 20006(W)

5.8

Problems
be whitt, Iloi&' wit h the mean value IJ, and the variance (1~.

Exercise 5. 1 Question 1 Show that the !;tocha.'llic pr()C(fu {Xd, defiU('d by

1Jroo""" {Xtl

dt'finNI by

'"ando,,,, w/llk. &.'1

wlwrt, (. is a constl!.Jl{ ami {E,) is w hitt' noiS<", i.s nOIl-stntionruy

XI

I,

and consi(i<'r the fo llowing.set. of tim('

112

STOCI IASt Ie.: PItOCESSE'i

PRODl.D1S

143

Question 1 Calculate till' mcnu \'1\111(' E!X t ], the varialJCt' VarIX,], and thf'
COVlil'i"lwf' CoV[Xtl' Xt~J.

3 US(' .., = 1 in (5. 160), i.e. , Y, = 'V XI' Ilud (ktermiue for which of 1I {he \1lriallC(' of }'t will be Ie&; thall the variance of X,.

Question 2 Is {X,} a s tationary process? Exercise 5 .5 Question 1 Rewrite 1I1f' prO<.."(>S.'i {Xt} defined by

5.B
(he following ARl\IA (2, I) process ( 1 - 1.27B + 0.81 B 2 )X t = (I - 0.3 B )~t

such that tXt} is ('xdusivt'ly dt~rib('d by past values of fel}' If {e,} is wbite noise, Xl is said to be (>xprCS-'.C<i in an l\ IA form. Question 2 To which category of ARI ~ I A(p,d.q) dOE'S ('nch of t he followillg proc~ belong? ArC' they stationary/invertible?

1 Ch{'('k if lht, process is stationary.

2 Sk('tch lhp impu\M' a'Sponse function for rhe transfer from for k ~ to).

C.I

to

3 O('t(>rminc the corresponding freq ucncy response function , ami thE' amplitud(' fumtioll.

;jAt2+ C I

''"

Conside>r the seasonal difference

Exercise 5.6 Lct tile following AR..\IA(1.1 ) process be given by

4 and find the> impulse-, stcp.. , and fr('(!uency-response functions. and sketch thc amplitude fUllclion.

(I Wil h E[etl
= 0 and

B ) X,~ ( I - OB )',

2 COIL';ide>r the> summa t ion

VArlEt] =

(1;.
impub.(" step G'(w ).
IUld

Question 1 Rewrite the process into all AR forlll. Question 2 Rewrite the
procc~"

frequency-respoubC fUIl<:t ions.

int.o an t-,lA form and find all e>xprf'~."ioll

Var[Xd
Exercise 5.7 L<'t {Xa} be a stationary pro('(~,; wit h the Con~ider a new process P',} defined by
autocO\1irillll('('

Expollcntial smoothing is given by

)t

oX, +o-(1
- (I

fUliction 1 ~

n)Y,

o)Xt _ 1 +o(I-O)2 X t 2+'" +aXt

0-

thE' 1l.Ulplitud(' function and sketch it for

= 0.1

4 BftH('
Question 1 Iktcrmi nc t he !\uto('OVurillll('C fUlletioll of {l',} IL '; n function

0 11 ) 't fI.

n('w time series Zt is dt'fincd as

Zt = X t

Yt

,x(k).
Question 2 In the following. it is g iven t hat

flInplitudl' fUllrl ion related to the lrausfer fro m X t to Zt 1.\nd sketrh

0.1.

X,

nX, 1+ >::"

101 < I

(5.

proc~~

{X,}
(I

(\('fillOO by

wlwrl' { ~d is whitt' Iloi>;('. nttermin(' til(' aul()('()\"fLrian("(' function of

B + 0.5 B 2 )Xt = (1

..j...

0.5 D)e-,
1.

Pi}.

is whit(, lIoisc with E[t,] = 0 and Vllrid

144
Question 1 Examine whether {Xd is stationary/ invertible. Charocterile the process. Question 2 Determine the autocovflrinnce a nd autoC'orrdnlion fUllctions for {Xd. Sketch the autocorrdation function for lags :$; 6. Question 3 Skctch the pnrtialRlitocorrelation fu nction for lags :$; 4. E xercise 5 .11 COllsidcr the class of multiplicativc (p,d,q) X (P, D, Q). M!llsollal modcls:

CHAPTER 6

estimation, and model checking

0(D)4'(B')I7'I7:' X, ~ 9(D)8(8')',
wheT(' {Ed is white Iloil;('. Question 1 Sketch the autocorrelation function for thl'
i;('ll.'>OIHti

modd:
5. we intrO(\lI('('(1 n "ariety of stocha....,tic models which are often dH,c'ril)(' :;todlllstic phellomena of dynamic Sy~tCIIIS, (waler levels, lIIail etock iudice'S, Hir t('mpcratures, etc.). In this chapter, met hods for suitable lI10dcb for a given phcnollll'na bll.'.,ed on obsf'I"vt'<1 'i\dll tx- dcscril)('d. The main part of theM' methods- especially the for ~ 'St illlll lioll aT(' dir('('tiy relatcd to linear :;tochastic proc~. modf'>[ bu.ilding procedure can be divided into the three stages us in FiJo; lIrl' 6. I. The 1II0dei identification is lmscd on data from one or .""ri('S that r<'prC'M'nt. observations of the stochastic process which

Quest ion 2 Sketch tire alltO{'on dation funct ion for thc sc/lOOnal model:

!aID;,,"

all

Quest ion 3 Sketch the autocorrelation ftUlctioll fo r the seasonal model:

(sJ)('(ifyi ng t he model physical insight

,
(of t ht:' model rmramet(>n;)

Applkations u ..,i ng t he model (Prf'di<'tioll. ~imuln l ion , etc.)

14G

I OJ:;NTI FICATION, ESTIMATION , AND MODEL (,IIE('K ING

EsTIMr\TlON OF rOVAIHANCE ANt) COHHi-: L. ATION FUNGI IONS

1']7

is being modelled. The primary tool for identification of the pro<:l'S.-; is the e,1 )(>{'illlly for the pr~ containing periodical component:; the ~timl\t('(1 SjX'<'trlJrn is an important supplementar,Y tool. The method for estimation of SI)('('t rlt is dii>('tL';s('(IIHtl'f ill a dUilltcr 011 spectral analysis. &tillllUion of lil(' romria.n<'e and correlation is treated in Section 6.2. Tills section also d(':!i('ril~ ho1ft' considt'rations of tht' rorrelatioll functions can he used to determine """" ..1 .,10: starti ng vulues for I), d, and q in nn AR L \lA(V.d,q) prOCC&'i. As illdicat~'(1 ita Figure 6.1, the identification process is usually an iterative process, nud til(' proc'('(lurc ha.'i to lx' r('pt'lltro :-;cveral times before an ad('(}uf\tc mo< l ~'1 ill found. Scvera.! tools for idcntification are therefore closely r(>Iated to model dl('('king, ftnd thl'S(' tooL.. nre thus treated in the S('Ction on model , ,,,,.,k;"1 When the model structure is determined ill Stage I, the model paTflineten; be esti mated. The Illethod of estimating the parameten; in the AlUlA r is treated in 5('(tioll 6.4. Finally, some tools for model srlf'('tion and checking are dcscribC'd in Sections 6.5 and 6.6, respectively.
t'Slinllul'd ('OvariuuC<' o r ('orrelalioll functions. For lllallY pr()C(.'S.";('S

E[c(kJl = N

:V

Ikl L , , O(k)

(Ikl)
I

O(k).

(6.2)

that the estimator (6. 1) is (l bia.<i estimator for l(k), bllt it is that the ~timator for a fix('(1 k b ("O/ui.~tcnt (since E[C(k)] ..... ')'( k )

'" ).it might bE> t{'mpting to apply the l':Stilllator

C'(k) = N

~ Ikl , L, p ', -

.\" Ikl

Y )(Y,+ lkl -1').

(6.3)

.=:; itt

II unhia.'>t,,<i whell we agnin (as no fim! or(\('r approximAtion) llf'giffJ from Jt bt'ing t'!.limatcd by Y. Howcver. Var[C(k)] = 0(1 / N) 1981 , pp. 150 and 32t1), Witt'rCHS Var[C"(k)] = 0(1 / (N -Ikl),
P~tillllltOl' C(k}, in contrast to C(I.:), is a non-negative definite (!If'(' ThrorclIl 5.3 on page 10-\) just IlS lhe theoretical llutocoVaI' iallc('

6.2

Estimation of covariance and correlation functions

Estimation of nutocovnriancc and autocorrelation function, the !>MUII] (:orrelatiOI1 function, Ilnd til(' cross-covariance and cross-correlation r""";011 is described. We will 8SSUIll(> that we have observed a realization of a stochfl.'itic {Yd, i.e.. t llllt \V(' IlUve a time series Y I Y2 , ... , Y,\'. Furthermore. Wt' assume that the underlying process is ergodic (and hereby can b ,;:~~: !';tationary), so that til(> ensemble mean and the meall over time are (' d. Sectio n 5.2 Oil page 97. This is the assumption that ensllr('t! that ;nr, ...,~ of the stochastic process can bE> based on a si ngle time series, which is th<realization available of tht, stochastic process.

~ V

L
fl, . ""

(,.(mh(m + t')

+ ')'(m + r + t'h(m -

r).

(6.1)

6.2.1

Autocovariance and autocorrelation functions

First we consider the autocovnriancc func t ion. As an estimator for the vaf'iance function, we apply

....."',1,.. ion of the estimates implies that there call OCClIT systematic in an f'Stimatcd llutocovllTiall('{' fuuction. which. cspccially at larg(' DOt n(,(,('S.'l<l.rily im ply that the observations corresponding to these Often one will .'i('(' n pattern at large lags which is also low law.-. a nd I his is only due (0 the fnct that the e.timates of . . Me correlatC'd within thelll;;(']v('S. T his fact is s ketched ill Ih(' followin g pngl\ which sho\\':; the theoretkal and estimated . fun clions for an AR(I ) process and an AIUtA (2. 2) process, Whl'n.v 500. \\'f' have as It s]}('('illl case tltt variance of the estimator for the

C",(k) ~ C(k) ~ N

1" Lt_'1

N-Ikl

(Y, - Y)(Y,+ ,kl N

2"" V",[C(O)I" N
k

L
-00

O'(k)

~~

21~
k

L
:lO

p'(k).

(6.5)

for jkj 0, I, .... N - I. F'urtlwrmore, Y = (L:I-..-I ytl/N. In (19SI), u detailc'd treo.tm('llt of the propert ies of the abo\'e estimator autO<'Owlriall{'(' fu nction is givell. If we Ilegleet that I' is t~tillll\l('d hy

1'{O) and p(k) is lluto(."orrdaliOll fHlletiOiI for the process. AR(I ) Jlroc('.'i~ with Ihe co('fficienl w(' have p(k) = .j)lk . In this from (6.5) Ihat
1 , [1

(/)1

~'l.

(6. 6)

1-18

I DENTIFI CATION, ESTlr-.IATION, AND r-.l0DEL CH E:C KI~G

EsTI~IAT10;>l

OF COVA IUANC AND CORREI.ATION FUNCTIONS

149

1.5

(41 dOtiC to 1), lhell the variancc of the process is not determined wl'll,

iIr relatively large values of l\'.

1.0 time series contain nOIl -stalionariti~ or very ~ Iow \'I!.rialions compared ~Ilpling frequency (for an AR(I) this corresponds to dJ close to 1), NCODlIllendnble to r(,lIlove tlu.."SC trends by applying a suitable filter. e.g.,

0.5
0.0 +-----'~...,.,.....,c.:.:.-.:..:-~~-'-" -:.:.-~--~~"r-->

(6.9)

10

15

20

25

30

Lag
(a) An AR(l) process

[. 1.1]. Wi' hav(' uscd a =- 1 (Box IUld Jenkins 1970/ 1976), which
. .oo.d> to a sillgle differencing.

.,.rrl<,/i,,,, junctiorl as
0,1., , ., N

till'

e.~limat('d

atdocovariancc junction we find the estimated

1.5

ii(k) ~
1.0

r, ~ C(k) C(O)'

(6.10)

1. In practical si t uations one will ilL most calculate the

0.5 0.0 0.5

""o)"l;''''1 lip

"

"

"

, , ," ,"
"

to IlIg k = N/4. Both flotations uSNI in (6.10) fo r thc of autocorr('lnlion func t ion at lag k a.re oftell sccn in the literatUl'e. propcrtit.,., fo r p(k) nrc largely the same 8S we IlIwc discuSSt'd for the of C(k), and especially there can occu r a high correlation between t'alU('$ of tile cJltimated autOCOnl!iatiOTI junction. The remarks on JIIo.....J Ruto("()\'ariance fu nction related to Figure 6.2 Clln, thcrdore, be
for til(' autO<'Orreiation fUTlction.

10

15

20

25

30

Lag
(b) All A RMA(2. 2) proces"

N -~) both the est imates of the flutocO\'ariance and ::~~::::~;,,(for will be normally diM ri b ute(\: but the mean valuc ruld the
in g('ut'ral d ('l){'ud!'llt on the t heoretical autocovarial1l::e function in prru-lic(' impossible to cluculate. Albeit, if {Vi} is white noise we follOWin g.
6.1 (P IWI'E In'II::S I N CASE OF WlilTE r.;OISE)
noU(- tht melJn mlue and varianct an'

Figure 6.2:

Theoretical (solid linc) alld eatimatea (da.,hrd lim;) au'oco,,'",~

/lIl1rtiOIllt for an A R( I) pmcp.", fa) lind an ARMA(2, 2) pr'O('PJu {b}.

This result can be compared with the \'ariance of th(' C'StiUla.lt't1 variallt:(' N independent normally distributed obSt'rvations, wher('

E[P(k)j '" 0 , Va<jp(k) j '"

I'

0,

(6. 11 )

20- 4 VIlr 1 S I ~ N'

:2

~.

k I'

o.

(6. 12)

If we compare this with (6,6) we sec that the cquiL'lIlf'111 rmmlH.,. oj "'"!'p'.ntI~ ob.~(>nlati01ls for the AR( 1) procl'!>.'i is

baaP.:1 0 11 tll(' Il.-'iymptoticai normality and IIIlder thl' hypoth{'!o;iil

0111'

" ~ .v[~l 1 + r9 .

w(> al;,o ~ that if tIl(' ollt>('rwl ions in tl1l' A R( 1) prO('(~~ IIr(' ~lronp;l)'

Wlwu

("t\lIlllak(' an (approrimatil'r) 91)% ('Qnfiden~ illlcnJal liS jud1(i llg <..('\,('ra] lag \'U.luCl'l on(' has to bear in mi nd that

150

IDE~TIF'ICAT I O:-l, ESTIMATION, AND MODEL ('m;C K]~O

"'I I ~IATION OF COVARIANCE AND COIlRELATION FUNCTIONS

151

('ven if the p r()C('SS is white noise, about 5% o f the \'al liC'S will h(' out;;idl' t hia confidenc!> inter val. Tile estimates in the partial autocorrelation /u1I('iion arc d{,tf'rmin('(i UII'<;('<i on thl' estimated autocorrelat ion function ill the exact similar manlier ~ for

0.6
0.1

the t hrorctical functions; sec Secl ion 5.5.2

011

page 119 or ApPl'ndix B.

"

I
" " "

6.2.2

Cross-cova riance a nd cross-correla tion fun c tions

\\'e a ...,.,lIlll(' that we observe two timt' S('rics YI \ ... , }r,,_ and X \, .. " Xl\'. estimatOl" jor lhl' cross-covatiancf' function we lL<;('

C,n(k) ~ N L (X, - X)(l', - Y)

1=1

1N-k ",

"

" "

"

.
-0.-1
- 0.6

I.

" " ,

"

Cxd-k)

~~

N-k

L (X,.. 1=1

X)(Y, - Y)
-

for k O.l, .... N - 1, and whefP X = (2:1' I Xtl /N W10 Y (E, I The estimator (6. 13) ih bia&'d for fixed valucs of N but consistent. As for the autocovariancc fUII{'tion, ncighborillg valucs ill an "'';"" a"'' cross-co\'arianC(' function lIlar be strongly correlated. F\lrlhrrmorc, the con"Claliou in cadi of the two time ~eries. {X t} (Lnd {Yt }, may inimdlliZ JHdtern in lhe estimated CT(M~'COtJ{Jriana JUTlctiOU., evcn t hough t h e ' cr08s. . rovll.rinncc fUlldioll is 0 for Illl lab'S' For two such mutually' processes. it hold!> that
-" N

IV

1.3: 1'.\timaUd Cf"OSs-<",orTl'tatio" octween tu/O mutually utlco,..,-elated AR(!) . BlimY' (da.~hcd linc) (Llld aftrr filtt'f'i1l9 (solid line).

Cxx(O) and C} d O) arc the l'SlimatOO variances of X t a nd }'" re;I>('C'" As 8 It'!>! for zero cross-correlation, we haw' the following tht'Orem.

Cov[CX\'(k), Cxdf)[" N

1"x.dmhn{m + f
0

~::::~;.~",~cro~ ,~,~",corT"elation bettl!e1l

k).
Hnd fl, i,e.,

III

two mutually uncOrTf!latcd nOn/wHy di.~tributcd hcnJiny

time series

If {X(t)} and {Y(t)} A.r(> AIl( I ) proceb..-;('S with codlkicllts

E[iiw(k)1

0'

(6. 18)

,",x(k)
we

g('t from (6.14) for k = l

.01

V"'[iixy(k)I"

~,

(6.19)

'!hOi at Ica.qt one of the l1t"QCe.qSes is ulltite ?loise. If both tlie 7"ocesscs noMe thn. U!C haL'c (6.20)

This im plies that IIll fulloco rrclatiOIl in (,I\('h of [hc two till\(' l;CriC'S can a larg(' cr~~ . . eo"nrianc(' bClw'e1i tht' two timC' !.t'rips which af(' 1I0t ("()rrt'intM. 13ased 011 lhl' <'Sl imMed ('r<h'!-('ovarianc(' fUllclion, til(' functioll is (,~tililatcd by

Fo llows frolll (6.11).

.;c; , (OJ (',,.{OJ .

C'n(k)

....Jble m with autocorrelalOO lillie scrie; is also S('{'11 in th(> crOl->..';funct ion. T his is ilIustmtPd ill Figure G.3. 8&,*,<1 on two Illlttwl.lly (no ('fos,<; . . corrdlltion) AH(l) prof'('SS('S with ct = d = 0.9 Il.lld the eslimal<'(1 eross-correlation function is ill w;l mte<i by t il(' dasll('d 6.3. WE' oh!;erw that Iht' shifting positivC' and lIl'g1l..tiw valUC!i of

152

I DENTI FI CATION, EST IMAT ION, AND MODEL CII t;{' KI!liQ

153

the nutocorrdtltiOIl functions (see (G. 15 are also observl.'o ill the csti llllllE'd cross-correlation fUllc t iOl I. If Oil{' wishCll to test if two time smes are cross-corrclutf'd. olle should H:Ie till' fo llowing procedure.

",,,,,;all) t he choice of differencing based on the nntocorrelfllio ll fUllctiolis 1~1. which is especially dul' to Box and Jenkins (1970/1976).

Identification of the degree of diffe rencing

f).5 .a
Oil

Test for cross-correlation

page 125 we considered AITh IA(I,.q) pron:'S.';es, wfilll'lI as

= 8(B )I' If t he pro('('SS i!i stlltionary. the> cOfT(>5ponding theoretical

i)
Det('flll ill(' II

model IfJ(B)X t {etl is white 1I0iM'.

:::z

O( I3 )t for the {Xt} time scrips!io thai.

{X,} gi ven by

"",,'.00" fundion satisfies t he fo llowing differcll(;{' equation (see (5. 100) 126l1l1d (5.10 1) on page 127):

ii) Based Oil the lIIodd rwrform a "prr-whit('ning" seqU("n('(" of residuals Ct = 0 I (B)t;!)(B)X t

all

( B)p(k)

0,

q+ I.q+2, ..

(6.21)

iii) Per for m t he similflf "prc-whitl'ning" on {Yd givell by U', ~ 0 '(B)( B)Y,. iv) Test whet her the (Tos.<;-corrdatioll P~ll,( k) = O.

roolS fo r the chumctcri.'St it' ('<!Imtioll ure rliffcr(>llt , the solution to c:ao lx' wriuell ill t he form

. .. + ApA!,

k = q - p+ l,q

p+2,.

(6.22)

If we Com par(' the proposed pro<'('(lure with til(' v8rian('(' (6. 16) for Art(l) model, it is seen thnt t he proc('(iurc illlp1i('S that op b('COIIJ(..'S 0, Ilcr('by we obtain a n undisturbed est imatc of t he vuriunc('. Tilp solid linc in Figun' 6.3 shows the estimaH'f1 cross-correlatioll ht't the r('Siduais of both time S('rics. and we observe an ('!;ti mat(>(\ t",,~,co,,, II",i" funClion that corresponds 1Il0rp to t he true o ne, IlIuuely, Ihnt {X I} a nd are mutually ulleorrelat(.'(l.

daim for stllt ioll/lrity ell.'SUrf'$ that AI, ... ,Ap all lie within the unit I also be noted that if one of the roots is dOH(' to the unit circle, I will o nly very slowly decrease toward zero. Similarly, 1It~ 'n in the lru;t i'>cctiOIl, n root clos(' to the unit circle implies t hat of the prQIT;.S for relatively large values of N is badly d cterlllillt'C:1. lIf'Til'S {l't, t I. 2, .... N} contains 1I0n-:-,tationarities or very slow compared with lh(' sampling fr('((uelley, the est imated a utocorrdation wry slowly to zero. III Box lIud .Jenkins ( 1D70/ UJ76) it is t hat .'Such 1\11 a pparently no n-stationarity s ho uld be removed ~<aI'I,' order of d ifferencing
(6.23)

6.3

Identification
......<ly d('!o('ribing {H't} by lUi AR\ IA(P.q) model. lIellce, we model ARl MA (I'. d , q) model. th(" CtbOlle (,o'lSid ~mtio1/.~, we select d as lhe lowest arTier' of uhicll till' ('stirnated autocorrelation dCl.'Tf'..a3f!s sufficiently mvid practkal situations tl~is correspond!'; to d ('(I lial to 0, I or 2. sitnations it will not be udequalc to perform a differencing. This C8s(' if, e.g. , from phYl'i i('al cons ide mtio lls, Q IH' {'an a&>tllne t hat Ie stationary and tbe model b intended for ~i lllulation . iustend that th('f(' are eno ngh observatio ns availnble to estiml1tf' a ll of the procC'Ss. F urthermo re, Oil(' ('1111 a pply a method which is mal-f('('ordt'(l obSt'rvat ions (~ , e.g., Huber ( 1964) o r f1. 11Il'till ~1'1I8l;)). sill('(' n model with roots dOM> to tIll' unit c ircif' is vpry IJU('h efTOII(,OtlS oiNrvaliolls .

The purpose of tilp identificat.ion of a ll A Rl t.. IA (p.d, q) modd is to d",<"mla the model order, i.c .. to d('tf'rmine \'tlluC'S for p, d, and (I. A" indil'lItt'<i Figure 6. 1 all rmge I 15, the identification is based o n in fo rnmtion frolll in conjunct ion with coll:'sid pratiollS about th(' physics. 1I0 Wt'VN, for phe nomena it lIIay be im l>Ol;:;i ble to a pply a ny information from th(" A!I examples of such phenomena we have !>tock pricN>. mail loads. In such cases the identification is 1)8.'iE'(1 solely o n t il(' info rlllation frolll In cases when' it is possible to a pply physical knowledge, o n(' should ct.'ftni nly not d isregard .'Sueh knowledgf' but combine it with informatioll data. 1 \ preliminary identification of an ARl t-. IA (p.d,q) model b based on t he estima ted nlllocorrpll1tion fun ctio n, wherp 011(' compar('S th ",:::: auto<:orreia tio lls with knowlf'dp;e of t il l:' ('orrt'Spo nding th{'()retical al t ion functions for :;tatiou.o.ry pr()('('S.o;('S (d 0). The preliminary I ; 'I."t;fi~~

151

IDE:-JTIFICATION, ESTIMATION, AND MODE L C IIF(' I<I"

IDE:"'TIFICATION

155

6.3.2

Identification of the ARMA part

Charoclf'ristil'1I for the aulO(;l)rrrlaiion /unctiolU 0/ ARAtA pnxe.sSf.J.

\\'e now assullle that we have ehosen a s uitable d. The next J.,lep in tlip preliminary idplltifieation is to ic!<" II! ify values of p uut! q ill the AIU, IA Part based on thc c:.li lllutcd autocorrelation functions for IV, := 'VdYt, and naluate which combination of p and q C8n yi('\d theoretical autocorr<'iations that tH'! approximate the estimated 811tocorrplatiOll functions. For this evaluation. tbe former deriwd ('haracteristic:; for the Ihooretkal 8utocorrclations fUllctiona !>ho uld be applied. These ciJaract('ristics are sUlilllIlIril,(>(1 in Tablt' 6.1. As a su p plelll('nt to the evaluatioll of t he e;timated autocorrelation func>tions, olle can exploit the following theorem fol' l<'sting the hypoth('sis 01 whether n lime !;('rie; can be described by u pur(' AR(p) process or br !>.IA(P) pr<x't-'ss. .
TIIEOREM 6.3 ( Pn:OPERTY FOil AH PIWCESSES)

ACF
Dam pc'{l ex po ncutial and/or s ine funniolls

PACF H
4JH1=Ofork >p

IACF

pi(k) = 0 for k>p

Damped exponential IlIld/or sine fllnctiolls Damped expOnelland/or si ne functions after lug
t ial

p{k)

0 for k > q

DOllliunted by damped cxponcntinl I\nd/or sine functions Dominated by dumped exponential and/or sine functiolls after Itlg 1) - q

For an AR(]J) p rocess it ho[(is

J)alllprd ('xponelltiallUld/or sine fUllctio lls aftE'r lag q p

p - (/

8 [';;,,[ '" 0 }

Vruc[,,] '" -

k=p+ l ,p+2, ...

Iw nnnua ll.... correlated. Esp('dally for larg(' lags. there can be -::::::~ol';'}''ltt(,TIl in {he estimated autocorrelation after thc thooretintl ~ hu.., faded om, d. thc di:-;cus,.,ioll ill til(' pr('vious sect ion.
R\i\Y

where IV is the nUmb(T of ob$fTt'atiOt1S in th.e stationary time s('rie.~.

Proof Omitted
Sff'

Dalliells (1956).

THEORE~I 6.4 ( PItOPERTY FOR MA PROCESSES)

For an : M A(q ) process it holds

EIF(k)] '" 0
Va,[P{k)] '"

~ [I + 2 (;;'( 1) +

q+ l ,q+ 2.

where N is lhe rlumbt'r of obserL'alion,~ in the sta/iolla1'Y time .~Fli{ \.

Proof Omitted (1916). follows from
801lI('

approximat(' r("Sults due to

$kk, (k > p)

F\lrthermore, it {'lUI be shoWJI that the Cbtilllut('S of 'ji(k) , (k > q) art' Approximately no rlllally dblributt'(l. and thu:-" le>ts for processes ar(' bascd on {he norill al di!.tribution. It is much Ulor(' difli('ult to dC't('rmin(' the m()(it1 ord('r for mix('d In thnt ('a.'R 0 11(' ha.<; to mak(, a "good P;1It~~ for fI, mod('l order. I\lId lIlodd ch{'("k r('v('als that a re-id('lItifklltion is ntXxil,{1. It is imporlnlll to bear in milld that wlw il ('vlItl1aliug ('Stilllatt'8 toC'orrdntioll rWl('1 iOlls. tht' mri;UI('(' of the p;.;tilllllt('S IIlfly b(' larAt

(Sample autocorrelation functions for cloud cove .. ) Ibnulatimls (If cloud COn'I' is sought. In or<i('r to id('ntify th{' order tlw 811hx'orrl'lation I\l1d partial autOC'Orr('\nti()1I ar(' cstimatt'C1. of cloud COV('T txprt ....s the per('('ntage of {he sky whkh In the pn'S('nt study hourly ol)S('rvalions of total oct"" an' used. AetulIlI.\\ til(' cloud covcr i!> mctu;uT<-,(\fL." lUI o aud 9 ..... 11('1'{' 0 cOIT('sponds to (,()lllpl('tl'ly clE'al' sky. 8 to .....".1 sky, and the ol}S('r\'llt ion 9 illdicat~ thllt lil(, cloud cowr which IL.,urul.v is till' ('fL.'>(' ill foggy w('ather. The observations aubjt,(t iVt' ('\-run.lt iun. frona.January 1. 1!)::i9, to Dfft'mlwr :U. 19i3, obsPrwd at art' 1lS\,'(1 in t'StimRtinp; tht' ... mll]llt' autocorrelation Flgun'6A on th(' twx( pngt'. and tllC partial fl.utocorT('IHtioli G.!). Thi.'i ~iY('s a total of 131,.196 hourI.\' \alu('S. amount of data. 'Sting should b(' uSC'd wilh ('are. It is St'('n, ,..~v .. , I 1If'('IIlS to l)(' si~nifieant ly diff('TI'u( fr01l1 I.('ro. A('('()rdinp; it ill ('tmdud('(1 that ill tIl(' cia."" of AIBIA(p.q) pro<."t'S...c". cloud ('U\,('r IIIIL.,t hI' th"'('rilxxl by lhl' AH( I ) pron's...... PI'OCes6 iM. 8b m('utiOIl('<! ill Exam!>I(' :'.1 lUI Im~t' 101, 1\ tin-t Sine'(' th(' sam])lt, SP{l('(' of thl' doud ('(WN obt;('r"'utiulls lII(1dl'l wCluld ht, to lUI(' a fif'lolt orcit'r Markov

""'P''''''p';.,.

156

IDENTIFI CATION, ESTIMATION, AND MODEL CHECKINQ

~11 "' 1 ,\TIO~

OF PARAMETERS I N STANDARD MODELS

1 ,57

t!("OnOllwtrics, like for s tock market index values, the variabk-s often through tin)(' roughly following fl mndom walk. Tcsting for re\tition ~uch index mlues ('ould most. appropriately be cloll(' by test.ing for n
us coll!oidcr a simple example of coi ntcgralioll more d OM'ly. H a time !ihaws a Iilll'ar trend and Zt = ( I - B)X t = \lX j is stationary, thell to he nn integrated process of order one, which Wf' write X I E / ( 1). M'ri~ XI shows a quadratic tr('1I(1 I:lnd Zt = 'Ill Xl is stationary, i. .. !<aid to be inlegrat('(1 of order two, which si milnrly is wrilten as

ci

10

20

30
lag k

10

50

U'

Figure 6.4: Sample autOCOTTelatioTi functiQTI for cloud cover nco!' COJJ('Tlhagm. NoliN" thf uponentiai decay.

o
N

~:~:,~~I mriahlcs ar(' o ft cll introduced to modelulld eventually predict I in the process Xl - If X t is Ull I{l ) process, an additional variabic an 1( 1) process and X , - n Y, is stationary. The process (X t - oY,) 1(0), and X t and Yf arc said to be roilltegmted. If we now want to mo drl for thl.,* two mriahles. instead of first differencing both o f t he and OWIl building a model, the COIll;traint implie'd by the stationary ClOIllbillatiOIi X, - o}', !l{'('(is to be incorporated ill the model. is a means for correctly tc:.ting hypot hf'M'S concerning the 1 )I.'t\\-1'<'n non-stationary processes having unit roots, i.e., integrated 0Dl'. s.~', ('.g., Johall5("11 (2001) and Engle and You ( 1985) for It. further In {'ointcgratioll analysis.

ci

10

20

30
lag k

10

50

(Cointegratio n ) data in Figun' 6.6 Oil t ht> uC'xt page. It is cI('arly SL,,{,11 that s liutlld I)f' ditf(,l'euccd Oll{'(' to obtain stalionarity. This th.., hond rat(', Ill l h o\l~h it is less clear. Uy (:omparillg till' Wt, it iI'I M~ II lhat til(' mOlle.... d('IHlllld dpc rea.'j(>s when the bond and \ it'' \'('rsa. T hlL';. it i~ C'xpectcd that th~ ~writ'S art'

Figure 6.5! Sample partial autocorrclationfuncUonff)T' rioud cover nt'(I!' Co,........ Notice that ;Pu. : : : 0 for k = 2,3, .

Itli...ation of parameters in standard mode ls

IIH'thods ror C'SLimatiou or par/lmet('n! in AR. MA, nnd art' d l'sc-rib<'d. Tht' methods IIIny be interpreted as a geJlf'rd ynamic IiIlf'ar models of tilt> mClhods d('S("ribed in Chapter 3 stal ic li ll{'ar modt'ls.
SO il \{'

chain. S('(', e.g., CrilllllLit Hllri Stirzltkl'r (1992). Ac t uall~' thUi i;u('(~"S.o;fully used. to modt'l the Yllrinl iun in cloud ('()\'pr, ,.('{' \ huLtro, and Thyn'Rod (H)i\.')).

6 .3.3

Cointegration

Cointcgratioll is a propt'rty

or timf' ~'ri(>S mriabl<'!' oft('11 used ill ".,,,,,,,,,,,ut

,I

DI;:F I N ITIO;-'; 6.1 (('OINTECnATIO~)

~:~::::~,:or /lutoC'owlriance

If two or mort' s('ri('S Rr(' t hptllselv('S non-stnliollltry, but fI. Iim'ar c" n,t,in.ac of t il(,1Il i~ !>tationar\'. tl1('11 the St.'ri~ IlW said to be ('ointegrat('(I.

arC' found by s ubstituting {'!';l imatoo va.lu('S o f ftlllctioll into l ht' lilrorl't icru r('intions and til(' \'1I1u~ of the au tocorrelation or nutocovnriltllcC' 8uilt.t'(lut'nU.... , solving thCS(' (>Quat ions with resp<,d 10 t ilt, pl\ralll-

monlf nl

f',~timldl.~

158

I OEKTIFICATION. ESTI MATI ON,

',n~f,'\TIO ;\, O F PAR..-\ M t; T E RS I N STANDA RD

MO DEln'i

159

l.A.'vcls
12.1

Differencl'!!

6.3 (Moment estimates: MA(2) process)

,.,," _, Pf<X'("":-;

1'2.0

0.10
0 .05

we

Obst'fVt'

the following:

J
~

1!-9

p(l) =

rl

= 0.6

and

p(2) :

r2

0.3.

(6_26)

11.8

'"
11.6

1 L.!'>

1975

1985

0,05 '::::;;--'---:+.:: -------:-_

1975

function (M'e E<IIUHion (5.65) all

P(O)

".n

"W
~O_16

0_01

1'( I) 1'(2) 1'(1-)

(I I

+ (h(h
(6.27)

:;;

" 0 .18

~ O. J.I
0.1 2
0.10

(I + Of+ O~) 0, (1 + O? + (I~ ) o. for k ?: 3.

01 + 0102

1975

1975

198<1

a)
b)

1+0f + O~ = rl

Figure 6.6: The u1l1X',-.le/t plot sliolL's ob.~(,nJatior~ 11'0111 1974:1 to 19H7::lof noalmollCJI (m2), 11,/1 ('", the log tm 1ls/orm atiml 11(1.5 be.cr, IIPJJli('d to s/abiliz,' vurianCt'. Tile upvcr-right 'Plot ~h oll!8 til{" liiJJcn:mY:d lJe';e.~. The IoU'fr-[rIL sllQws obllcrt'l1tiol1$ from the sallie time JH'riod of 1M OOn6. rott, arul tilt' plot shall'.' lilt difJNtncet1 st'rif'$.

B,
W('

'",,-noM

to 8\.92 Wlwll Il) b divid<>d by b),

obtl:\in

01 =
T he method is very ad hoc, Mid it is d ifficult to say ~ lIy thing gt!wrnl t he pro pl'fties of t he estima to r. F\lfl ht' rm o[l' , it is pos.<;iblc to find, rt'lll.tions betwccn paralllett'rs a nd \u lliC'S of the nlitocorrt'lMioll fUII,tion; this ca.'>t' D ill' should look for relat iOIl:' wit h t h(' lowest possible IIlR t he a utO('or relatioli functi()ll. ~l oll1 ('nt estimat('S ClUl S('rv(' fL-; ~ood . vlll ll ~ fo r ot her esti matio n met ho ds. W(' hn"l' for lilt" AR(p ) IHYHH.~eJl Yule- Walker estim(lt(',~ by w iving t hc lincar cqmlliolL<;

1'1

02
(12)

1'2(1

b)

Wt' g(,t

th('I'th ordl'r ('quat ion

0_
va\Ut,>; of 1'1 and 1'2 1'.'\' 1I<'C'ept thc solutioll that yields an PI'Ot"",-" for jj.l - O.Ji217 and, Ilt'ft'hy, 81 0_6$83.

Prl)) (ii(~' )
P(2)

fi( I )
I

ii( I)
(

Pr p
2)

Prl'-I)) 2) (-' - <>"> )

I

t hl'

IJlOIl\('lIt l"StimalC'S

are often doubtful fOf non-A n (p )

l>fOCcsses,

ii(p ~ l ) Prp

- ,.
1\

lS estimator for linear dyna mic models

cohHid ('r lh(' AR(p) model (6.28)

with r{'Sl)('('t to 9 1.02, .... 9 p For till' At A ((f) fwd A R M.-1 (p. q ) lJ1"V(C.~.~f,~ lilt' l'quat ion:. 1>e('OIlIt' nOll-lilwur (''Illations, I.<'t us illustrate thl' mf'thod of mOllWUls iu au

uoiS('.

,,'f' n,,,,.,um thal

1'.'"

haw' thl' ohs('rYHtiolL'; VI,

160

I ORN' lltI CATION. EST I MATION , AND MODEL CIIECKI\G

E.iTnIArro ~

OF PA ltAME:TERS I N ST ANDARD MODELS

161

If we introduce

X;

= (- ) ',

I, ...

- y/,p),

t = IJ+1,P +2, ...

OT ~ (" ... ,,) tll('11 (6.28) can be writtclI in till' lin(;(lr form

Similarly to the approach described in Chaplcr 3, we will write the model (6.31 for aU the observations. We inlroduC('
y T = ( Yp-tl, .. " YN ),

X T = (X 1,+I,"" X ",).
T =(e: P+l, ... ,EN).
J-kr('hy the model (6.28) observations:
CUll

bt, wriUen in a compact Jiut'ar

forlll

for all
(6. 10)

Y = X() +E:.
C'omplU"P with (3.33) 011 page 36.

The LS estimator is fouud as in Chapler 3 by minimizing

5(0 )

(Y - XO )T( y - XO ),

'C:ba.~, 3, lilil'l\r in
I
'V ,.

) 't

(6.11) since tht' X matrix ('ontains Y variables.

(nl<'IIH vf\lll(' = 0),

a nd the solution is given in the followin g theorem.

C(k)
o.

til

LS ES rlMATOH FOR J, INEAR DYNAMIC' MODHS) Th(' LS estimate for (J iTt thf 1;'len,' model (6.35) is fOlmd by 8Ql,.jng til(' equation
THEOREM 6.5 ( Til E

L, }~ Y; +k ,

on IN\g(' HiS that the )'ule- WalkfT estimates are determined

and as an estimate of

0-; = VarIE,] we apply

o
_2
L
~--~

(6.42)

5(0 ) N - p

.....

L,."."
N

<, (0 )
p

2 .....

LS 1'3timatn (d>tcrmillt'(l oy (6.10)) nnd th(' }'Ule- Waikt'r are ruymptolically identicaL
is important to

Proof As in Chapter 3.

U'

Example 6.4 (LS estimate!\!: AR(2) process)

II!'.

estimAtor

eou:-;idl'r th(' AR(2) 1110<1(')

L<; flot a linear fUllction of til{' ol)<;CrvatiollS !-oi nC(> th> ('onillin:-; } ' variables.

ii

II not 'A'hit(' f1oiM', ll](>11 8 will oftcu be to th( lrn(.' valUI:' of (J for /\' .. x).

1l01l-("()Il~is'('llt (8 will uot

162

I DENTI FICATION, ESTlMA'IION, AN I) Mum~L

C'liECKU;;,

TIIEOHEM 6.6 ( PROPERT1E.-" FOR TilE LS EST1MATOH FOR DYNAMIC MC'O,. .,

A.~iil.lme

The prediction error m ethod

f.~fimatjon of .~totic models in Chapter 3 is obtained by millilllizilLg of squared re;iduals. Tbe residuals ca.n l}(' writl(,11 as

kd

is nonllally distributed white noise. then

i)
ii) iii)

if is con.sistent. ii i., asymptotirolly efficient. if is asymptotically nomlally dist,ibltlcd with

a.~ymptoti('.aJ"

(6. 18)

If {EI} is not normally (iist7'ihutcd then the estimator is not efficient.

E!li IX] is liw ('ollditional tll(,<~l or the prooiction of.:! gi,ven X. tht' LS nlimation of dY1Wmtc models, as ('xt'llIpllhcd III the AIl(p) the re:<iuuab call be written
(6.19)

Proor Omitted.
An estimate for is given by (6.38). This estimate ('au hc u~1 ill (6.-13 by calculating the varill.ll((' mlltrix belonging to the C'Stilllal(>. It should be noted that it is not possible to extend t he description to inelud{' MA modds Ilud, at t he same time, use the IlofHlH.l eq\llltioll~ (6.37) ('!o;li lllltte the panl.lnet<'fs. This is due to the fact that c, is unknown a nd rt'Sic\uals, which should otherwise be included in 311 extended X matrix, are function of O. In the following SOllie alternat ive methods will be illu~lratt-d In contrast, the LS method CAn be applied 10 til(' extended fillffir

""",,,;""'0 for the param('lcfS is agaill found hy minimizing the sum of

0:

~id\la h..

plldit/ion NnJr' method, this princilJie i~ 9f1tcmiizcd to (among AR\IA (p,q) processes, i.e., models in the form

(6.50)

class

~( B ) )~

wdB)ltL!

+ ... + W",D)Uk.1 + d1h(t) + ... + dth(t) + Ct

(6.51)
nrtJ'-lIit1dy calculate the one-step prediction:

wl]('(e {el} is white noise and ullcorrelated with {1I"d. Here {u. d and arc deterministic functioll~ Imel. hence, (6.44) can bt, writt('11 ill t.he I
fO l'm

(6.35).

; + L 0,,. ,(9 ),
i, I
I

(6.52)

Finally, it should ])(' 1I0t('(1 thnt there are t wo ways to tr('llt the mP1l1l of the process P"'d, {UI,t), .... {f1k.d: t'it her by subtracti llg (1J(' "'~:;;: IIlrans hrfore the e.limation so that the modrl (6. J.1) dCS('ril)('S til{' v around the mean or by subtracting the stationary \'l'Llur. as illm;trat('(! in followillg example of a model in the extended dass:

(6.53 )

lb., cairulnlions a."~ullle that at time> / there arc infinitely many pa.<.,t In ,)nu'! imlapplical ions. one will o ft en usc either conditioned or ~.,.timntes us dCS('ribed in the following.

Alternatively, we call cOllsidt'T the reformlliation

( 13 ) ~'t =w( 13)u,

+ (( I )/Ll'
..... (B)Ut

w(1)I1,..)+E,.

i CGndltlcH>ed estimation
iK "()n\('tilllt~ cal1('d

which can be writtcn ill t 11(' form

4l(B)Yr

+ d + c/.

,AI"aI,.",
method

Alld
w('

the ('O"llitiOJtf(llta.~t squan's method (CLS) i..('(\o\ter (1983)). simply put

Cp
-p

which is A model in til{' clns,., (6.1 1) with (f(t) = I ). EqUlItioll (6.17) ;,,1;.... from (6.15) in that it is linear in all the paranlt'ters.

.. =

Cpt I .q ...

O.

(6.51)

164

I OENTIFICATION, E:Sl' IMAT ION. AND MODEL C II ECK INQ

ESTI.\I ,\TION OF PARAMETERS I N STANDARD MODELS

165

and hcrcnJtcf, ~ It I can be calculated for l = P prediction errors arc ca.lculated recursively as

I,p

+ 2, ...

tii llce

tbe

prroi'lion procedur(' using the forwnrd shifted equntiolls (6.60) can be

&0 caJcuiatt' t"r(B), which is then w;K.'d in (6.58). For a morc detnikd

. of the procedure, set' Box and Jenkins (1970jUJ76). unconditioned estimation. we have Analogous to the LS met hod in Sc<:tion 6.4.2, we find t he pmiictioFl error estimate I\S

fj =

argn~n { S(O) =
_2

(6.63)
011('

t" l(B )} .

l = p+J

can alternatively divide by N minus the number of estimated

In general the criteria is a non-quadratic function of O. A11 estimate for is

u;

q~ = N - ,)'
6.7 ( PROPERTIES FOR TilE PREDICTION Eililon MET HOD )

S(ih

minimizing (6.56), as well as (6.58), one may apply a Ncwton-Rrtphson (or itrmth'c minimization, sec (3.3 1).

An altcnlHtive esti mate is obtained 8.'1 5(ii) divided by the IHlInl)('r of '"alions utilized in (6.56), N - p, minus the numl>cr of esti mated ,"".,,,,,,,''', p+qorp+q+1.

(f,}

i~

white noise, then

6.4.3.2

Unconditioned estimation

U ~ymptotical normally distributed with E[B]

= f} awl the t'ariance

This method is somNimes cruled the unconditioned least square .. mdllOd. unconditioned cstimalion, the eslimates are found by
H/"K~ ian

(6.6 1)

matrix, H , is determined by

where

{h,,) ~ 80 iJI)
I

8'S(O)

_.

(6.65)

k 8 == 9

,,(0) ~ 8[,,,0 , Y" ... , YN [.

T his conditioned cXI)('{'t('(1 value can bc calculated using back-Jore('a.~ti'19. will be described briefl y ill the rollowing. Assulllc that {l~} is generated by the statiollA.ry lIlodel

H iN til(' (~tlrt'atlHl' oj 5(8) in the point () = ii.

:11l1li1..."",,, or the number of observations) 1000 valuPS of l't defilll'd by tllf' AR\fA(I.I) proccs:s
(1

(B)Y,

O( B)<,.

It is also possible to consider (YI } as bei ng generatro by the bac/.:u'aro model

(F )I',

8(P)e,.

+ 0.7B)I;

(1 - OABJ<,.

T his backward shifted model call be ap plied ill the cnlculation of till' shiftcd pr('(lictiolls E[l',1 6 ,}'\ .... , l'.'II ],
1 = 0, - 1, -2, ...

simoe E[e,18. yl, ... VN] 0 for t ~ O. B('{'flUSf' til(' model (6.60) is " .,10. til{' prl'<iic'tions will bt., very clost, to 0 A.fter a givCl1 step. t = _.Q.

:'jeo"i'"'

DeXt page shows thp contour CtlrV('S .'0,'(6) = 8(411.01). ",. Bub... o( 100 ....alues of 1'1 wa:,; used whilp all the right WBII U8ed in the minimization. It can be seen thai the (unction lies at th~ t rue value of the pa.l"a.metenl, ushq; tbe whole Meri~. However, when using only 100 ...._m does not lie near the true value of the paramet.en.

166

IOI~NT IFI CATION, ESTIMATIO:"ol, AND MODEL CHF:CK INQ

N = 1 00

EsT1U ATIO'IJ

or

PARAMETERS IN STANDAIlD MODELS

167

1000

is the mean l'alut: for ( }/,I Yi _l. (J .U;). predi('tioll errors become

"
WI'

"

-I

'A

,
h -t
(6.71 )

Sill(,(, t is normally distributed ,

111\\'('

h O.{]'_~) =
-0.6 - 0.1 - 0.2 0.0
MA param('l('r
Fiaure 6.7: Contour
CUrt'!:S

(1,

v.!rr
v 2rr

1~2 cxp( -( Y, - EWtl Yi

ex p(-,1(8)/20;)

), O])2/2a;)
(6.72)
011

=
-0.6 - 0.1 -0. 2 0.0
1\1,\ parameter
DOW
(1(

1 =

write the conditional likelihood function (conditioned

Y p)

lor S(8)

S(I.Od determined from 100 Q11d 1000

'Iimulalcd t>alIJCS of the ARMA (1, 1) proces.~: ( 1 + O.7B)Yi = (I - OAIl)t.

II

J(l'lIYt_I, O ,a;)
(6.73)

6.4.4

The ML method for dynamic models

Commry to lhe methods nH'ntionoo so far , t he ~ IL method ( ~hlxim llm hood) require;.; an assumption of lhe distribution. In gt'IiNai, we will "".un. that fed i.~ normally distributed white noise with the variance Varlet] = We consider the ARMA (p,q) process

thE> logarithm yields

(6.71)

Y, +~I Yt -1 + ... +,,1',

PUfthermOfe, we introduce

+ 0 11_

+ ", + Oqf"t_q
c is a cou:-;iant. Differclltiatilig (6.7 1) with respcct to
to Z('ro giv('S

(1;, and setting

OT = (4)1, ... 'P, OI,'" ,Oq),

( Y,,)~ ! , yd , Le., Yi ('outains all obscrvatiOI1~ up to time t.

y;T

alog/,

(N
2

p) l i N
(1t

un{
""...m i... ohtained for

- '2 + ,' , (0) = O. 2u" '" ~

c 1,.",.. 1

The likelihood function is the jo int probability distribution func t ion for observations for given valil('S o f (} and i.e.,

u;,

!(yN[ y .... J, O,CTc)J(Y... _1 (J ,(],)

(6.75)

~ ( IT
I

/(Yd>';

,,8,o;)) / (Y,,18,.,).
(6. 76)
th,. AIL ('.~limall' () is oblai nc'd h.v lIIinillli;dng

1'+ I

This is t il(' gellerai formula fo l' llt(' lik<'liilood funct ion for titllt' S('rit'S W{' 1\......,lUlU' that there lire infillitf'ly many past }' valuC'S, II.' fo r the' ""'~- ('rror Ill('thod. we ('I\U ('alrlllat('
\ 'j,_d O) =

Ell,

Y, 1. 0

,+

L O,, , .,

,(8 )

S(8 )

, L

'~(8).

(6.77)

I=pl

168

I OENTIFICATION, ESTIMATIO'l. AI"O MODEL CIt ECK I:'<OG

EsTI~tATIO :>i

OF' PARAMETERS IN STANDAnO MODELS

169

where the AIL e.stimate for u; i:-J obtained by

(6.,") Compared with til(' prediction error Illethod, it. is Sl.'C1l that if the assUlIlptioll 0( normality d ocs not. hold , then the cstimat<'S (6.77) and (G. 78) can be int<:rprC(l"<i as pr('dictioll errol' estimatcs; SC(> (6.56) and (6.57) on page 1(iI. Si nc(' t ht' above memioned estimates relate to the conditional Iikdihood function (6. 73), thty arc called conditioned l\'I L estimates, A nl('t ilod for calculntiug the exact likelihood function is babed 011 tht' back.forC'C8." t illl method , which wa.'i described ill the pr('violls &'(:tio ll .
TIIWRD1 6.8 ( PnOPERTIES OF THE

= ii,

w('

ha..,t>

OS;;)

== O. aud hCr<'by = _N
- I'

a'10g LI
uu
JlJ)2

-? 68 -

(S _'(9 )8 S(0 )) 1 00' 88

IP.PI~ the ol)b('rved curvalurl' of til{' logarithm of t he likelihood function

tI II."
ML
ESTI MATOR)

lUI

approxinlal io n to Ihe Fisher information matrix, we obtain

i) fj is consistf'nt.

V.. [8[ '" 2 5(0) H -'

N-p

ii)

0 is asyml)iotically flonlWily

distl'ibuled with E[O) = 9 anit t'ariant~

H HI j.!;ivcn by (6.80). Since S(O)j(N - I =

Extended model classes

17;, the result follows.

whITf' the matrix H is itetenni"cti by

pn,Ii,"c", eITor lIlethod a nd the ,\ IL method can be used directly lIlo(kl das..'it's. For the considered rnC'thods the idea is that giVC'1I
6 (j

{h ll.} = aOlao~.
iii) fj is a.rymploiicaJ.ly efficient.

0'5(0)1
(6.83)

Y",

,(0 ) = E[I',[ Y,_" O[.

(6.84)

Proof Omilled. Follows from the gencrnl prolX'rtie; of the ML estimator Pawitan (200 1), Ols.')on (2002), or Thyrcgod and ~ tadM.'1I (2006)). The variance (,lltimatc in (6.79 ) is also based 011 the as,'!lIl1lptioll that l\ IL estimators, it holds asymptotically that

,prol,I,',,, is in any case to calculate the conditio ual nlt'an E [Y.I Yi._ .. O].
~,~:~:: with d('terministic input)
l'Xt('lldf'd

Illodd and the principl<'S, consider the tmnj( I' (6.85)

V",[8[ - 1 ' (9)

where 1 (0 ) is the Fisher injormation matrit:

.~::~:,~;; IIN(' /ldB) and

~

fl2 ( IJ) art' rational traoJoofl'r we will demand that f/2(O) = I. (For f'xamp!e o aud 1I:2{ IJ ) , O{B)-IO(U) til(' pn'viollsiy cOll.-;idcrf'd
gt't

1 (9 )
Since (see (0.76))

_ E[02!OgDj
Of}:2

(6.85) we
.

the condil i01l1l1

I t!!

f'XI){'('t(,(j

vahl<'

- HdB)u .. 1 /(2 (B)

10gL - 2- 1og5 (9)+,

.v

J1

H,(B)", + [1 -1I2(B) IJW! - H. (B)u,) H 2 (8) I HdB)u, + 11 - H2(8) IJYi

(6.861

170

IDENTIFICATION, ESTIMATIO~. ANI) MODEL CliECK1NG

SELFMIOc-; OF TilE MODEl, ORDER

171

Since H1 (O) = 1. I h(' conditional cxpcctl.'{i mille will dt'p('nd onl.... 0ItI } ', ]. } 'i: .2 ... as well as t h(' kllOwu va]U0> of {ud With givell inilial coa.. dition,.;, WI' C1\1l sUl'lM'qUenliy calculate the prooiclion ('rror by lL-;lug And jWfcby. a:; WI' haw 8('('11 ('arlirf, nppiy either the prcdietion error m"h,",1
or the ~IL method for l"StimlH ing til(' ullkuoWIl parameters. 0 , ill H t( il) II,(B). For tilt' ARMAX model (d{'SCri bNI in C'hlltf'r 8)
tP(B) ) 'j = w(B)u/

The a utocorre lat ion functions

IUt()("orrc lation function /:Iud lIlI> partial aUlOcorr(>lalion function an' applit'(l to determine reasonable gU('S.'i('S for (p, q). whidl ar{' d('S('ribed 6.J.2 Oil p/:\~t' !54. ....h.,"'''o', the autocorT('latioll fUllctioll:-! for the residual:; li('r\'P as good for how tlj(' e!-.lilllllted mo(kl should be cxlcu(Ii"(I. This is illu:;tmt('(1 ill that fit n gi ....{'11 sta!:( ' ill the ilerutil'e modd building procrd1ll'f', we the following model:

+ 0(1),

i_'i",,,"<i

w,' haV<' H,(B) ~ ",(O)/(B) ""d 11,(0) ~ O(D)/(B).

( B)J',

O(O)\\',

(6.91)

t ill' ('I;timated autocorrcllltioll fUl1ction for the residuals,

{HId, ill(li-

that II\(' residuliis

O(U)Y""

('an

1)(' described by the following:

",( B)",

+ [O(B)

(B)[ l',.
(j)*'

'(B)\\"

0'(0)".

(6.92)

With gi ....(,11 init inl conditions (most oft(,11 Ytit I = 0, 111 I ::; 0), Oil(' may proceed 8b d{'S('Tilx'(l pn"'iously Illld calculate till' 1",,,.0<1< ('rror frolll (G .83).

I(B)O(B)( in (6.91). " 'l' see that the multiplicativ(' model

o'(B)(B)Y, ~ 8(B)O"(B)"

(6.93)

Please note that models in the form ( U)Y, ~ d,J,(t)

+ ... + dtft(t) + O(B)"

an ob\'iotls modd to t'xlllllin(! in the next !it('p. Alternath'ely to (6.93 ) HbvioliSly !'>('l(,{,t th(> correspondi ng ordinary (non-multiplicative) Tilt' (kscrib('fl prO<.'edlll'l' is l'srecially usefu l for sea.'iOlLfll modd:>.

where II(l) up to I,(t) are known functions and d l up to itt an' "nkn'' ''' parameters, can also be estimated by the prediction error method or the method.

('ontains n IItnuiwr of tools for testing the model. All example is

m-t fur !'>ignifiNlIIce of t il(' t"Slimatoo panulleten;.

Ukelihood based infe re nce

6.5

Selectio n of the model order

The model building is often all iterativc proceduf(', sinC(' a model '~:~~:::~ lin estimllted model wit h a given structure will ro rnmonly indicate ~ models. Th(' model building fin;l stopS when all likely models han' ht~'n and the m()(-jt adequate model is found. A very important problem is to determine the number of paranwtl'f!l. with the ch()5('n model structure, is the most suitable to describe til(' vn.riatioll. The Iluml)('r of {),'l.rrunet('fs is caUed the mood ordr:r. If 0<'" . the most adequate model alllong tbe cJa.. ....'i of the ARMA procc..'!(.'S, tiw com-;ists of d{'tennilling (p. (I ) In th(' following SOUle tools and te;ts for d<'l('rminillg 111(' model ord be described.

m.".n.l.,-,,-,. thl' maximum li kd ihood method, lIS described previously,

Il met hod for obtaining an of ltmult'tt'rs: it is I\. \I1ethod ror lin objcctive rt'a."K)l1 ing with data. l'htir(' likelihood fUllction. not just its maximiZ('r. that captures infOf l1l llliOIl ill the (hl.Ut about 11 ('('rtain pUI'aJlit't('r. The Iikdihood ahwl S('n'e til(' hasis for u rit'h fa mily of methods for sel(l('ting tbe i lJl()(i('l, 8t'('lhm Wt' dt~'fib(' methods for t($lillg hypoth('S('S using th(' like. T!l(' bfit;ic i(/ra is to d<'(ermim' t 11(' maximum likelihood wI(h'r hoth Ill(' lIull nnd lIitemutin' h." l>othe--is. For a mor(' rig ious w,' r('f('r to, e.g .. Pawitan (2{X)1). Tb~'f('v.od <lnd ~ l a(l'i('l1 (2006), ( I !)1:j I ).

U8Pc1. T he likelihood principle is not just

172

IO ENTIF ICAT IO N, F-STIMATION, ANO ~IODE I.. ('HE(' I(I~O

St:I.ECT10N OF TilE MODEL OIWEH

173

Consider tilt:' hypothesis flo : 0 E 00 against the alternati\'e where dim(Oo) = rand dim(O) m. The likelihood mLia is defined as

flo:(} E U \ {lo.

A(Y)

SUPOEno

L(O; V)
L(O; Y)

sUP8EO

Clearly if A is slIIall, then the data arc seen to be l1Iore plausible ull(kr alternath'e hypothesis than under the null hypothesis. li enee the h)'POIII". (Ho) is rt'jected for small valucs of A. It is somet imes poo.<.;ibl(' to transform likelihood ratio into a statistic, the exact distribution of which is known 1-10, This is for instance the ca..<w for the General Linear ~ l odel for Grn","" data. In most Ca&C!I, however, we must use the following important result the asymptolk behavior.

'""fUdiJ.

i
'!!---1--2---3--'---5--6---7TIl(; loss junctioru dependency
011

= number of parruncten;

the ntlmber

0/ parometcr.'I.

T H EORE~I 6.9 (WILK'S liKELIIIOOD RATIO TEST)

For '\(Y) defined by (6.94), then tmder the null hypolhesis Ho (as aoouj, nmdom. variable -2 log '\(Y) C071VC1yes in law to a \"2 mndom variab/('

(m - r)

dcgr('c.~

0/ freedom.

j.t'.,

-2Iog..\( Y) _ \2(m - r)

.....'i"'''.

liD follows a I-distribution with f = N - I) (f degrres of frcec:iom, N ill till' lIulTIl){'r of observations uS('(i in the estimation of 0,. If we have a m('an mlue, then the degrees of froc-dolll lx>rol 11e f = N - I - p-q. w(' can form the test stAtistic

under Ho.
Note, that the model must be estimated under both liD Imel thC' ulleu,",tl'Il hypothesis. A vl.\riant which does not requ ire Il.I1 evu luation in both the Wald test ; l;C(', ('.g., Harvey (1981). (6.98)

110 i:-; F( I, f)-distributed (sec Section 2.7 on page 23).

6.5.2.2

Test whether a parameter Oi is zero

T.... for lower model order

,,"vk.w; :-;('I"tion we introdUCl.'(i the o!um _ orsqunrrd ,W duaiS]or Ihe los8 for a ~iv('11 set of parallll'tcrs 0, by ,......1.v9-<-o-

In Sections 6.4.2 through 6.4.4, it is shown how LS, ~ I L, It.nd predictioll esLimutcs of t ilt' modl,1 parameters can be found. In All cases, determini ng the (asymptotic) mrilulcc of the estimatl'S for each of the have been giw'l1. Since the estimators can be as:mnwd to be "',II,pl.ot .ioIi normally dUitributcd, we have n tool for testing whetlwr a panuurtl'r 8, .. Civen an estimator 8; with the (asympt otic) varianC'(> j1~" 1I111'.'h:::~ found as t he diagonal element of the ('Qmriance matrix Var[O], see 1 6.7, and 6.8. We want to test WhCtI'C1' 0, is zero, i.e., we W8.nt to teM th(' h,po''''....

S(O,) -

L>1(O,),

(6.99)

Ho:O,

RgRin:;t

H1 :O."# O.

Based on thf' asymptot ic nornmlity we illtroduee the t('!o.t :;tatl:-;ti('

T

0,

I !::~~',:~'
~
M('

index i i~ the num~r of est imated p<"'l.rameters. It will also hold , a~)\'(' Illollf'] is extended with one extra parameter, I.e., to i + I thl'n S{Oj .. d < S(O!). l3ut it is nlso obvioll.'j that the "gain " from one IIIOr(' parameter bC'C0111('S ]('S!; as the number of parametcrs Thill m:-;kclched in Figur(' 6.8. that w(' IlIl.\'e .v observations of A storha..<;tiC' pr()('('S.'; which can be by flo Imralll(-tel"S. 81\.-;('(1 on th('5(' obscrvaliou:-;, models containing have be<>n ~till1l\t('d, where 11 0 ..:; Ttl < "2. F'urtherlllor(', a sub:-;et of til(' 711 pnrRmf'ten!, A11(11h(' 711 parAmeters ar('

li- '

174

IDENTIFI CATI ON, ESTIIIIATION, AND MODEL CIIF'X'KISG

175

a subset of the 112 parameters. The corresponding loss fUllctions I'm.' dVlloted SI and ,'h., Illid it holds asymptotically that

Cross- validation
_"w,ti(~

(5, - 5,)/(., - n,)

S2/(N
Il2)

E F (II'l -

nl

N -

f/2)'

of the modt'l nrt' t'XlIIllill('(l Oil a dataset which has not been estimation. Specifically it is examined if the \'ariam:e of the prediction 2 is of the same order of llUlguiLudc I1.'i for the observations llsed for ii,
~

Thl'test ;,tatiMics ill (6.100) ('an be used to te-.t in the genernilinhlr as described in Chapter 3. lIo\\.'(>ver, it can be shown that (6.100) C'lt.n nl .... ) used for All. models. and ill general. the test is regarded 8." heing ad<'<IUHtc AR..\lA mo<leb.

16 :J.N .-Mt-t..-l "\OI(.!..1

\
\,........

\ ~k-.;J.....

.MJA,

1...11Jlf

Residual analysis
~""aJ a..~umption

rl'glt.rding th('

AR~IA

process is that {Ed is while

6 .5 ,3

Inrormation criteria

Thu.;. if a model is ndequate, we have to require that (E1(9 )} is while The n".~idtJal analy_sis consists of testing whethrr (e,(9)} can be assumed or {f"I(O)} . Thi~ plOl cnll r('v{'aJ l10n-sllltionarities and outliers (e.g.,
for change in signs, If fet} is white noise (wi th meall vallie 0), El on !l.wmge will clUlng(' its sigH ewry secol1d time, i.e.,
P{cilanges ill sigu from 1J
Oll('

TIl(' literature proposes various different methods and proce<iurl-s for tl\('

mirullion of th(' most "ad('(IU!l.tt' model order. Probnbly the 1Il000t ,o>u, "'''nIJ used is based Oil Akaike's In/ormation Criteria (AlC). which hi d('filwt\ lUi AIC = - 210g(lllax. likelihood) where
11 ,

+ 21l"
t'flSf'

....,.'0"' measurements).

is the 1l\1ll1ber of (.~tillll.ltl'd model paralllct('fs. In the A nt-.IA (p, q) Inodel Wf' get (including an additive constant) AIC = N log

of

8; + 2(p + (I),

to f}

=~.

where (j~

L"i white noiS(', cach of the chnnges will be independent alld wc ha\'e
~ulllJx.r of chauges ill sign E

S(9) /N. The lncthod ba.<;ically consbts of choosing (11.1/) that AIC (P,(/) is minimized. Akllike's information ('riterill ila."i II "'"d,'nq

13(N - J. 4),

(6. 104 )

esl)('('io.lIy for Inrg(' N, to n{'("('pt too many parameters (Llit kepohl 1001). An alternative criterion is BIC (Bayesian In/ormation Criterion), including lUI additi\'(' (Oll~tant. i!o.

SIC

Illlogii;

+ (p + q) log N.

is the Ilumber of residuals. For large values of N, lhe binolllial ('"lUI b", approximat('(1 with the lIorlUal distribution because we - 1.1 /2):::: N(Ni I, "'4- 1 ) for large values of N.

In the autocorre lation fu nction. If {Ed is white noise it holds

Contrary to the Ale. the BIC yields a consmtellt (.'Stimate of the tII()dt'l The information ("riterion (6.103) is sometimf'S called. Schll'(Utz'.~

[I,' ....

6.2) (6.105)

Criterion (SBC).

6,6

Model checking

After haviug del('rlllill('(1 th(' modd structure, (\{'tcrmined tl\l' lIIod",) and estimated the model parameters. the next step should be I 1Il(J(It'I. By dR'eking the lUodd onc can cxamine whetl]('r t \l(" ob.~.",..~ can be dcscrib('(1 by the estimated model IIdequat('ly. Often II lJIodf'1 wi11 revt,td th"t II Illodd is not satisfactory, but at t.ht' saUle t iml' IIltel"llntivt' Itlodrt ordNS. It is thus oftel! St."<'11 thal model dwC'ki ng i!'l on nil t he obvious model choices and t hat th(" outcome from lUo<lpl is u...-;'(I ill fiudiug til(' fi nal and moe;l satlsfoctory mod!'l. III tht followill~ !o.('(UOIlS. SOI1l(" of til(" mOM cOUlmonly lISl"(1 HHt1,odII dU'('kinp; til(" Illodpi will !)(' di;,(u~M'd.

this. o lle ('lLII plot, e.g., 20" limits (rorresponding to an approxiconfldt'nee interval), and t hereby tcsl if the illdividual wtlues of . are significalltly different frOIll zero. W{' cannol observe {(t(8 )}, bllt only {Et(9}}. It. Crul be shown II the 1I1~{'r limit for the variance of Pt, g)k). Especially for small th", vananC(' ("nn be signifiCiUltly smal\er. L mprovcd estimates of or p~,(9)(k) Clln bt' found in McL{.'Od and Li (1983). lack-of_fit. t.est. We define

~ (v'VP,( I))' + (v'iVii,(2))' + ... + (v'iVii,(m))'.

(6.IOG)

(6. 105) apl>roximat<'S a sum of squaro::l independent N(O, 1)8tocha8tie variabl('S, i.e., Q" E y2(m).

176

IDENTIFICATION, ESTIMATION, AND MODEL CII'E'(",",,'

IIlo.l> '1,

ell EC' 1\ I NC

.77

Sinc('

w('

base the test on !:(6), we obtain

~r---------------------------~--~
6% limit line
25% limit ti.",

where n is the number of estimated parameters ill the model. A 'c"~"",I,,1e' ('hoiC'(' of m is in m~t ('llses 15 SinS 25.

e) Test in the cumulated periodogram. For the frequeuC'i<'S V, i = O. 1... , LN/2J we cnJculnte the periodogmm for the f("Siduals as

.....
."

.. ..... . -, .......... .. ........ ....

, "

..

"

which is a description of the variations of fed in the frNluC'IICY domain I (v,) dI'Jlot{'S the JUIlOIlHt of variation of {e,} that is related to tlw (<,,,""'1 IIi The pcriodogrnm will be discussed more do:scly in S(.'('tioll 7.1 on pagC' Th!' sClIlcd ('wflul(ltl:d fJl'f'iOO(Kj1YlfII is

0>

02

0.3

0.4

0 ..1

j,

C (v ) - .,L:C:C:=--', . , - [ IN/2J ~ L .. , l(v,)

which is
A

[L~., l(v,)J

.J'
V,

---

5~ limit [inl' 26% Jjmi~ tint'

non-d('('rC'flSing fUliction defined for the frequencies

= i/N

O . 1... ,lN/2J.
For white noiSE', {el}, we get, as discussed in Chapter 5, thnt til(' is uniformly distributed 011 All frequencies. Thlls, for whitC' lIoil'i<' thl("'.~~::~! periodogram is constant. Furthermore, the tota] variation from IV 0 is No;, and thlls the thcoretica.I periodogram for white noi:-.t i;.,

l (v.) = 20;.
The thcoretkal flCCulIlulated periodOb'Tam C(v.) for white noi.o;(' U! a stmightlillf> from the point (0.0) to (0.5, I). If {ed6)} i~ white will thus expect that C(v.) is located around this straight lin('. A interval for th(' straight line can be obtainffi in a !;imilllr IIlfLlUl('f to Kolmogorov-Smiruov t('St for di8tribution, sec Chapter I of COllrlldscn or Hald ( 1952). The confidence limits in the Kolmogorov-Smirnov distribution are determined so that they include all the 8(11;) Wlltl{'S at 8 probability (1 0'), ('onditiOIl('(i that {!:t(6)} is white lloiS(', fLS 1>111 " ",,81" Figure 6.9. The lines arl' drown at It distance Ka/.;q above and i)('iow the rt'tical line, where q (N - 2)/2 for N even and q = (N 1)/2 for N Approximate ,,3111('8 ror 1\(> arc given in Table 6.2.

,,,,,6,10

0.'

0.2

0.3

04

0.5

j,

The cumu/utrd perlooogn.I.1n for all illadequatc model (top) and 11"
(bottom).

CoeJficit"t., irltJoll't'd i1l mlrlilattll9 Pf'Obability limits for test ill the

"o

0.01
1.63

0.05
1.36

O. IO
1.22

0.25
1.02

"

178

IDENTIFICATION. ESTIMATION, AND MODEL C II E('I\ I ~Q

. l ; "''': STlJDY: EI,ECTRICITY CONSUMPTION

179

Case study: Electricity consumption

('On ~jdcring
\'il'

" ,"'N'

a "fcal tire" {'xamplc laken from Jenkins ( 1979). In this a[(' looking for a model for e!cctrieity consumption with the of pr('(liction. in a hllCI' exrunplC' in Chapter 8, the 1II{}(!t'\ will be to un input-output model , which exploits the monthly meau ambient in order to improve the model. Th(' upper half of Figure 6.10
CIlt'rK,Y 118(',

plot of the rnollthly

t,

and

nil

input series. X tt cOlIsi:-;ling

c';""Ix>nding monthly mean tcmpcratun..'s. The lower half of the plot

("o rn'Sponding rouge-mean plot. l'8Pgf'-trll'HIl plot is used to cxamill(' whetilf'r a transformation in thl'

,,
~ ! K
~

1957 19511 1959 1960 196]

1962 1963 1964 1965 1966 1967 1968 1969 1970

Y,

1'1 _

{{ V ;' - I)/ !.
InY,

(6.11 J)

,"
\051 1!l58 lM!l 1960 1961 1002 1003 !!1M 1965 1966 1001 lOO!! 19(;9 191(1 Electricity ..... n~umptlon

W ~'i('ld a constant variance. This cla.~ of transformations is called t"m,~for7llatiml}f, or variance stabilizing trnllsforllm\ ions S('(' Dox

( 11"1). T Il(' tilllt' "ot'ries is di vided SC<lucntially into groups consisting of .. to 12) observations (dC'p<'llding all the length of the tillle S(!riell, or if varilllioll."! ('qllill the ICllgth of t he season ). Subsequently the range

alld lUcan "o,lut ill ('Il('h group arc calculated, alld corresponding
Tem petatu'..

p kJtl('(i in
r"lIV;~'

lilt"

range-mean plot.

S
0
0

'.
! ! !

0 ~ "

!
! !

'.''J '.'

, !.

i
0
0

" .. ,

1213

iR independcllt of the mean Vl).iucs, no transformation is .\ I ). bul ally dcpcndC'llcics 011 the mean value indicate thllt a ... hould 1)(' com;idI'rM. If there is a (p()i<.iti\'{') lincar dCI)(mclcncc. transformation (,.\ = 0) should be applied. unci if t hen' is all or downwards curw\turc, th('11 >. = -1/2 and ,\ = 1/2 should be
~_., .

.;

,
~ ~ ~ ~

. in-Iy. Figllr(' G.II on the following page iudkates Box-Cox for various range-meau rdu! iom; rclal ionsh ip ill Figure 6.10 belwN'1I lile range and the menu f'l(~'l rieity eOIlSlllllption indicat('t; that a logarithmic tram.formation OOIlIIidt-rt'(1. Rt,l!;flrcling the tCIllIK'rature series, there is no 11('('(1 for

"
60
00

6.1 2 Oil tile' Ilext page, a smnple of the scrie; collsi.,<.,tillg of the
13.0
\1<."" 13.!.

"

8()

00

HlO

125

M('ftfl

of tht' ('\('(tridty eOIl ..,11Inptioll and til{' est imated a utocorreitllioll lh( ('Ill ire ~;(' ri{'S is lihoWIl. T Il(' autocorrelation flllJ('tioll is charac JMriod of 12 months, and the value'S d('Crea.<;(> to 0 slowly. which

Filure 6. 10: Plot of the f'it'ctrirny consumption and tempemluf'(- aTld till' : :': ; : : iug nl1lgemeon plots (sub-lIcrie-s of sitt' 12 hallt' been applied if! thi' " plot).

that tlw Sl.'ri(o:. is non-stationary and that a differ('lwing io,; n(,(",f'SAAry.

,.'1"" fUll niou.

shows a Sin n pi' o f t he M'rif'S \lltl \~ and the corr(';;pond illg It io,; s(,(,1l that the diff('r(' ncing has remuved t h(' ill 1111' original S(ril.'S, but n 'ot'a."<1II1t1 diff('rt'l1("i ng is 11('('(\ed in Ill{' slowly (i<.>cr{'asing 11111()f'orrdal ions at lag 12.21, 3(i, .

' OV

IU~.N'J' tFlCATION, ESTIMATION. ANI) MODEl, ("H"C;;;::~

EI,ECTll[CITY

(,ONSlJ~IPT[ON

181

No [mils.

_----:-;.;;" _-/ --

Residuals

>. - 1/2

In

_----

>'~-1/2

///-------

Residual 8utocorreiatiOlI functiol1

Ol~~~~------------------____~

Mean

Figure 6.11: Box-Cox tmns/QnllfIlu)IIs.

"
Al1tocorrclat iOl I

Series

r"H''',,",
Rf1lidlJaL~. ~ .

Lag

0
"
o

and rorT"t!6pQflding CltdocQ'Tl'/atioli jUflctiQn jor the modd

em tlu- u'lwi( writs.

o o

" o
Ca) I n (~'. )
o

"

"

6.12(<,). the results frolll applying a S('l\sonal differencing V l2 Appart:llli-", t!J(,rt' an' no need for further differencing the scri('S, vV 1:lln}", om be assumed statiollnTY. TIl{' largest values of the 1111('tl011 for Q'V121n Yt arc S('('11 to be at lag 1 and 12, thus followin g i'llnl('[UT(' is applied: (G.112)
aUhworf\']atiOIi fUllctioll at lag I, rl = - 0.27, for VV 1:lln
I. al(

o
o

o o

Y" ....'t.

e
o

,
o o

8 o 8 o 8

: ~~,l,...oo,-.--"""""""~
o, o

as a starting "Hlue for O. Similarly w(' uS(' tilt' autocorrelation "" -O.:}:l, H.'i A. ;,tflrtill~ value I - 0.35 for 9 . In both C8.';t.'S til(' CUrrI'Spout! l'i()S('iy to the mOIJl(>ul e:;tilll8.t(' of tht' pure ~IA(I) pW("('S.". rbJl('"{th'cly. \"lIill('s art' now flpplicd for pnmlllNer ('stimation by the IIIN hotl. BH.W"<I on thC' cntire ;,('rips ( 160 obsl'rvatiO!L'i) we find lUodel

(1

O.30.tO.08S)(1

0.55 oU"l BJ"l)t

(6.m)

iT?
Figure 6.12: DiJJFrf'''1 onlFrll 0/ diJJn"Nlciug 0/ !lu .'FriF~ lor thr t"lfr/ririly tlon arid thF oorre. pondiug (lulocorrdation junrlilm.,

O.rt21'\3 2 .

SM"il"S of fl':-;iclual" and rurrespolldill~ nutocorrelation fUIl("1 ion secu. Ihill the antocorrdatioll i.... larv;t-r than two stflll<iard

182

IDENTIFICATION, ESTIMATION, AND MOOl~ I . 0 "'>:(."

183

d('vialions Itt hlg 6 lind 13. The number of wilues outside 2 standard d" . ;;",.;~ is considerably larger tha.n it should be, and it is concluded that lilt, mrxiel not aclt'quatt'. F'urliler improvemellts of the model have not bct:-n I . .' In order to examine the rcason for thc inadC<luacy of til(' modd til(' i serit"S Ims bCt'n rlh'i<i('(i iuto two sub-series of 61 and 96 rI.'sJ)('('tively. and model has been fitted to each of these series. T he re~;ults are:

DKK ttl US'f).

1 IOS-I 2 1052 12 931 3 1028 13 922

5
10:.\5 14 919 1013 15 956

6 1029 16 910

7 998
17 912

8
980 18 919

9 976 19 906

10 967 20 885

11
95.'>

First half

vV' 12111 }'/ = (I + O,14 %0. 12 B)( I - O.83O,O IBI2k.

- 2 q~

= 0.0237 .

:2

3 E.'itimate the autocorrelatioll function (lag ~ 5) fo r {Xd lind sketch

Second half

(N

96)

v'V 12 In Yt = ( I - O.73008 B) ( J ~' q~ -

O.83O-O~ lJI2kl

4 BI\.<;{'(i on the estimatf'd autocorrell\tion function , investigate whethcan be a<;''iulIl('(1 to be white noise. &;tahliRh u model for the \'l\riatiolls in the exchange rate. Use this til(' aH'mg(' {'xeilnllge mtC' ill week 2 1. in this chaptl'r the I\ut()c'ovariallce 1\1. lag k is est imated based .. ,:r III using
011

00?55' . .

II is:-;N1l that t he estimate of the seasonal component is rclat.iwly N"~t.. comparing t he two sub-series. 011 the ot her huud, tile coeffici{'ills for the parts 1m' wry dif[l'I'PIiL The explanation is most likely that this "", .. not is varying in ti me. It is rccolUlIlendt.'d, depending 011 till' Appli("at icm, IIpply eitlH'r IlII IIdltpl ive t'Stimation procedure or a direct mode lin~ or tim(,-VRrying coefficient. The methods to do so Me d iscus.'>('(i ill Another po~siblt, explanlltioll could be that the inpm series lx-hflV(>!i~":~~~: in each series, but this does not seem to be an obvious feature. III E on page 23 1 t he iuformatioll fro lll the input series will be ('xpluitt1J in inpllt-outpuL lIlodel . An analysis of the estimated autocorrelation fllll("t ioo each of the sub-hCriC'S has ~hown that t he models for each part art' Concerni ug the model checking. ODe should also apply otll('r the residuals, e.g. , a Portmalll('/Iu lack-of-fit tc:.1 IIncl t('st in til(' JX'riodogralll . Thus, the prC'vious exam ple is Dot complete ooncf'rnill~ checking. However, the example illlL',tratcs the IJ06:>ible nC'Cd for and difftreudng wil('l1 build ing models of real life phenotlleull.

C\ -

~ 'L:: (X t -

'Ii _A;

,-,

X)(Xt+k - X )

",,,,,.Jo\

",,,,,.r,,,"1I!

.,wh;te noiS(>.
Sketch the allm\'f'(1 area for the pnrulll('terM, i.e., lhe set of (1)1, -l ) AR(2) process is stationary. that I)art of the alluwed area of the parameters where the function shows dalllp('(1 harmonic oocillations. a pro("('S;; the following Itutocorrclations have been estimated:
0.1,

6.8

Problems

Exercise 6.1 The following time S('rics {Yt l is supposed to be the Danish kron(' (Oi\K) US do)!!\!' (USO) ('xdllU!gt' rn1(' (DKKflOO$) over a period of 20 W(>('iaI. number in Table 6.:.\ shows til(' average exchange raW of th(' r
w('('k.

,..\

-0.4,

T4

-0.2

Question 1 Sketch t he tim(' !'OE'rics. Is the time series stationary"! Question 2 Find {\ t rnnsformlltion that gives an acceptable shll i(Jllary &'ri~ and :;kcl<:h the gellC'ffllC'd liwe S('rie!) x,.

......mt {'1-;timnl(' for (1'~) bn.st'd on the ol>scrved values of t he functioll , lL.'!SulIliUK tiUlt til(' prO<'(':-;8 can be approximated by

184

IDE NT IF'I CAT IO~ , K<iTI/o,IATrON, AND ~IO OEL (,H':(,,, ...

185

Table 6.4: Sampir l/artilll (lu/ororrriatw1l.

Estimated It(I,riance (lnd Sam/Jle (lli/ocorrl'liltioll junctions.

k
I 2 3

u
-0.-1
0.19 0.01 -0.Q7 0.07

k
6 7

u -0. 15
0.05 0.00

k
II

.k>
0.18 - 0.05
0.09 0.18 0.01

Sample autQ('orr!'\ ution in

Va...
70.7 52.5 96.3 0.99 0.11
0.45

2 0 .99 - 0.01 - 0 .02

3 0.98 0.04 0.97 0.02 0.00

5 0 .96 0.02 - 0.07

8
0.91

8
9 10

- 0.10
0.05

5
Exercise 6.4

12 13 14 15

- 0.0 1

0.96 0.00 0.06

0.95 0.Q7 -0.01

0.04 0.01

10
rl

dny~'

prices (dRY I Ihro ug h day 10)

W(>ft':

Bru.ed 011 oi)M-'n'('(\ valll(':!) from an .\tA(2) process, we havl' found that

- 0.6222 and "2 = 0.2222.

Estimate th(' parameters of the process.

233.219.205. 171, 182, 193,202,206, 195. 179 autO('(Jrr('\utiolls for the o bS('rvlltions, and their 1st alJ(1 2nd
1m""" ar(' gi,'cn in Table 6.5.

Exercise 6.5
For H process tllf> sliltlple itutocorrelations r l = 0 .57 Iilld r2 foun el. It is assllmed that the procCl:iS CUll be deserih('(\ by

Formulate H. Sliitahk' modl'! for tile proces,'l and estimate the pathe model.

The ohst>rwd rates of dny II and d ay 12 arc 184 and 1%, respccOllt an update of till' prediction for doy 13.

Exercise 6.6 The C'S limatcs of t he partial aulo<:orrelation function of a

ill

pr()('(~'i

arr

Tobie 6.1. The totlll nllm})(>r of measurements behind the N =70. Find It sui tabl(' modd (including parameter values) in th(' cla...s of
Exercise 6.1

"",i",,".

Compar(' til(' calculated predictions with the actual rate o f day 13. 202. IL-.ing suitable> confidenc(' bands.

models. Consider the following AIUIA ( l. 1) prO<X'SS

r-""'"

the CO 2 conc('utratiOIl of the flue gas air has been registered TIl(' laM 5 registrations .....ere (i n ~):
78, (.1, 73, 16, 79
BIt.~'(l Oil 100 reh rioSlrations the following estimates Rut<x'Ovariauces for the d('vintion frOIll 75% have been found:

wll('re

{Ed hi

whitt' !loiS(' with Var!Et!

:=

(12.
C(l) = 1.58, C(2) = 1.13,

Question 1 Writf' th(' onC'-step prediction X/HI' as II function of th(" past of Xt. Determine tbe vuriance of the one-step prediction error. Question 2 Write till' k-stt,p pnxlietion ~t(+kl( and deLcnnin(' til(' thE' prediction ('rror.

C(3) = 0.90

:::~~~

a lllo(\("1 for J'('gistrations and ullac!l'qlU\tc (wil h r('~pf'(t to

usc :>uitablt' te:sts to verify the available information)

""O"oal," Ill(' unkllowll param('t('rs of lhc idcntified process.

Exercise 6.8
A ('OIIII>lUly !I lL'! analyzed thc dail.\' price of shares ZI in of 300 days. Thl;' aV('rR.g(' rat(' W!l.'1 198.
fI.

,on,,~utl ~

tilt' next 3 vnl\l('S ami cI'tNlIlim' n. 951,;{. eonfi<ieu<,{' band

186

i DENTI"IC A'J'ION. ESTI MATION, AND MODEL

""" .K'"

Exercise 6.10 III order to establish models for improving the prt"(iictions of the water it is found that deviations Yt betwccn the hourly calcu lated Waler level the observed wutcr level can be described by a Iincar process of tht form
(1 - 0.88)( 1 - 0.28')(1 - B)\" ~ EI
where ~ ( is wh ite noise with the variance it is found that

CHAPTER 7

Spectral analysis

11;.

l3ascd all about 1500 ob""m".~

(;; = 0.31 dm 2

Question 1 Based all the ob:;crvations of Y/ !;hoWIl below, use the giV('n , in order to provide a prediction of t he wl.lu(' of )r/ corrc:,;pondillg to I :-:: 12.

t (in hours) Yt (in dm)

1 2

3
- 1

56789

10
-~

-2

-3

-I "

Table 6.6: Observed u:aLer

It"t~ls.

Question 2 Provide a 95% confidence interval for the prediction.

cOII(,pms the estimation of the spectrum for olle or more t itn(' ('fiJl('d the sample spectrum. In S('('lion 5.4 on page 113, it was that the sp{'('t rum for a stationary stochu..-;t ic proce>S is a freI.....';'" dc:;cription of the llutoCOVllriall("(' function. Correspondingly. flpt'Ctrllln is a frequency domain descr iption of the {~tilllated aufunction. T hc5pcctnmt contains f\ description of the Vllriulions (ioutai'i, w\J('reM the corresponding deM:ript ion in the time autorovariancr fu.ndion. Spectral analysis is thus applied when propl'rti('s of n phenomena arc invCiitigated. e.g" when a time p('riodicit ics. If marc than one time series, or signal, is aVllilable, anal.\"l,is call he applied to o btain a nOIl-panlllletric description of, amplitm\1' and the pha.-.e. T his will be used as a possible approach to ......,[" ftLlIClioll models with mu ltiple input in Scctioll 8.5. 7.1 and 7.2 concI'rIl tilt' est imat ion of the spt'Ctrum for one time 5Pctioll 7.3 concerns the estimatioll of the cross-spectrum betwC('n eeri~~. \lainly non-J}ommetric methods will be considered. The of the 1HJmmetl'ic methods (also called time domain methods) ~'lrllm hi d<"i;('rihed by parameters in an ~timau.'(1 model (e.g., ("()('fli('i('llt in all AR(p)-modcl), a nd the SPt'Ctrulll is found ""lpOnol;,ng tll('oreli<'al spectrUIlI (e,g" (5,112) on page 129) for the

"'''''''''Y

lib""""

""'rum il-l dn-.<:Iy linked to linear SYl-Items and processes s iuct' it is ion of the autoCO\'llriflU('l! function in the frequency domain. da.....,kal ref('rt'Il(,('S to l>Iw('trRI analysis let us m('lllion Jenkins 196R), I3l'illinger ( 1981), P ricstit'y (1981), and Papoulis ( 1984). Ir ....." ' . su('h Il..'i the bil>I)('(.'trum ar(' described in, e.g., Subba Roo

in 8ft'tiOI! :l.I (Thoort' lll 5.6 on page 115) that the' l>1X'CtrulII for ""'''''" {'an be d(,termiHed by Fourier l l'RlIsfortni uj.; th(' M.llto<'OH("II('(', it :;('t'III..'; ohvious to apply the following l':-;timaU' for

188 189

tile splrurll

Harmonic analysis
N-l

2rr

k= -(N-I)

L:

C(k)c-' ,

'w) S

11".

fuudamelltfli fn.'q Ilt'nci<'S ill (7.1). we have the following orthogonal

WIlNt' C(k) is tile t>:'!timlltc (6.1 ) on page 146 of the autO('ovariftllc(" fUIl("tiOO based 011 N ob~.cnution....; l'\ ... _. Y". If
we 8&UlJle

w- (I
0 < l' = q < Nj2 IJ = q=O. (orN/2forNewn)

that {V,} hfL'i llit' mean O. tht'll we can write IN('.4.I)

It.."

2: sin (w" / ) ros(wqt) =- O.

!
~

.v

for !til p. q

(7.6)

L .'nl:,."I),", '("',,1; ~ { N/2 0< p ~ q < N/ 2

o
1)
(1 = 0 ,
I ~21fN "v" L ,

t"

(orN/ 2 for Nevell)

I"-,

OIl thf'Sl.' ort hogonal rdutiOlls W('

.,,'/2 1

S('('

that for N even, we can write

) ', -:0

which

W(- ('llll

formulatc as

1,(",)

2:N [

(t. I,c"'("")

r (~I,
+

"

L (UpCOO(Wpl ) + bpsin(wp t) + aN/2cos(7rt) ,

N

(7.7)

... X. and th(' ('(x'ffi('icnts nrc giw;-n by

"n(,.,,)),]

IN (W) is called til(' periodogrom or sample spectntfll. and l'ml 1)(> hy either (7.1), (7.2), or (7.3). The lX'riodogram is ddlnf'(1 for all "" in but in ordN to aehiC'\'C' in<ieJ)ClldC'lIce bctw(.'(>11 1,,,(;";) at difrC'rl'lIt v(llIlI's (mor(' about thi!. later), it is fldvisablC' to calculate til(' Il<'riodogrllill the so-cnll{'(l jrmdamrrltai jrequcnMf'.',
27rfJ

.t,.,,,',,ulI.
o .v

00

-'"l': N L I - \'

,,

ap

~~L}'ros(Wpt) .

,, ,,

p = I , .... N/2-1

2 .v N L}'tsin (wpt),

JJ = 1, .... N/2 - ]

(7.8)

11

0.1, ... , IN/2J.

10

It is!K'C1l fmlll (7.2) that tllf' s,a mplc spf'(:trum is proportional amplitude o f tile Fourier trallsforln('(i time scriClj; YI , . , )'"

thl'.
WI'

There IIlso ('xists an illvt'rsioll formula. Corresponding to (7.1)

C(k)

I:

Iv(w).""dw.

Ikl $ N

I.

For k ::ll 0 it is S{'('II thllt I ...(w) ~hOlL'H hOll' the empiriral"(lrianCf' j.~ on /Ill' dilfrrr:tll jnqU('1rif's. It should be noted that alternati\"{' p",",...~ the fl\('"for 2 in tilt' ,",ouriC'r tmn ....form are often S(>(>II in tllt'1itl'rntlln" for throt.gh (i.5); rf. the uiS(n...."joll in ('hnpu'r.1.

tIll' COt'fficil'nt, (lq' call bl' foulld hy llIulliplying eos(wql) on both and subs('(]u('utiy SHunning owr I. Equation (i.7) is fl. finite '''IJ1'f' M niutioll of the oh!.l'r\'ations VI, ... , Y.v. It should bt, thf' .\' obS\'rvutions ('orr~pond eXllctly to IV coefficients ill the 111\' nlH ]llitud{' or th~ p'th lmrmonlc as (p 0:1 -";/2)

II,

190

SPECTRAL "'ALY".

PEIUODOGRAM

191

it cnn be shown that, (ngain the orthogonal relations are applied)

, ",
t

N L..,(Y, - Y ) =
1

-,

",,/2-1
p=1

"' , , ~ Rp/2 +

V",[A("',)I ~ E[A(w,,)1 ~ V"c[LJ(w,,)I

E[I1(w,)1

,N

0N/2-

which is kllOWIl as the Pa,'sl'llal identity. The identity illustro.tes that vuriotions in YI can 1)(' divided in to frequencies, Whl'I'(' R~/2 is tile 'o" trib",,;~

of the vnriau(:c from the p't h harmonic. Oy compari ng (7.8), (7.9). a nd (7.3), it is SCCn that the prriodogram lx- d('termined based 011 Il harmonic analysis by (p =f N /2)

~ { :;' ':
0

{ u2 N

a""2

" " O"V/2 ,

(N

(,V{,IL)

p=o
""O,N/2,
(N ('ven)

p=o

NR'
I,v(wp ) =

8/'

that 11 (..... ) and 8 (wlp ) are ullcorre\l\t{'d. Furthermore, both A(w,,) " 'ill 1)(' :lOrmnlly distributed and. thus, also ind('pclldem. When 011 page 18$ as

7.1.2

Properties of the periodogram

Oy appJyillg (6 .2) all puge 117 we find

E[/(w)1 ~ 271"

N_' (I - ~ Ikl) ,(k)c '"

k

L.,
.(t.'

I)

/ , (",.111, / 11? is a slim or lWO mut.ually independent, squar('(l, ltiIoari,bu", mriahl(':o, nnd. l hus, that. 'N(w,, )47r/11; is dis tributed as.\:l (If f[('('(lom ; cr. Section 2.7 on page 23.
'lpocial ea.'-o('
w('

= J(i.JJ) ,

for N

--t

00,

hav{'

Wf>

i.e.. th(> estimator of the pNiodogmm is asym1JiotiClllly unbiased. Fuocth,,,".. have an important theorem.

TlmOREM 7.1 (PROPE I{I'I..:s IN CASE OF WH ITE NO ISE)

p rj) be ,wrmally dUilributrd u!hile noise lial'ing following hold.J

/,(>t

1l6ria1lCl'

uf.

(7, 13)

that

th(' I'fHiance of the eshmate docs not go to zero fOT N __

;)0,

i) {1(wp)},p=O,1. ... ,IN/2] arc iltdepertdffll.

ii)

Ulima/or i.. not l'Qfuistenl. By usi ng more observat ions we obtain

iii)

-, J{w,;p,.. -,

J( ... ~.r'" E

in thl' perio<iogram. whereas the prccil:;ion or the estimates docs

fI'SUits arC' based Oil til(' assulllPlioll that {V!} is white noLsc. n con~i:-;[en('y probl('m wit hout this IlSSmnptioll. The is 8 Iill(.'/lf (~()lIlbinnlioll of N samples or thl' !1lJ\oc()\'ariallce. \\'c Rlclo,..,y that VnrlC(k)] = O(l /N), lind ;.inc(' til(' tcrm is indu(\t'(l

\::2(2),lJ=/:-O .V/ 2,forN ,,2(1).p= O,N/2.

ctlCft.

10

IWu."'.,,
~.

If lhe M.~ltmption of normality dot's nol. hold, llu-n th(' thl'()1Y"11I i.~ lllJ1Jmrimation.
Proor I f \\'{' imrodu('('
A(wp )

Kd

V"'I" (w)1 ~ 0 ( 1),

(7.11)

,v L1i(()s(w,)t ).

, 8 (w,,) == L 1, ~in(":,,t) ,,

dillt til( variance d~ not go to I'rt} ror N -- x and. thus, 1otI",.m I (w) j.' 'lOt fl c.on.'1i.\tent C.~till1fl/or for the s~trum f(..J). ~f' \1U"iation of th{' vaiUl'!; in Ih~' p(riodogram j.; iIIu.stral~1

192

S PECTRAL ANALYSIS

Ttn: l'ERIODOGRAM

193

20

15

", , ,, ,

" "

10

"." O.,~'i''_______________'~-~;c~.'=~

, ,,
"

" " , , '" " , ' " ., , " ,I " " , ,I I'

,\

,, ,

,,

..-...__..______-.____
0.50
I

0.00

0.25

11'~)

1170

1790

HIIO

1830 1850 1870

1890 1910 1930 1950 1970

Figure 7.1: The pcdodogmmjor the AR(2)-proceS8: 1, - l.Ian',. (lOO obsen:ation .). The smooth t'urvc is tht' thOO7"(.iico.l ~trtJm.

T ime

+0.64 1'/

to l""",~ r
in F igur<, 7.1. In the following section we will introduc(, SOIllt' wh ich will imply COI1... islent estimates of the s p('('t.rum. If we assume a weak dependency (which impli('>; thAt values IU'Ixin.f!, time-distances only nre weakly correlated), tile n Theorem 7. 1 on page l~) be extended to Ihe following. THEOREM 7.2 (PROPERTIES 01-' PEIUODOGllAM ESTIMATOH)

NumlxT uj obsc",ed sun S11(}U 1971t

pt."

halj yetI!' in the JJCI'iod from Jv./le

",,,,,Iifi,,,,,,,,.
Jikoratun" it i... "';)Illetimes sren t hat instead of few) (J(w), respectively) lilt' mapping of / (11) (J(v), respectively) II E [-O.5, O.5J, where ill flppli{'(1. Introduce

f (w)dw = f(l/)dv,
Let

2;;11

{l'd

hold.~

be a weakly dependent 1,rrxess with the spcctmJ demity f(;";I . that

&,",~i"K rl'la/ion

b< tween f(w) a"d f(lI) is obtrun('(1

W=
27t1l

f (w) 27t = f(II).

1(..,;)2"11" = / (v).w = 2"11"11.

(7.15)

i) {I (w p )} ,p =

0, I. ... , IN/2J.

are approximately illdepeFld('7It.

ii)
iii)

W::l
H~:~

EIIPI>rOX,

\2(2),pO,N/2.!orN pven. \2(I) , p O.N/ 2.

( 19~1 ) or S llUIIlWIl,Y (19rill).

,Perlodogram for the number of sun spots)

Eal'l'rox,

shown in Fib'1Jr(' 7.2 is oft('11 S/.,(n in til(' Iil{'ratur('. It sho.....:> a All oh\'ioliS lX'riodi("ity corrct;ponding to about 11 yf>il.rs
in tilt' plot The p\'riodogrrun for tilt' ,;('ric::; in Ihown ill Figun' 7.3 on till' fullowinp; !'ap;!'. Tlw figur(' shows f'xbt SI:'v('ral]wrio<iicitie:; othl'r than til(' II y('ar e~d{'. largp \lU"iahilit,Y in t.ll{' ('!.tillmtt's of th(' pcriudogmll1, it. is "'''''ome whit-h of tIl(' vahw" an' aetnal c~cI("",. In ordl'r lO confidl'nn> iut('l'm] for SOUl(' of tiJr Jl(,llk valu!'s in Figurl' 7.a. 7.2. Sin(j \212}o_lY. - 0.10:1 awl \:.I(2}ow. =- ~i99. \\"{' iut/'rval sho..... 1! in Tnhl(' 7,1 on th(' nexl IlaW' int('rvnl" art <\ f('f;ult uf tlw lantl' WU'i8hilit~ of tht vnrihhlf' h viug olily 2 dl'grf'j'S of t'rrrdow. SolO'
SC'('II

it!

Proof Omitted. $('(', e.g., Brillingcr

~ R e mark 7.1 The theorem Ctlll be applied ill A..'''''iC'S.<;inj( the II1l("('rtainty of tilt' p<"i.. ".~ Especially, it. !'ihould h(> nottod. lhat Var [2 / (w,,)/ f(w,,)J ::::: 22 I. wlwn e ndjlOillts arc lIot l'ollsid('f('(1. ami ther('b,- VflTll{w,,) ! f(;.,;,,)1 = I ('ontrnT'y, WI' huvf' for the (,lid points Var[ I (w,,)/ f(w,,)1 - 2. whkh implifW Ihl' IIllct.'rlainty is lwi('(' H.<; Ifl.rgj' for tht endpoints A..<; for til(> rC!>t ofth('

194

SPECTRAl, . , NAI.YSI.

bOO.S"T ' ' ' EST1\ IA T ES OF T IlE SPECT RUll

195

x 1000
IlO

11.1 years

T he truncated periodogra m
with the periooogram, (7. 1) on page 188, is that it <..'Ou ti ns too wlu('S of the estimated autOCOnlrio.n<..'C functio n. T hus, it is obv io us to
t runcal n i 1'O 'iodog mm
M

GO

" ~
"- 10
0

J '" "'
~

'" "

i
~

i(w)

~
211'
k

L
_M

C(k)<-'' .

AI < (N - I),

(7. 16)

20

M
~

iii tht' truncatio lt point. The t runcat ed periodogra m is a Iiut'a r of.\I + 1 values of C(k), Rnd t hus (Priestley 198 1, p. 433)

'" '" "'

0.091

v"'lir",)) ~ O( M /N).
tinw we haye
0.28 1

(7. 17)

0.5

F'rcqm'llcy (in cycles p(>r year)

Figure 7.3: The pcriodogromlor tile sun spots !;'('I'''ies.
Table 7.1: 00% oollfidefl(:~ inten.>als lor f(v) cQrresponding to the pr:ot mluf'.\ oJ
SI(Il

.) -I L 211'
M

EIC(k)Jc~'

_ AT

2.
k

f: (I - [~)
.\1

(7. 18)

O(k)c ,-,

for N ---

SllOt.

f{w ) ,

fo r AI ---

00,

II an a..'l)'lIlptoticHlly unbiased e;;t im l\to r for f (w).

Frequellcy (cyclc/:ycar) 0.0 12 0.090 0.102 0.1 17

Period

(year)
85.3 ILl

I(v) x 104
3.GB

90%. confidenceinter m l

8.84
5. J!)

9.8 8.5

1.07

1 1.2, 71. tJ 1 2.9, 171.61 11.7 100.81 1 0. 1, 20.81

(7.17) il follows thllt if N _ 00 a nd AI _ oc in such a way t hat for S cx.. the v1Iria nc(' of l(w} will go to zero for N --- 00. Th lls of..;. we hav(> that 1(1.41) i., a CQ11.~isl{'nt estimator' for' f(w). It is vaiu('S of M which fulfill the a bovc statcmell1. All cxample is to p.\". or morc gl'lleral M -= Nt> ,(0 < 0 < I).

~:~',:'Of j{w). detNmincd by (7.16), is a spec ial cu..,;c of t he more

met hods to improve 011 th(' number of degr('("S of frt'{,<iom will be th(' following section. TIlt' mNhod ('oll:--ists of aYl'nlgin~ th(' .. Ix'riodogram with neighhor fr('(IUt'lIdfti for which f(..;) (or f(v)) do too ~trongly.

,im_

f("'l ~ 2rr

, L " C( k )<
(N 1)

N-'

'-,

(7.19)

I {

k [< M

7.2

Consistent estimates of the spectrum

[k[ >

At

(7.20)

The pr('vious sect ion illustrated that tl)(' pNiodogram l.>v (w) is nol It. ~tilll nlor fo r tile "'pcclrUIII f (w) Iw('llu.<;(' l.ll/ (w) coma.in~ -too many" of till' f"Stimntoo uutO<'Ofr('lnlion fUllct ion. In this sN't ion w(' will COIlSi:-;t('1lt l';;limalf"S of the sl)('('lnllll. Tlwreforl'. tll(' varimlC'c will "..deI.. hut at the ('ost of an ill{Tl'f\....(' in bil\.<; (lIon-c('!Itrality).

Illso h(' formulated ill til(' frequency domain and is tlwn called lllirulow. 1'h(' sl)('ctml window is o btniu('(i by taking the Fourier the tilll(> window, i.e., (d. App('ndix (' on I>age 361)

~ {""(At + nOl}
211' sin (Oj 2)

D.,,(8).

(7.21 )

lOG

SPECTRAL ,""~I ".

v..

""," " , , "'1 ' ESTIMATES OF T il E SPECTIlUM

197

Fouricr transforlll of {A~}. "itullng (7,5) from pagt' 188 for C'(k) in (7.22) yields

[(w)
Figure 7 .4: Din'chlce kCflIel. The .fpectrni window for the tnmcalcd p(-riod(l{Jmm.

I:

1,(O)II' (w - O)dO_

(7_24 )

8lDoothoo spectrum can 1)(' obtainro e ith('r by applying weights to all/ocol!ariarlf:f jrmctio1l (7.22) or u.<; fl cOllvolutio7l, (7.24), which f(w } C()rr~pond f; to K. 'Io("lt.lly" weight(>d m(>an of t he periodogralll

wher(' DM(O) is the SO-('(lll(xl Diric.hlet kemel o f oreier AI, ,<;CC Figllfl' 7.4. It should be nO!t'd t hat Vii' (0) has a peak ill zero as well a:s "" "," .d"IU",

~~~:~~:~~, of ..v, ~ most {'Ollllllouly

mi nor pt'aks. Punhcrmorc, it is seen that D 1II (0) llIay lake 1I('gativt' Sinc' lIluitiplicntion with tl)(,' lag+willdow ill (7. 19) ('or r~ponds to i2 t hc frcquclley domain (see next M"Ction). the negative vnlut~ illll.ly f(w) CIUI llPcomc negative. This is obviously not dcsirah\(' i)('{'cUl!,;(' f(w)
non-lIt'gati\'C fu nction .

lwtod,,.,

Ilst'd !I\g-wiudowf; l,UC g iven in Table 7,2011 For prru:-tical a pplications lhc&e lag-w indows yield morc or iPs... the JX'rhap:-; wit h tbe ('XC('pt iOIl of t he t rullcated periodogram, The ar(' shown ill Figurc 7.5 on the following page. window hus 0. [('(:tnngular s]K'('trnl wiurlow.

11"(0) = {M I2'

,1M " 0" 'IM

otherwise

(7.25)

7.2.2

lag- and spectral windows

"""",,,,.dl,ng lag-window is obtained by an inven;e fourier trall.'lform

In this SC('tion "H' will consider both lag-windon's (time domai n) lind v:i"dow., (fn.'<ltlcncy domain). A lag-window is idt'ntifi('(i with a sequence of Pd. whit'h fulfills

~I
2Jf

1",'M
,,/.\1

('lkO

dO =

~in(d-jAl).
(1rk/Al)

(7.26)

I)Ao= 1
ii)..\k

>'.k

iii)"\/o

O,lkl > Al ,
Wf'

~~'~~ru variall('('!; for lIlt' difff'rent lag-windows arc given in lh(' 7.2. It is S('('n that if 0. ..,mall van'anre is desired, then AI Tht, ('OIlS('qU{'IlCC b lUI in('f{'fl..sc i ll bias (non-ccntrality) for the

when' 111 is the irwu:ation l)()int. Corn'Sponding to n lng-window p.0:}

/(w) =
F'unh('fIl10n'
10'.'('

have the smoo/lU'll

".rt._

b{..... ) for 1lip windows. disJ"{'gardi ng the truncated periodogram

. is giVl'II ill Table 7.2. h(w) -

2,

,L

N- ,

>,C(k)e--'.

Elj(w)1

/ (w) '"

A~i!"(W)

(7.27)

-(N-I)

inlrocluc(' til{' "vcclralwtnliow (>orrl'Sponilill)l; tn

of (' i1'l XiVI'll in Tnbl{' 7,2. It is M,('II t Iwl for valu~ of w whe[e
[twK.l )
Illllxillllltll,

,,
L
(.V

Jr.w} tends t o ullder('Stimate f(w), since

(ami opp()~il(' for vnlm'S of...; whe[e f{w) has a [po.'>..-;ibh'

11"(0)

A,,<,
I)

.Ok

that if a .~m(Jll bi(l.~ i.., df.~j,.."d, then ,\1 .~lwIIJd lit laryf'.

198

SPECTRAL AN.A LY".

COS!'lsn;/'iT ESTIMATES OF IIiE SPECTRUM

199

0.5

R('('tangular
Tl.Ikey- 1L anning
P nrZl'1I
~

--- Bllrtlf'tt

" II

~ ~

c-

.,;
II

"
N

"
N

" II

"'~ ~

"'!2
~

II

"

.,;

.,;

"

"

.0

W (f)
2A1

R<'Ctangular
Thkf'),- llallnill,J.!;
PnrM' 1I

--- Onrtlcu

-O,5A1
Figure 7.5: Some lag- (lTld spcclml windQlI"~.

.,,,"
~

. . -. ------ " - - - - - ." ~

~

VI

" - . ~

VI VI

~ IC'I

V,

VI 1\

<Ie"'!

1\

~:::!.

..
" 1\
0

V, 1\
~

"
0

" ::;-1 ------

-" '----"
<:>

..,
0

..:.tl_
0

"
<:>
~

VI 1\
~

~-

Example 7.2 (The general Tukey window) This ('xnmplc illu!->tratl'S thl' relation octY.WII lh(' trllu('al('(l ,.,,';odo",'"'' all important class of windows, giwll h,v the generui l'tu.('.II,t'illdl)fD. lime domain the ialt(>f ('an be wriUen as

'"

---- ---0

" _'N .~ -!:.

~

~::.
II

'----"
~

.,;
.,;

II

-<

-<

II

-<
;
:..

---------
0

"'

-<1..:.:1< ----------'----"
-<
<:>

"

'----"
N

'--------v-----

II

II

-<

-<

-<

II

2(L+2a('os(~~) Jkl~.\1
Ikl > AI,
Wh(>fl'
(l

S il l ('Ilsurps thnt ..\l > 0 for all mIlle;.; or k.

IUld

~
S::

'S

'I'll(' gCIlC'rru ThkC'.v window

~lilllat('

thl' TruncatiOIl point .\1 vi.,ld the

c
2. " Lk=

"

:i >. 0

'" "

.~
5
~

~ ,

.:

.:

" " 0' :

,;

AI

200

CO~'iISTENT

ESTIMATES

O~

TI lE SPECTRUM

201

0'

Equtt>aiffll degrres of freedom

/01- some winllows.

Equivalent dcgre<'l:l of freedom (II)

J(w)~2rr

1 (

L
M
,\:=

(1-2a)C(k}e 'w,
AI

Estimation Tr uncated perio<iOb>T3111 Barlif'l l

iV/AI
3N/AI 2N/M 8N/3A1 2.516N/AI 3.7O'JN/M

2~

It

(
I:

L
.11

Da niell

C(k)e

,(,.,0\: ,irk)

'Illkeyllauning
1'1Ikey- ll am lll itLg
PardI!

+ (1
+a
k

2(1)
k

'" L
_.\1

C(k)p ,...,,,

t
AI

u mlird the equiwl.irllt dq1;rt'C'l of freedom

C(k)e-'tWqirkl )

(see Tuble 7.3).

-aj;

(w

~TJ +(1-2a)];(w)+ah

(w+ ,;/).

n) % ('Imfidcnce inte>rvru is obtained by

where j; i...-; t h(' trullcated I)('riodogralll ddincd b:y (7. 16 ) Oil pagt' 19.'), iliustratCl3 thllt th(> b p('(' (rutn c all iw det('rlllitlf'd citll{'r hyappl.\"iug

I, { \ , (V) ... /2:5 "j(w) f(w) :5,\ '() V 1-0./2 } ~ I -

0',

window ill t he lill1(' domnin (7.29) or hya locn lly wf'igbkd m('an t rw \cl\h>o J>('riodogram (7.:\0). ,.. Remark 7.3 If we scl{'('t a = 1/4 for the gencral Tukey wi ndow W{' get t he T"i",-"a.,", window. and 1;Clccting u = 0.23 yields the Tukcy-llammillg lIIiudOlI'.
p {

"J(w)

\ 2(v1l -0./2 -

< J(w) < "j(w) } '"

- \ 2(v)"/2

I _ 0

'

(7.33)

7.2.3

Approximative distributions for spectral estimates

(7.34)
th(' !;pedrull1 on II lognri lhmic scale, we achieve t hat adding "---" . '" a ('(m~tant to the' smoollll'd spectrum y ields the confidenCt>
Excf('ise 7.4 o n pal1;c 2 12.

From Theorem 7.1 on page 190 we baw that t ile' estimnt('1; of til{' I:;:~~= are \ 2(2)-dislribulC'd. In the following theo rem, the corno;;pomli,;g the ~lIloothcd srwctrum an' given.

THEonEM 7.3 ( P ROPERTIES FOR A SMOOTllEn 5PE(.'Tnu.l) Let thnY' be git'tn a smoolhrd .~pectrum

1 J(w)~271"
k

L M '\,C(k).-"'.
,'(v).

,\1

ror the general 'l\lkey window) window i!' ddiliN ill EX3Illplf' 7.2 011 pi\~f' 198. Prom

that (appruximKtl'i.\')

It holds that

vJ(w) f(w)

v.'["~~)l
VOl
~

2"

Eapprox.

t.he VRriI'Ul<'t' prnportioll (hMW('('1I hiM and variIUU't')

2V

"

, ___ M L.A

).2'
.\1 k

2/_

202
Th(' degree of fr<'NIow
IJ

THE CROSS-SPECTIWM

203 15 10 5 0 -5

b ('I\i<'ulatro from (7.32). WC'S<'t

Ai=
~
l'

-AI

k_

(I
,If

2(1+2acos(~~))2
2n)2

'" I:
!If

(I

+ la

COOl (~~)

30

GO

90

30
(b) AI

GO
100

90

I-

+ (1
~ (I -

2.)1.",,( ~:))

'a) No j)arun window 15 10 5 0 -5

2a)'(2M
2

+ I)

+la

"006("' ~)+1] [1+2~

1+2:OO6G~)) hO
2n)la( _ I) 30

GO

90

30

GO

90

+(1-2a1' a(
= (1
2n)2(2A1
2

(0) .H "" 75

(d)M ""50

= (2 + 12a

+ 1) + ,j il 2{l + AI) + (1 8<t)M + (1 + IGa 2 &,).

Ba)M_.:+- (I

15
10

alld til{' varirul{'(' proportion b(,(~olll(>S

.,.

+ Hit12

!!a)

:~I

60

90

-5

"

,
30
(f) AI
~

60
5

90

(e) .\1 :: 25

The 1Ukc.\'-Hallllillg window is oblaill(,(\ for a = I l l. and til(' prOl>ortion for this willdow h("('orn~
Vnr

1<W)] [few)

11"""'''''''' point AI.

f".,lilllllkJ. and inmc.ated tllliocov(.Iriaflt:e /Tlnction A Parzcn window is wmi.

/01'

va11'ow

t>allit'.$

3M = 4N'

TIlt' f'quival.cut dl..'gft'('S of fr{'('(lom for the Tukf'y-Hanning windull{ is (cnkulntcd fmm (7.32) and (1.36, which u; the ~fl.m( "'tltH' as Tnhl' 7.3 on the prCViotL'-; page. Example 7.4 (Retaining the variance proportion)
In ll. given case we have appli('(i a Tllkey~Hanning window using lb., trOll

poim AJ.r =10. WI;' clffid(' to apply /l. Pur,wll window in:;tN,d and cipt('rmin(' the trullcation I}oint so thltt the variance proport ion same. We delloh' thl' tnu\("ntion point for the Par'l.t'll Rnd tht 1\...y.I wiudow, Mp and .".'r, r(.~I)('("tj\"t'ly. T1w claim for rquru \"RriJUIce impJi('l; the following rdation hHwl't'1l .\/p and .\fT"
~.\II'

:!tad,..,

(Estimation of spt.>ctra) . froUl Olbjer, lIolst. JlIld II06t (2005). We consider Yc - I.:JJ}", 1 " 0.6UI _ 2 ='t. In Figure 7.6 the tbcor('ticnl f11lwi ion. t III' (~t imatro untoc(J\'urianc(' function, Itlld the ~oo"""i.u .. .. ruuc-tion lIIuhipJi<'d by Ii Pafz('n window hnving for ,\I (nuUlhl'f of ohlsermtiom; 1(0), rue 1;110"\\"11. 1Io<1oo...n (whie'l! corrDiponl"i" to FiKUrt' 7.1 on page U12 apart l't'ali ...atiOlI) /lttd til' !;Itlootlwd "p<'<'lrUIll con{'~pondjng to ill Fij.\llrf' 7.G (but for higher valllt'!; of .H) are shown ill thP rl)lIowill~ plIJ{e.

.v
/lull the ciaim ili Ilt'nh~.. uhtAint'l:i for ,\1 p ~ 56.

~""''' ... m

j" Il'-;('(}

(lome!"

!lilt!

to cI('!o,("riix>d lite Iy:lali01U bdl1'({ll two procf'.~.'(' . corr~pomb to thl' ("ros;.-conlrian('{' function in

20 I

SPI:;CTRAL

CROSs.-!:iPECTIlUf.1

205

25 20 15 10 5 0 0.0 0.1 0.2 0.3 0.4

0.5

25 20 15 10 5 0 0.0 0.1 0.2

(b) AI

to as the comriaTlce matriJ: at lag J.:.

defined as the Fourier tram,form of th(' cross-

f.n(w) = 211"
0.3 100 0.4

L
4= -00

'Yxdk)f' ,... 4

(7.38)

(a) Periodogmlll
25 20 15 10 5 0 0.0 0.1 0.2 0.3 0.'1 25 20 15 10 5 0 0.0 0.1

,1I"J. Notl' that in gcncral 'Yxdk) is Iwt an ('ven fUllction and, is a compifx funrliol1. The inverst' r('lation is
(7.39)

0.5

0.2

0.3

0.4

(c) /LI = 75
25 20 15 10 5 0 0.0 0.1 0.2 0.3 25 0.' 0.5

(d) /LI - ,,)()

Foun'cr tmll..~fOF"f/llLfion pair is gh'en by Eqtllttiolls (7.38) and (7.39). forlltnlHl ions arc :sometimes used ill lilt' literature (the factor 211" I cousidrrt'(l for w). Plea8C note, that fxd-w) is the complex of Iydw). proP<'rtit'S of the bivariatc pr()("('SS are dl'S('ribed by thc spectml

25
20 15 10 5 0 0.0

I (w)

(Ixx(w) fvx(w)

IX\'(W)) In(w)

(7.40)

,,

awl

hd. . . ) are
~) ......e

,.. '!Y.\(

the s pectra for {Xd and {Yi}, rcspe<:tively. have

0.1

0.2 (f) AI

0.:1
5

0.1

hx(w)

211"
k

L
-00

00

'Yxy(-k)e ,... 0, = f;dw),

(7. II )

(e) /LI

Figure 7.7: Estimation of the spech"tllll: (a); The JHriodogmm. (b f): "I)('("/nlm (Parzen window).

('oOlplt'x ('olljugatt' of f. TIJt' pr('vioUl; results can be generalized p1Vfy ~'iO. ('onsider I he ntdimensional process {X I.t, X ,1, .. 2 thf' {'(}\'Ilrillll('(' m!ltrix at lag k

the time domain. in thi'! S('('tioll \\'e consider the' ('f~~.spt'(trum fOf stochastic process, while the c:;tilllatiOIl of the cro:-;;..sp(><'tru lII is Se<:tion 7.4. \\'(., consider the two stochu.:,1 ic proccsSt'S {X t } Ilud {\ i} for t b. O. .... Since thc procc&;('s 11ft' 01;('[\'('(1 at the ~alile time in.'"itIUl(~. we bivuriall 8t()("ho..,/ic ImX"(".'.~, {21} {(Xt. }'i)'I"}. 1ft 11(' hi ....arilllt stationary, thell the pro!X,rtk"1-l of the cowuiance for th(' bimriatf' d('S('ribNl b.v a SC'CIU('nc{' of utilI ric~. where tlj(' IiIlltrix III lllp; k is 1"\"\:lk) ( '!) y(kl

r~

h.,(k)}.

i.j = 1. ...

,111.

(7.42)

transform can ht, formulat('(1 ill terms of man-ices

I

/ (w) =-= 211"

,

L
-x

r k(>

ik",'

(7..1;J)

"""txdk))
'h)

(k)

206

SPE(,TllAL ANAL',..

CIlOSS-SP EC'TnllM

207

7.3.1

The co spectrum and the quadrature spectrum

SiuC't' i'.\,\,(k) is generally nol an even function, the cOfl'{'!)ponding spectrum ix>cOlileS t\ complex function

ito."""v (at the frequency w) or til(' coherence S]1CCtnnn is de!ined I (Some' authors r('fN to IUlxdw)12 ns tbe coherency.) It is to interpret Iwxdw)1 as the correlation coefficient between th{' oomponellt s at the frccluen('y w in {XI} HIllI {Yi}, and thereby wc

ixy(w) = 21.,.
,

L
ro
:>C

ro

'Y,n-(k)e-..;k
1
'X>

(7,50)
WlUai

27r ~
k ...

~ ixdk)cos(wk) - i - '"' 'Yxy(k):.ill("'; A,")

-!X>

27r L.

10=-">0

oorrt'illtioll coefficient, the value of Iwxy(w)1 i!; a mc&<mre for the ('omponcnts are Iincarly relnted at the frequcncy w. \\'t' definp the gain spectrum by

when' co, (w) ami qxdw) are the real and complex part of fxd...;). Here, cxdw) is called the co-spectrum, whereAA thl' comp\{'x part opposite sign) q,,()' (w), is called the quadmture spectrum. Pleas<" no\(' CXy(w) is an ('V('11 flluclioll while (IXY(W) is all odd flll1('l.ioli. practice is to plot t he spectrum only in the interval [0, rrJ.

.n

(",) ~

If" (w)1 ~ OXy(w)

Jx(w)

ix(w) ,

(7,5 1)

proportion \ll'tWC'C1l the cross-amplitudc spectrum and the spec-

&be input process. Til(' ga.in spectr1lHl hm; alrcn.dy beeu describccl in y.ill be furthl'r dis('us,'>(.'d in Chapter 8, which considers linear
&y8tl'IIIS.

7.3.2

Cross-a mplitude spectrum, phase spectrum, coherence

spectrum, gain spectrum

L" til(' gaiTt iu til(' !oiystt'lIl from thc input {Xd to t hc ou tput is shown that

TIJ(' polar form of In(w) is determi ned as follows:

Gn(w )
wher('

l1in(w)l,

(7,52)

equals the numn'ical value of IIt(' frequency response funelion;

(4.21 ) on page 71 in Chapt.('r 4.

and
tflxdw)

arctan[-qxy(w)/cxd w)].

tlme delay with noise)

......... pl'tK'f>Nt d('finN\ by

The fuudion nxdw) is called t he crus.,-amplitude .~pt'ctm11l, And cruled the pilose specinJ1lI. il foUows from (7.18) thnt (/J" ". (w) is ",nd,"I<,rmI from all IIdditivc multiple of 27r, but. usually we defin(' .,d..v) ill til(' i {. 7r, 7rj and the vn.lul' is unique by defining tP.'~ (w) as the anglt' (in till' . [-7r,7r]) b('twt"l'1l the positive regioll of the eXl' axil> and the lilll' tlmt 'l.('rO wi t h the point (r ,y , (fXY)' Tht, cross-ulll pli tudc 8p('('tI'1II11, nxy(w}, may b(' intt'I'\}I'ct('(l as the p,ooud of tile aml'litlld('.~ of th(' components at t he frC<III(>IL('Y w in {'~I} P 'I }' Th(> pllll,<;(, SPl'Ct l"UI11 lIlay bl' iuterpreted as thl' IIH'rllRl' dlflTll/f an l){'tw('('11 th(' sam(' eompon('nU;. F'urthl'rtllorc, th(' COTllfJk;r co!tcrulcy is defin'(\ h.v

Xt

-' !"I.!

1-, =

XI

+ E'l.t.

(d E No)

')'xY{k) = ('ov lXIo}o j

d
t2.H"j

CovIt" I.I ,tl.l_dH

wn( ...... )

-{;:

208

S PECTIl..,\L ""'L1,.

I EST"",mON OF '1'111::

C ItOSS-S PE(,TR U~1

209

Estimation of the cross-spectrum

,\' observations of tile bi\wiate proces

{(~:) } , t~I,2,.,N.
in Section 7.2 on page l!J4. a consistent esti mate of the spectrum ":~~~:: ~pl't'trum) is obtRined either by u weighted estimate of the .. fUIl('tion or as a weighted integra] of the pcriodogram. Both ('lUI be applied ill estimating the cross-spectrum. /irst mt'thod is b&';('d 011 the estimated cross-covariance fimction. C u '. ~timat(' of the cross-spectrulIl is found by (AI < N)

i.t' .. we obtain

Co-spectrum:
QuadmtIH'W' .~I)f(:/rul1'l:

I
271"

L
II -!If

AI

A", Cxy(k)e

,w,

(7.53)

F'lLrtilPrmore, we obtain
('1l),',NUllplillld(> spcdnJlII:

J'O (w)

211"

L
1< =

AI

"kC,\,) (k)cos(wk)
M

Plia,sc ..prciru1Il:

In ord{'r to determine t he rohen.'Il('e ~ p<'<'t r\lm and the' gnill. th(" for {X,} and P~} bav(' tu Iw cakulntNi. It is S('('I\ llml

" k=-M cxdw} - iqxy(w).

-'2-

AI

AkC X) (k)sill(wk)
(7.55)

,xx(k) ~
Ilnd tllil:;,

{
0

q,

"'>do... {A~} and trullcatioll point M are chosen

for k 0 olh(>t"wiS('

1i

Ilk) = { o

2a:

for k

-0:

of Iwn,'l88,

f\S for the (aUlo.)specSection 7.2. From (7.51) a nd (7.55) it is clear how the co-spectru m. and t h{' quadrat.ure spectrum, qXy(w), are es~ im ated. on f\"dw ) and q\"dw) the estimates of the cros:Hunplitude and ar{' foulld by using t he theoretical f{' lal ions

fX(W)=~f_
"k

L
""

.....

,..\ .\ (k)t...;k=~~

:2

fd ...:) _ :L
K

q'

(7.56)

Tlwreby

WI'

gel the

("QII('"

n("f~JH'("t1"um

(7.57)

Iw.\dw))
!lne! t 1If' 9(111

we 11('('(1 em estimate of the

G n (w) - .'-- " 1

rhl' cuhcrcncy rt'\,-aL'I that tilt'n- is nut a totaJlilK'ar depe,oo."", duc to tht' ltoUil> ('Olltponent {f"2,,}. The gain shows that aU ..... W>dlsUubed through .be _ . Com.... with ExA..plo 4~

.,a e
..1..0 2

, ,

iii,> (w)

~ r.,;,l~ xby);:(w~)=
,jlxy(w)j,(w)

(7.58)

" ,

(7.59)

210

SP I~CTRAL ANAL1",

211

A lag-window, and Clipccially the truncation point, should ideally be so that it hmatches~ llit' decay in the co\'ariancc fUllction. Thus. it IIlRudatory that the same truncation point is u!>cd in the Clstimation
cn:lf,:O-!>I>CC'trUln as in tht' estimation of the (auto- hl>('('tnull. The estimate of the CroSo:;-8PC<:lI'UIti will he biased if the IUflXitullrn cfoss-covariance futlclioll is not dose to lag O. This bias lIlay 1)(> uliing a time aJ.ignmclIt of one of the time series so that the 11l1l.ximulll cross-covariance fUllction is I\t lag O. The second method is based on the cross-lH'riooogmm, which is d.,~ ..!CI
IXl "..v(w)

that tbe spectral density for {X,J can be writtC'1l

a,.<;

JAw)

2 (12 s iIl (kwj2) =....L 2 .

2rr k 2 :sin (.,;/2)

D~(w) = siu(kw/2)
ksin{wJ2) a Diri('hi(,t kcrnel. This function oC('IlT!j for examp\(' when wOl'king "'"",01,,, time windows. that if...; .... 0 and k --- no in such a way that k...; con!,tant. till'll:
811

= 271"

N-\

Cx),(k)('- ,,,,/.:

-(N-I)

D" () uJ --

'in(k"'/2) kw/2

As in Section 7.1

011

P8.g<' 187 the cross-pcriodogralll may lw written as

Consider the r('('tllngullU' window in til(' lillw domain

f .y.,(w)
~

2;N (t x,e-"') (t ",e'"')

t
I 1=1

niT g(t) ~ { 0

for

It I ::; T

othcrwise

{x (w){i (w),

,. .pn",i.oo for this window specified in the frequcncy domain.

wherf'

b'(w)

_1_

.j271"N

L
t= I

Xtc - .... t

domain the general Thkey window can be written

and {;,(w) is ttJ(' complex conjugated of {y{w). As for the expression in th(' time domain, (7.53) on the previolL'i cun cIl\(;ulatt' til{' cross-sp{'{'trum by

Ikl,; AI Ikl >AI

Find the vn.ill('s of 11 which ensure tlmt
).,1. ;:::

0 for all valUe:! of k.

where WN(O) is the spc<.'tml wiudo,," corresponding to thf' lag-windOW' sec Section 7.2.2. Now the estimntes of the CO-I)pectrum, ampli tudtetc., can eusily be found lL;; deM'ril)('(1 for the fir.;t mt'lho<i. For a detailed description of the properties of the e:stimf\tor.>. ~ (1981).

1 - 2.T
k

L
. .\f

CkCO.I(uJk).

w=rrj/M,j=O.l,,,.,/I!

tlw l';1){'("traJ e,liumtc can IX' writlen as a moving average of trun('at('(\ Jl('riodogram llsing the general 'J\lkey window. tht' TlIk('y-lIanning window is ohtuillt'(\ for (I = 1(.1 while the window is ohlllilled for a '-- 0.23.
fI(X'('i

7.5

Problems

nun is Cl:;timatcd as

Exercise 7.1 Question 1 Consider tbe 1II0villg av'rage

/("')
Determine' thl' \1I.riance relation
,"rian" nlation for tllf' 1\lkey-i-lanllillg wiudow. and 0(1111)<\1"('

x,

!( +/

1+ "'+(:1_ (0\:. ~

;.
CIIl ptt."W I!Y.J.

wlwre {Ed is ru;...,uJlI('(1 to 1)(' wilit(' Iloise ",.-jth Illl'an \ruuc 0 aud

212

S PECTRAL AOA.L\1I

213

Question 4 In 8 particular case a 'lUkey-Hanning window with /11 40 bC(!1l u!;t.'(1. Now 8. Parzcn window is want(.'<i instead. Determine fo r value of AI the variance relation is unchanged.

Exercise 7.3
For procCSSC!S in continuous time it is well known that thE' autocovaria!l<'f' tio ll '}'(7) is a continuous function and the spectral density can be d(,l<-nllbo using

few) = 21

100
; 00

1'(r)e-'...... dr

-00

271" An example is the Bartlett window:

For continuous timt' processes the lag-window (the window in tlJ(' lim(' is a continuous function , >'(r), and the window in the spectral domain by 1 k(w) = '\(T)e .... rdT
'00

,
,,

Let T = I and find the corresponding spectral window. Exercise 7.4 Figure 7.8 shows a smoothed est imate of tbe spect rum of an AR(2) with 1 = - 1.0 and rh = 0.5 based on 400 obscnlltions (simuiat'<I). A window has been used to smoothen the estimate, alld the corr{'ialioli is considered ulltil lag 48. Furthermore, the thooreticai spectrum i~ Deduce w he t her t he theoretically s pectrum is insidc 0.11 807t interval. ( Hint: The dblance between 1 and 2 0 11 the y-axis iog lO (2) - loglO(I) ~ 0.3010 on the logarithmic scale.)

.,
l
I

, ",
t,

IJ"

,'\

,, ,
S

"",,,,poaI

IlS

lS

7Jw. ulimate4 ttmoothcd spectrum/or' tM AR{2) process is shown with

. Tht .(olid li,u; OOJTt'1l11Onds to the thco~tirol spectrum.

Exercise 7.5 C-Ollsider the bivariate process

X II = f" 1,! Xu = PIX 1 L + {JlXI.L
I

+ f"2,t

where {f" l.d nnd {f"2.d Me mut ually IIllcorrclated white noiSl' P'("""'~
variance

q? and qi, r('spectivcly.

Question 1 Find til(' cross-covariance fUllction of the pr(){"('l<s. Question 2 Calculate the co-, quadrature-, cross-amplitude, lind p h.~"' lIP density fUIl(lions. Question 3 Skf'tch the cros.<;-amp]itucle and phase spectral d{,lIsit~ and .h/rJ, 1. Comment 011 the resulls.
fUf

CHAPTER 8

system s a n d stoch astic processes

5 it WfL,) shown that a gC'llcrai linear process eflll IX' characterized as systelll having white lIoise 11." illllilt. In this chapter. we will collliicler where thl' input is not llcct'SSarily white lloiJ;(' hut, in general, a IJ' ",,,,.,,,,,,';, Jlr(X~. 'The Chll-l>tl'T is based on the theory discus;('(1 in btu also 0 11 the' discussion on linear dymllnic sYhtems presented in

..10-<""

we will restrict our attention to ~illgle-inpl1t, single1I10dels. In Section 8.5 this will ~ gC'Tlcraiizcd to multi plepUl (~ II S0) models. Chapters 9 and 10 consider lIlultivariate proN'sSf'S, which con t ain lIIultiple-input, multiple-output (r-.U)'IO)

fiCMt

~'<'tions

till' introductory (,xl~lIIpk'S in Chapter I, the case on page 3

thE> indoor nil' tl"ltl lwfIl.ture in a tcst building relllte;; to the types of
I ill this chapter. Till' Olltput is the indoor air temperatu re an' the h('ut supply, t l1C' ambient air tCIIlIX'Hiturf', and the solar

~'YP<>' of' """,It,I,,,,, e, '''''''"Iy described in Ljullg (1987), SOderstrom ,and Johansson (1993). A classical reference to in put-output r~ tionl modeling is Astrom and Eykhoff ( 1971 ). All introduction in C'hnt.fidd (2003).

...

II-!tIonsl'ip between input a nd output processes

Ii

.~inql('-input, sillyie-oulpltt system (5180 sy.9lem) witholtt

X(t)

Y(t)

Syste~
Input Output

....... j""',. . ~i"gle-oulput lIy.~lt'm without f10isr

that tht, s~'sl('m is linco.r and timf-intJaritmt; cf. ChapIN I.

216

LI NEAR SYST EMS AND STOCHASTIC

IPI\"c, ,,,,.

dEL,' TIO'SII II' I BETWEEN INPUT AND OUTPUT PROCESSES

217

S uciJ a system in di8cIl'le time may, according to Throrem 4.1 On pagl' be cham('tt'fized by the impulse respollSC fmlction {hd. !mel tIl(' Outmlt obtai ned 6S thc convolutio n

Equation (8.2) foll ows dir('("l.iy. ~:Cl uati on (8.3) follows from
"i\

dk) = CoviYi, l" +kl

)'1

L
' =-(X) oc

h.XI _ i

~cOV [,~~ h.X'_ .. , f:~h'X'+k

=

,]

&peciaHy for causal s)':stems,

L L
- 0::..) _ 00
-00

00

h,lt) CoviX t _ " X' +k -J)

Yt =

,'"

L h,X

I_ i ,

L L
oor - oo

oc

h,hJ''Yxx (k - j

+ i).

which dClHom~t rates, thM t he present \'a!uc of t he output d(){'S not future values of lhe input. Furthermore, we will !,L<;.'mme that the system is stable alld that {Xd not depend O il OJ}, i&._ {Xt} is an exogenous variable. The tfl.<;(' {l'd hus feedback to {Xd will be discussed in Chapt.er 9, which d('alll multivariate tillle series.

d"'''',<I .

(SA ) follows from the same arguments: The slationa:ity fo~ iows frOIll

that

(\](>

system is stable nnd thnt t ile IIlput process

IS

stationary.

8.1 from (SA) t hat if {XI} is white lIoise, lhcll

(8.5)

8 .1.1

Mome nt relations

-::;:' of the crQSS-covarian<."C fU liction mny he applied as all estimate II rcspon.-.e function, conditioned that the input process is white

We now assume that. {Xd is a stationary process, and lhus lh{' M'('olld represelliatiOlt for {Xd is given by the mean \'alue ILx and the function "'Ix..\(k). TIl(' following then ho lds.
THEOREM 8.1 (~ I OMENT RELATIONS- DISCRETE T I Mf~ SYSIEM) TIle outputjrom tile stable system in (8. 1) is a stationary pfl'.JCt'H Pi}

.....,,,,, time we get the following.

8.2 (~ I O\fDIT RJ,:I.ATIO NS COi'lTINlfOUS TIME SYSTEM ) p rv<H!; {X (t)} be a ,9tali01tary process with mean J1.x and au

lerued by /Jle second moment representation

~ II" II" (T). Till" output from a continuous, linear, time-invariant,

8ptt'm ,

u-ith impulse l'eslJOnse junction h(u} and output {Y et}}, is by /JIC second moment rrprcsentation

'=-00
1>'Y(k) ~

L L

00

00

h,h,1xx(k - j

+ i).

"'" -00;=-00

"'In -( T) =

Furthermore, the cross-covariance junction between input and output II by "'Ixy(k) =

the nl).~s-C01Jaria1tce jWlrtion between input and output is given

L
00

hnxx(k

i).

')Xy(.)

I:I: I:

Ity = I'x

I:

h(u) llu ,

(8. 6)
(8. 7)

h(u)h(l!)')'II"X(T - v + u) dudv.

h(u) ')'.\\(T -lI)rltJ.

(8.8)

. FOllows frOIll t h!' SHill!' arguments as a.p plied for Theorem 8.1, (2003).

2 18

L INEAR SYSTEMS AND ST OCHAS'! J(' "1<,0<' _

" ' "lorISl IlI"

OJ-~TWEEN

INI'U T AND OUT PUT 1'1l0CESSES

219

.. Rema rk 8 .2
FroUl Th('()r('IIlS 8.1 and 8.2 it follows tbat if the input pr(){'('ss is
1\
I

.,>o"d;",gl:"., we hnve the following relation bctwff'1l the cros:;-~w'('trurn, the cross-covarillJ1('{' ftllwtioll, 1.\\'(k):
.~k

proce;.." then the output process will also be a normal proc~.

8 .1.2

Spectra l relations
FrOIl!

(8.15)

Let liS fin;l ('()Hsidpr systems in discrete time. thl' system can be wriLtCIl in the z domain as

Throrem 1.7

Oil
~ ~

'"" h,..,\ \ (I.: - t)e 2rr A: '"" LL-00( __ 0<; where N(z) is the lron.tller /tmdion of the system. Equation (8.9) '",""M to til(' ool1\'ohllion (8.1) 011 page 216 in the lime domain. Dy 00,,,;<1,.,;';' definition of the z-tnUL<,fonn, it is set'll that (8.2) can be writl(u as

~ ' " h/c ,..,f "" 1'.\x(k 2;'1" 6 L (>S_ ...:> k, 'XI

t)c ,.... ( A:

I)

that tilt' n o;..-;,-spl'('tnllil bl'twl'Cll in p ut Hml Ollt.put is giv{,l ) by

If
WIi'

illst('ad choose to ap ply the frequency /'esporlSe jr.mctio1l

(8.16)
_ noi,H' t ht'n fxx(w)

1t(w) =
tht'll (8,10)
('1111

L
kc
-:)0

hke-I;oJk = H(c tw ),

ol/2J'!', IIlId
(8.17)

bt' written
/-LY = 1t(O)J.lx.

According to D{'finition 5,6 a ll page 1 15, we hn\'e t hc follow ing i)('lw{,(,11 the slX'Ctrllll1 fn-(w) and the alllocow!.rianC'{' fU llelioll "I )
I = . " 1n(k)e- .... k . IYd..u) = 2. Lk= -00

f'Stima te o f fxd..u) (pO!>~ihly lIonnHlized) call bt' applied as fln the frt'< llll'ncy res pOllSE' function 'H(w) = I-I (e""). prl'S('nt the oorr~poll ding rNlllllil in continuous time. Accord4.16 o n pug!"' 92, the system can be formulated ill the s domain

Y (s)

fI (,)X(,).

(8. 18)

If we substit.ute 1"n(k), g1WII b.... (8.3)

IYI(w)

the tram;fN fUllction for t he system. The frequency responsc

OIl

pag' 216, \\'(' ohllli n

1i(W)

fI (iw)

(8.19) (8.20)

II d"')

H {<w) lI ( '''')Ixx(w).

(8.21 )

betw('~:on in p ut and ou tplll is givell by

which

('xp r ('S.';('S

t il(' rf'lation betu'ccn the spcrlntm 101 til( illPld

th(' QuIJ}tI/ pm("('s.~.

Ix d..u)

f-I (i...J)h .\"(w).

(8.22)

the ,~p('('trom

fUI"

Equat ion (1'1. 11 ) Ihlls "'O",,"pe

E<llllitioll (H,7) 0 11 t il(' prt'('t'( litlg 1>811;('.

that til(' abo\"(' forlllula.,; ill COlljllllCl ioll with the formulas ror syst< 'IIlH in writ,s, parall('I, and f('('(lhtl<'k (Theon'lIl U 1

220

L INEA R SYSTEMS AND STOCII ASTI C "1<001.,.

"lIT'"' ' WITH MEASUREMENT NOISE

221

X(t) System
Input

Y(t)
Output

L: L:
'In (k) =

h,h,'YX x(k - j

+ i) + 'YNN(k),

(8.25)

hi'lxx(k - i).

(8.26)

Figure 8.2: Single-input, .single-ou/put system with floue.

X t and Nt arc mutually independellt we get

page 85) arc useful in calculating the theoretical spectra for mort' systems. Finally it should be noticed that Theorem 5.7 on page 117, whkh is determine the spectrum for an A(u' IA process based 011 the tran;;f('T is only u special CIL'iC of the above, namely, where the input pro(>('Ss ill
011

7Yl(k ) = Covil-'/., l',+k]

=
=

lIoise.

L h,X + Nt, L + Nt+k] , , , LL , h,hj CoviX Xt+k-JJ + CoviNt NtH],

Co\'[
t_
hJXHk - ,
t _"

The frC<lucncy rt'Sponse fUllct ion, which is a com plex function, is split into a modulus and u.rgUlllcnt

'lxl,( k ) = CoVi X t , YtHJ

= Cov [XI>
where G(w) and ~(w) are the gain and phase, respectively, of t il(' , the frequency w from the input {Xtl to the output {Yt}.
=

L h.Xt+k_. + NtH]

II.X Hk
.]

Cov[Xtl L

= Lh;1'xx(k-i).

8.2 Systems with measurement noise

.. Now we consider a system where the output is affected by !loiS(', as ' in Figure 8.2. Apart from the observation noise Nt, the syst<'m ill corresponds to thl' system in Seetio .. 8.1. Since we havC' added .\", system, we get

~~;.~ .':I:'::"::;lc_hanged.

with T heorem 8.1 on page 216

WE'

see that the formula for the

spectral relations are gi\'en in the following.

SYSTEM WITH MEASUREMENT NOISE)

( SPECTRAL RELATIONS-

{Xd, {Nt} be spceific(i as in Theorem 8.3. Then we have

In (w)
lI (c-'-) JJ ('~)lxx(w)

l'/=

L
-~

h;Xt_,+Nj

+ fNN(W)

(8.27) (8.28)

= G'(w)fx dw)
f.n (w)

+ fNN(W),

\\'c assullle t hat Nt and X, are mutu(1liy wlcon-eiatcd stationa ry processes and we want to examine the noise influence on t he ''''I'UII-oI relations from Sf'd ion 8.1. In t his sed ion we will consider only systems (and prot('S~ ) in time. In thb en.<;(' the tTlOIl1l'lIt rdations are w; follows.
TI1 EORE~1 8.:1 (r.. IOM I-: NT RF:l.ATIO~S- SYSTEM WI T H ~!EAS t , " '''''<T '' Let us COflBidcr a lined" system with measurement noise (1.$ . " .cifi"" $in(t' {Xc} (md (."'d an' a..~.~ltmed to bt- mutually IInconliatrd

lI (c'W)fxx(w) = 'H(w)fxx(w).

(8.2D)

thp same arguments WI in &-ction 8. 1.2.

applied G'(w) = 'H(w)'H(w) lI (e-) II (e'W) It is thus sccn !loiS(' is prcscnt, it hold,!;
'H(w) f'O'(w)

I xx(w}

(8.30)

222

LINEAR SYSTEMS AND &'TOCIIAS-, Ie P'Oc I.~.

. ..,. .TII',CAT'O", OF TRANSFFR FUNCTION MODELS

223

(for a.ll w where j!(x(w) =f:. 0), i.e., tilt> frequency res]JQnsr jWlction is the between the cross-.SIH'cintm arid the spectrum for tile irlrld prrx:r,~,~, This i... of the lII()!,t important results within til(> field of linear s)'btem.s as it allows the est imation of the gain ,'nd phase of the systcm hy sUbstituting " ';"'... for ixdw) lind Ix \(1.1.1 ) in (8.30 ).

<,

OlD) olD)

tV,

XI

8.3

Input-output mode ls
Dio.fJfOrII for

..,(B)B' o(B)
Ih~ tmrn<ft'f'

Y,

The model (8.2-'1) mny coutRin an arbitrarily large numb<>r of parflmctcrs, and Jenkins ( 1970/ 1976) hn,'c, in mllllogy to the AlBIA mod('h;, writing tltl' cau~al vcrsioll o f (8.21) on page 220011 tJlP form

jlmc!ion model (8.37).

}',+OI}'I. _I+ +OrYt_r =WOXt _I>+W1 X t

l W.X/ b

II,

,+ .. ,

.+N;.

simplify 1I10dt1 (8.37) by ~t.' tt ing 6( 0 ) = ~(13). Tlwr('i}y we obt.ain model (Autoilcgr('S;ive-to. loviIlK Average wit h eXog('llous input)

where b is an intC'gcr valucd time delay. In this section we will consider different. types of input-OtltlJUt namely, tnmsje1' junction models, difJl'1'ence equation f/uxlrls, and oul'p,,'.'" models. First, .....C' give 8,11 introduction to t he ltI (o t gell('ral model, lht' fUllction model.

,,(B )Y, ~ w( B )X, ,+ O(B )<,.

(8.38)

...,(13) and 0( 13) are p'lh, 8't h Wid (I'th order polynomials. rt'5pcctIN- ..hift operator B. Ucrc bis..a.lI illleger ...-n!UM tiwe dcbw. CfL'I('!'! of till:' AR~IAX model arc the ARX model (0( 13) - 1) find Illipuise Rcsponse) (IP( B) = O(B) = I). The Allto. lAX 1I10dei i~
tt'ft'rrro lO us the CAn~IA model (Colltrolled AR~ I A).

8.3.1

Transrer runction models

\Vhen we consider transfer ftillction modc!s, \,\1' ItSSUIllt> that the noise. be d(.'S('rib('{1 by all AIU\IA model (possibly with seasollal compOIH'utS).

"(]IutP",t error models

N'TlJr

modd (DE modd) il, a model in the form

w( B) YI = 6(D)Xt b+tVt
(8.39)

tV'+-PltVl I +"'+'r'p,vt p=~1+81~1_1 ++ OqCt-q

By introducing

orB)

I +0, B+,,+o,O'.

w( B) wo + WI 13 + ... f w~ B ~, .p( B) = i +.pl B+ +'r'pBP. 9(8) = 1 + 0113 + ... + 8'1 Bq.

we call write (8.3 1)
Ill)

ill

Ill)

modd for the noise term. The Illodt'l is most. oftcll applied

hy thc output error Ill('lhod (O EI\I ). This will b(' c1abon\lcd ill
OIII~timation .

IIntifiica,ti<m of transfer function models

..,( 8 ) " Y, ~ 6( 8 ) B X,

+ ~( Oj ,.

O(B).

.".;"" of an illpm--output model (tran~rer function model ) is dOlle pmf' rdwy. First , fl transfer fun ction from { Xd to {ytl (the ...".'" COIllI}()IlC'Ilt) iii ici('nlificd. flud then 1\ model for the lloiS<' is

si nce tV; = o(B)N r The polynoliii:!ls (8.3G) lIIay he t'XPflll(kd 10 S('!\."IOnai compon('nts. e.g., OrB) lIIay 1)(' repiacro by 0(13)8(B.'i). -rhe weB) Bb / 6( B) i.~ falkd thr Imn.~fer function wm1JOllCnt. Box and J enkins ( 1970/ 1076) rd<'l' to this model a .. a t1Yul-~f('r model. 'I'll<' term tromfer f""r-tioll i'i often uS('(i for linear ~ .... st('111... in 1'1 domnin. The 1ll000i('1 is abo r('fprr('(1 to <I::i til(' no.-.hnki7l.~ 7IIodd.

'10".", w(' will considt'r single-input, Sillj.\ll'-OlllpUI Imllsfer funct.ion

::,:~~ 8.5 lIlultiple-in put syst('IttS will 1)(' ('onsici(orro. ~ for iti('ntificatioll of a callsal (physically rmlizablc) lill('ar tOlkIWinJ,! b til(' Box-J('nkin..~ approach. Notice, thlll it b fI.'..,.,ullwd and Output S('ri('S art: 1llC'!UJ-eorr('(h'1:l.

224

L INEAR SYSTEMS AND STOCI I;\ST IC "I<OCE,..

.... , .' 1'IO' ICA1'ION OF TRANSFEH FUN CT ION

~IODE L !:i

225

Let US consider an input-output model of the type (8.31). The fin;l I is to identify T, 1:1, and b. This can be done based all the cstinlltted respo.nsc fUIl(,~l, since, after such an estimation, we 8elect r, 8, and 1I!:io we get 1\ description in t he form

illlpulS(' re.pouse

h(8)

hlB) ~ w~B) B'

51B)
The estimation of tli' impulse response fUIlction is based on th(' foct that i input is white noise, thell the impulse res~ fllll.tion can2~~ ,.".. " . "'!!! on the cstimat~ -.ross-ro, u niance functjon: (8.5) Oil page . III olher words, lilP impulse response fWlction lIlay be csli llla t('(i method called I,re-whitening, i.e., based on "pre-whitening" of the input .

~I
o
k

cr.

~I-'---------k o
Wo

foc
a) A suitable A1U,IA model is applied to t.he input series:
~IB)X, ~

9(8)0,.

wll('rl' 01 is white lIoise.

b) Wc perform a pre-whitening of the inpuL serie!>
Sum,. example! of tile tnmsjer function:

c) Till' Output-series (Yd is filtered with the same model, i.e"

""""''' JlHldwn h(I3).

a t_I.

Md taking expected valuE'S yields 1'od(k) = hk")'c>o(O) = hkO"~.

d) Now the impulse response /unction is estimated by

h,
Proof It holds that

C.d(k)(C IO)

C.dlk)(S.,

. . of r , 8, and b is built o n (8.40) und the corresponding relations. Some cxamplcs al'(, showll in Figure 8.4. a time d(']ny, b, the transfer estimate will start at k = b. Unving 8, and b w(' can est imate the r(,IlLIliniug parameters in the model

wi B) B'
5(D)

X,

+ N,.

(8.41)

and it follows

p,

9( 0 )-'",10 )1';

~ 9(B) ''''I B)[h(B)X, + N,j _ hl B)a, 91 B) '"IB)N,.

_thod of moment.~ or t he 1)n:dictio71 error method. t radi tional ARMA ~ in C ba pt{'r 6. Aftt'r t his id{'nlifkation, the total model iR

' ~::;~:~~~;" {Srd, . . 'C sub.sccIU(,lltly estillia te a

226

LINEAn SYST EMS AND STOC HASTIC PItOC'",.

" 'UUf]P'LF.-IN'l'l'''I'' MODELS

227

.. R e mark 8.3
The BoxJ c nkins approach de&ribed above is not adequate in thp 'oll! ow.
si llla tiOIlS:
C'()rrl~pondillg

Fouri{'f trall~ro nn atioll hi a lincar Opl'mtor, it is st'Cn from (8. 13) spectral relations are

Multiple-input system3. In these cases the ill l)ut pr()("('S.'>l'S arf lnQl,l correlated , and the corrdation between the input proc('S.'>{'il makf'b t il(' pre-whitening method wery problematic.

(8.44)

H! (w)fx",x.(w)
irl]Jut 111"OCeSS is riot of the A RAJ A type. In th is ('n,w 11 0 f('l\SO rmhl(' model can be fotUld for til(' input process, and hence the d{'l;('ri bcd I for prcwhitening can not be used.
Procedures for ~ lilll 8ting the impulse response fUllct ion... in lh(~(' dt'SCribOO in Section 8.5.

+ .. . + I-J ... (w)fx ... x ... (w)

(8.45)

I xy(w) ~ I xx(w} H (w)

(...;') b all m x m spectral matrix (8 straightforward generalization of pag(' 2(5) , a nd H (w) is a vector colltainillg til(' m frequency responS('

H I'"
th81

.11",.
(8,46)

8.5

Multiple-input models

Until HOW we hayc consid ercd sing le-input, sillglt'-Ollll)l]l lIIod('L'i. consider t he U1ultip](,-input. si llg ]t'-out put (~ II S0) model

- n framework fo r estimating t hc Lrull.sfcr fli Belions bast.'CI 0 11 of 01(' (alli,,)sp('('tra and cross-speclla.. Equa lio n (8 .46) facilitates a domain bast"d method for finding the impulse response functions for 1 S)-stl'IIIS. By ui>ing t he ilwen;e Fourier lntnlifo nnation . Wl'ilre til(' impulo;(' r<'Sponsc fUllction from {X . d to {Yt }:

l'j =
'

L
.:x;

hl,.XI. 1 _.

+ ... +

h",.,Xm.l ,

v,

2'

j" .H. (...;)e"-'Ird..J,

k= .... -I,O.I, .. .

(8.47)

where {X u} .... , {X,,,,d ar(' 111 'xogenous) inpUl vnrin bl('R. ",hidt SUJl1('(lllot to depend Oll p . .tJ. Th(' observational noiS!' pt()(.(,:-;... {N,} ''''''''. to be mut ually uncorrelatcd with a ll Tn input variabk'li.

......,;fica.;o," of multiple-input models

orIId"";"'g method d('s(: ril:wd in Section 8.4 on page 223 is vcry usefu l hilt it is not suitable for multiple-in pu t. systems. In t hat u.....' tht, rrC'<lltl'llCY domain met hod d C'S('ribed in Section 8.5.2, i.e., t::,tinmtC'<\ :-'I}('('tral matrix i x:dw) aud thl' ~limated vector of /'0 . \\'1' find a ll <'S timnte of th(" frNlu('nry response function as

8.S.1

Moment relations

tl llIhiplyillg (8..12) by {Xu}, ... , {Xm.d and taki ng th(' ('XIWl"!lItion the moment reJal iolL';
~

H (",)

(8,18)

1' x ldk) =

L
''''' 00
~

II1 'Y:r,:rl (k

i) +. .. +

L ,=

h",.,").r .r,~ (k

;)

'IX;

illlpuis(' r(''''!)()IIS(' fUllction i!; foulld by (8. 17). . til(' illlpuL'i<.' f('SjXl Ilst' fUllcti on can be found by linear rcgrt~'> ollly til{' first. q valuC!i of the illl]lIlISl.' ]'{'!;POTlS(' function al'C of

\\'(, haw
~

O,\.,(k)

.~

li L

,,,),,,,,,,,,,

(I.' - i)

+ ... +

;=

hm.{'I.r ... .r ... (~

;)

h!(B)X u
~IO."'"

+ ,.

hm(B ) X",.1

+ Nt
(8 A9 )

+ IlllXu 1 + ... + h1q)(u q + ... + h ...o_\".1 + "miX.... ,. 1 + ... -+ hrnqX",.t q + Nt

+, ,

which is !-o('('11 to bt- a gCI\('ralizlllioll of (it I) on pag(' 216 and (h.2(i) on

228

L INEAR SYSTEMS AND STOCHASTIC PROC......

229

which is a general linear model with

Moment estimates
dominMing the impulse response fUllct.ion, h. u can be used to the equations that are used to determine the estimates for the , ill t he transfer function component. Hereby, we obtain the so-

xT =
8
T

(XI ,f,"" Xm,l_q)

= (h lO , .. , h",q)

Ct::::

Nt.

Now the impulse response values can be found using t he LS m('lhod Equation (6.37) Oil page 160). Given N observations, the linear relation in (8.40) can be written for observations as

r'

,,:::~~~;{'l;timatcs.

,we have

Y = XO +e
where

= 110 +hl B + ...

thi!. yT =( Yq+I, ... ,YN)

WI'

get. Wi) = 110 and

;5 = -h;/~

=(X q+ 1 , ... ,XN)

= (C q +l, ... ,CN)

er

and hence, the values of the impulse response fUllctions cau be found by the norTllal equation (6.37). This leads to LS estimates 0/ the impulse response function val u~.

.-titt.."" for dynamic modcls t hat was dcri\'oo in Chuptcr 6 Illay be &1.1 mod els, wh ich can be written in the form

(8.59)
Cal;{'

lIowever, in many applications the values of the input. proces.<;0, Me autocorrelatcd, and this leads to near singularity (due to collinearity) of and, hence, a large variance of ii. This problem can be treated by . considering the Ridge estimator

for, for examplc, the ARX und F'lR Ulodels. The method and of thc (.~t i lilator arc discussed in Section 6.4.2 on page 159.

'P,,,,,II<llio" error method

4 'rror ffi l imate may be found as discussed in Section 6.4.3 on

where A E [0, IJ is the Ridge 1K1mmeif'r. The optimal value for A is the value Amin for which 8{A) is reasonably constant. TIllS method is in general useful as an alternative to Icast squarf'S where the expla.natory variables are highly correlated. The term ro,Iin.. is often used to describe the situation where the explanAtory varinbl<'fi high ly correlated thnt. it. is impossible to come up with reliable estimates the least squares approach; see a..Lso Christensen (2002) or Hastie, . and Friedman (2001).

(8.60)

(8.61)
appropriately. Tht' input proeess {Xd is u!iually assumed to If t his is not th(' case, ......e IUl.Ve to condition on t.he observed The conilitional fllt'(lrl is calculated as given by Equation (6.86) for thl' trcul.'l/cr junction modd fl n d by (6.89) fo r the ARMAX

8.6

Estima tion
{f"d is n01Tnally distribu/t'd whit{' liaise, we obtain thl' ~ I L

The estimation problem ha.'! been discussed ill Cha pter 6 also for having inpnt variables. In this section Mpects related to input-Output will bE' discussed. In addition, we will discus:; a method tl..ro for i models, namely t ilt, OUlpU! ('rror method (Ofo, l ).

&be parrunNers in t he

Mill!' WflY 1\.."0

for tht' prroictioll ('rror

' Cha,'"'' 6.

230

L INEAIl SYSTE11S AND STOCHAST IC' 1'1t0".",.

1I()o,;e

CII EC KI NG

231

8.6.5

Output error method

<'rCnT

\ Vhell a pplying the output enVT metJlOd we consider the outptll (8.39) Oil pagc 223. ulld the parameters are e::;ti mat('(i by

.1

15

12

when. '

Lag k

, (8 ) _ 1\ t -

v
' I -

w( B) S' 5(D)

v '" ' t-

P ICIL';C lIot(' lh81 N t (9 ) is t he simulation error since it rep r('SCllt!'; JI1I' lioll betw('('11 Y, and t he output from t he model where X I is input cOJl};id crillg t he nobt'. T he varia nce matrix for the parameter esti ma tes arc ofte n wr,V ,\;lffi ,aItI calculate, J;CC SOd erstrom and StoicCl (1989). F'urthel'lIlOl'', tli(> moot wtlidalion led llliqu('s can not be applied , e.g., test for white notS('o I the me thod sho uld be used o nly under spl.'Ciai condit ions.

Th,. empirical rry>lIs-00""tiatioli

jUflCCiQII

octween th{pre.whiten'!ll;{;lput

tmd tilt Jiltf'Tfd~ulpll l IIt'DW {t:~, }.

8.7

Mod el checking

As a final step we p('rform a model cheek (or model d iagno;.;tics) alS in Chapter (I, but w(' ('xtel1d t h(' diagnostics wit h a tellt for U'lu"fh r r the f'.OrrY'lations bt>twf'en {f"1} and {XI} are zero. If {EI } is w hit(' 110i-;(', we acoordi ug l{) T ht'OIT'1lI (1.2 a ll page 15 1 t hat

Il00''''''' oft(,n theN' will Iw 11 cross-correlation between input series, t hat til{' identi fication is mOI"C unccrtai n, a nd t he separately '''',p" l" n'spoIL<;(! fu nct ions (by pr('-wil itelling) may give only a weak orciN of each of t ile transfer function compoucnls. In s uch DeCeSt;ary to IIpply :-,c'vf'nll diffE'fE'llt mod els and base t he model tests. ('.g. , tc:;ts for single pan UIlNel"S /UI{I tests for lower model

3",,-~~ ~~

ii.dk)

appro><.

EN (0. ,I,).
I

An np prox illlUl(' 5% l('v('1 te:,t is dOlle by Il('eepti ng t he hypo til<'Sis (against. a two-sicil,<1 alterna.tive) if px(k) is wit hin t he 2q -lf'w1.... The Portmanteau tt'st, which illcl udes lIIore valu('S o f tilt' ,,'''H, orr'' funct ion, is based on t he t('S t stat ist ics

(Identifica tion of a tra lls fe r func tion mode l) of the l'xampl(' ill &<-tio n 6.7 o n pa~ 179. In found t hllt tilt t'lt'('lril-ity t OIl.'illmplion could bt' dcscrih<xl -d;".,"';Jo,,,.1 lIlodl'i:

r. ...

Q~ = N L: ii, x (k)'.
k= O

'"

t he information ('oll!l\i[1('(1 in tlu> ohl-iervat ions of t he 8. tnms f('r fUllction ilIolk'l. III oIder La idelltif. .t.hc. , tilt, illpu! s('!"it'S. {X ,}. T h(' model obtained

w hich under n hypothesis fo r p~x(k) = 0 and p~( k) = 0 is HPI " oxld \ 2-tliiltribul{'(i wit h "In + I (r +,~ + I ) = m - r 8 dcgTC'<'S of frt'('(iom. the model OOlltl"ol ind icnt<'S that a r('-ident ificati on is llf'('(kd. T ilt' t('StS fo r whet her a paramct(>r is zero a nd for lower lIIod('i unciHl.ng('(1 from til(' d iscu'i.'iioll ill Sectioll 6.5 011 pag(' 170. In I ll(' discussio n We' have , on.... ide red models wi t h ollly Ollt' but it is fOfllmll y .straightforward to extend the moo('b to illcilltit'

O.27D)V l:lX t

(I

(S .67)

8ltered. uH ing the saUlt' modd, i.{'.,

(8.61')

....!Iat\on funrtion bct 'Neetl {o ,} and {lJ,} iH estimated. Ftsure B.S.

232

LINEAl! SYSTEMS AND STOCHASTIC P'OC",,_

..,,"DI<'T[ON IN TRAN SF ~~ R FUNCTION MODELS

233

The crOh...'rcorrelntion function is proportional to tht' impuiS(' function, since

". The ('Stimatoo cros.<;corrciation in Figure 8.5 on til(' prt'\'iom, pa!l;('

lnrge llpgath'(' value at lng O. whf'fPty; the
remajnjn~

11k = p"fl(k);:::-.

".

values arc wlthJiaJ

2(1 1f'vels. This impliftl that an increasing tcm{X'raturc will 1'(':o;IIIt in ciC'Ctricity cOllsumption flild viet> wesn.. Thili tall hI' dt'S('rib('(i (d. on )lngc 225) by ttl(' following modI'!

IIll-i = "':oX,

+ Nt.
70

As a fin>t gUC:i:> of thl:' Ntru('lUrt' for the noise term, Nt. the on,('-[lin,,,,,,, model (8.66) for In }', may Ix> applied
'V'iJ I 1.A't

(I +OB)(1+9B I2 )er.

Comoining (8.70) (Iud (8.71) yieldN the following transfer functioll

"
II

, I

II

III

'V'V 12 III Yj

WO'iJ'V'12Xt+(1

08)( 1

BB I2 )e/.

Ba."Ni on p"a(k), I.\lId uJ, we CUll culculntl' a mOIlLl'lIL (''itimate With lhTh t'Stimat(' fLo.; Il ~tarlillg wtlu(' in a prroiction ('[ror m"tb~ paraJll('I('r.-> in the total mod('1 (1:1.72) call be estimated. W(' filLd (fM N that tr'V 121n Y, O.OO222 to.OOU20 v'V 12X,

an,

12

121.

"ii;

- 0.61 B)009. (I 0.0171 2.

+ (1

O.750 12 ).10.09 f"/,

For til(' ollP-dimensionalmodr[ in Section 6.7, we fou nd t haI and it is thus sccn that by using the iuformation in til(' Wl' lIlay further n't.itJ(""(> !ht' T(~idtlal varianC('. If we write the' modd l)('lW('('1l {Xt } and PI} in tht form

Plot of {E!},

(i~(k) (Uld p<>~(k) fOT"

the model in (8.73).

'''[',.,,_,
t+k

III }'j

O.OC)222o.OOO20 X,

+ \",.

the model, due LO lhe logarithmic tran..... fonn. may !H' illt<"fpretOO An incr('8SC of I"C in the l('mpefnlure in a given month impliea in the energy COnStllllplioll of {).22lJ\:, :;.. 0.02% ill that salll(, ID(''''~ put)' f(X} we have thr linrar approximation l ' - l u With Y == cxp(uX) Wt' g("1 (Y lo)/Yo = n(X Xo) In ordpr to ri]("ck till' lIIod('1. Wi' have pl()tt~1 tlw n~iduals, autocorrelation for til(' fl':iiduab, fiE, and th(' crON"l-corr('iation r('Siciuab; and til' pno..whit('II("(i input )l('ri('!.. ji,u(k). in FigUre 8,6. \\'1\('11 romparing Ill(! t'mpiric-ru ('orrf'latiol\ Il.Ul('ti()lL~ with the 211 is fIj.'1I that w{' cannot rejt'("t that {tt} may lX' wblw hOist> or tba& BHlit'"t.! Ull thi:; Vo,(, <-'Onciudt' tilt' moot" is ~ti"'ir8('tory.

Gmf'ml a..HumptioIU in prediction using ill/lut-output models.

oedllc1:io'n in transfer function models

prediction, in t('rlllS of minimizing the sqUllre of the prediction M in C'haptC'r 3 and Chapter 5, as the ronditional mean, that the lime of prediction is t a.nd tha.t we want to predict at tinw Y t - k , FUrther we assume that the values or the input availabl(> up until time t + j, see Figure 8.7. Th(' pf"f'didion hon'zon

231

L INEAR SYSTEMS AND STOCI IAST IC P I<'O(:F~,"

_~'M'[ O' I N T RANSFE R FUNCTION MODEL S

235
Ej. E 1_ 1 ...

is thus k.
T he general assumptiolli> ill Figure 8.7 have SOlllC important fop('C iai

Yt- L' .

i\ lId

Xt+ j. XI+ ) - I, ..

Sf{'

known , then also

and

.. arC' known. Thereby we get t he following expression for the

i) {Xtl may be a <:ontrollable input (which we determine ollf'sIAve;).

fore it is possible to predict Y,.+k tinder certain oolltrolling strategies,
Nt+k
Nt+k t

given the choi'(' of future values for t he controllable input.

.-. L
,0

1/-';1+1.: ,.

(8.80)

ii) {Xd is an input variable which is know n up ullti l t ime t (i.e., knowledge of XI and Y,. follow each other in time, e.g., as for temperature in Exumpi(' 8.1 on page 231).
In order to make the prediction as indicated in Figure 8.7, we model (8.37) in the form

k -]- 1

Xt~"

X1+klt+j =

L
;=00
.] - J-I

1P;1/t+k .'

(/.:

> j).

(8.81 )

this into (8.79) we get. the prediction error for YI+.\::

,. - J

}'/+k

L h,X/+k _. + Nt+k'
i,O l
Clln

L it" L

Ie

1?,'It+k.,

.~

. L
+

. , L
, ,0

\'.I+k-;

V'it+k- .

where {N, } is all Alu\ IA process, which

be writt(,11 ill r.. IA form

Nt =

L
"o0

\:.Iit_"

{Ed A.rf' tl..<;''jlllll{'d to be mutually uncorrelatcd, we get t he '_rinnN oj Iht' pn,diciion cn-or:

where kd is white noiS(' wil h variance, Plirthermore, we o.,,,ume tXt} can be described by an AlU lA process. An AR~ I A pr()(('b,.,> call be written as
00

(7;.

(8.82)

condition for t he above stat.ement is thNt {Xd can be described

model.

X,

L ;I/t "
,,,,,0
W('

where {l1d is white noise with variance u~. By taki ng tbe conditional means in (8. 75),

we observe fl t ime <it'lay 1)('1W('('1 1 in p ut and output. delay is b tilll ' ulli ts then h o, hI, ... , hb. I will be Lero, and frolll
tbe pn'di<1 ion efror

.ppli<-~t.i()ml

g(,L the prrdieiion

(8.83)

: L: h. E[X ... . [
Ie-jt

00

iodin ion
II

h,Xt+le-ilt+J

~O

L hi Xt+k ._ k-j

00

+ Nt. . kit

('[fOr:; dqX'lld on t he !loiS(' process only when the prroiC"thull or ('(llIil [ 10 the time delay pl us j. In the important j - 0 and t he information on {Yd a nd {Xd follows c~th 8.701\ pu,I.\I' 2a3}, we get frolll (8.83) that l he prroiction error on till' H()i.';(' pn)(:elo~, {Nd, when t he p red iction horizOtl is It'Ss

til(' tilll(' dI'I HY.

lh'ldlcl;loD in a transfer function model) or Example 8.1 011 pa~t' 231 fur tilt' model which bet'Ml('ll tli(' monthl:.' elt'<'tridty ('ulisurnptioll, l t. and air temperature. Xt .

T ht' predict ion errors b<'COUlf'

V,-tk-YtHlt

..
, L
=()

h,( X ttk . - X t+k.i;l+j)-t .v, .k

.V /+ kll

236

LIN EAH SYSTEMS AND ST OCHASTIO PROC","

"'OD!'C'I"ON IN TRANSFER FUNCTION MODEI,s

237

The standa.rd devialiQII, given

Table 8.1: PJY'didions of the electricity consumption with the prediction (lead time) 0/ 1,2, ... I 12 hour,lrrnn the time t = 9G bwed 011 the '''', ' ,','''''''..;; model for the electricity consumption {lJ and the tmns/er junction model (3). (1) Uni\1\.riate (2) Univariate
(3) 'i'rllnsfcr

i'~

percent of the expected eledricity

at diJJtm:flt prruirtion hOrUon& Ilnd modelA.

Vnivariale model
2.55 2.63 2.71 2.79 2.87 2.91 3.02 3.09 3.16 3.23 3.30 3.36

Trans fer function model

(predicti o n o f t CIll I>era.-

1hlllsfcr func tion model (temperature known )

1.72 1.84

Obscn'Rtio n

number

prediction of the d. consumption

129.1 11 8.8 111.9 102.2 97.7 106.7 110.4 119.9 135.6 147.8 154 .6 147.0

prediction of t he temperature using

11.0 12.4 13.7 15.9 17.0 17.6 16.5 13.7 11 .4

97 98
99

100 101
102 103 10 ' 105

115.6 \0;1.5 101.6

109.6

113.6 12:1.'1

2.49 2.62 2.70 2.18 2.80 2.93 3.00 3.07 3.14 :t21 3.27 3.34

138.8
1.'i2 ..') 157.3
l.'i 1.1

106 107
lOB

9.1 8. 9.4

a.s..!
3.62 3.70 3.78
3.~(l

3.51
3.60 3.69 3.78 3.87

1.95 2.06 2.16 2.25 2.35 2.44 2.52 2.60 2.68 2. 76 2.96 3.07 3. 17 3.27 3.37
3.47

3.94
160
o

3.95

150
140
o
o

we haw p;ivt'll til(' prNIi('tioliS of the el<'Ct.ricity cOllsumptiDlf' t . 00 using different prediction horizons. The
prediction of I hI' ('llI'rgy consum pt iOIl ( 1) is ha.'"ied on the prediction of thr trmpcraturl' is based on the model (8.67). by tbe trrul.... f<'r fUII(:tioll lII(}(it'1 iR b&ed on (8.73). The ~::::': 1\1"(' u.!i<'d to prt'<iict the electricity consumption, ~ are sbowlI ill Figurt, Is.B. comp&' till' stl\lldnrd dl'viations of the prediction error and Dlodt'l:-;. 0\1(' to thl' logarithmic trtl-llsform, is givt'll ruI 1\ pl'rc(,lltag(' of the expcet{'(i electricity

130
120
110

.0

,lie,.;,,,,,,,

I()()

o. o

00

~+---~6~6---7~ 2 --~ 78 ----8'4----00r---9~ 6 ---I~O~2--~

Figure 8.8: P"l:di ctlQII.Il irl Irrmf('T function model. Th(' QlI('-dimffl..~jO:..al irl Tablr 8.1 ill ~hQu'fl 1IIi/h circles ulld tile transfer fUIic/ion mood ur
rro~~rs.

".""fer rUIl(~tioli modl'i produclug residuals wilh sigllifi(lee Example $. 1), W(' sc'(' t hat tilt' !-itandard deviation for tbe one-diUWIl"jollal model and transfer fWlction Thbl is lw(:auSl.' III(' 11Il("'('rtll.jllty of the tpmp('mmodel d. (8.M2). In the last column of Table !<.2 ........... deviation of thl' pnodktion error assumillK thai in the period withuut uncertainty. Here. only the pI'OCftIII, {Ne}, is prt'8t'nt.

238

LINEAR SYSTEMS AND STOCHASTI C PI<OC_

...,"I(',,"ON IN 'I 1{,\N!:iFEH F UNCTION MODEL~

239

B.S. 1

Minimum variance controller

In this section we oon.<;ider the iml>Ortant special ca.'K' of prrdictions in fUliction models wbcrc we nrc able to select the input signal and control the output signal. Dy solvi ng the prooictioll ('(Illations with the tont rol variable (the input in t his case), we obtain the so cllllt'd variance controller, which leads to a controller wlwre the varia nce.' wlllltl'd reference value is minimized. We oonsider the following model wit h a time delay k from input

( O) Y. _ w(B) 0' X O(B) , 9(B) "

'""<nd

> '"", ~

(w(lJ)

( 8) - B (B) O rB)

, o(B) W(B))

X,

+ 0(0)

or B) Yo ,

w( B),,'(B) X + orB) l' O rB) 'O(B)"

,Ido,,,til,\' (8.88) is applied.

(8.92)

( B)Y, ~ w(B) B' X,

+ 9(B)"

ooII.k.,",,,, that the optimal prediction is dmructcriled by minimizing

where we furthermore assume that {XI} is a controllable signal which d{'teTlIline at time t (j = 0, shown in Figure 8.7 on page 233). If we write (8.81) at time t + k we ha\'e
( B ) Yr+k = w( D)X/

*'

+ O(B)!+Ic'

th{' prediction ('rror. Si nce we are able to determine the input X I SO that the variance of the output with rt"Sp('(;t to a gi\"{,11 is minilllil'.('(1. \\'ithout loss of generality, we put the referencc and we get the minimum variance COTltrollcr, characleriv.-o by slIch thllt H1 1 = 0, i.e., the minimum ,raria7lce conlrolle,' stmteKY) is determined by
""I,~'

9t

l3y rearranging we get

(8.93)

w(O) 1H = rb(B) XI Y
~--X

+ 41(13) f"Hk

0(0)

w(B) O\( B) , +(I++":'k

it is ;;('('11 that the error is an ~ I A(k - I ) process, and thus """"''' ofl~ using the control error (8.93) Ix>cOIllCS dt'tNlIlill('(1 from (8.93)J = (1 +

tt'? + ... + \"L,)u;,

(8.94)

where the latter is found by polynomial divi~ion. Allcrnalively Y I +/< elm be cxprcsscd as

tht' variallce of the k-fi teJ) prediction in the ARMA process applying (8.93) it is oft('n sim pler to directly write t he k-step 8u1Js<><11If'lIl1y iwlate XI_ This is illllstrnt(, ,<! in the following.

where "P-(IJ) and ')'( IJ) can be determined from til(' id('71lity

l1li......,"" variance controller)

O(B) ~ ", (O)$(B) + B'0( 8 ),

with

(8.95)
\\-'e find

1,"(13)
')'( 13)

1+,, +
1'u

~'k 1

B' , ,
u P 1,

+.,.

I 1'p-1

-EiYH,lYh Yt-l, ... ,X"Xt

= -4j)Y,
+WO X t _ 1

" ... ]

where we assume thnt q :5 k + JJ 1 (otherwi* the ord{'r (If "'1( U) . TIl(' idt'lll ity (S.SR) is called the diophantine e'Ilwfi(m. The opti mal pr{'(lkl ion of l ~+.", given l 't, l'i I, . . .\ , ..\ ,- h . b\' (R.87) 11.':> or B) ",(B) < ( B j' , +- c>( B{1

+w,X,_:1,

(8.96)
(8.97)

- -fiY,+ljl +woXt

+ WIX I _ I .
I - ....X.

+....x. + (... - .... )X,

240

LI NEAR SYSTEMS ANI) STOCII AST IC

pn,oc,,,,,

....,"'''TIC'' MODELS

241

Siuc(' ....OC wAllt to minimize lb(' v/lrialwc of the output. (I\round tllf' point 0). we rcquir<' that '-2;t = O. Thu!'i, the minimal varilmce

Yi

"",,,,,,.l,;on models
AfU,IA mod els til(' a.s.sumptiou tlll\t the time seri0.1 is stationary non-stationariti~ may be {'xplaiucd by balII)(,lling- also call(>(i an intervention. For exam ple, one observed in tilt' pl'ict' of meill when Denmark t'ntered the Common l\htrket 19i3. Another such cxamp\(> is the change ill the WN'k ly sales of a due w an advertisement campaign. lIere the IIdvert.isement III(' int('rvt'ILlion. ".,,~,,';on modl'l is [\ model which can dcscribt, the changes in an duc to ('xlcrnai pile nolllcna, whidl are then considcrcd a..,: illjlllL

ixx'omc:;
XI ::::;

..!.. (~}'I + (""I ~) XI

"'0

1 - twt Xt .

2)

In sI)lIlC liitunlions, however,

Csing tlLis l'ontrol :-.lrategy the vnriance of tl\(' output

bccotrl('S

a} = ( I

+ fP~)a: = (1 02)a:.

These typt~ of regtllators ure further des("fibed in Astrom (1970) and PouiS('1l ( 1988). The nex t example shows that the frullous PI-('ontroller ( p"O""r1~ integral) is obtailled as a minimum \f8.ria nC(' COlltroller for a particullU" Example 8 .4 (PI-controller) Th{' discrete time PI-mntroller can l>c writh'lI
XI = X I I
0'

~:,~;;:~'; 1II(){lels the input is n quulitatil't' tlariable whereas the

function lIlod<'1 is II quantitative v(lfiable. intrc)(ltice tl.<, inte'''f.lt'ntion function:

t = 10

+K

[( I

+ ~) Y,

-# to

(8. 105)

x,
This controller is ohul.ined as the minimulli ....arianc(! controller
~ _ ,(B) \' . , - ",(BW(B) ,

is also seen t hat till:' intervention ftlHc t ion is defined as a Jd(l B). where 5 is the :summation opcrntor. "",",,,,,';on model cun he writtell in the form

Y,

",(8)

,(8) 1, + (nl'"

0(0 )

(8. 106)

for a Inodd where the lime delay if;. k = 1. (I B). Sinee ttJ(' order of )(8) I' - I I, we S('(' that the order II from the ici('utity h'[( 1 + h T) 131 = B I (O(B)- tP(Bl).
01
4'> . .....

K(1 + ~)
h

Ol - 1>2

II thto illt("rvI'lltioli function. The polynomials 4>(D) , 9( B), 6( 8 ), <if.fiuf'd AA for til(' transfer function model. way as for the transf('r fUllctioll models. the polynomials can to illclud(' S('asonal components, e.g., we can s llool itute 4>( B) 4>{O)~(B ) . Th(' term w( B)ft/6( B) is called tile intervention til(> model (8.106) may IX' eXh'lIded to include severnl inteT:;::~:I\ltd hereby account for severn! tYJ>~ of interventions t hat

The tcchnique of writing down the prediction of future vall'''' " of t he control variable may lead to \'cry powerful approacht'!! fOl" control. The predictiolL'i art' based on a stochastic model for system, and hence, these methods nrc often cn!led model ba..td

In Palsson, r-.ladscu, and Sogaard (I99.1) Rdvanced arE'developed. The&' conlroll{'rs contain mC'thods for control sy:stems and systems with ullknown time d('lay.

Ixtweell the intcrv(ntion model (8.106) and the transfer is rou:-;i<ierable; however, for I he issues of identification I hey the intervention ftillction takes only the \'lI..lu('8 0 and I. fun("tiou, we ('stimatcd the impulse r('io,l}Onse fun ction, and for the sul>se<"luent idelltifkation. For lin illterveution may 1 )(, dir{'('tiy intcrprNed as all impulse r'Sponsc fUB(lion arC' v('ry close in tim(', thl' (fin;t ) idl'ntification (".an Ob thp oh:.t"rv('(! t.im(' S('ri~. For this, Figure 8.1 011 pAge 225 a furth('r di);{"lIssion. S('{' Milhoj (J!)86) or Shoo (1997).

LI NEA R SYSTEMS AND ~n OCII ASTI C PRor_

,."., H)N MODELS

243

Example 8.5 (An int.ervention model) Let {l'd be the obM'rvoo price of lIleat in OI'III1lMk. ami kl iuterwutioll lha1 d{'SCril)('S wht'n [Wllmark joined the' COllllllO n 197;$. i.e. ,

I t January 1973 It = { o otl1l'rwiS('

In Milh<,j (1986) the followiug iuh'rv('ntion model has b('('u applied

{I
n ) hd ~

i (.,

iMl
Time

2tIO

50

00

Sol)

iOu

fsu :100 Tilll('

2SO

where {cd is whitt' noh" . Pl.Irthe'rnlor(', ht' fo und that 0.6:1, - 0 .21. Hnd a~ 0_033 2.

iii

.:J.) ""

(b) PhasiFl9-Qut period

MroMJ.Il."d Will .'Jilllulalcd intensities for r.hf>1I.'in9 fIIHi,llw.sifl9-01J.1 time

intervention modd s !In ' 1I,,('fll l also ill mo re t('('hnical a ppli('lll iOIlS, ('ru;{'S o f som e anomalies in t he data which lllay look as o ulli('rs for sam ples, a ll inlervclltioli lIIodel m ight be useful for CNtiuHllillp; the d(':;pi tc these anomalics. Sinc(' missing da t a most o ft('11 i ~ a hllg(' t imc series a n alysis, t ll(' tiM' o f illlervcmion m odels in such CfL'i('S is mor(' reaso na ble solut ion. Examples o f t he usc of illtNV(' utioll models fou nd in Box and T ioo ( 1975). Tilt"' fo llowing exam ple also illustrates t he poU.nlinls of uJ;inJ!; models in a tcch nical ap plicat io ns. Example 8.6 (Chewing gum Aavor release) In this f'xll.tllple , Il mod('1 for th(' release of flavor compounds will latcd, ill which an int('rv('ntion 1I1011!'1 is applil;'(Il0 I . i C'1l('wing g1lm can bf' UJ;('(1 to g<'t rid of bad bn'ath situ( the giw;o; good taste and tlJ(' ('xpir<'(1 air \){o(-om(>S fr{'Sh. IlifonnatloD rt'l('fL'i(. hu.<.; hrell gathf'f(oJ wheT(' EUl iutNfact' WIL" lL';('(llo dir(,(:lly from til(' /lost'. During the ch('vdng !X'riod the rclease df'p<'uds 011 :-!I"Vf'ral art' diffl'rt'nt (rom lilt' fw: toJ'J; Iwl iug ill the- J>t-riod ll.fwr the /liso caJlffi phasing-out p('riod. TIll'rcfore, two diffNl'ut time ('oll .... id(r('(i: tilt' first Oil(' (o\.'('rillg till' dlt'willg 1I('riod, the covcriug the entire phu.."in,ll,-out IX'riod. For ('ncb window, dbl IUI' tllf' qllalit.ati\ii' input, dt's(rilwd by lllf' inh'r\'('lltioli wh('rc it is a."Sulll('(1 that llit' inl('fv('ntion wk:t't. place at the lim!' wiu(\ow. TIll' intf'rY('ut iOIl modd applif'd is the s<Ka)lcd OE ........ in (h.39). with X t --!> ('(IIIW toO I t _,,. By applyin,r; the output Ilwlhncl, th~ parameter CNtinUllftl art' obtained 88 equatioD whcrt' ."1,(8) Us defined by (8.63). The COIDponcu1.8 in the

lw:-;t fit proved for both 1)01.\"IlOluials a nd b I. l&abilil.\", which can be controlled by finding 1\11 till' l>oles, the Yr in tlK' mudd will approach a stfltionllf".\' (we arc Ilegicding N,) CC)lln.'lH r;nion given by

(8.3(j) 111111 for til(' OE estimatio n ll11'tilOd , til('

\"
......

~ I ( [ )~ w( l )
t

b( I J .

(8.[07)

Ik'ell frOIll (8.2). Til(' conv('rgcllCc rai(' is h iV('1l hy lh(' limp tra.n. ...(r flll1("lion and the largest tinl(' ("onstant determincs oollstA.nls lin' nlated to the roots of till' transfer functiOIl . imnd M ,Ill' roots of Ihl' dt'IIOIlIiufltor, 6{z 1) = 0, wilen' rt"ftJ ur ('OlUplt'x. For a r('al aud 1)()Sitivt' root . (p, < i). ls foulid II.."
T,

ill-.
1',

(8.[08)

r"""'....

,bo.

and 8.9b show llu' IJ1ci\.<;ur(xi H'ivl\.<.;P ]lmfill', along with n IIlO(\('], for both til(' dwwing ]Jl'riod lind t.iI(' retpof'tivl'iy. Tit., ('stimat('t.1 parllllll'l/'rs il1 ttl(' model nr(' ",,1m... illlt'u"ily and two limt' ("mL~tallls, QIl(' d('S("ribinK Ii\(' chewing IX'rilMlallti another d(,;';(TihinJ!; tlil' pha."ill~~Ollt fur IllI' brf'luhillg. Ihal b. IIl1' ltmnl('tNs ill till' aod thE" thf('(.' t't\limat('Ci l)afRlllf'l('rs ('/UI togf'th'r provide 1dI",,,, of the wholt reif'8.M' prO('('88.

244

LINEAR SYSTEMS AND ST OC II AST IC PROC",",

245
is white noise with the mean value 0 and variance It is an function defined, such that it is ('(llUll to I, if in week t an campaign is undertaken I\nd 0 otherwise. on the previolls experiences of all vert isemcnt snies, it is fo und that
II = 200 units/ week -"0 = 100 units/week 400 (units/week)2 01 = 0.3 units/week -0.9 units/week

8,10

Problems

(1;.

Exercise 8 .1 Assume {Xtl is n stationary process v.ith spectral density /;r;(w). prQ('('SS {Y,.} is defined by the li near filter:

W I -:-

11;

where t he up's are real numbers. Question 1 Show that the spectral density for {ytl is

pa.... t 7 weeks the following sales has bccn registered.

Week Sa1{'!-;
Question 2 Set
(!p

J 238

2
2'17

3
230

I
208

5
215

6
196

7
207

= I j(k

+ I)

for 1) = 0 , 1, ... , k and show that. t il(>

density for {l'/} be<."omes

has been no advertising eampnign in the above mentioned period

of the previous campuignl> is assumed to be negligible.

J,(w) ~ ,;n'(w(k + 1)/2) JAw) sin 2 (w/2) (k+ I )2

Question 3 Given {Yt} (from Question 2) a new process {ZI} is dl"fi',"'"

Calculate the expected sa.1es ill wl,.'f'k 8, 9, 10, and 11 assuming t hat

"",pa;on is IIndertllken during these weeks.

1I0W planning nn advertising campaign in week 8 he wants an as.<;(';V;llwnt ovel" thc effect. Assuming that a sales und('rtC\k(>n in week 8, you should calculate the expected sales in 10, and II , and l;l)('('ify II 95% confidence band for thc future sales.

The importer is

Zt = -k-- ' " Yi.- p

'

'+ I

p=O

L...

Find the s j}('('tral delL';ity for {Zt} cXjlr('S.<wd. by the l; jlf( tml density Question 4 Take k 1 and find the impulse response fUllctioll for filter frolll XI to ZI' Question 5 Find the frequellcy responsc funct ion ror r('Sp()n din~ to rcspou&' runction ill QU<'htiOLl1. Sketch the amplitude and pllNl8. composed filtl.'r a low or high.pa...... fiitl' r? Question 6 State a simple change in the comPQM'd fi!tl'r ill ortlt'r to phase shi ft, wililp the am pli tude is lllaintained.

Exercise 8.2
The importer of a given cOTIIlllodi ty has invcstigated the sale of the Hild 0>lwcililly t h" C'ili.'('l of a particular advertising campaign

magazinps.
Intervention alla1y~is has shown that the wf't'kly !-ml~ cnn \K'

the following lIIo<ld :

CHA PTER 9

Multivariate time series

.. ,ch"p" "'" \\'{' hA,\,e mo:,tly considered univariate time S('ries. Howewr, 7 Wf' illlroducoo the \)h'8riatl' pr()('('ss nnd the relnted co\'8riance page 2Q.l fl,nci spe('tml JIlfllrix in (7.10). In this S('Ctioll this is to Illulti\'l\rillte or V('('tor limp M'rit'S, and we s hall .sec that t he for Illultivtl.riate lime series closely correspond to what wc have . for univariate time S(>ries. 8 we im rocluced the transfer function model which describes the the Olltput time scri(>S {l'iJ lUI a u1\('l iou of the input time scries i,..,nnlat;on in Chnpter 8 wns ba.'iCd on the important a&iumption d()('8 not. depcnd on {Yi.}. In wdillieill tenus this implies that

for 1\ feedback fJ'Om {l'd to {X t }. In other words an open-loop sy~tem. For physicAl systcms wc say that there is a ......,"pl)('tw('('11 the input and the output. in many ca,<;e:. {X!} does cil'p<'nd on {Yd. ami we are not able of the variables as Illl input and the other as an o utput; the ju... t inter- reiated. In t{'('h nical t('rnUI stich systems arc often "''''''Psy:stellls. In th;:; case the system should be described as a
....';;;d 1I0t allow
HI rif,~.

":~~i:",ft~PC~.":.~p<'C:ti\"e it is importaJlt to notice that in this chapter

~

. and 1l01l-("IUlsal variation of S('veral \'8riablcs. In , whkh is easy to illustrate, we now have the situation shown on the Ilext pagt'. in Figure 9.1
Y, = hdLJ)X, -+- N 1".
(9. 1)

x!

h2 (D)Yt

+ NH

(9.2)

an {'xlunpl(' t he temperatun' in two attached rooms, Room A htitially we 1l.."SlIlUe thnt til(' tf'mperaturf' is the !'ame in both reason, the tempernt\lrc i flcrcll..~ in Room A (e,g., due to the t('mpCfnl.\If{' will Ill~l i"itnrt to illcn'Il..';(.' slowly in Room B Iwt W(,(,11 the rooms, and vic(' vefAA. most lilll(' i"iNi{'S Me flon-eatlhnl. If you for in.<;tnncc considf'r th(' pric-(' of c-hkk('n nwal nnd the amount of c-hickcns on

248

r..IULTIVAllJA1'E TJ~n:

IrAT"" ,RY

STOCIIA~fll(' I'HOCl~SSE~

MW I'HEIH MO;\mNTS

249

l'

(9.6)

x
N,

H,

l'

(9.7)

Figure 9.1: $chmw.tic diagram of a linear modd for Ihe bivmi/lte PfY)('('U (( ' dl'jillf'd j" Eqrlotioll (9.1) on the precedillg page. The inputla the IiIHIlI' the st dal ron-eiatcd process {(Nu N2.d T }.

tJw rI1uitit'(Jriatc imn$Jer junction from the in put (Nu, N,1,t)T

(l-i.X ,)T. NotiC'i' tlutt t il' lrElH~f('f fU llI'lioll from Nl.t to X I is

H 21 we will introduce the concepts related to multivariate \\'c ~h al1 see that til{' lheol"Y ill the lTlultivariate case turns olll 1I,a;,:htfmw,,,d generalization of whUL we have seen previously for tillle seri('S. W(' shaJI fls..'>mll(' that t he system which gcncrates thc proC('!'iS is li near lmd time-invariant, a nd h(,[ICC, it is known that 8. Illultivariate im pulse re;r)Qllse fu nction describing the dynamics. IO<h."ing the linear multivariate (or vtttor) process, the lIlullivariate VAR\lA ) will be introduced in Section 9.3, and the concepts of and invert ibility will b{' introdu{'t'd. Theil nOll-stationary IUCKieL<; ()'IARI )' IA or VARIr-. IA) model will be treated. The very of ('Q\"arianC(' fUllcliolU; is ililroduced, and it is shown how the II '''''. mia" ,.,. matrix functions can be cnlculated in the multivariate 9.3.1. Section 9.5 is cOlisidering predictions in multivariate ......'. Finally, identificatioll , cstimatioll and model validation arc in Section 9.0. III gen('ml the subjects in this chapter are ordered dIt' !i~milaritil'S wi t h the ullimril\tc ('8.S(' in Chapter 5 and 0 can be

t he market , a lHuilivl\riatc approach is needed. B N"llllH' Illl ilK'H'ILo;(' in will motivnw til(' fnrm'I'S to produce more chickens, and, as n (o"_q'" large amounts of chicken lIIellt Oil tbe market will CfllIS(' 1\ d('(' rI'IL,-;( ' in
and vi('(' v(' rsa.

E xample 9.1 ( C losed-loop transfer functions) ConSid{'r the mood d{'S('rilx'tl by Equations (9. 1) and (!)'2 ). thi), 1110<1('1 j,.; (.'\'I/,Vl.d T and the OUlput is ( Y" X d1'. For {'ath of input and output th('rl' cxi.s t~ a lran:sfer fU llct ion; h {' IIl"(', four tml1.-'.f('r fUllct ions ('xist. thi ng t il(' ~tUlW l('('hniq ut' a... in TheoIl'm 1.1 2 0 11 page "'Ii. while the change in til(' ~ign, w(' nrc now able to fi nd . e.g. , till' trauli(er rrom .VI.I to \",:

0'

stochastic processes and their moments

f1.ud ~ iltl illld\' W ( ' {'o!lld fi ml t h(' r{'maininA thn.'C traJL'.fN """<1:10,,",, lIow('v('r, it is oftl' ll ew;ier to find thfi' tramif('r fUllction." ~:~
V(' (" t Ol'

notRtioll . Do ing t his, the modd (9.1 ) (9.2) ('an be.

lie

our f1.t t('nt ion to s tnt iolllll'Y vp(tor proces.<;(.'S. Consider t he jOintly stat ionary \{'('tor procC!;sl { Z t = [ZI,(, ... , Z""I]1' , t = As III lh(' univariate CI\.<;(', t ilt' stationa rity implies that the mean

;:~~:::!:~~ill:: . P:h('ll\ IhM ('wry uui""ri"u.- component pr0ct'fi8 is sta.l ion "laHollary Pro<'l'!IIW'fI ill not IIf'CC'O<11l1.rily a joillt ly "'ation"ry

250
is ('Ollhtaut, i.e"

~tULT'VARIATF: TIME SERIES

9.2

L1NEA1~ PltOCE5SES

251

Tm'X)RE~1 9,1

E[Z,[

=" = '":

()
11",

Th e covlll'iarw.: and eon'Ciation mallir, jltTLctions are positivI' sl'lIIi(il'jinite in the St'1I$f' that
(9.8)

, ,

L L o; r (f; -tJ) a J ~o
.=1 )=1

(9.1 1)

and that til(' cro:;s-oovarinncc bNwC('n Z.,t and Zj,. for all \'ahles of (i,j) is a function of only t.he tilllt' differellce (3 t ). Helice, the lag k covariance matrix

for any m-dimensional t)('ctOI','1 (0], a z." . ,ak)T. mill similarly fm' p(k),
P roof Follows from the fact that fol' ally set of r('lll \'1\l nt,:; at , 02 , ' , , ,01;,

"

r (k) = C'[Z" Z,.,[ = E[(Z, - ,,)(Z,+, = E[(Z" = r "(-k)

"f[
(9.9)

and any set of limc poi nts, we have

(9. 15)

,,)( Z, _ ,,)1'[

r(k ) is c<1I1('(1 the covariance t1!ldru function for the yt,(tor proc{>5S. Notice that

For two jointly s tatiollnry vedor processes Zt and W , wp defin(' the lag k cross-cot,ariance matrix betw('('11 Z and W us

0,,(1') O,,(k)

O,,,,(k) ) '2m(k)

(9.10)
which us
II

rz.... (k) =
9.2
linear processes

ClZ,. W ,..i

(9. 16)

,,,,,~,(k)
wh('rc "Yij(k) = COV[Z',I. Z}.t+kl. llcn('t', for i = j, ")' .. (k) is the autocovariancc [uuNion for {Z. ,}, and for i of- j, l'iJ(k) is the {Toss-covariancc functioll bctw('('11 {Zi,d alld {Zj,d. As introduced in Section 5.2.2 the cr<>I:>.Ycorrt'intioli function bctW('('1I {Z"t} and {Z},d (i - j) is given by (9. 11 ) wherea.-; for i = j, we obtain till' autoeorrelation fUliction. The lag k c01"71lali071 matrix is defined by
p (k) = [p,,(k)[

fUllclion of k is called lhe Cr'Oss-cot'ariance mal,.iF function.

\\'(' !'ohall foclls on lilll'IU' v('('tor proccs:,es ill the till1e dOlllllin. In general, a multivariut(, linear process can bc iuterpreted as the output. from a lIlulti\'ariate linear sy:..tem with muhiVtlriate white noise input. Lt'. tiS then first defi ne the
(,Ollf'(>p t.

Ot-:FI NITI ON 9.1 (~lrLTIVARIAl'E WIllTE NOISE)

A process {el} is said to be III-dimewsional white ,wise, if {/} is a sequencc of mutual uncorrelated identically diSl ribntl'd raudom variables wilh zt'ro ml'lIn and ('onslallt covarianCt' matrix :E ,

It follows that for white nobe, the ('ovariance matl'ix function is

(9.12)

which as a functio n of k b called til(' conY-,l(~lion m aim function, ami by defining D diag["YlI (0) "" , ')m..,(O)], we havc the relation
(9. 13)
I n thc lilcratlu'(' tILl' ('ovflriall("c and correlntioll IIU1trix fUllctions arc also

r .(k) =

{E 0

if k

;f k"O

(9.17)

eall('d tILt' 1I1It()('O\'luia.n('(' and lHltO<."OlTelntioll matrix fuuctions. A~ ill tILt univariate ('fL'i<' we h!).\"(' tILt, following.

rl~p('('ti ... t"I~'

a"

U]2 Uh" )

C'

a"

a1

a2~"
0",

0':"1

Om~

C ' n.:"
0:.!1

a" Um 2

a,,,, )
(12",

(0, 18)

n.n,"

252

MUL,TIVAIUATE TIME SER IES

9.2

LINEAR PROCESSES

253

lienee, the cl('lI1('nts of the voctor white noise are uncorrelated for different time points, but they may be correlated at the same time poillt (<''OlItt'lIlporanootisly

on page 100 fot' each ~'qlle]]ce of ""iF In the multivariat.e CIlSC t hc critcrion in (5.39) must be fulfilled for each of the sequences of '/j"l' St.aliounrilY ensures tllll.t the influence of historical values of the noise proct'SS goes sufficiently fast. to zero.

correlated). Thl:' fundamental process, as for the univariate case, is lilt. ' lirlear process.
DEFINITION 0.2 (THE LINE,\R MULTIVAR.IATE PROCESS )

A lILultivariate linear process {Y,.} is a process that can be written in t.he

form

Example 9.2 ( Bivariate linea,r process) In the bivariate case (m = 2), the rnndom shc)('k form is

(9.19)
where {Et} is white lIoise and ~ is the mean of the process. i-l.lltlumnOff', 1/Jo =- I is th(> m x m identity matrix. lienee, 1l.':I ill Chapter 5, the lineur process can be defined as the output frOIll fl linear coll\,"oiutioll between a matrix weight function and the vector white noise. Tht, S(.'quencc of matrix weight fUlictiolis {1/Jd is often rCf('ITed to as the 1/J

Y.. ,) _ (,." (B) ""(B) (".,) ( } '2,t - l/'21(/J) 1;'22(0) E'2,t

(9.23)

weights, and (9.1 9) is referred to as the random shock form. A lincar I)rocess defined as in Equation (9.19) is also called a purely stochastic process (or purely nOIldl"terministic process) compare with Dcfinition 5.8 on page 103. In order to eMe the flotation but without loss of genenliity, we will Msume that,... = 0 ill the following. By introducing the lineill' operator

C()mpill'(' with the lram;f('f function in Example 9.1 on page 2,18.

--~

Given the existence of an inverse operator 1r (B) so thot

~ ( B )"( B ) ~ I ., ~ ( S ) ~ .. - ' (B),

(9.24)

til(' linear process (9.21) cau be ..... ritlcll ill the form

.. (B) ~ J +

L: ..,S'. , ,

00

(9.25)
(9.20) where

C(luation (9. 19) can be formulated (for 11, = 0 ): Yj = "'{B)Ef'

tr (B) = T + (9.21 )

L
;",,1

00

71',

Bi ,

(9.26)

Her(> ",(B) is rcferred to H.'i the transfer function of the multivariate proc{'S$.
TUEOREM 9.2 (S TATIONAlliTY FOR I.INEAR MULTIVARJATE PIlOCESSES)

,, (B) is referred to as the 1r weights of tile multivariate process. Equation (9.25) is cruled the i'UI('r.~e fonn.
DEFI NITlO:'>l 9.3 ( I NVEIlTlBILITY FOR LINEAR PROCESSES)

TIle linear process Yt = ",(Bkt is stationary if

Tht;Ylinenr process 1r (B)Yj =

Et

is said to be invertible if

..(,) L: ..,,'
;,,0

00

(9.22)
COnvNge; for

(9.27)

convcryes for Izi

?:

I.

Izl >

1.
valu~

Proor Omitted. Since slfttiouary prOC'('S.<;C!i ar(' cham.ctcriz(>(l by thl" fa<'t. that th(' \:ll.riance is finite. 011(' nUl interpr<'t the result by comparison with (5.39)

Invertibility {'Il..<;urN; that the influence of pH.'>1 lIufficicntly fast. to zero.

of til{> prON'SS gOffi

254

rd ULTIVARI A'I'E TIME '

\1 t:LTI VAIU,\T F.: AR~ IA rno('J;;''iS

255

9.3

The multivariat e ARMA process

Th(' li lwar pr()('('SS('S ('onsid('rro in the previous section are uS('ful for SOIll(' results !llx>uL Illu ltivariate 1itochaslic models, but typically thC'y are useful fo r fittin g to ob8('f\'('(1 data since they contain too many . As ill th(' univariate case in Chapter 5. pon.illlony can be' adli('\'f'd colll'>idering 11i(> mix('(1 wprp;.;emation using AR and ~IA polyumninls urollsly.

Omitted . 1I0 ..... t'vl'r, till' r{'!\lllt is (,MilY::;(,(,1I by considering the fact the AR..\IA (p,q) procC!;S (9.30)

Y,

= (8)

. I

9 (B)

= dc[((B)) 9(B)

adj 4>(0)

(9.33)

- we have t he following.
9, I ( 1:-I\ERTIBlI.ITV)

ARr>. IA PIlOCESS) Th(' multivariate (or vCC'tor) AlUlA procc:;s is obtain{'(1 hy genc,col',', ,5.t a ll pag(> 125 in til(' proc('s.i lIIod(']
DEFIr-;ITIQ:"I 9.1 ( :\lnTIVARI ,\T F.

. .,,,".", ProN'!IS

is i'll'ertible if lJI roolJi

of d('l(O(Z-I = O. with re.'f1Hxt

"oillli" the unit cirrlt'.

where (Ed is wh i t~ lIois('. We 81m]] refer to (9.28) as lin A RMA( P . Q ) where P is a Illlurix with dements Pij and Q is a Illatrix wilh <'i" ""'n'" Sometimes thc prOCl'S.'i is called /l, Vecto r ARMA or VAn~ IA pro('('S!!.

...Db" thll.t for univariate t ime serics a given autoco\'ariance function to ma rc than on(' AR~IA(p,(J) proccss. To ensure a uniqu(> lhl' concept of idcntifiability is imprn;('(1 in til{' model select ioll.

""Lt',,,,

By lI1iing til(> sh ifl. oJl('rator 0, the ARh IA (P . Q) procl..'SS can lx(B) Y, = 9(B)<,.
wh{'re 4>(B) is fI. III!1lrix of autoregres.sivc op('raton; whQl,t' ('11'1Il(',lIs polynomial.s in the backward shift operator B of degr('(' Pi), lind matrix of moving t\\'('rag(' o p('rators whose ('l('m('nts 0.)(8) ar{' "",1",,..010 th(' ha("kward shift operator 13 of degree q.j' It is noted that . 1111' Or)(!rutors 1/>,,( 8 ) and 0.. (8 ) hun' leading t('fIllS which urt' unit\". wi,,,. ... off-diagonal clements hav<, leading terms which are SOIll(, pO ..... I'r of U. Alternativcly l ll(' lIIaximullI order of tht' polynomials is u.,"('('1 Sill'l l multivariate AJUdA(p.q) process C'lUl be wTitten ab

~I,,,,'",e AR~IA

9 .1 (Identifiabilit.y) models, left-multiplying hoth sides of (9.30) by an DOIH.illgular matrix or a nUltrix po lynom ial in B yields a class of i(I(-'nlical ('ovarilul('(' lIlatrix structures. Therefore, the stationarity :::::'~",~c:rit('rion havC' to be somehow ('xtI;'Jld('(1. TIJ(' solution is to minimal movillg a\'('rugc order and minimum autoregrt'SSive a nllllpl(,j (, discussio n w(> r('fcr to Hailllllll ( 1970). ...

Tilleor,ot'cal covariance ma trix functions

","un,, that tli(' process Y, is sta.tionary and that the mean is zero. 8how how to cuiculllli' r (k) for pure autoregressive. pure mO"ing miwd flutor(>grt~": i ve.movi n g svcragc modE'b.

( B)Y,
where (IJ)

9(B)<,.

writ(, til(' pur(> autongn~.,iH' mod!'1 ill lht' form

1 + , B + ... + ,' B'

1 + 0 1 B + ... + Oq I3
q

(9.34)
oltlpl,.'n.g by
r(O )

0(1::1 )

y,T and

lIsing the ('xp(.'('talioll operator, we easily sc<:

For lJ = 0 we obtai II t ile \'('('tor AIl(p) process, wl1'r(><"I..'\ fur p til(> v('("tor ~ IA ((J) pn){('s,,,.
Tm:OIlE.\1 0.3 (S'IA IIO:,\AlIlTY )

r( - I )Y'

r(

-,,); + E

(9.35)

multi",I" 'n" hy
Au AIlMA(p.(/) J}rYJ(:('.~s i.~ Matimwry if all roots of dct(q..( 'l"'fJN'ct to z, iii 11';(11'11 tlf( unit drdr.

y;T~_ . (k > 0) wc gN
(9.30)

256
9.3.1.2 Pure moving average models

MULTIVAIUATE Tl~11:: SEHIES

9 .3

Tm: M l1 LTIVAR1ATE A R ~ i A PHO(ESS

257

The pure moving avcmgc model is

Pu(k)

"",(k)

(D.37)
For this model the covariance matrix fu nction is givell by
(9.38)

(9.3D)
Note t he similarity wit.h (5.65)
011

- I

- I

page J 18.

9.3.1.3

Mixed autoregressive moving average models

Similarly by multiplying by ~~k and taking expectations wp gN

" ] 0 O~-L~-r'---~

o
- 1

o
- I

r(k) ~ - rtk -

1)<1;; _ ... - rtk - p)<I>;' + f' Yt:(k ) + ' }, (k - 1)0]T + ... + [' h ,(k - q) O~ , for k 2:: 0

(9.40)

-I

where r Yr (k) = C[ Yt ,e'H ]' It is S('('n that, 8.'; in the univariate ('f\.Se (compare with (5.97) 011 page 126), the identification of mix{'(1 AJTh.1A models is complicated by the fact that the first q covariance matrices follow no fixed pattern, but for k > (I , they satisfy

---',.,.1]10 ;.,-.._

o
-I - I

'I'

rtk) ~ - r (k - 1 )<I>r _ ... - rtk - p)<I>;

(D.41 )

which is th' same recursive equation us for a pure autorcgrCStiivc lIlodel. Thb will b(' used later in the definition of the qconditionffi partial correlation matrix , which in theory can be used for identifying mixed proccsscs.

Figure 9.2: Theoreti(;(J/ corrY'iation fun ctioru cm-responding to the t11rf!~ bivariate models.

Example 9.3 (Theoretical correlation structures)

In thj:-; ('xample w(' will consider thrC'(' models nnd their throrctical corr('lUI ion structures. TIl{' nto<i(" h; art> fin;t order moving awrnp;e, fin;j order autor('gn~ .. i\'c, I'lnd a mixed fin.t. order l'luton'p;rt>Ssi\"e--fiTbt ordl'r mo\'ing nV{'fag(" model. Model A: First ord(,1" moving fl.Vf'ntg('

Model C: Fin;t. order autoregressive

fir:-;! order modng avera.ge

I+OAB (
0.38
I -

0.38 ) (Zu) (I -2.98 0.10 Zl.t 2.48

E = (_;

-3. 1D ) ( ' ' ') 1 + 2.98 t'2,t

~)

~ (I . 1.78 2.18 ) (,u) E~ (1.0 0.5) (Zu) 1 1 III Z"l.! 0.7B

2.1
O.~

1.1)

Ifbe tooun'tiC"Al c:-orre1ation matrix runc-tions for thl' Ihr('(> model:-; are plottro
10 F1gurt> 9.2. It can be eeen in the ha\lt! the following cbaracti8tics:
figtlTf'

that the thn'(' rorrt'ialiOIl function....

Model B: First onipr I\ulorpgrffi...i\"(' I (

+ 0.98 0.98

2.48 ) 1-1.98

0.75) (Zu) (,u) E= (1.0 0:/1; 1.0

Z21
>=:

2.'

Model A Cut-oft'ia tbecorrzhtloo unctions at lap I. In general a moving awraae ~ ..... order 9 wiI1 show cud fat k > f

--1Y

258

~ I U LTI VAR I ATE TIME SEnlES

9.3

T ItE f-!l'LTIVAJUATE ARMA I'HOCESS

259

Model B DfllIlPNI sine wave bellavior. In gellcral for pun' {Uitorcgrf'Ssivt' p rocCM('S til{' fluto('Orr{'llltion functions will bdla\'c as 1\ mixture of r('al (,xpollelltii\l~ Rnd damped sin<' wan"';;. See also Exam plc 9.4.
Model C For fJ22 all clqx)I\cntiai dl:'Cay is obs('lw'(l, but thc df'C8y stnrt!> in lag 1. ratli('I' than in lag O. In g<'llt'ml for mixC'd proc~'8, the fin;1 few lagged cOIT{'lations will follow 110 fixed paUl'fII. hul thE> tails will rcilf'Cl th(' ~lr\l('turr of th(' nutoregrcs."iv(' part of til(' model.

From the denominator we see that t.ilt' modcl fol' (} "d must include a S{'('()nd orcier ltutOrl'!;:ressive part , i.e., p -, 2. A linear rombinllliOIl of two whit<' lloL'>e Pf'O('{'SSCS is again a white noil;(' proc-cs.s. Sin('{' then the numerator contailll; an ~ I A (l) rolllponent plus a white noise compow'lIt, !II' to[1l1 pt'O('ess will be an MA ( l ). III {'oll('!usion. (!J. 16) can be written as

(9. 17)
where {{d b
II.

It sllould be noticed t hat for lIIultivariate processes the cOl'l'eillt ioll betwC(!1I til(' individual !iiguliis will be reflected in the fiu toeorrelation fu nction for the individual sel'i('S. Tills is illustrated in tilc following examplt'.
Exumple 9.4 (Bivariate AR(l) b e h aves as univariate ARMA (2, 1 This (,:<81npl(' ilI\1~trat('S that the auto{'orrelatioll for one of thc signaJ:-; ill a tw()-iliIllPnsiolial AIl(! ) Uloticl Ix-lu'I\1'S 8l> the autocorrelation for a univariate ArL\1A (2.1) model. COllsider t h<' bivariat(' A R( I ) lIlo(\(oI:

whit(' lIoi!;e :;eqU(,Il('C with variauce

nl-

9.3 .2

Part ia l correla tion matrix

As with til' univariate case the partial (.'Orrplation matri x is a u5('ful tool for idelltifying purc A R mo{lt'ls, and we can usc the following definition (S('{' also T iao and l30x (1981)).
D EF INITI ON 9.5 ( PAI(I'JAI. COrtHE LAT ION MATIUX)

Let

lIS cOII~id{'r

til(' lI1ultivariatC' All(k) prOC<'S...

I +911 B (
" D

(9. 12)

(9.48)
[hell t he part ia l corr<'iation llIatrix <PH is defin('(1 as the last matrix coefficient ill the AR(k) process (9. 18).
COIl!iid('r t he following m ulti\'ariate gCllcnliizatioli of the Yule-Walker cqua[iollS in lInnormalized fOI'III.

Thc {'ovarinll{'(' for lll(' whit(' lloiS(' pro(('S.'o; is

Consici('f now
Wrilt ('11

()Il('

ofthC' hvo signals.

l'i

} 'i,t.

The model for {l'i} ("ftni>l'

') + (".,) 2.1

(9.43)

" ~ (1 0) (" "') Y'l.1

From (9.12) we s('(' t hat

(9.H)

~~~l
I)

fll )T
rIO)

r (k - 2)T r(k

-I)")("'' ) (rll ) )
"'"
. . .

r(2)

. .
.

(9.19)

r( k

r (k - 2)

r iO)

"'"

r(k)

The M~IUClltt of partial corrcliltioll llIa(.ric('S is thus found by solving (9.4D) for stlccessiv('ly highN orders of k, 81ld n lIIethod for i(i<'ntifying the pure An(*1 procl'SS is buill on the following theor('m.

" ( I +..l:.t B
~'.ll

1 +911 U

-0" B) (''')
C2.1

T HEOHF.M 9 . 5 ( P A IU IAI. AUTOCORRELATION MATRIX FOil Fllr

AR PROCESSES )

a mwtit'al'ia/e AR{p)

p1YK'i'.~.~

it holds that

Using (9.44)

WI' S{'i'

that
(I

rPkI.: -F 0 . k::::;" rPu, O . k > p.

}"
I -

+ 0:22 B )ct,r

912

n.. :.u

l -+(,-I+~) B i f!pJ I d>r.l~-llt/t2)B2

(9.46)

Proor

Follow~

from

[ilt'

discussion ul>o\'e.

~(jU

!VIULTIVA IUATE T I ME !::iEItIES

9.4

NON-STATIONAHY I-.IOOELS

201

In contrast. t he partinl correlation matrix function for an l\'IA (q) proccss

9.4

Non-stationary models

will take values different from 0 for arbitrarily large k. A recursive fonnu lA. for
calculati ng the partial correlation matrix
eRn

be found in Wei (2006).

9 .3.3

q-conditioned partial correlation matrix

In the analysis of time series, nonstationarities such as trends and seasonal behavior arc often seen. As for univariate time series, trends arc often dealt with using t he difference operator (V) , whereas seasonal non-stationarity is dealt with by introducing t he scll.sOIlIl1 difference operator (V .. ).

As in (he univariate C8l)(', for a mixed ARMA(p,q) process. neither the correlation matrices nor the partial correlation matrices have a cut-off. T-1ow('ver, as shown in Section 9.3.1, for k > q, they satisfy

9.4.1

The multivariate ARIMA process

r (k) ~ - r(k - I).{ - ... - r(k - p).;;

, (9.50)

For an m-dimensional multivnrinte time series some of the signals may show some sort of nonstationarity, wlJ('r('as sollle other signals are stationary. This calls for an individual differencing of the vuriotlS signa1s.
DEFINITION D.G (TilE MULTIVARIATE ARI!\ IA( P , d , Q ) PROCESS) The process {yt} is ea11('({ a multivariate integmtc.d (or summarised) autoregressive moiling uverage or, in sllQrt, U11 A RIMA ( P , d , Q ) rJrfx:ess if it call be written in the form q,(8) W , ~ 9(8)<" (9.53)

This is lIsed for defining the q-eonditioned partial correlation matrix as the last. matrix to<'fflcicnl in

(9.5 1)
forh_q+l,q+2, .... By writing the generalizcd vCl"8ion of the Yule- Walker eqlllltiolls for Ii. = q+ l ,q+2, ... ,q+k, thesel'quations can be solved forcPrk(q). It issccn thuL a mix('(1 AR~ I A(P,q) process call be suggested in the idelltifkatiOll step if the q-conditiolled partial corr('lation matrix cP[k (q) has a ~e llt-off" at log k > 1>, wilerI' q is the smallest vulue for which a ~eut-off" is seen. In practitt, however, the use of the q-t.'Onditioned parlinl correlatiOIl matri. ces is not very useful due to approximations and estimation uncertainties.

where {Ed is multivariate whi te noise. The ARM A part of the model is as described previollsly ill (D.29), alld d is II v(.'Ctor with clements ct., where d, is a non negative integcr describing t he order of differencing on the i'th signal in yt, i.e., the vector W t has t ilt' (It'lIlellts w.,t = Vd'v"t. Alternath-ely, if we define
8.

mu ltivariate differencing operator D (B):

( I - 00)"

9.3.4

VAR representation

D ( O) ~

(9.54)

All VARMA(p,q) prO(."ft;SCS can be reprcscnted by a VAR(I) process. AssUIll(! that P;::: If + I then the VAru.IA(p, q) process

(
v:e urc able to writ!' the

o
ARI~IA

o
model as
~

Y,, +cJ>I Yi _1 +"'+cPpYi - p =Et+ 8\Et_1 ++ OqEt q

q,(O) D (O) Y,

O(D)<"

(9.55)

can be written as th(! VAR(l) process

wh~ the zeros of det(cP(z I)) and det(O{z

I)) arc inside the unit circle.

-q"

C,)
Z2 .t Zp .t

""
-p 1

0 0 0

0 I 0 0

0 0 0
I

C '-') (I)
Z 2,t.1 81

9.4.2
Et

The multivariate seasonal model

(9 ..,2)

q,"

Z P:t 1

(JP'_I

wh(!re

ZI .t =

Yi. ,

Seasonallllodcls arc lIscful for d(.':';Cribi llg, e.g., diurnal variations or anllual variat ions in time seri('S. Very oftell we fOClLS 011 a single seasonal pcriod s; however , in many cru;cs more than olle S('UIj()1l11l lX'riod is 11('('(\('(1. For install('(' ror hourly variations of heat ron:-' lImp t iOIl in 1\ district heating system we most likely need a diurnal I)('riod , a WI'f'kly period, IUld an annual seasonal pt'riod.

262

l\ \ ULTIVA lllAl'l!: TIML

263

Remcmix-r that in the univs.riatt' case, the Dlultiplicative seasonal of the form

( 8 )" ( 8 ')'J d 'J!, I, ~ 9(8)8(8')0,

where {cd is white noise /lnd lind 0 arc polynomials of o rder 11 rcsp('(;tiveiy, !Iud (j) find 9 aff' polynomials of order P and Q in til(. , lag o l)Crator 8'. As in tile IIllh-arintc case multivariate models tt.rt' most Often t1luliiplicatit'f' models. The lI108t illlponant attraction of lIIu ltipiicatiw is that they VNy oftC'1l achic\'(> parsimony in r(' p rl'M'ntflliou (i.e" I of needed panHIlf'ters) compared with 1!oll-multiplicalil'e modP]~ in iudividual parameters Ilrc introd uced a~ all relevant lags. in the lllulti\fJ\I'ialc case the 8etl1;Ollal components ar(' o[t('11 limil('d to of tlJ(' diagonal dements of the 8utOf<'grcssivc or Illo\-i ng avcrage IIlfllri('el., us consider till' following ('xamplt' of 1\ multiplicati\'c lIluitiwlriate lIIodel with only o ne M'&;onai period .~:

(9.59) tllIIl the predict ion error is unbiased, thc f'Ol'ariance matrix of

;",,,,.1"'''"

(9.60)

to find nil objE't'th'c funCTion we Sf.'{'k a reasonablc scalar function Assuming normality of t he prediclions, lllinilll izi ug del V (A') is eqllivmillilllLdng the volume of the hYP('r-ellipsoid definin g the probability t he pr~.'di("li on , which is minimized for
(9.6 1) optimalTJfY'(lirior (or forccu..<;t) fo r Y'H b{'<:01UCS

((I - " 1l)~1 - 'I'" ~ ('I

9.4.3

B')

(9.62 )
pn'(lictor i~ thlL<; the conditional mean . Furtlu.'rlllor(', it is known of t\ normal process the conditional mean is li near sec also $.1 5. Hpn' \\"(' will limit our allPlltion to the li near ca.'>('. where the ,,..1.;" ' ;0" is as.<;lllllcd lincar. collsidC'r any linefLf multivllriatC' model AR, AHi\IA, ARHI!A, etc. find!1 formula for the cO'v1lrinncc of the prediction crror , it is useful t h., t\l A form (or the weight forUl ) of the model
("fiN'

T im e-varying m odels

For rllllltivl.ll'illti' models a tilll('-vlU'ying mean cnn be introduced as lion 5.6.4 on page 131; this also includes models with ('()\'flriatl's. models with tinw. .. \lI.rying coefiki('llts can be introducro as in SfftioD on page 135. This s uhject will not be co\'ered furtll{'r ht'r('; bow{'\"t'r, time-varying lIlodcls arc introdu("('(1 in Chapter 10. and adapti\"{' dealing with slowly varying models arc described in Chaptcr 11

(9.63)

9.5

Prediction

is whit(' n()i~C' with covl\rio.llCC E . in S,,(t ion G.7.1 011 page 137, knowlt>dge about all previous values qu;",Ie"" t t) kllowlPdge of all previous values of El, and si llce {Ed is W{> havl' for k -:; 0
fork > O (9.6 1)

Now we shall oonJ;idl'r how to predict fUlurc value-; of th(' n:':::':::~::::,.:: process {Yi} based on past o~r\"flti()lls. It is ru;8ullll'<l ~ilat the process lip to time t, i.e. , {Y,. sSt}, arc ollS('rv('(1, and now to predict the value of Yi +k' H('II("I', til(' prediction horizon or lead Furlhcnnor(' it is a...."IIITlPd that lli('n' arc no mi:-;.. . ing ohs('rmtiolls. 10 of missing obsC'rvat iOlls for mllitivilrillt(' tilll(' S('ril's. Wl' ri'ft'r to thl' described in t'itlwr St'('tiou 9.5.1 or Scction 10.5. DIlSft! on the o bs('rvat ioi..U> Yj . Yj I ....... we S('Ck to pr('(lit{ Yi ... 1r and w(' thlL'! illtrodu{'t' lht' pT"fdictor

find t il(' prc'dict ion at Origi1! I with lead time /':, we write
(9.65)

c:unditionnl lIIean 011 (9.65) while t1~illg (9 . Gl) leads to tht'

Y,

,A'I!

A,E(

+ V'A + 1 E/

+ ....

264

MULTI VA RI ATE TIME

2&5
of tht' procc's.'j at l

By subtracting (9.66) from (9.65), we obtain t he prediction error:

el+klt = Yi +k - y"+,kll
=e/+k+Vh Ef+k_l+"'+k
11+ 1_

+L

th(' mod!."1 is writt('11

Yj

- .pI yt

+ er 1 -t

OLe j

("onditional (>xpt'<' tatioll operator aud using I h(' rules ill (9.70) .
mhJ(' for {ilnC I

Now it is readily seen t hat t he covaf'ia7lce of the prediction entlr bt'(~oOl.

+ 1 is

rt'a.diiy

Ht'('11

to hI.'

(9.71 )

licnce, the covariance of n ono-step prediction is always V O) = E.

Remark 9.2 The yul lies of t he luatTices a re often lIl06t easily found by a CO",,'" gCIlcra liz3t ioD of Remark 4.1 o n page 7 1. It is noticed t hat us ing th' matrix as input in the convolution sum (9. 21 ) OI} page 252 providt,s tLt. Vl\l uC8 o f t he 1/J mat rices as o ut put. lienee. we simply cxJl\sid('r au aI''!>"'''' formula tion o f t he process IIlId ta ke
for t= O fo r tf- O as t he input for the process. T hen it is seen t hat t he cniculntl'(l output the needed sequellce of 1jJ matrict'S. The p raetica1 calculatio n of l he prE'<iictions (foreca..<.;ts) i:; most eo"h' CIt! by app lying t he conditiona l mean operator d irectly to til(' pn}('('HO as a (multivariate) d ifference C(luation for the point in tirn(' \\'iI('r(' Iht' if! considered. Noti<:t.J t ha t in orde r to lise the conditional IIlcun o,,,,,"Ul rn ultipJicativ(' polynomials in t he backward shift op('mtor IIlllst 1)(' just 011(' polyno mial by multiplyi ng out the po ly nomials . The rules for eval ua ting t ile conditio nal means are
Oll t

l)(' eakuiAt('d hy l'itl\rting up th(' Oll('-st<'p predictions at. sa,v,

.p'".,i"1< til(' first \u lucs of Et equal to :.Il'rtl Rile! liJcll SIl('('(>!';,.;i\'(>l~' (approXilllll,1 illK) E, h~' thl' prroi{,tioll {'rror

(n.72)
S(,('II

thm the covariauC<' of till'

Sf''t1

Ollt'-S\('p

prt'<licliotl is

it is rl'adily

that we obtain (9.73)

for tlJ('

ahow it is S('(' n that \\'C obtain tlJ(' following sN of rl'CUIT(,llc(' tp IlUltri<'t',.,:

'!PI

- tPl + 8 1
tP l"1

1/12 =

noti("('(l,

!JOWP\W,

that in moot Cfl,.<;('S the pl'('(liC'tioll hori7.0n

"'''n" Klt'lts.

E(l'I _j! l'I . l'I ). .. 1 = l'I- j Ell'I+il l'l, Yj II ... J = }I;+il'

E(Et JI Y/, Yi E[ej+Jl Yi , Yi
I ... ] = '-1

j= 0. 1.2. j = 1, 2.. .
] = 0,

values ror some signals

of lIIuitivaril\lp time St'rics vcry ofte n t he measurements of o ne individual signals an' missing from lime to time. In t hat case from Theorem 2.6 can be u~('d . eXlUtlpk illustrates the practical lise o f pl't'(lietions and the tllI'orrm in fl simple setting.

1, 2.

"

... J~ O

j = 1.2, ...

"'''';0'<'''''

T he ru les in (9.70) arc ulmost t'xltetly as described for univarial(' &ct ion 5. 7. Let li S consider a n example. Example 9.5 (Predictiolls in th(' multivariate ARMA(lt 1..(( Wi fiNl COiL';i(\t'r pr~'("Ii('lioll~ in the llIultivarial~' AR~tA( n. ...... UIU( that tlu? pHKt ....., {Yj} i:. olll'l'rvt"(1 up to and illciudiDI Ilhlll""l "Iways \\X' 11. ..... 11111(' tha.t tlM'lUtUa.l tim(' 1:. t. In urder to

~oIIo.b.. signal values) cities 8 monitoring of the air pollution il'i laken. WId if a
limits it. (>xJ>C'C'tt-.:i ill thE> h(>Ar futuTf' 8 warning i.-; giwlI to providE' thm warning, tilllt' 8l'I'kti 8Ilaiy"j,.; is wry

266

~IULTIVAIUATI~ '11M\::

' ''"',T,,,,'ICAT'ON OF Mt !I: I' IVAIIIATE ~ I01W, [,S

267
all pr('vious observatiolls).

In thb (,XHlIlpl(' it i<. H.-iSllmro that a mod('l for til' \'lIriatiull pollutnuts i" {'Stnhlish('d. Using t he' l'!->lablish{'{1 modd and the rmINt data, prc'dictiolls of til(' pollutants for the next hours art' ('nr("1I1at ~ ..J. prolmbilit\, for tht' pollutant: to 1,('("0111(' larger than sump (kfilu'(i \'l\11Il'S is Inrgl'r than, say, 20'/{ a warning is gin'll. CarskuS('n (1990) Ita.... 11'0('(1 time S('ries wlruysis to st'l lip m()(I..ls ami l\02 for a partkular location (Jagtvej) in Copenhag(,ll. III till" flillowing WI' will fl.o,;SUIII(' that nll'a....UIT'lIlellfs ill general art' ('\'('r~' third hour. Furthermore, we will assume that the "",~"""'. .. '\0 Ilud 1\02 mt' dl'1>("ril)('(1 by a simpl(' first or<ipr bi\ariat(' """".....
lll()(i('l.

nOlation wc will avoid to ("ondition

0 11

E[lit
= 17.6+
Oil

d + ~(Y1.t H
~~(19

",

Ejl"u+l])

13)

21.$_l()().811g/ m3NO.

this t';;tilllntt' b
}"I.f+t1

Var[\"z.,.
23 - 212

L('t

liS

introdu('(' }'"t ..... ~02.f -1':0<0, and

1"2.1 '"

NOt -/'~O.

30

b till'll:

( or using
(l

(V"' ') + ('LI) 0,1 -0,1) 0.8

0.9
}'Z,t-I !2.t

matrix notation,

Ilw pr{-clict('(1 valu(' i!:i abo\'{' lOOllg/ m:l. 1 illll!-.tmtt'~ Ihat th(' <"or'll\liol1 b("lW{'('1l XO and NO z il; projN:'tioll formulas to provide a rather pf{'('ist" estimation of Ill(' ..,,,,,tion of NO:;!. Sillec tht, (':-.limale il; at the actual point ill time, is oft('n ('ull{'fl lh(' fil/a (..~lim(llt' of lilt' IIlj~Sillg obst"rvntioll.

lcI<!fItifi,caltion of multivariate models

Wh('f(' Ej is whit('
Iloi~

with lhe cO\'uriall(;t:

", an) (a" a' ,

30
=

( 21

21) 23

Filially W(' llllV(' /J'.j()l - 18]1p;/m1 81l(ill'lO = 79]1)1;/m:l X()w Wl' shall oonsidl'r thl' predictions cakulatNi !-ohortly TIl(' 11\0:-;\ rt,("(,llt 1II('a.-',II'III('lIls art ohtaill('(1 111 9 a.lII. c,<a~~:;;::': timf' point t in the following), and the obscl"\-atiom; an' ti 9311,1!;/m3;-.;:O. Til{-' pn'(lidioll of thl' con('('lItration/pollution at 12 noon (i.e t + I) h(>('olllt'l-l

O.D (

(16) ~ (13.0) OJ's 11 Ij.6 01 -0.1)

whkh cOITf'Sponds to 61 jlg/m 3N0 2 and 96.6 jlg/m:l;.;O TIlt' ('(jvllrillll(~' of til(' pn'(liclioll is 1:. This matrix tuut.aiD8 of the one-step prcdictions of the individual polilltfUltl> and llt'Lwt't'1l t il(' pn'did iOIl.-'i. Now Ihe time point is (t+ I) "'" 12:00 lloon, 67 \I~/III:1~02' whili'tlli' olN"nlltion of NO, ... I i.-'i " ... d,,,,"". with til(' lIli'il.... \II(llwnt l'(luipulf'llt Sow \w will illuslrah' h.)..... rhl' formulas for th.' lim'3f projod IIN'(I 10 t~tilllltt(' tilt' mi~'1illj( (mlt"t'utratioo. 1.('., to t'Stimatto

have con;;ideroo the theory fo r multivariate st ochastic processe~; COlbider tht (,/\..<;(' of modeling, /lnd it is i\.';Slllill'(l that un observed it. availahle. ....';n1, modeling of lIlultivarinlc time series is silllilar to the methods 'm,'"'''';'"''' lIlod!'ls dt>SCrihed in ChapH' r 6. It follows the same Box prindplcs ru. sketched in Figure 6.1 a ll page U5, i.e., the prinei('.,amalio". (HId 1IIrHhl rhcrki"9 arlO ill gelleral us fo r timl' ';('ri('>;. In this S('('tio n we s hall focus o n the ide ntification step. for cstimlltiou /llId mo d('1 dll'<"killg o f the idt'lltified models and til('n, if inad<'quaciC'S a.r(> disco\'croo. a fUfther cyde of ,l'stilllfllion, and model dl('('killg is ('arritx l out, and so on. ullS('f\'('(j Illulti ..,tri.tl' lillll' ';('ries Y I Y 2. .. .. Y,,, the underlying froill the pattern of its slllnple rorreiatiou a nd partial corn::-How{'\,cr, illilil'lily It proP('f tra nsfo rmat ion and differencing of tht, HlllltivH-fiatl.' time series is introduC{.'(1 according to exac tly -;~~:::'~~ l Prt'viousl.r in S('('lioll~ 6.3.1 H-Ild 6,7. lienee, Vie will assume II: . tmn.-'iformatiOIl and diffel"encillg is selected for each of the , timc seri!'s. ttl(' i(\l.'ntifi("/\t iOIl is till' samplC' correlation alld partial correlalio n
W('

"i"",

N)1J(II'ia7lce flwtrir III lag k. C(k) , is

Ai

O(k

.\

L (yt ,,

~,

Y)(Y, ~

1')"1'

(D.71)

268

MUI.TI VARINrE TIME

Irr"!A.T "oN OF PARAMETERS

269

for k = 0, 1, ... , N - I. FUrthermore, Y = (L~ I yt) jN. Notice thl' with the estimator for the univariate CfISC in (6.1). The sample correlation matrix at lag k, R (k). is then ObtAined bv the theoretical relation ill (9. 13) or alternatively .

".".,oL.

Identification using pre- whitening

as described in the previous section, that identification should on the correlation and partial correlation matrices of the stationary pwntually obtained by differencing the o rigi nal time series. literat ure aiteruath'c methods for identification arc suggested. One frequ('ntly used approaches is first to fit univuriate lime series of till' signals of the lIlu ltivariate time series. By this approach we noise residuals E,f of a ll the univariate time series. Following this of all til(' signals, the cross-correlations between the sequences are then u&'d to find a multivariate i\lA part, which then links &hE' individual series. This method is suggC!lted ill, e.g., Jenkins and ) and J enkin.s (1975). However, the tIl('thod is not recommended. Dw,,,,"cthe problems by considering the true multimriate AR..~1A(,}, q)

ro,(k)

c,,(k) ;Cjo(O)c,,(O)

where C~j I\lld r,} = Pi} arc the clements in C and R , rCSptttivC'ly. As seen in Seet.ioll 9.3. 1, the theoretical corrclatiOIl matrix of a avcrage model with maximum order q has a cut-off after Jagkl '> q. evaluation of a point at. which a cut-off takes plocc, uS(' may be' ""ode. 8P1)roximate formula which is embedded in the following throrem.
THQREM 9.6

<I8)Y,

O(B) .

The estimated cross-cortl!lalion between two m11tually unco,nlatcd

can be WrittCIl on MA fo rm Y, ~ <I<- '(B)O( B) , dcL(<IB))Y, ~ adj <IB)O(B) .

IIQ~H)
AR~IA

(i.e" p;}(k) = 0 for all values of k) is asymptoticolly normally

havi,l9

E[p.,(k)[" 0

(9.78)

df'not('l'l the adjoint matrix. lle nce, it is concluded that ally model may be written as

<I<,(8)Y,
where the tlieoreti('.ai autocofTt'lations p .. (h) may be estimates T;;(h).
Proof Omitted. See Bartlett (19-16). The sample partial cOnY!wtion matrir, 5 ", i.'i C'StilJlllt('(1 a!' tht the last cocHicicnt matrix in tile autoregressive model of ordN Ir.: previously in Section 9.3.2. For identification of pure \'('('tor autoregrcs.sive modf'l~ thl' (act. pMtial correlation lIIatrix for a v('('tor 8utoTl'gr{';;.'iiV<' ltl()(kl with order 1) has II. cu t-off after lag lkl > p is used. l b determi n(' t ht point 8 useful result is that till' f'Slilrlatt'li of the e lemellt.'i of the pnrt ial matrix have an approximate s tandard error 11m g ivC'n tlml k :> thc sam(' as in T heorelll 6.3 o n pflgf' IS\. In principle for mixed models the q-conditioJl(.>d parliall::::7t::~: eau be u.'l for identification. lIuwf'W'r. in practi('(' it is mtlll'r this quantity for idelltificatioll, and ht:ncf', moT(' ill'rntin:' pTI)((..-ltln!II for {'xtu'tly fIS for uni ....ariak tim(' :-'('ri('S.
t~.~limalfd

9,(8)_.,

by Ih,irM
auton'grC1'>Sive matrix 411(B) is diagonal wi th identical clements, all to the dl'tf'Tminant of 41(8) ; moving av .... rage lIlatrix 9 1(B) is of maximum order ql = qq. i.'i the lllllXilllum order of the adjoint matrix in (9.78).
X

III

"::J.::;::,':c~I~'.~;;';'~ of the

that th is method leads to a mis-specification of the AR part

i\1A part of the model. is lhflt fitting models to the individUAl !'ignals and then for llu' cross-correlation leads to a mis-specified model. This .,''''''11 in Exampl(' 9.7 on page 275. thl' method bf\..'K'd on correlatiOIl matrices described in Section 9.6

Iionatie." of parameters

:~'~::':bl' ord('r of tile moo!'!.

we are now r(,Rdy to estimate tbe In this ,,t ion wc will describ(' both l('fiSt squares and Ilwthorl~ for parameter c;tinlfllion.

270

~'I ULT IVAIlIA1' E TI~ll"

....,,"""1110' OF PARAMETF;ns

27J

9.7.1

least squares estimation

\"llriancC' of tile c:-.limator is

Lellis cOll,..,idt>f [he multivariate ARX(P) model

Yi + eP,Y, 1+"'+<Pp Y' _P= WIUt_I+"'+ W"u , ,.+,.

whef"(' dilll( Yj ) = 111 nne! dim(ud = s. Furthermore <Pl .... q,p em. m matrices containing the nutorcgrCStiivc parameters. W I, ... ,W, Arc". lll11triccs, and Et (m x l ) is lIlultivariate white noise with covMian('(' 1:.
FiN;t we call write

V",[O[ ~ E 0 (X T X )-',
denotes the Kronccker product.

(9,88)

of a dynamical model such R.<; t h(' A IlX model. the properties are d. T hrorCIII 6.G 011 page 102. C8."t' wherc , '" 1\'"(0 . I: ) the ~timators for 0 and E presented are .....,ill"""n likelihood estimates.

Y."
0'

9.3 (Nonzero mea n val ues) bt, noted that nonzero m('au "nluCh ('nil b(' treated by extending

V ;"' = XTO+e'{
where
T A.,TT T[ [4>. '" .. ' '<''1' ' W I , ... ,w,.

""

~
I

T TT Y, IlP. + "~ 'UT t .) w ) + t

) 1

"

(9,89)

tilt' multivariate ext('llsioll of (6.17)

X t ami (j ill (9.81 ) should be clllmgrd I\("cordillgly. It is noticed that 011 page 162.

0=

Equation (9.81) is a multivariate gcncral lillcar Illodl'l . CiVl'll N o bst'rvntio!ls Y 1 .. , Y N a nd the needed input variahll'9,

extended lS method for multivariate ARMAX models Spliid method)

previously, the LS methods crumol be IL<;C(I in the case of l\lA txx-au:;{' that tilt' mo<il'1 in tlmt ClL<.;(' is nonlinear in the parameters. a ult'thod which circumvents this problcm is s uggested by S pliid lIH'thod is based 0 11 r('1X'1\t('(1 liS{' of IIlUllivtLriate rpgression. ":::~::~'('OIl\"{'rgf':). but not to thc l\I L estimates. Expericoce suggests ~t arE' rC3.';Onably d OS(' to tht' r ...IL <'8ti m alt'S. f':'itimal e of tht' paTlUlll'tl'rs 0 could be computed by filling a '''''''doll ami taking (h 9 q = 0 ill t he fin;t reconstructioll.
Ollt

y =xo---
where

(Y'~I)
:
T Y .\'

_ X _

(XJ+')
:
XT
fl

(';<1)
: ,

= ... =

Not i('e til!' l-oilu ilarily wi th (6.3.5) 0 11 page 160. The LS Ctiti mntor is gin'lI by

As Stn ('Sli mst!' of 1:

wp ("all take

E~

~Y - XO) (Y - XO ) ~ (N p)

~T

L
tV

1>+1

,,(0)<, (0 ) (N TI)

,.-

bole"',n =f(Yv.[O)

the unknown paraml't('rs, al ld y,.,.. "'" ( Y 1 , Y2, ... , YN') denot.e (N' S; N). J?or all tiltll' scri('!; d ata the conditional (conditioned o n Yo) ('lUI be writtl'1l

=f(YN'[YN' I,O)J(YN' tlYN' ,, 0) '"f( Ytl Yo , O),

wc just
",'i.'HlIllP

(9,90)

Th(' propertiC'S for tht' LS t':'itimatQr in th(' multivariate

""".,1",..0101

that 1h(' following conditional density is

1U{)(lci when.'
1.

t '"

i\" {O. E ) arC'

jj is j{)i ut IIIl1hiVluilltl' uormal with

- 2.)m det ~""1 D +,) 1/2 C'xl' [ 2 I y. - t+I T TI - I y: - t+1 ~~ 1

(9,91)

272

l\I ULTIVAH IAT F' TIM!!: .'

...",,,"1'1('" OF' PARAMF.TERS

273

wiJ('l"{' we have int roduced the

Oll('-st('p

prediction error (the

iUIIOY8 ti

its variance:

on)

ML estimates under stationary conditions

set'll that in 1\ statiollary ~ituatioll (e.g., 110 mi~ing ob:;crvaliolL';) '. reduces to the following likelihood fUllction

Yt+1 =- 1'1 +1 - Yi +l',

R t + 1 = Var

[Y;+I J =

Var [Y, +dY, ]

E" 1+111

L(8;Y,'\')::= r1 ((21l")nIdf'L E ) 1/2exp

,,

[-~YtE IY,]
(9.98)
, ~,

Assllming that the white noh,<' ill til!' model is normal, tiJ(' aoo\,(' fo,'", mLd is valid for an linear models, i.e. , models which ar(> linear i ll pr(' \' ioll~ tlu.' noise term. This is dlll' to til(' fact that any liHt'M eombinat.ion of dL<.tributed random variables is agaiu lIorm81. Dy combining (9.90) nnd (9.91) we get

=- [(21T) dcl E]tn

N/2exp

[_ ~ > '/E IYt]

L (9:Y.v)

Q27l")'" de! llo) - 1/2 exp [

~Y/R,

IV,].

with t he notation above we now use the symbol 'E for the steady of t he inllovation. lIence, the problem is simplified to calculate predict ion ('rrol'S = ~ ,t- I = Yt - Y flt \. which is easily done multivariate ARl\IAX model. ,prol>I..", is most cOllvenienlly separated ill the CI,L';CS 'E known and

Yt

where Ilml R, are t he inuovlllioli and the corrcspolldi llg vHriall('(' i'th observation, rcsp('{'tivt'ly. NOli('(" lhat in the CliS(' of lllis~illg observations the ]>rt'(li<'tion in (9.92 tli(> associated variall(,(, must be calculated appropriat('\~\ e.g.. 8.'; d,o"";ib< 5<'<-tion 10.5. The maximization of the likelihood fu nction is cqui\'nlellt to n, ""'imJ~ of log L ( 9 ;YN') =

Y"

,m.Hm, of 5 1(0) whcre

For this

Cll-~',

1l l/lXilllizIltion of the likelihood fll llction is t,'<luivaiellt

(9.99)
"du,~

the sam ple covariance of t he predictions error, i.e.,

- 2 ~ (logdet n, + v.
I ",

T R; 11'; )

+ (,01I."it.
fu n('tion
5'1

D (9) =
is wl'i tten

~ L Yt Y ?
1",1

(9.100)

and the l\ IL estimate of (J is t l1(' argument which maximizes this, Lf>., (9. 10 1) 9 arg{m;:XlogL(O;YN') } .

the di"tribut iOll a.....;umptio n is postponed, this method

is called a

mdhoti. and t he cost fUDctioD is hellcc 51 ill the case where 'E

As an approximation of the \1l.riallc('8 of t he paramett'f f' limat('s In thin C/\.<.,t, t he maximizat.ion of the likelihood fu nct ion abov{'

to minimiza tioll o f
where

S(8 . E ).:;

{H ).,

a'IOgL WYNo)1
()().f)()J
8 8 '

~ N ]og det.E + ~ " y ;'l' E -1 Yr 2 2 L...,

1=1

(9. 102)

with rt.~ P('<' t to 'E gives

For the mrudmizatioll a Illl merical met hod. such as til(' ~,.wl",oJ prOC'edllN' de;cribcd in Cha pter 3. must be used.

iJE

as

(9.103)

274
which ('(luals zero for

t-. lULTl vAIUNr p. lUtE

CHECKIN G

T herefore the problem call be reduced by imposing the constraint E Sul:lbtituti ng this constrai nt. into the above cost function giv('S
l i

5,(8 ) = 'i logdCl D (8 ) + 'i

0'

L y,TD -'(8)Y, ,. ,
1860
1870

N _

1800

1890

1900

1 Iogdct D (O) + 2" 1 tr [( ~ N y,y;T ) D 5,,(0 ) == 2 1 N = 2"logdet D (O) + 2" tr 1",

= 2"iogdetD(O)

AmllwJ/y !rnded shus o/lllll..d:m! (Z L,,) olld millk (Zl.') olter logarithmic

+-2 -

Nm

Si nce the last term is constant, t he t-, lL estimate is found by n, ,;n;,n,;rlllj

cost. fUllction

5,(8) = logdetD(8)
Again this is a predictiorl ~'TIJr method if th(' as..'Illlllplioll about tribution must be postponed , where, in the present case, the ('()ht given by 52-

is to vcrify thaI the lUullivMillt(' white noise /.L<;'0;1II11plioo is not bv the observed st.'qut'lLt(' of rcsidunls. Jiowever. while it is for 1~\Odd eh(>Cking to work with (>ach series of residuals individually, it lIIight b<' difficult to make in<i<'pendent ch(."Cks due to a betw('('11 the individuHI ~ign als. When large correlations I and iuv<">tignting II transformed sct of residuals in to tht, original residual S('(lucllce. Suppose CO\'Rriancf' nmlrix for thc st'qIlCIl<"C of multivariate residuals suppo:-;(' that A is the dillgollal matrix of ('igenvalues of :E , m&tri:'( of the pigl'uv(,(tol1J. so that

9.8

Mode l checking

I;Q

QA .

(9. 107)

The tools for checking multivariate models arc the same a..., for models a nd transfer fUllction models in Chaptcn; 6 and 8. IIt'm(' !oC'q ucncc of residuals thc tools from Chapter 6 should be w;("(i, . each combinatio n of twO ~w e ll ces of residuals the cross-cofreltion
lIS explained in Chapte r 8 should be applied. If any problelll is
.
8n adj usted model s tructure might. sohre it, as illustrated in Figure

""..Ifo,nu";.,,, j = Qil will ('St!lbibh a scquenN' of uucorrclated {"()IIlI)()II{'llls f;1 hnve the variant' '\, which is the fth value on A . \0\\', control limits of. sn.\'. 2~ call br used for making thf' prl.'Sl.'IIC' of Illrge rt'SidunL" ill tht' transformed sequencc of

page 145. An important aspect of Illodd clK'Cking is the analy:;is of tJlf' fundamelltal !Uld important part of the model chccking is to plot t hp and compare the individual rt'Si<iullls with some control limits us ing t he variance of this sequence of residuals. If la rge rcsidtmis actions must Ix- taken. In some ('/.L'lCS it Illight be useful to add a ll . model t:ompollent to cxplain tht' large residuals. IIow('vt' r , When'!LS there is just ODe set o f Tl."Sidllull"i to ('xamioe lInivariat{' analysis, ill multivariat<' madding t here will be Ollt' S('t 8SSQCiLLlOO with {'SC'h tim(' SC'rit'S.

multiva ria t e ti m e series : M us klat a nd

the lII()( h1 struelllrt' of [I muitivnrllltt' tillic &eriC's will be

parsllwtt'r:-; of tilt' tuml('1 will bc estirnlll('d IIsiug th' rut,thod . TIl(' bivariate' riul(' st'rit'i' fLlml.vscd consists of (ZI,d a nd lILink (Z.!.d skins trnti(-ci frow lI u(Lsolls liay 19 11 . TI\(' two M'ri('i' Ilft t'r lo~arit hlnic tmnsfol"1111l1iOIl

9.3.

. ....."d t rl'ud in tht, mll~kml

llerie,

:>I f it'S it is dt'C'"i(iI"d 10 d iffen'lIct' will. huy,.,\\w, I~ Idt lIuditf.rt'IICM in til(' a nlll.n;is.

276

MUJ,TIVAIlIA'I'E TIME

CHECKING

'U (

and 1Jarl.ial colTI'lation matrices S" based on

R.
- 0.33 -0.33) 1.00
0.65

- 20

Coo
- 0.61
2

( 0.22 0.03) ( - 0.29 0.10) - 0.25 0.26

( - 0.13

(0.QJ - 060) 0.71 0.30 ( 0.21

- 0.01

0.30) - 0.13 009) - 0.36

Figure 9,4: Cross-correlation function of tht' re.ridlWl.! from the unit'ariale along with i~ 2 standmt/. e1TOr limits under llie a.'l$Umptio ll that the Ullcondalcd.

0.1 0

0.18) 0.01:1

(0.09
0.14

It hn. . ))('('l1l;howl1 in Jenkins (1975) thAt the two !>.{'fi('!'I are r('pn';'('lltt'(i by lIlt' tlillVarial (' model,;:
(1 0.6:)1)

( - 0.08 0.26) 0.22 - 0.26

( - 0. 17 0.21 6 0.32)

0.10) (000 0. 10 - 0.12 (0.00 - 0.02) 0.02 0.21

( -0. 11 0.11 - 0. 27) 0.15

(I

(I - O.M)f'1 i 1 O.x2B + O.22U:.! +0.003 +O.2$U )(lnZz.1 - 10.79) =

+ (I.GO + 0.319 1 + 0.0613 5 + 0.3806 )\7 In Zu

l

0.23B1

-0.3 1
0.21)

( 0.25
0.17

- 0.29

with ('!-itilllah'd r('!-idnai vl\rifUl('t'

a? =

0.082---1 and

oj =

0.0681.

froOi tht' IIllivl\rial(> moel"ls i~ shown in Figure !J.t. By if ('urn'lalion fmletion the following cnll hI' notie('(i.

Thl' cf(lss-oorr('iatioll function bct'i\-'t'('n thl' n~idllal!< E"":';;~~I:~

""",,"1.>1 approach is to ('()nsiclt'f til(' method ba&'d on til(' 11. .. cit-'scrihl'Ci in St'('lion O.G. Hent(!. in order to identify
.....''''., tILt wrrl'intioll and til(' partial correlation mairices of and dilfc'n'II{'I'(\ ;,(rit;o. an' IL"I..,(J. The matric{'S an' shown ill

TIl(' ('Orrt'intiOI1 in lag 0 and I is po..;itivl' incliC'atiuf( that p()puh!lioll ilL(f(it.'it,'S ill .nar t. the mink population ;,' om....
IUld

t + 1.

'I'll(' ('orrdation at lit/-(

I i." Ill'gntiw inuiruting that if lliniioll in{'I'Ills{'!>. ill .war t, th(' 1ll1L... krat I}()pulntiun

+I

Sincc thl' population of lIlUskrub lil'pcllw; on the pOIPuloll~ vict' Vt'rsl~, il. is uol possible to da...sify (jill' of the IICries as OtlWf 011(' liS OIHPlll. Tlwrdor(" fl bivariatc model ',,,du,dIoDl
1lt'(III'(I.

standard I'rrur limits 1I1llkr the assumption that liL' are l/JS 1!v'G2 0.13, giving II 95% conficlcnc(' By looking III tht' Lllatri("('S it b lIoti('('(i that the values of ~~:~'vn:Lal ri('('S arc' !<mnlkr thnn 0.26 aftl'r lag 3 indicaiing a modrl with mnximulll (lrd(')' p = a. , of th(~ illocic1 M(' ('st iUHLi('(1 u!<ilLg maximUIIL likelihood H." 9.7.3 r('Sult ing in III(' 11l0l11'1
otle

1 tht 1111'1 holl haM'<\ Oil pre-whitening as d('NC'ribcd In lLs,'(I. tl1\' OhM'j'\'('d nOS8-C'fJff(,latiolL function indicates tha&
AH~IA((i.I)

o.n:w .- 0.26582 0.1228 J ) 1- 0.7178 O.21[,IJ'.l +O..lliB3

mnch'l ~hould hI" 1L~'Ci to link the two series. Using this IIIOcil'l ill "b'-"inro.

278

279

and tilt' c()vllriall(,(, matrix

2 CalculalC' and sketch the nutocorrelation and cr05S-cOTrciatiOIl

E _ (0.0623
0.0147

0.D117)
O.OG38 .

for the proccs.'i. Calculate Rud SkClCh the impulse response (unetioll for the transfer
to

A~ IIlI'ntion('(1 previously lIuivarillte modf'L-;

fittt>d to eaeh of the

{Xv}

lead to tht, models (9.108) and (9.J09) on page 270, which aT(' n ' fourth orci(>r iIIodd, re:sp(',(,lin'ly. By using (9.78) it is S('('n thut fitling ~f lh,(> I\J~\"e 'btinUHf'd tl,lird order mo<ie>1 will lend to a Sixtb moclcllf UUlYnflate> Illodeb nrl' htt(.'(1 to ('11.1'11 of I h(' , serif'" 'fl,"L" ' .~. till' lllis-sIH'eifkat ion of till' modd obtained by fit.! iug univariat(' each of till' S('ri('S.

9.3
number of houses sold ill Illonth t, and let }', dCliote the number lIew building projects in momh t. ex'rrisc rollsidefl> the relation \)(>lw('('11 X t and Y,. It is expected that of initialed new building projects will dcp{>nci all the number of in t ll(' lIlrn.t re<::cnt past, aJl{lll('rK~ we will initially consider X, as pro<1..':;.'i. Th(' investigation is based Oil monthly obscrvatiolL" frOIn January 1965 to Decem\x-r 1974. pha.o.;e an AR~IA model is RuNl for the lime series for number sold in month t, The tilile series of residuals resulting froUl is ('ai1N\ to,}, i.e., a prc-whitening of the input series has been T he time' series Yt is filtered with the same ARMA model and tillle Sf'ril'S is called {/h} IoIkw';".ge'ross-e'orrelation fun ction between {o,} lind {fid has been

9.9

Problems

Exercise 9.1
A bivariate process is giv{,11 us
X l./

{xet.

==

(lllXl./

1 +OJ:2X2,1

1 +f"1.t

X 2 " =02I X I.I

1 +O'nX2,1 1 +f"'l,(

where {"~\'d and {E'l.d are white noise procC5:sCS with m{'an mille V3.fl!}JIC(, 0'1 rul( (12' rcsp<!Cti\'{'ly. TIl(' coval'iall{,(> 1)('1\\.'('('11 'J.t I'IlId ':l.f

' I'

o
0,262 0.274

2
0.103

3
0.079

'I

Question 1 Find a suitable r(,(,lln;ion formula for dl't('rrllillill~ thl' fUlletiolls 1'12(k), ,11(1.. ), 1'21 (k). and ''I22 (k). Question 2 C'on.-;ider now tht, process

-0,0 11

5
Xu = O.GXl.I
XV O,IXI.t
1 1 -

9
0.1)4

10 0.053

O.5X2 1_ 1 + f"1.t
O.5X 2,1
1

+ --"J.I

0.080

0.043

0.054

-0.0-13

for the variances of fit and PI is found as where kl,,} and {f":2,e} arc 1il1lLtmlly ullcorrcla\('(! whit(> nois<' ,n""" mean VIlIIlt' 0 and varianc(> I, US(' t he result fouml Ilhovp to sketch the autororrC'!atiOIl IU1<1 cro:-..'rcorr('\ntiou fUllctions for tll('

a~ = 45.60

IU1(1

a~ = 13.27

foc Ikl '55),

Lrtimntc the impulse response function and make a sketch of it. Find a model to describe how the number of initialed lIew building 011 thl' Ilumber of sold houses.

Exercise 9.2
Consider the hivariatc AR(J) proccss

X,.,) - (1.0 -2.5) (X) + (") (Xu 1.0

I,' 1
I
],t

2,0

X ',l./

~1.'

y'_wo+wIBX f I +6n t

where

{e1.J} I\ud {=2.d arf' whil'

f'J.t

11Oi~e proc('~S('S hOI h with ''',;~...

estimat('!; of the

paI"!}Jucte>T'S

of this model.

corrdatioll 11('1\\.'('('11

and :2.1 is O.i5.

Question 1 Establish re<'ursiOlls for dl'tcnni ning thl' """"""".";"",,,.

('onu'jnll('(' fUlldions of the pro('(~'i.

('lUi i){' done if it turns out thal lIumber of sold houS<'S nu ml)(>r of initialed new building projN'ts duri ng the past

280

l\IUI,l'IVA lliATE; 'l'I MF; ;::::: (],t,'),d is white !loiS(' with covllriauce matrix given by

Exercise 9.4
Question 1 A p rocess can be described by the bivariate .M A ( I ) prC/<'erte

Xu) _ (I --60,,8 (X u 8
21

which

('lUI

bt> rpprPSeIlH'(1

ill

matrix form
1\ multivariate An_MA(p, q) first k - 1 values of the I>C(llIcncc of matrices arc needed. the recurrence relationships for the 1/J malrkes for t he multi!"'"fl.\P.'II model are tl)(> following set. of equations:

&0 calculate the k-lotcp predictions in

8.6

wl1(>f{'

Var[etl -=

(0"11 (112)
(121

un

ASSlLm(' that only X,;;u is IlU'asU(('<i, and let us illtroduc(' Vt for the mCllStIrlxl output, i.e.,

1/J]

= =

1/12

- 4>1 + 0 ] -tP,-rP1 - 4>2 + 8 2

Y, ~

(0 I) X,

The proeess tYf} is t lif'rt'fore a univariate process. Show that {ytl CUll be described by a univariate ~ I A(l) pnX'f'SH.

Question 2 COIlJ:;id!'r
process

!lOW

IU10lhcr process described by til{' b;,~,;; ...,

Or in matrix form

<I>(8)X , ~ ' ,
Show that thf' variation of Y, defined by

ump,le 9.4 on page 258. For th is !'xflmplc the vruuc of the coefficients of the univariate model in (9.47) is readily seen. In this exercise '. equations for findin g th!' pltr1\11lctcr 8 of the MA part of the varianc(> of tht' combined white noiSt> proc"ffi'> {~d

(7l

Y, =(O I)X,
can be described by a univariate AIU.IA (2. 1) proces.... Question 3 Derive equations for finding the parameters of tl)(' pr()('t':';.<; 8.'j a function of the parameters of the bivariatc proc('SH. Exercise 9.5 Question 1 Consider the process

I 213 ) (X,,) ( 1 (- 1.250 I 213 + [) X'l :t = - O.5B

Is this pr~s stationary? 11Iv('rtible?

1 - [)

0.58) ,.,)
(2,(

Question 2 Find the CO''A.rinllce matrix functions of tlJ('

1 ~

p~

0.58) u) B
E'U

CIlAPTER10

space models of dynamic systems

sy"t('lm. are often described in t('rms of differential ("(llIl'Itiol18 (in time) or in terms of diff{'rell(,c {'(Illations (in diM'rde time). The state (or the state tP(~clor) i!'l a:.ct of numbers I I (to). J"2(tO), ... ,x ... (f o), h.v the fact that, together with pOI-sible input-signals and noise at time f ~ to, they uniquely determine the systelll at times ddlTm inistic $y.9it'1nS (i.e., !lySll'II1S without noise), the state tbt' initial vailit's IIl'Ccssnry to determinc the particular wlution to or diffCI'ellc{' (>Q1l<ltiolL. The (-'olllpictc solution and the set of ions will dt'tCfltliuC' the future ('voiutioll of the system without For stocll(utic systt'm.~. tile slate v(,ctor at n giyen tim{' contains R\'ailablc for the future evaluation of the SYi>lC'lll. The state (or vt'(.'tor) is thlls 1\ (fiTht order) Alal'kov process. A system having 111 an m 'til order sYBtem. pn'violl.'i dmpters hoth ullivnriatc and multivariate input-output l)ft'n pr{'S{'ntro. Such 1\ mo(\('1 r('prl'st'uts an c.rternal description . since ouly lilt' input to output relations arc modelled. The U. :.1I.id to pro\'ic!e a moor\ on input-output forlll (abo called external function forlll) of the syst('lIl. Spill'\' 1110<11"\ intro<lll('('(1 in this ('hllpl('r defiuC'S an internal thl'sysH'lIl. This b obtain('(1 by defining a state vector stich ~""n;c" uf til(' systl'lIl ('1111 1)(' (J(>sl'ri\)("(1 by It first order Markov K alman filter will be introdu('('(1 as a method for estimating and IIllm('(lsur('(1 !o.lal(' \'('(""1Or. [n &'('tion 1.1.3 tile h('nt dynamics of 11:;('(1 to introduce the concept of state space models. models lire ofteu IIs('(1 in trlod(,l"n eOlLlrol t heory and some of da....si('al hooks in this area nre Astrom (HI70), Kailath (1980), ami S6dcrstriim ami Stoica (1989). In t'{"ollolilics and applied h(lok Hal"wy (1!)!){i) is oft('n lIsed as a reference. The Kalman introduced in KnlmAn (1960). Lntrr on fi\t<'ring were covered in the hooks by Jazwiuski ( 1970), ~Ia.yix'<'k (1982), and ~I('inhold (19"3).

281

STATE S PAC E MODELS OF DYNAM I(' ,

LINEAR STO('I IAST IC STATE S PACE MODf::L

285

10.1

The linear stochastic state space model

fl

the Markov property becomes csscutial there arf" advantages in

AIL,\lA models in state space form, e.g., in estimation proble ms of with missing observation!;. This is discussed in Sect ion 10.5. .. ,h",,"te ;;1)3CC formulation may be obtaiued di rectly from considering

In discr(>tc time we define the linear stochastic state space model by containing two cqlllllious; namely, the system equation .

problem wh ich is being considered. This is illustrated in the

a nd t il(' observation equation

Y t = CtXt

+ e2,t,

where X t is an m-dimensional, latent (not directly observabl(, Slo<I"",,,.o, vel'lor. Her<' Uj is a deterministic input vector, Yt is a \lector of (measurable) stochastic o ut puts, and A t. B t and C t are known I suitable dimensions. Finally, {CI,t} and {e2,d are random \leetOTh

(A falling body) is tak('11 from OlbjN, Hobt. and Holst (2005).) We consider only by gravity. i.e .. til<' position of the body. :::(l} may be
the following diffl'rcntifll
('qLlalioll

y.

(10.10)

fors=t

for .9 f. t

detemlint' the positioJl of th(' body z(t) for t ~ to, we 11e<'f\ ""l<Iitio"' . (An m'th ord('r difff'rf'lltill.l l'CjuatiOIl HeedS In initial If we S("1 ('('t the initiul ('ondition for the position :r1(tO) = z(to) , ..,I""ity .l:2(l0) = z'(to)' we cau ohtAin t.1ll' !;tale vector

C[

C2,t, C2,. -

) _ {Var (e 2,d = E 2 .t
0 for all

fors=t forsf.t
x(t) -

(",,(t) (t))
y.

(10.11)

C(el,t,e2 . ] = 0 ,

s,t
two stall' \,flriabl(~, W(' obtain the foilowiug state sl>:\Ct'

As tllf' notation indkates, the matrices A t, B t, Ct. E 1,1, and E.!.t Hlay on time, but for most application!; they arc constant. In tile stllte spa('(' modd (10.1) (10.2). the system pqnatiOl" ;:'~~ c\'olution of t he system states whereas the observat ion equation d can lX' directly observed (or measured).

Gl):n ~ (~ ~) G;):O + (~,)

t) ~ (I oj '(I).
Initial conditions. w('
~('l

(10.12) ( 10.13)

Example 10.] (A state space model of an ARJ\.1A(2.1) Con~i(kr till' proc('s.'. {Yi} ghclI b.y

th(' !!Olution to (10.10)

(10.")

}i

T
I '}'I ,-I tP:!}'j.2=f"t'i-OIEtl.

d,(t)

-9 (I
I

tol'.T2 (to)
- to)
,

wil('rl;' {!tl i!; whitl' noiS(' with variance This ARf>.tA(2, 1) prOCt"Ss elll1 b(' written ill a .state SptLCC

(1;.

z(t) =

-"2 y (t

+ J"l (to) (t

- tn)

+,(,

(to)

(10.15)

now bl' formulated in discrete tim(' by fi'tting t = kT, T = 1. In smmmtry, 1 k IIl1d 10 = k 1, amI we get
;rdk)
.TI(k- 1)

+ :r2(k
I)
y.

I)

( 10.16)

whl'rl' Var[ed 11 i.~ !'('II Ihat ( 10.8) and (10.9) rorrespood ('liminati ng X 1,1 nod .\'2.1 Tht, ~tlltl' ~pm"(' d('SCription for AR.\IA modeL~ in gt"nerai is St'('liou 10..1.

(1;.

x,(k) ~ ",(k Hk

(10.1 7)

aIIOctoed by other uisturham'('S

l){'l"iid~ gravity, e.g.. by thp to add thf~' dis! nrhtll\(1'S ttl til(' t'(luutiom; (10.16) and we can write the syslt'm in a NtOC'hMtic IItah- space forlll

286

STATE SPACE MODELS OF DYNAMIC

~RA:""":R F UNCT ION AND STAT Io: SPACE I'OHM lJ LAT10NS

287

I I) (XU I) + (-i) ) (o (XL' Xv

1 XZ.' .I
I
wllt'r(' u,

Ut

(ell,) ,
CI2,/

( 10.26)

is white uoiS(' wi th (""ariamC' E and H ,(,) - C(, I A )' B ,

(10.27)

!I is Il d('l('fmini.<;tic iUpUl to tht' sY:-.l<.'Ul Siud (t-II.I.

8tochn,..,tic disturban("('S. Using XL! and X2.1 W(' can calculah' thl' and \I-Jodly (If tht hody III tilll(' t. ('olllparl' (10. 18) witll (10.1) 011 Th(' !>.n,l<'Jll is ol)l-('rvoo hy lhe fact that the ~ition of lh(, hody i~ Th("oI' obM'rvatiolls M(' 11\(1\... urOO with ('rror (~,t. aud IlI'rt'hy till'
rquntion
1)('(,Oill(':oi

H ,(,) ~ C(, I - A )' ' K

+I.

(10.28)

K (which we williat('r denote IlS t he stationary Kalman gain- d. and E is dctermin('(\ bn..'>('t\ Oll E I , E 2 A , and C. since
K = APCT( CPCr

+ E 2)-I.

( 10.29) (10.30)

z,

(I 0) (:",) + e,.,.
.1\2.1

CPC

+ E2

Compnrl' with (10.2) 011 page 28 1. The i\ l\I JlIllH filtN, whidl will 1)(, introduced ill St,etiuu pos."ibll' to r(,(,OlL~trl1d (or ('Stimate) and prook't til(' prn;ition a nd tht' hody lm.'>\.'(\ 011 till' IllI'HI'iUrCIllt'llts of the body 's POSilioll ( 10.

determined from lhe Rir,alti cf/uation

(10.31)

the stat(' spac-e model

10.2

Tra nsfe r function and state space formulations

X' +llt = AXti t

Yi. = CXIIt
\\'C' (olL:-.id('r lh('<;iulc SIKJCt' mood

+ BUt + K el I + ,

(10.32) ( 10.33)

X t + 1 = AXt

Yi

+ BUt + el.t, eXt + e2.l'

(10.25) is the rorrl'Sponding transfer function form. It is obvious, is kllown. then WI.' can lis(' t il(' ol)$(.'rvation YI to calculate e l can IX' calculated from ( 10.32). SubS('quentiy we can calculate

Y2, ('tc. Thus, the SCQuell<'f' of X 'II 1 can be calculated from

wherC' feu} and {C7.,} l\r(' mutuall~' illdepcndC'nt whill' uoi.<"{ ( 10.20) rorrC'l'iponds to ( 10.1) ou page 284 with the following ('xI'l'pl iuus: noise term is ('ntering at time t instead of the (im(' f+ 1 and (i1) the ar(' cOII.... lnm. It is M'('U lhnt {Xd Wid {Yi} arc stalionary PWC('SS('8. Applying t he z-trnnsform yields , X (,) ~ AX(,) Y (,)
~

m.'",, l'I

I, l'i .7, .... In the :-;cction on the Kalman filter. we will that X n I is the c0l1d i1iollUI lUcan of X I given "l'i -I, Yi - 2,' Odingll,., p ;~ the (stationary) conditional covariance of X I given .; h('I\('{'. we will tiM' til(' notntion E:,~ _ I' SUb-litH1(' tIl(' prNlietion in (10.32) lind (10.33) by Zt. We hereby ltate Sp8.C'(' model
ZI ti

CX (,)

+ Bu(,) + el (')' + e, (,) .

AZ I + BU t YI = eZt +e"

+ Ke t

(1 0.34) (1 0.35)

By ('liminnting X (z) in ( 10.22) Y (,)

( IO.23) we gel

C(d - A ) ' Bu (,)

+ C(d -

A )' le,(,)

+ e,( ,).

o 'all,," tht, imlOllfJliofi form of 'he s.y!;t('llJ. The name is due to the D ohl(' procC'Ss equals the innovation (OJl('-stcp prt->ciiction el'ror),
,
",,";'1 "'.Y

This form is c8.II('(1 the' inl11tl-olllptd or tlll1l-<;j<T fundion form. If {V,} i~ stntiolllll",V (A stable) the noise terms ill (I O.2"\ ) ('l\IllK' into on(' singl' tl'ntl (Goodwin lUl(l Pn.vlIl' 1977), yielding
Y (l')
C( J

Yi

Yilt

( 10.36)

condition ZI is known, all futun' VRluC'!l ('an IX' recolL'>tructro blL,,('(1 Oil th(' oh:-.trv('(1 output (and inplll, IItl !iillCC
1,2 ....

A)

Bu.(.:) + lC(zI

A)

+l

:E(r)

(10.37)

288

STATE SPACE MODELS OF DYNAM IC SYSTEMS

10. 3

I NTERPOL.ATION, ROCONSTHUC'rION. AND PREDICTION

289

Thus, for a system written all the innovation form, the only unknown q unutity is the initial condi tion. As illustrated it is possible to find t he corresponding transfer fUll ction form base<1 on lilt' stAte 1>pace form by eHminating the state vector. Corre;IXllldi ngly, it is necessary to select a stall:' v('Ctor if one wishes to find 0. state IjpaC<' description ba.'I('(i all a. transfer fUllction. In princi ple there IU'l' infi nitely many possible ways to do this since t he state space form is not unique. This is ilhL'itmll'd ill the following example.

i) P rediction of u future vnlue of {Xd . i.e., k > 0;

ii) Reconstruction (filtering), estimatio n of X L, i.e., k = 0; and

iii) I n terpo lat ion (.mlOothing). k < O.

Il1ld

~ti m ntiO!l of a past. value of

{Xd , i.e.,

Example 10.3 (Two exa mples of s tate s pace fo rms) TIJ(' AR(2) modC'i: Y, + rPl \t- 1 + li>lY'-2 = E, cau be written in tll(' state

In this section we will limit oursch'cs to the' (".llS(' where the proces!>eS {Xd {l'i}, as ....ell as the rdll.tions bt"twi'f'1I them, can be described by the linear stochastic state space model

spoce form

X t -= AX, 1+ B U' _1 + el.,.

( 10.40) (10.41 )

(XL') ~ (., 1) (X,., ') + (1) <, 0) (Xu).

X2.1
-2

yt = eXt + e2.f, ( 10.38)

XU

_1

(1

X 2 ,!

o r ill th(' st!lll' IiPIlC(, form

XL

X'u )

(x:.,_,) + (1) <, (i' '''') o 0

X 2 1
I

( 10.:19)

which was introduced ill Seclioll 10.1 . III ord('r to make the notation simpler we coru;ider t he matrices A , B , and in (10.40) (10.4 1) as being constant ma t rices. Similarly, we considcr Vllr[el,d = EJ and Var[e2.tl = :2 to be COill;tanL The results prl'S<'ntoo ill the following may, however, be generalized to time-varying systems. Fina.lIy, we assufIle that the system in ( 10.40) (10.41 ) is observable which ('IlMlres that nit the states ill X t ('nil ht" eslimMed . An m'th order system such as (10.40) (10.4 1) having one output P"t scalar) is snid to be observable if

( 10.42)

There an' infinitE-ly many other state space descriptions, and thus it is more adequate to obtain a state s pace description by selccting a canonical form, C il/lfl\('leri7..cd by the fact that the transfer function uniqtleiy det('"nin('.~ the state space dC3C';plion. The equations (10.38) and (10.39) are examples of cano nical forms which each determine a state space de;criplioll for the AR(2) model.

f or a s)'stem without. any noise o r /lily input, u " (10.12) ensures that the fOllowing m equations

Yi = CX t Yi+l = CXH1

= CAX t

10.3

Interpolation, reconstruction, and prediction

('an be solved with respect to X ,. T his lHf'ans t hat, based all m observatiOIlS of the output, we are ablt, to calculatc the m-dimCllsiOIll\1 state vector (which <'Q uid not be Illca.'iIlrE'd ). For stochnSlic sy~tems ('("ml ion (10.42) ensurcs that thf' sta te vector can be t'Slilllat('(1.

In the introduction to Section 5.7 on page 135, "'e defined t he terms pI-cdictioli and interyJOlation in the case where, based on Ii stodllL<;tic pro('{~s {Y, }, we sought to Cbtill1at e futUfl' vnlllcs of the process, e.g., Yi.+k. This s('Clioll collcerns the same issues, namely, those where, ba.<;C(i 011 all obM'n'cd ~lOtha.'iti(' process {Yi} , we Sft'k to estimate H. thusly correlated process {Xd. u't tlu'rl' b(" giv{'n a Sftlllt'IIC{, of observations Yt = {Y., s = i, t 1.... }, a.nd a pro('ft<s {X,} which is ('Qrrl'latoo with {Yd. \\'t' now (,()lIsid("r t lil' d.,crY'ie timr r.,timation problem that is ('QllcNnro with tllf' cakulation of a n ~timl\l(' of X 'H gh'('11 Y t
\A.-I> ........ , ti ... i ............
I ~. _

10.3.1

T he Kalman fi lter
X , giVl'n t hl' oh-;('rvarions Of {UI } and {Y,} in t ilt' stnte
~ 1>SlC('

tb., Kalman filter yields t ht, opti mal r('('()l1slruction and prediction of tllf' stnlf'
model

290

STATE SPACE MODELS OF DYNAM I C

If\;TFR POLAT10N, HE('ONSTHUCTION, AND I'IlEOI('TJON

291

The foundat ion for the Kalman filter is given in ThoorC'1II 2.6 on or 2.8 on page 27 in the section on linear projectiQ'lS. iA'l there bf' two v('('tors X = (X"""Xm)T and Y = O'I ..... yn)T, and let the (m dimensional vector (.k ) be normally distributed with Ill('an and ('0, . . . . . .

introduC(' tht' p71:diction trrors

Xt + kll
YtHI!

X t+k - X Hk l,

(10.19) (10.50)

Yt-+ k

9tHlt.

(I'>')
I~x
rC:;pt'Clively.

l; ),

'

Exx

x)
-

""",,lllng to (10.48) we )l;ct the COWlriancc

E r:", = Var (X t+kIYt]

=
=

It then follows from Theorem 2.8 that X IY is lIorulIIlIy distributed Illetln wiltle

E [( X t +k

XHilt)

(X tH - X/Hit) flY!]

(10.51)

E(XjYJ Var(XI Y j

1-' .\

+ :ExyEy~ - ( Y

I-'d.

E [XL kIIX;:kl,IYt]

Exx - E ,n :EY~, EL .

Var {XI

kltIY,].
lIlSing !illca!'

If the assumption of normality does not hold, then t il{' ast;umptioll 01 lK'iug normally distribut{'(1 is no longer valid. How(>\'er, (10.1:1) and may still bt, applied, since th(' projection (10,43) call tiwil he Im"",re" tht, projection of X Oil Y , which has the smallest Y[lriall(,(' IUtlOIl~ aU projcctions (d. Sect ion 2.8 Oil ])ngC' 24). 'fhe theorems afe abo ,,"lid for the conditioned vector (k iZ ) wilt"n> (10. -\.1) can be writt(>n
E[XY , Z[~E[X Z ]+C[ X . Y IZ [Vnc-'[Y [Z[(Y

("onditional 1Ill'IUlS nrc calculated ","",ml 2.6 on page 21 thllt

projections, we have

(10.52)

,p",;II",';o" prror X ,

E[ Y ' Z ,I.

Ioo..,..

/<"1 alld Y, arc ullcorreiat<.,<1 (Of o rthogonal). funher assullle that t'hey arc independent (equ ivalent to t he a.-;sumph,,) we gl'l

Vnc[XI Y , Z [ ~ Vac[X IZ[

C[X . Y [Z ] V",-' [Y [Z [ CT[X . Y Z ].

V,U' [X'+kll:Yt] = Vl)r {X I+/.:I']' ..,<he. with ( 10. 5 1) im pli('s

(10.53)

It is SC('n from (10.45) that the information gained from Y is ;,mall lIIorc of the following issues hold.

I) The {,ffor (Y - E(Y IZ J) is s mall.

2) The collditional oovnrianee C[X. Y ' Z j is small rompnr<'C1 In the tional variance VarI Y IZ J. W(' now introduce
which i8 a vector illcludiug all observations up to tim(' l assume tlmt we know U l .... , 'U/. III addition, we introduCl'
,.'",,,,,,.,...

E:~klt

= Vnr[Xr+kIYd = Val' [X t+klt].

(10.54)

manner we call ;;how that

Ei!'l' Var(Yt+J,YI ] = Var [Yi k,']. E:!/.:It = CIX t./.:, Y, +kjYt! = C [X t+kll' Y,+klt].

( 10.55)
(10.56)

optimal 1'{'('on.o;lrl1etioil of X /It is found by using ( 10.45) through """"'y, with X = X " Y = YI and Z = Yr 1.
10.1 (Ih:CONSTH1'C'I ION) I'f'<'Ullsil'll('lion of X I t i.s obtuim'.d by

wh('re X /It is rallC'd the tT'cor,,~tr"tl('liQ" of X I. and X ,+k j (f.... p,.("dirtion of X t + k . W e know from The(lr('111 3.9 that if W(' ("h.M'II(" tllf' (,xJ)('<"wd mlue of tilt' sclllflr('(1 prt"{lkti{)1l error. t il(' o11lirlllJ
and pr"f'dklioll are obtflinrd b.III/I(' ('(mtiitional mean.

(10.57)

d Vnr

YiIY, d( Yj

E[Yi IY, d),

292
which can be written as

STATE SPACE MODELS O l~ DYNAMIC SVSl,..

I ~Te;RPOL ATI ON,

RECONSTRUCTION, AND PREOICTION

293

lIO.54) "( 10.57) yields

Xtit=~lt_ I+ E:'~ l(E~~_l) - I(YI - ~lf 1).

TIl{' covariance of the rc('ollstruction error become$

1::':1 1 1 = AE:I~ AT E r!11 =

"" 1+1 1 1=
~ :rll

+ E 1, T CE:':qtC + E 2,
E "z
t+ilt CT '

(10.70) ( 10.71) (10.72)

Proof See above.

""" tio>" (10.60) -( 10.62), ( 10.63) (10.64). and ( 10.67) (10.69) can in tL~1 to re<'on!:>truct (tnd predict XI and the corrCliponding covariancc
W(' ha\"c, thus, dcriV('(1 the Kalman filter in discret e time. The results in the following.
10.2 (TilE KAL ~tAN FII.TER)

Ir we introduce the Kalman gain

Kt =

E:I ~_I (E~I~_I) - I ,

the optimal recoru; t ructioll and the corrcspondjllg covarilUlC'(' CAn be

lintar r(-'('oll::>truction X* and prediction Xt + 1il of the system H IO ..lI is obtained in terms of a reconstruction (updating)

.K;'t =

Xt lt _ 1 + K t
1 -

(i - ~II_I)'

(10.73) (10.74)

E lit r..r: _ """"" Lotlt _

K~ "

T' l Lotl t _ ! K I

The formulas denote how information from past observations, Lt' .. be combined with (he new observation Yi to improw til(' CMtimat(' of It L'I further dear from (10.60) through (10.62) that in order to ,al,ul", reconstruction and the corresponding variance we need to cakulllt(, tilt' prediction of X t and Yi and its corresponding variallt'('S and ro',"""""_ From ( lOAD) and ( 10.41 ) on page 289 it follows that the QfHstep becomes

(10.75)

~+ l lt :::;' A~lt + Bu/ , ~+l l t = eXt +-1 1/'

Thus, it holds that for the predietion error

X ltl1t = AX; jt

+ BUt,
+ E l,

(10.76) ( 10.77) ( 10. 78)

E:: 11I = AEf~ AT

E r!llt = CEf~llt CT + E 2

1 = X t+ 1 - X, t !1 X t +11 1

A (XI - X III)

+ el.l tl

ronditiOfl

AXli! + el.I+I.

XIIO = E[X d = J,Jo ,

Ef~ = VarIX'] = Yo

~+ l lt

= i t! -

1'1 t 11 1=

(X+! l

~ t1II) + e2.1'~1
directly. rf'C un;ioll

(10.79) (10.80)

= eXt +llt
Prom this it follows thllt

+ e~ .ttl

V'lr (X ,tl,tj
Var (V, tlll]
C

A Var

(XIII] AT + EJ,
T

CVarfXt+ III] CT +E1.

[XI , III' Y,

IIIJ

Var [X t-..III] C

or each new observation Yi we first IISC Xt t 1 and apply the r('('ursioll formulas (10.73) (10.75) 1111 .... ... (10.76), ~ :.lot i" " of 1:: 11 -1 via (10.78)) and till' prerlktion formula. _ (10.71'\). If w(> give thb a Ba...,C'Sian interpretation, then X III I are t h(' 1Itt"1UI !\nd C()\'AriallC'e. rP.-il)('("ti\'<'iy. in th~ prior distribution will 1)(> furth('r elaborated in til{' lU'xt S('('tioll .
WI' sU('('('Ssiv('ly

291

STATE SPACE MODELS OF DYNAMIC

I :-;TERPOLATION. HECONSTRUCTION, AND Pltt~ I)I('Tl0N

295

Example 10.4 (The Ka lman fi lter and a falling body) Uo!> l'Ouhidel' the falling body in EX81llpif' 10.2011 pnp;<, 2.... 5. can 1)(' written ill til(' stall' space form
L<'l

the ),alman gain ignores thE:' v('locity at t f!<Iui\'alcm ('skull,Hom; for l = 2 yield

I.

~" ~
Lo

211

/11 (2.909 I) ~ (32 1 1 3.1

0.631 0.125) ( 0.219 0.274

whcr(' XI,I b tlu> pO!-;ilion And

Xl,!

is tht, vplocity. For till' prOCb.'1

El.I =(~ O~l)'

Let Ib a.,sume that tile pusition of til(> body is ol>:;('l'w(/ with high and that the \'dodty of th(' body is Ob!;(-'fVI;"(I
int('rval. TIl(> ol~r\'ntioll ('<Illation I}('("()mes

E"
211

(0.6 13 0.249)
0.219 .519

E':li2

1.661 0.798) ( 0.798 0.&19

obtain their stationary values reiativ('ly quickly. For examplr,

\'alu<, for til(' Kalman gain bccollH's

(~
wit II variance

0) (X,.,) + ('''')
I

Xv

/'22.1

0.526

K! odd = ( 0.19.]
K, ........
=

~)

E"
t:::;: 1,3.5" ...

~ (~ ~)

0.519 0.091) ( 0.181 0.1],1

whprf' 11 ~ 2 for t -= 2.1,6, .. and 1, = 00 (i n pnwtk(' a IMg('

that for t odd we an:' disregarding til{' wJocity.

previotL'; ('xalUpi(' the variance of the observations was timt'--varyiug. exam]". wh(>r(> the C matrix is tilll(,'-varying. is the following.

A-; "tarting valu('

W('

apply

EfiO
From ( In. 78)
wt'

0 (1 0

II')
II (
])

gel th(' covariatH't' of Y110

0

10.5 (Estimat ion o f sales numbers)

IIIP yt'nrly I'>ai(>s of a giveu produ('l nul! Irt } 't denote the in mouth I JIIodulub 12. From expericllce it is known how di...trihutI'd during the months of til{' y('<u. Let p, d('lIoH' the yt'lul.\" !'j1l1l'S ill mouth i. is illt{''SH'(1 in succ('S~i\'('I'y estimating til(' yearly s.alei; in an inn!'fl."" Of dl"(-rea.-;e in snll'S. From (xl){'ri(,Il(,(,. the year]" >nudum I Wll-lk, L(....

E" (I
110 =

~) + (~ ~)

\'Urian('(~

It is now pOI>.'iibk' to rakulllt(' til(' Kalman gain for I for the rC('()nslntC't ion and till' prediction.

E ,C;' (10 0
=
! I

0) (J/II 0) ~ (10/11 1/x. 0) 0) _ (10/11 0) (" 0) (10/ II [ 0 I /'X. 0

I 0 I/oc ()
ex. ()

f'q unt io tl l)txolnc:.

~ CO~ll ~)

(~

0) (" ")
I 0 n.1

Kalm811 IiIlI'r. til(' ('Ompany ('an now ~1I('t(SSiHly e"limalt' tilt'

:,W(j

STATE SPACE ~IODELS OF DYNAM IC,

I~TERPOLATION, BE("ONSTIlUCT 10N, AND PHEDIC'TION

297

10.3.2

k-step predictions in state space models

..1ft~
. 1

On ly one-step predictions ar<' given by t he Kalman filter. The k-step prediction in the state space model (10.40) and page 289 can be calculated hy applying t he recursive formula

obtaiJling the obsf'1"1)ation 1'/., we C8.n c8kll18te the posterior using Baycs formula (10.83), but.Jir.lt we have to calculate thE' distribution P{YiI X t.Yt_l}' Let Y t1t - 1 be t he prediction error

Yjt I = Y; - Y,lt -l and t he correspond illg rCClII'siV(' formula for the cov(Uiwtce

= Yi - C X tit - 1 =

(I O.87)
1-

Yt - C AXI_1lt

C Bul .-

The fo rmulas (10.8 1) and (10.82) follow di rectly from the slate ;,pac."t>

A, B , XI III I and Ut_1 are all known, the ohservation Yt is to o~rving V, lt- I' i.e., (10.83) can be wriLtel!

10.3,3

Empirical Bayesian description of the Kalman filter

P{X,: Y1.Y'I}
'X

P{ XtlY,lt-l,Yt-l}
P

( IO.88)

A Kalnltu\ filter is a recursi ....e pro(:roure for calculating an C!;ti lllllt e for Xc 011 'he observations Yt = (Yi , ... , Yj ). Givcn t hese obflerwttiOllS, about X t ("all be mad e by applying Bl'l.Ycs' formula. We gct

{Ytlt-d Xt,Yt-,} p {X tlYt-d

= eXt + e2.t we may wri te Ytll-I in ( 10.87) /lS

Yi lt I =

C (X l -

AX

III-I -

BUt_I)

+ e2,t,

(IO.S9)

where PIA B} is the probahility for the event A givell tht t' Wllt 8. From a Bayesian point of view P{ XtIYd is t he posterior P{ Yt IX "Y,_d is til{' likelihood distribution, and P{X liYI_ } L'I the distribution. We assume that at timps t- J, the information of X I _ 1 (a'."",i,,,OI," is given by
(X I-dYt-l) '"" N ( X I I II-I,

IIX "Y, I] = C ( X I - AXt _ Ilt-1

BUt_I)'

(lO.90)

.t!X1,Yt

I] "" E 2
is t hus

(1O.9 1)

E~:r lit

I)'

di.~ t rib1Jtio71

i.e., the posterior distribution for X l_I' We now considcr the time l ill two steps: I) Before obtai ning the observation Yt . 2) After obtai ning the ob!;crvntion Y I Ste p 1 Before obtai11iny the ob8f"rtJation 1'/., the best est iulIItc of XI dircct ly frOIll t he system equation

(Y11 t-dXt,Yt.

I)
AXIIII_I- BUt_,), E 2)'

(1O.92)

"' :" ( C ( X t

"'''ple, Wt' may a pply Bayes' rule (10.88) and calculate the posterior but it is much simpler to apply the following procedure to thf' <'Onditional mean a nd cond it ional variance in the multivariate (Theorem 2.8 on page 27). It holds t hat

""'b"ti"."

(1O. 93)
And
Wt'

get th' priQr distribution

I)' I

that

(X tIYt_d-N( AX,

BUI . ., AE:

lit

, A I ...

EI)

(I O.9')