Professional Documents
Culture Documents
Matrices: A review
Toeplitz and Circulant
Matrices: A review
Robert M. Gray
Chapter 1 Introduction 1
2.1 Eigenvalues 5
2.2 Matrix Norms 8
2.3 Asymptotically Equivalent Matrices 11
2.4 Asymptotically Absolutely Equal Distributions 18
v
vi CONTENTS
Acknowledgements 75
References 77
Abstract
t0 t−1 t−2 · · · t−(n−1)
t1 t0 t−1
t2 ..
t1 t0 .
.. ..
. .
tn−1 ··· t0
vii
1
Introduction
1
2 Introduction
2.1 Eigenvalues
The eigenvalues αk and the (right) eigenvectors (n-tuples) xk of an
n × n matrix A are the solutions to the equation
Ax = αx (2.1)
and hence the eigenvalues are the roots of the characteristic equation
of A:
det(A − αI) = 0 . (2.2)
If the eigenvalues are real, then we will usually assume that they are
ordered in nonincreasing fashion, i.e., α1 ≥ α2 ≥ α3 · · · .
5
6 The Asymptotic Behavior of Matrices
A = U RU ∗ , (2.3)
As the result is both important and simple prove, we state and prove it
formally. The result will be useful in specifying the interval containing
the eigenvalues of an Hermitian matrix.
Usually in books on matrix theory it is proved as a corollary to
the variational description of eigenvalues given by the Courant-Fischer
theorem (see, e.g., [15], p. 116, for the case of real symmetric matrices),
but the following result is easily demonstrated directly.
We have
Tr{A∗ A} = Tr{R∗ R}
n−1
X n−1
X
= |rj,k |2
k=0 j=0
n−1
X X
= |αk |2 + |rj,k |2
k=0 k6=j
n−1
X
≥ |αk |2 (2.12)
k=0
Equation (2.12) will hold with equality iff R is diagonal and hence iff
A is normal. 2
Lemma 2.2 is a direct consequence of Shur’s theorem ([15], pp. 229-
231) and is also proved in [13], p. 106.
called the Frobenius norm or Euclidean norm when the scaling term n
is removed) — will be used here ([13], pp. 102-103).
Let A be a matrix with eigenvalues αk and let λk ≥ 0 be the eigen-
values of the Hermitian nonnegative definite matrix A∗ A. The strong
norm k A k is defined by
If A is itself Hermitian, then its eigenvalues αk are real and the eigen-
values λk of A∗ A are simply λk = α2k . This follows since if e(k) is an
eigenvector of A with eigenvalue αk , then A∗ Ae(k) = αk A∗ e(k) = α2k e(k) .
Thus, in particular, if A is Hermitian then
n−1
!1/2
X
−1 ∗ 1/2 −1
= (n Tr[A A]) = n λk . (2.17)
k=0
√
The quantity n|A| is sometimes called the Frobenius norm or Eu-
clidean norm. From lemma 2.2 we have
n−1
X
2 −1
|A| ≥ n |αk |2 , with equality iff A is normal. (2.18)
k=0
10 The Asymptotic Behavior of Matrices
2. k A + B k≤k A k + k B k
3. k cA k= |c|· k A k
.
(2.20)
The triangle inequality in (2.20) will be used often as is the following
direct consequence:
k A − B k≥ | k A k − k B k |. (2.21)
The weak norm is usually the most useful and easiest to handle of
the two but the strong norm is handy in providing a bound for the
product of two matrices as shown in the next lemma.
XXXX
= n−1 gi,k ḡi,m hk,j h̄m,j (2.23)
i j k m
X
= n−1 h∗j G∗ Ghj ,
j
2.3. Asymptotically Equivalent Matrices 11
and therefore
X
|GH|2 ≤ n−1 k G k2 h∗j hj =k G k2 ·|H|2 .
j
2
Lemma 2.2 is the matrix equivalent of 7.3a of ([13], p. 103). Note
that the lemma does not require that G or H be Hermitian.
k An k , k Bn k≤ M < ∞ (2.24)
and
(2) An − Bn = Dn goes to zero in weak norm as n → ∞:
Theorem 2.1.
(1) If An ∼ Bn , then
Proof.
≤ k An k ·|Cn − Dn |+ k Dn k ·|An − Bn |
−→
n→∞ 0.
(4)
|A−1 −1
n − Bn | = |Bn−1 Bn An − Bn−1 An A−1
n |
≤ k Bn−1 k · k A−1
n k ·|Bn − An |
−→
n→∞ 0.
(5)
Bn − A−1
n Cn = A−1 −1
n An Bn − An Cn
≤ k A−1
n k ·|An Bn − Cn |
−→
n→∞ 0.
2
The above results will be useful in several of the later proofs. Asymp-
totic equality of matrices will be shown to imply that eigenvalues, prod-
ucts, and inverses behave similarly. The following lemma provides a
prelude of the type of result obtainable for eigenvalues and will itself
serve as the essential part of the more general results to follow. It shows
that if the weak norm of the difference of the two matrices is small, then
the sums of the eigenvalues of each must be close.
= Tr(D).
Applying the Cauchy-Schwartz inequality (see, e.g., [19], p. 17) to
Tr(D) yields
2
n−1
X
|Tr(D)|2 = dk,k
k=0
n−1
X
≤ n |dk,k |2
k=0
n−1
X n−1
X
≤ n |dk,j |2
k=0 j=0
= n2 |D|2 . (2.27)
14 The Asymptotic Behavior of Matrices
or, equivalently,
n−1
X
lim n−1 (αn,k − βn,k ) = 0. (2.29)
n→∞
k=0
−→
n→∞ 0
−→
if |Dn | n→∞ 0, yielding the following corollary.
2.3. Asymptotically Equivalent Matrices 15
∆
Proof. Let An = Bn + Dn as in Corollary 2.1 and consider Asn − Bns =
∆n . Since the eigenvalues of Asn are αsn,k , (2.34) can be written in terms
of ∆n as
lim n−1 Tr∆n = 0. (2.35)
n→∞
The matrix ∆n is a sum of several terms each being a product of ∆n ’s
and Bn ’s but containing at least one Dn . Repeated application of lemma
2.3 thus gives
−→
|∆n | ≤ K ′ |Dn | n→∞ 0. (2.36)
where K ′ does not depend on n. Equation (2.36) allows us to apply
Corollary 2.1 to the matrices Asn and Dns to obtain (2.35) and hence
(2.34). 2
16 The Asymptotic Behavior of Matrices
and hence
n−1 n−1
" # " #
Y Y
lim exp n−1 ln αn,k = lim exp n−1 ln βn,k
n→∞ n→∞
k=0 k=0
or equivalently
n−1
X
−1
n |αk − βk |2 ≤ |A − B|2 .
k=0
|A − B| = |U diag(αk )U ∗ − W diag(βk )W ∗ |
= |diag(αk )U ∗ − U ∗ W diag(βk )W ∗ |
= |diag(αk )U ∗ W − U ∗ W diag(βk )|
where we have defined pi,j − n−1 |qi,j |2 Since Q is unitary, we also have
that
n−1
X n−1
X
2
|qi,j | = |qi,j |2 = 1 (2.42)
i=0 j=0
or
n−1 n−1
X X 1
pi,j = pi,j = . (2.43)
n
i=0 j=0
which will prove the result. To see this suppose the contrary. Let ℓ
be the smallest integer in {0, 1, . . . , n − 1} such that P has a nonzero
element off the diagonal in either row ℓ or in column ℓ. If there is a
nonzero element in row ℓ off the diagonal, say pℓ,a then there must also
20 The Asymptotic Behavior of Matrices
be a nonzero element in column ℓ off the diagonal, say pb,ℓ in order for
the constraints (2.43) to be satisfied. Since ℓ is the smallest such value,
ℓ < a and ℓ < b. Let x be the smaller of pl,a and pb,l . Form a new
matrix P ′ by adding x to pℓ,ℓ and pb,a and subtracting x from pb,ℓ and
pℓ,a . The new matrix still satisfies the constraints and it has a zero in
either position (b, ℓ) or (ℓ, a). Furthermore the norm of P ′ has changed
from that of P by an amount
= −x(αℓ − αb )(βℓ − βa ) ≤ 0
Note in particular that the absolute values are outside the sum in
lemma 2.4 and inside the sum in lemma 2.6. As was done in the weaker
case, the result can be used to prove a stronger version of theorem
2.4. This line of reasoning, using the Wielandt-Hoffman theorem, was
pointed out by William F. Trench who used special cases in his pa-
per [20]. Similar arguments have become standard for treating eigen-
value distributions for Toeplitz and Hankel matrices. See, for example,
[7, 28, 3]. The following theorem provides the derivation. The specific
2.4. Asymptotically Absolutely Equal Distributions 21
n−1
X
lim n−1 |F (αn,k ) − F (βn,k )| = 0. (2.45)
n→∞
k=0
Note that the theorem strengthens the result of theorem 2.4 because
of the magnitude inside the sum. Following Trench [?] in this case the
eigenvalues are said to be asymptotically absolutely equally distributed.
which implies (2.45) for the case F (r) = r. For any nonnegative integer
j
|αjn,k − βn,k
j
| ≤ j max(|m|, |M |)j−1 |αn,k − βn,k |. (2.47)
so that
aj − bj |aj − bj |
| | =
a−b |a − b|
j
X
= | aj−l bl−1 |
l=1
j
X
≤ |aj−l bl−1 |
l=1
j
X
= |a|j−l |b|l−1
l=1
≤ j max(|m|, |M |)j−1 ,
which proves (2.47). This immediately implies that (2.45) holds for
functions of the form F (r) = r j for positive integers j, which in turn
means the result holds for any polynomial. If F is an arbitrary contin-
uous function on [m, M ], then from theorem 2.3 given ǫ > 0 there is a
polynomial P such that
n−1
X
−1
= n |F (αn,k ) − P (αn,k ) + P (αn,k ) − P (βn,k ) + P (βn,k ) − F (βn,k )|
k=0
n−1
X n−1
X
≤ n−1 |F (αn,k ) − P (αn,k )| + n−1 |P (αn,k ) − P (βn,k )|
k=0 k=0
n−1
X
+n−1 |P (βn,k ) − F (βn,k )|
k=0
n−1
X
≤ 2ǫ + n−1 |P (αn,k ) − P (βn,k )|
k=0
The properties of circulant matrices are well known and easily derived
([15], p. 267,[6]). Since these matrices are used both to approximate and
explain the behavior of Toeplitz matrices, it is instructive to present
one version of the relevant derivations here.
25
26 Circulant Matrices
and eigenvector
y (m) = n−1/2 1, e−2πim/n , · · · , e−2πi(n−1)/n .
= n−1/2 e−2πimk/n ; m, k = 0, 1, . . . , n − 1
Ψ = {ψk δk−j }
where δ is the Kronecker delta,
(
1 m=0
δm = .
0 otherwise
To verify (3.8) denote that the (k, j)th element of U ΨU ∗ by ak,j and
observe that ak,j will be the product of the kth row of U Ψ, which
is {n−1/2 e−2πimk/n Ψk ; m = 0, 2, . . . , n − 1}, times the jth row of U ,
{n−1/2 e2πimj/n ; m = 0, 2, . . . , n − 1} so that
n−1
X
ak,j = n−1 e2πim(j−k)/n ψm
m=0
n−1
X n−1
X
= n−1 e2πim(j−k)/n cr e−2πimr/n
m=0 r=0
n−1
X n−1
X
= n−1 cr e2πim(j−k−r)/n . (3.9)
r=0 m=0
But we have
n−1
(
X
2πim(j−k−r)/n n k − j = −r mod n
e = .
m=0 0 otherwise
so that ak,j = c−(k−j) mod n . Thus (3.8) and (3.1) are equivalent. Fur-
thermore (3.9) shows that any matrix expressible in the form (3.8) is
circulant.
28 Circulant Matrices
3.2 Properties
The following theorem summarizes the properties derived in the previ-
ous section regarding eigenvalues and eigenvectors of circulant matrices
and provides some easy implications.
n−1
X
βm = bk e−2πimk/n ,
k=0
respectively.
CB = BC = U ∗ γU ,
C + B = U ∗ ΩU,
C −1 = U ∗ Ψ−1 U
(1) CB = U ∗ ΨU U ∗ ΦU
= U ∗ ΨΦU
= U ∗ ΦΨU = BC
Since ΨΦ is diagonal, (3.9) implies that CB is circulant.
(2) C + B = U ∗ (Ψ + Φ)U
(3) C −1 = (U ∗ ΨU )−1
= U ∗ Ψ−1 U
if Ψ is nonsingular.
2
Circulant matrices are an especially tractable class of matrices since
inverses, products, and sums are also circulant matrices and hence both
straightforward to construct and normal. In addition the eigenvalues of
such matrices can easily be found exactly.
In the next chapter we shall see that certain circulant matrices
asymptotically approximate Toeplitz matrices and hence from Chapter
2 results similar to those in theorem 3.1 will hold asymptotically for
Toeplitz matrices.
4
Toeplitz Matrices
∞
X
|tk |2 < ∞ . (4.1)
k=−∞
31
32 Toeplitz Matrices
This assumption greatly simplifies the mathematics but does not alter
the fundamental concepts involved. As the main purpose here is tutorial
and we wish chiefly to relay the flavor and an intuitive feel for the
results, we here confine interest to the absolutely summable case. The
main advantage of (4.2) over (4.1) is that it ensures the existence and
continuity of the Fourier series f (λ) defined by
∞
X n
X
f (λ) = tk eikλ = lim tk eikλ . (4.3)
n→∞
k=−∞ k=−n
Not only does the limit in (4.3) converge if (4.2) holds, it converges
uniformly for all λ, that is, we have that
n
−n−1 ∞
X X X
f (λ) − tk eikλ = tk eikλ + tk eikλ
k=−n k=−∞ k=n+1
−n−1 ∞
X
X ikλ ikλ
≤ tk e + tk e ,
k=−∞ k=n+1
−n−1
X ∞
X
≤ |tk | + |tk |
k=−∞ k=n+1
where the righthand side does not depend on λ and it goes to zero as
n → ∞ from (4.2), thus given ǫ there is a single N , not depending on
λ, such that
n
X
ikλ
f (λ) − t e ≤ ǫ , all λ ∈ [0, 2π] , if n ≥ N. (4.4)
k
k=−n
∞
∆
X
≤ |tk | = M|f | < ∞ .
k=−∞
n−1
X n−1
X Z 2π
= (2π)−1 f (λ)ei(k−j)λ dλ xk x̄j (4.9)
k=0 j=0 0
2
Z 2π n−1
X
= (2π)−1 xk eikλ f (λ) dλ
0
k=0
and likewise
n−1
X Z 2π n−1
X
∗ 2 −1
x x= |xk | = (2π) dλ| xk eikλ |2 . (4.10)
k=0 0 k=0
k Tn (f ) k = k Tn (fr + ifi ) k
≤ k Tn (fr ) k + k Tn (fi ) k
≤ M|fr | + M|fi | .
Since |(f ± f ∗ )/2 ≤ (|f |+ |f ∗ |)/2 ≤ M|f | , M|fr | + M|fi | ≤ 2M|f | , proving
(4.7). 2
Note for later use that the weak norm between Toeplitz matrices
has a simpler form than (2.14). Let Tn = {tk−j } and Tn′ = {t′k−j } be
Toeplitz, then by collecting equal terms we have
n−1
X n−1
X
|Tn − Tn′ |2 = n−1 |tk−j − t′k−j |2
k=0 j=0
n−1
X
= n−1 (n − |k|)|tk − t′k |2
k=−(n−1)
. (4.12)
n−1
X
= (1 − |k|/n)|tk − t′k |2
k=−(n−1)
We are now ready to put all the pieces together to study the asymp-
totic behavior of Tn . If we can find an asymptotically equivalent circu-
lant matrix then all of the results of Chapters 2 and 3 can be instantly
36 Toeplitz Matrices
applied. The main difference between the derivations for the finite and
infinite order case is the circulant matrix chosen. Hence to gain some
feel for the matrix chosen we first consider the simpler finite order case
where the answer is obvious, and then generalize in a natural way to
the infinite order case.
t0 t−1 · · · t−m
t1 t0
..
. 0
.. ..
. .
t
m
..
.
T = .
tm ··· t1 t0 t−1 · · · t−m
..
. t−m
..
.
0 t0 t−1
tm ··· t1 t0
(4.13)
We can make this matrix exactly into a circulant if we fill in the upper
right and lower left corners with the appropriate entries. Define the
4.2. Finite Order Toeplitz Matrices 37
Lemma 4.2. The matrices Tn and Cn defined in (4.13) and (4.14) are
asymptotically equivalent, i.e., both are bounded in the strong norm
and
lim |Tn − Cn | = 0. (4.16)
n→∞
Lemma 4.3. Let Tn and Cn be as in (4.13) and (4.14) and let their
eigenvalues be τn,k and ψn,k , respectively, then for any positive integer
s
n−1
X n−1
X
lim n−1 s
τn,k = lim n−1 s
ψn,k . (4.17)
n→∞ n→∞
k=0 k=0
In fact, for finite n,
n−1 n−1
−1 X s X
n τn,k − n−1 s
ψn,k ≤ Kn−1/2 , (4.18)
k=0 k=0
4.2. Finite Order Toeplitz Matrices 39
Proof. Equation (4.17) is direct from lemma 4.2 and theorem 2.2.
Equation (4.18) follows from Corollary 2.1 and lemma 4.2. 2
Lemma 4.3 is of interest in that for finite order Toeplitz matrices
one can find the rate of convergence of the eigenvalue moments. It can
be shown that K ≤ sMfs−1 .
The above two lemmas show that we can immediately apply the
results of Section II to Tn and Cn . Although theorem 2.1 gives us im-
mediate hope of fruitfully studying inverses and products of Toeplitz
matrices, it is not yet clear that we have simplified the study of the
eigenvalues. The next lemma clarifies that we have indeed found a use-
ful approximation.
n−1
X Z 2π
lim n−1 s
ψn,k = (2π)−1 f s (λ) dλ. (4.19)
n→∞ 0
k=0
n−1
X Z 2π
lim n−1 F (ψn,k ) = (2π)−1 F [f (λ)] dλ. (4.20)
n→∞ 0
k=0
40 Toeplitz Matrices
m
X n−1
X
= t−k e−2πijk/n + tn−k e−2πijk/n . (4.21)
k=0 k=n−m
m
X
= tk e−2πijk/n = f (2πjn−1 )
k=−m
Note that the eigenvalues of Cn are simply the values of f (λ) with λ
uniformly spaced between 0 and 2π. Defining 2πk/n = λk and 2π/n =
∆λ we have
n−1
X n−1
X
−1 s −1
lim n ψn,k = lim n f (2πk/n)s
n→∞ n→∞
k=0 k=0
n−1
X
= lim f (λk )s ∆λ/(2π)
n→∞
k=0
Z 2π
−1
= (2π) f (λ)s dλ, (4.22)
0
where the continuity of f (λ) guarantees the existence of the limit of
(4.22) as a Riemann integral. If Tn and Cn are Hermitian than the ψn,k
and f (λ) are real and application of the Stone-Weierstrass theorem to
(4.22) yields (4.20). Lemma 4.2 and (4.21) ensure that ψn,k and τn,k
are in the real interval [mf , Mf ]. 2
Combining lemmas 4.2-4.4 and theorem 2.2 we have the following
special case of the fundamental eigenvalue distribution theorem.
i.e., the sequences τn,k and f (2πk/n) are asymptotically equally dis-
tributed.
n−1
(n)
X
ck = n−1 f (2πj/n)e2πijk/n . (4.26)
j=0
(n)
and hence the ck are simply the sum approximations to the Riemann
integral giving t−k . Equations (4.26), (3.7), and (3.9) show that the
eigenvalues ψn,m of Cn are simply f (2πm/n); that is, from (3.7) and
(3.9)
n−1
(n)
X
ψn,m = ck e−2πimk/n
k=0
n−1
X n−1
X
= n−1 f (2πj/n)e2πijk/n e−2πimk/n
k=0 j=0 . (4.28)
n−1 n−1
( )
X X
= f (2πj/n) n−1 22πik(j−m)/n
j=0 k=0
= f (2πm/n)
from (3.9)
n−1
(n)
X
ck = n−1 ψn,m e2πimk/n
m=0
, (4.29)
n−1
X
= n−1 f (2πm/n)e2πimk/n
m=0
as in (4.26). Thus, either (4.26) or (4.28) can be used to define Cn (f ).
The fact that lemma 4.4 holds for Cn (f ) yields several useful prop-
erties as summarized by the following lemma.
Lemma 4.5.
Cn (f )−1 = Cn (1/f ).
Proof.
n−1
X ∞
X
= t−1+mn e−2πijl/n
l=0 m=−∞
44 Toeplitz Matrices
n−1
X n−1
X ∞
X
= n−1 t−1+mn e2πij(k−l)/n
j=0 l=0 m=−∞
.
n−1
X ∞
X n−1
X
= t−1+mn n−1 e2πij(k−l)/n
l=0 m=−∞ j=0
∞
X
= t−k+mn
m=−∞
2
Equation (4.30) points out a shortcoming of Cn (f ) for applications
as a circulant approximation to Tn (f ) — it depends on the entire se-
quence {tk ; k = 0, ±1, ±2, · · · } and not just on the finite collection of
elements {tk ; k = 0, ±1, · · · , ±n − 1} of Tn (f ). This can cause problems
in practical situations where we wish a circulant approximation to a
Toeplitz matrix Tn when we only know Tn and not f . Pearl [16] dis-
cusses several coding and filtering applications where this restriction
is necessary for practical reasons. A natural such approximation is to
form the truncated Fourier series
n
fˆn (λ) =
X
tk eikλ , (4.31)
k=−n
which depends only on {tk ; k = 0, ±1, · · · , ±n − 1}, and then define the
4.3. Absolutely Summable Toeplitz Matrices 45
circulant matrix
Ĉn = Cn (fˆn ); (4.32)
(n) (n)
that is, the circulant matrix having as top row (ĉ0 , · · · , ĉn−1 ) where
n−1
(n)
fˆn (2πj/n)e2πijk/n
X
ĉk = n−1
j=0
n−1 n
!
X X
= n−1 tm e2πijk/n e2πijk/n (4.33)
j=0 m=−n
n
X n−1
X
= tm n−1 e2πij(k+m)/n .
m=−n j=0
Proof. Since both Cn (f ) and Ĉn are circulant matrices with the same
eigenvectors (theorem 3.1), we have from part 2 of theorem 3.1 and
(2.14) and the comment following it that
n−1
|f (2πk/n) − fˆn (2πk/n)|2 .
X
|Cn (f ) − Ĉn |2 = n−1
k=0
Recall from (4.4) and the related discussion that fˆn (λ) uniformly con-
verges to f (λ), and hence given ǫ > 0 there is an N such that for n ≥ N
we have for all k, n that
Since ǫ is arbitrary,
lim |Cn (f ) − Ĉn | = 0
n→∞
proving that
Cn (f ) ∼ Ĉn . (4.35)
Next, Ĉn = {t′k−j } and use (4.12) to obtain
n−1
X
|Tn − Ĉn |2 = (1 − |k|/n)|tk − t′k |2 .
k=−(n−1)
and hence
n−1
X
|Tn − Ĉn |2 = |tn−1 |2 + (1 − k/n)(|tn−k |2 + |t−(n−k) |2 )
k=0
.
n−1
X
= |tn−1 |2 + (k/n)(|tk |2 + |t−k |2 )
k=0
lim |tn−1 |2 = 0
n→∞
and hence
n−1
X
lim |Tn − Ĉn | = lim (k/n)(|tk |2 + |t−k |2 )
n→∞ n→∞
k=0
(N −1 n−1
)
X X
= lim (k/n)(|tk |2 + |t−k |2 ) + (k/n)(|tk |2 + |t−k |2 ) .
n→∞
k=0 k=N
N −1 ∞
!
X X
≤ lim n−1 k(|tk |2 + |t−k |2 ) + (|tk |2 + |t−k |2 ) ≤ ǫ
n→∞
k=0 k=N
Since ǫ is arbitrary,
lim |Tn − Ĉn | = 0
n→∞
and hence
Tn ∼ Ĉn , (4.37)
which with (4.35) and theorem 2.1 proves (4.34). 2
Pearl [16] develops a circulant matrix similar to Ĉn (depending only
on the entries of Tn ) such that (4.37) holds in the more general case
where (4.1) instead of (4.2) holds.
48 Toeplitz Matrices
We have
n−1
X
D(x) = lim n−1 1x (τn,k ) .
n→∞
k=0
The idea here is that the upper bound has an output of 1 everywhere
1x does, but then it drops in a continuous linear fashion to zero at x + ǫ
instead of immediately at x. The lower bound has a 0 everywhere 1x
does and it rises linearly from x to x − ǫ to the value of 1 instead of
instantaneously as does 1x . Clearly
1− +
x (α) < 1x (α) < 1x (α)
for all α.
50 Toeplitz Matrices
Since both 1+ −
x and 1x are continuous, theorem 4 can be used to
conclude that
n−1
X
−1
lim n 1+
x (τn,k )
n→∞
k=0
Z
−1
= (2π) 1+
x (f (λ)) dλ
f (λ) − x
Z Z
−1 −1
= (2π) dλ + (2π) (1 − ) dλ
f (λ)≤x x<f (λ)≤x+ǫ ǫ
Z Z
≤ (2π)−1 dλ + (2π)−1 dλ
f (λ)≤x x<f (λ)≤x+ǫ
and
n−1
X
−1
lim n 1−
x (τn,k )
n→∞
k=0
Z
−1
= (2π) 1−
x (f (λ)) dλ
f (λ) − (x − ǫ)
Z Z
= (2π)−1 dλ + (2π)−1 (1 − ) dλ
f (λ)≤x−ǫ x−ǫ<f (λ)≤x ǫ
Z Z
= (2π)−1 dλ + (2π)−1 (x − f (λ)) dλ
f (λ)≤x−ǫ x−ǫ<f (λ)≤x
Z
−1
≥ (2π) dλ
f (λ)≤x−ǫ
Z Z
= (2π)−1 dλ − (2π)−1 dλ
f (λ)≤x x−ǫ<f (λ)≤x
These inequalities imply that for any ǫ > 0, as n grows the sample
average n−1 n−1
P
k=0 1x (τn,k ) will be sandwiched between
Z Z
−1 −1
(2π) dλ + (2π) dλ
f (λ)≤x x<f (λ)≤x+ǫ
and Z Z
(2π)−1 dλ − (2π)−1 dλ.
f (λ)≤x x−ǫ<f (λ)≤x
4.3. Absolutely Summable Toeplitz Matrices 51
Since ǫ can be made arbitrarily small, this means the sum will be
sandwiched between Z
−1
(2π) dλ
f (λ)≤x
and Z Z
−1 −1
(2π) dλ − (2π) dλ.
f (λ)≤x f (λ)=x
Thus if Z
dλ = 0,
f (λ)=x
then Z 2π
D(x) = (2π)−1 1x [f (λ)]dλ
0
Z .
= (2π)−1 v dλ
f (λ)≤x
2
x < z and the function can be thought of as having input y and thresh-
olds z and X and it puts out y if y is between z and x, z if y is smaller
than z, and x if y is greater than x. The following result was proved in
[11] and extended in [22]. See also [23, 24, 25].
imum and maximum of f , and indeed these extrema may not even be
bounded.
h 2 i
= Cn (f )−1 I + Dn Cn (f )−1 + Dn Cn (f )−1 + · · ·
(4.43)
and the expansion converges (in weak norm) for sufficiently large n.
(c) If f (λ) ≥ mf > 0, then
Z π
−1 −1 i(k−j)λ
Tn (f ) ∼ Tn (1/f ) = (2π) dλe /f (λ) ; (4.44)
−π
that is, if the spectrum is strictly positive then the inverse of a Toeplitz
matrix is asymptotically Toeplitz. Furthermore if ρn,k are the eigenval-
ues of Tn (f )−1 and F (x) is any continuous function on [1/Mf , 1/mf ],
then
n−1
X Z π
−1 −1
lim n F (ρn,k ) = (2π) F [(1/f (λ)] dλ. (4.45)
n→∞ −π
k=0
54 Toeplitz Matrices
(d) If mf = 0, f (λ) has at least one zero, and the derivative of f (λ)
exists and is bounded, then Tn (f )−1 is not bounded, 1/f (λ) is not
integrable and hence Tn (1/f ) is not defined and the integrals of (4.41)
may not exist. For any finite θ, however, the following similar fact is
true: If F (x) is a continuous function of [1/Mf , θ], then
n−1
X Z 2π
lim n−1 F [min(ρn,k , θ)] = (2π)−1 F [min(1/f (λ), θ)] dλ.
n→∞ 0
k=0
(4.46)
Proof.
(a) Since f (λ) > 0 except at possible a finite number of points, we
have from (4.9)
Z π n−1 2
∗ 1 X
ikλ
x Tn x = xk e f (λ)dλ > 0
2π −π
k=0
so that Tn (f ) is nonsingular.
(b) From lemma 4.6, Tn ∼ Cn and hence (4.40) follows from theorem
2.1 since f (λ) ≥ mf > 0 ensures that
since
−→
|Dn Cn−1 | ≤k Cn−1 k ·|Dn | ≤ (1/mf )|Dn | n→∞ 0
(4.45) must hold for large enough n. From (4.40), however, if n is large
enough, then probably the first term of the series is sufficient.
4.5. Inverses of Hermitian Toeplitz Matrices 55
(c) We have
|Tn (f )−1 − Tn (1/f )| ≤ |Tn (f )−1 − Cn (f )−1 | + |Cn (f )−1 − Tn (1/f )|.
From (b) for any ǫ > 0 we can choose an n large enough so that
ǫ
|Tn (f )−1 − Cn (f )−1 | ≤ . (4.48)
2
From theorem 3.1 and lemma 4.5, Cn (f )−1 = Cn (1/f ) and from lemma
4.6 Cn (1/f ) ∼ Tn (1/f ). Thus again we can choose n large enough to
ensure that
|Cn (f )−1 − Tn (1/f )| ≤ ǫ/2 (4.49)
so that for any ǫ > 0 from (4.46)-(4.47) can choose n such that
which is (4.42). Equation (4.43) follows from (4.42) and theorem 2.4.
Alternatively, if G(x) is any continuous function on [1/Mf , 1/mf ] and
(4.43) follows directly from lemma 4.6 and theorem 2.4 applied to
G(1/x).
(d) When f (λ) has zeros (mf = 0) then from Corollary 4.2
lim minτn,k = 0 and hence
n→∞ k
which diverges so that 1/f (λ) is not integrable. To prove (4.44) let F (x)
be continuous on [1/Mf , θ], then F [min(1/x, θ)] is continuous on [0, Mf ]
and hence theorem 2.4 yields (4.44). Note that (4.44) implies that the
eigenvalues of Tn−1 are asymptotically equally distributed up to any
finite θ as the eigenvalues of the sequence of matrices Tn [min(1/f, θ)].
2
A special case of part 4 is when Tn (f ) is finite order and f (λ) has
at least one zero. Then the derivative exists and is bounded since
m
X
df /dλ = iktk eikλ
k=−m
.
m
X
≤ |k||tk | < ∞
k=−m
The series expansion of part 2 is due to Rino [6]. The proof of part
4 is motivated by one of Widom [29]. Further results along the lines
of part 4 regarding unbounded Toeplitz matrices may be found in [11].
Related results considering asymptotically equal distributions of un-
bounded sequences can be found in Tyrtyshnikov [28] and Trench [22].
These works extend Weyl’s definition of asymptotically equal distribu-
tions to unbounded sequences and develop conditions for and implica-
tions of such equal distributions.
Extending (a) to the case of non-Hermitian matrices can be some-
what difficult, i.e., finding conditions on f (λ) to ensure that Tn (f ) is
invertible. Parts (a)-(d) can be straightforwardly extended if f (λ) is
continuous. For a more general discussion of inverses the interested
reader is referred to Widom [29] and the cited references. It should be
pointed out that when discussing inverses Widom is concerned with the
4.6. Products of Toeplitz Matrices 57
n−1
X Z 2π
−1
lim n ρsn,k = (2π) −1
[f (λ)g(λ)]s dλ s = 1, 2, . . . . (4.57)
n→∞ 0
k=0
(b) If Tn (t) and Tn (g) are Hermitian, then for any F (x) continuous on
[mf mg , Mf Mg ]
n−1
X Z 2π
−1 −1
lim n F (ρn,k ) = (2π) F [f (λ)g(λ)] dλ. (4.58)
n→∞ 0
k=0
(c)
Tn (f )Tn (g) ∼ Tn (f g). (4.59)
(d) Let f1 (λ), .., fm (λ) be Riemann integrable. Then if the Cn (fi ) are
defined as in (4.29)
m m m
! !
Y Y Y
Tn (fi ) ∼ Cn fi ∼ Tn fi . (4.60)
i=1 i=1 i=1
58 Toeplitz Matrices
m
Y
(e) If ρn,k are the eigenvalues of Tn (fi ), then for any positive integer
i=1
s !s
n−1
X Z 2π m
Y
lim n−1 ρsn,k = (2π)−1 fi (λ) dλ (4.61)
n→∞ 0
k=0 i=1
If the Tn (fi ) are Hermitian, then the ρn,k are asymptotically real,
i.e., the imaginary part converges to a distribution at zero, so that
n−1 Z 2π Ym
!s
s
X
−1 −1
lim n (Re[ρn,k ]) = (2π) fi (λ) dλ. (4.62)
n→∞ 0
k=0 i=1
n−1
(ℑ[ρn,k ])2 = 0.
X
lim n−1 (4.63)
n→∞
k=0
Proof.
(a) Equation (4.53) follows from lemmas 4.5 and 4.6 and theorems 2.1
and 3. Equation (4.54) follows from (4.53). Note that while Toeplitz
matrices do not in general commute, asymptotically they do. Equation
(4.55) follows from (4.53), theorem 2.2, and lemma 4.4.
(b) Proof follows from (4.53) and theorem 2.4. Note that the eigenval-
ues of the product of two Hermitian matrices are real ([15], p. 105).
(c) Applying lemmas 4.5 and 4.6 and theorem 2.1
vanishes in the limit. Let ρn,k = αn,k + iβn,k where αn,k and βn,k are
real. Then from theorem 2.2 we have for any positive integer s
n−1
X n−1
X
lim n−1 (αn,k + iβn,k )s = lim n−1 s
ψn,k
n→∞ n→∞
k=0 k=0
"m #s
Z 2π Y
−1
= (2π) fi (λ) dλ (4.64)
0 i=1
m
!
Y
where ψn,k are the eigenvalues of Cn fi . From (2.14)
i=1
2
n−1
X n−1
X Ym
n−1 |ρn,k |2 = n−1 α2n,k + βn,k
2
≤ Tn (fi ) .
k=0 k=0 i=i
m
!2
Z 2π Y
= (2π)−1 fi (λ) dλ (4.65)
0 i=1
Thus the distribution of the imaginary parts tends to the origin and
hence #s
n−1
X m
Z 2π "Y
lim n−1 αsn,k = (2π)−1 fi (λ) dλ.
n→∞ 0
k=0 i=1
2
Parts (d) and (e) are here proved as in Grenander and Szegö ([13],
pp. 105-106.
We have developed theorems on the asymptotic behavior of eigen-
values, inverses, and products of Toeplitz matrices. The basic method
60 Toeplitz Matrices
n−1
1 1 1 X m
ln (det(Cn )) n = ln detCn = ln f (2π ).
n n m=0 n
63
64 Applications to Stochastic Time Series
two common linear models. For simplicity we will assume the process
means are zero.
From (5.9) it is clear that rk,j is not Toeplitz because of the min(k, j) in
the sum. However, as we next show, as n → ∞ the upper limit becomes
5.1. Moving Average Processes 67
(n)
large and RU becomes asymptotically Toeplitz. If we define
∞
X
b(λ) = bk eikλ (5.10)
k=0
then
Bn = Tn (b) (5.11)
so that
(n)
RU = σ 2 Tn (b)Tn (b)∗ . (5.12)
We can now apply the results of the previous sections to obtain the
following theorem.
Xn = 0 n < 0. (5.16)
Autoregressive process include nonstationary processes such as the
Wiener process. Equation (5.16) can be rewritten as a vector equation
by defining the lower triangular matrix.
1
a 1 0
1
a1 1
An =
.. ..
(5.17)
. .
an−1 a1 1
so that
An X n = W n .
We have
(n) (n)
RW = An RX A∗n (5.18)
since det An = 1 6= 0, An is nonsingular so that
(n)
RX = σ −2 A−1 −1∗
n An (5.19)
or
(n)
(RX )−1 = σ 2 A∗n An (5.20)
(n)
or equivalently, if (RX )−1 = {tk,j } then
n n−max(k,j)
X X
tk,j = ām−k am−j = am am+(k−j) .
m=0 m=0
Unlike the moving average process, we have that the inverse covariance
matrix is the product of Toeplitz triangular matrices. Defining
∞
X
a(λ) = ak eikλ (5.21)
k=0
5.2. Autoregressive Processes 69
we have that
(n)
(RX )−1 = σ −2 Tn (a)∗ Tn (a) (5.22)
and hence the following theorem.
∼ σ 2 Tn (1/a)Tn (1/a)∗
5.3 Factorization
Consider the problem of the asymptotic behavior of triangular factors
of a sequence of Hermitian covariance matrices Tn (f ). It is well known
that any such matrix can be factored into the product of a lower tri-
angular matrix and its conjugate transpose ([12], p. 37), in particular
Tn (f ) = {tk,j } = Bn Bn∗ , (5.28)
where Bn is a lower triangular matrix with entries
(n)
bk,j = {(det Tk ) det(Tk−1 )}−1/2 γ(j, k), (5.29)
5.3. Factorization 71
where γ(j, k) is the determinant of the matrix Tk with the right hand
column replaced by (tj,0 , tj,1 , . . . , tj,k−1 )t . Note in particular that the
diagonal elements are given by
(n)
bk,k = {(det Tk )/(det Tk−1 )}1/2 . (5.30)
b̄(λ) = b(−λ)
∞
X . (5.31)
b(λ) = bk eikλ
k=0
b0 = 1
From (5.28) and theorem 4.4 we have
Bn ∼ Tn (σb). (5.33)
n−1
(n) 2
X
lim σ −2 n−1 bk,k = 1. (5.36)
n→∞
k=0
From (5.30) and Corollary 2.4, and the fact that Tn (σb) is triangular
we have that
n−1
(n)
X
lim σ −1 n−1 bk,k = σ −1 lim {(det Tn (f ))/(det Tn−1 (f ))}1/2
n→∞ n→∞
k=0
.
= σ −1 lim {det Tn (f )}1/2n σ −1 lim {det Tn (σb)}1/n
n→∞ n→∞
= σ −1 · σ = 1
(5.37)
Combining (5.36) and (5.37) yields
75
References
77
78 References
[13] U. Grenander and G. Szegö, Toeplitz Forms and Their Applications, University
of Calif. Press, Berkeley and Los Angeles, 1958.
[14] T. Hashimoto and S. Arimoto, “On the rate-distortion function for the nonsta-
tionary Gaussian autoregressive process,” IEEE Transactions on Information
Theory, Vol. IT-26, pp. 478-480, 1980.
[15] P. Lancaster, Theory of Matrices, Academic Press, NY, 1969.
[16] J. Pearl, “On Coding and Filtering Stationary Signals by Discrete Fourier
Transform,” IEEE Trans. on Info. Theory, IT-19, pp. 229–232, 1973.
[17] C.L. Rino, “The Inversion of Covariance Matrices by Finite Fourier Trans-
forms,” IEEE Trans. on Info. Theory, IT-16, No. 2, March 1970, pp. 230–232.
[18] J. Rissanen and L. Barbosa, “Properties of Infinite Covariance Matrices and
Stability of Optimum Predictors,” Information Sciences, 1, 1969, pp. 221–236.
[19] W. Rudin, Principles of Mathematical Analysis, McGraw-Hill, NY, 1964.
[20] W. F. Trench, “Asymptotic distribution of the even and odd spectra of real
symmetric Toeplitz matrices,” Linear Algebra Appl., Vol. 302-303, pp. 155-162,
1999.
[21] W. F. Trench,, Absolute equal distribution of the spectra of Hermitian matrices,
Lin. Alg. Appl., 366 (2003), 417-431.
[22] W. F. Trench, Absolute equal distribution of families of finite sets, Lin. Alg.
Appl. 367 (2003), 131-146.
[23] W. F. Trench, A note on asymptotic zero distribution of orthogonal polynomials,
Lin. Alg. Appl. 375 (2003) 275–281
[24] W. F. Trench, Simplification and strengthening of Weyl’s definition of asymp-
totic equal distribution of two families of finite sets, Cubo A Mathematical
Journal Vol. 06 N 3 (2004), 47–54.
[25] W. F. Trench, Absolute equal distribution of the eigenvalues of discrete Sturm–
Liouville problems, J. Math. Anal. Appl., in press.
[26] B.S. Tsybakov, “Transmission capacity of memoryless Gaussian vector chan-
nels,” (in Russian),Probl. Peredach. Inform., Vol 1, pp. 26–40, 1965.
[27] B.S. Tsybakov, “On the transmission capacity of a discrete-time Gaussian chan-
nel with filter,” (in Russian),Probl. Peredach. Inform., Vol 6, pp. 78–82, 1970.
[28] E.E. Tyrtyshnikov, “A unifying approach to some old and new theorems on
distribution and clustering,” Linear Algebra and its Applications, Vol. 232, pp.
1-43, 1996.
[29] H. Widom, “Toeplitz Matrices,” in Studies in Real and Complex Analysis,
edited by I.I. Hirschmann, Jr., MAA Studies in Mathematics, Prentice-Hall,
Englewood Cliffs, NJ, 1965.
[30] A.J. Hoffman and H. W. Wielandt, “The variation of the spectrum of a normal
matrix,” Duke Math. J., Vol. 20, pp. 37-39, 1953.
[31] James H. Wilkinson, “Elementary proof of the Wielandt-Hoffman theorem and
of its generalization,” Stanford University, Department of Computer Science
Report Number CS-TR-70-150, January 1970 .
Index
79
80 INDEX
discrete time, 64
Rayleigh quotient, 6
Riemann integrable, 48, 53, 61
Taylor series, 16
time series, 63
Toeplitz determinant, 60
Toeplitz matrix, 1, 26, 31
Toeplitz matrix, finite order, 31
trace, 8
transpose, 26
triangle inequality, 10
triangular, 63, 66, 68, 70, 72
two-sided, 64
uniform convergence, 32
uniformaly bounded, 38
unitary, 6
upper triangular, 6
weak norm, 9
weakly stationary, 64
asymptotically, 64
white noise, 65
Wielandt-Hoffman theorem, 18,
20
Wiener-Hopf factorization, 63