STATIONARY DISTRIBUTION OF

MARKOV CHAINS
MICHAEL TWITO
July 21, 2018

1 Introduction
In this note we take a closer look on recurrent states and the regenerative
nature of the chain.

2 Claims
Claim: If a and b are two positive recurrent states then

ExTa+
ExTb+

Is the expected no. of visits to b between two successive visits to a.

Proof. By the law of large numbers for Markov chains we noted that
t PTa+ −1
1X Ex[ k=0 f (Xn )|X0 = a]
lim f (Xn ) = (1)
t→∞ t
n=1 ExTa+

holds with probability 1. We also have the special case of this for δa , that is
t
1X 1
lim 1{Xn = a} = (2)
t→∞ t
n=1 ExTa+

as b is positive recurrent, we may replace a by b in (2),and from (1) it follows

that P a+ −1
1X t
Ex[ Tk=0 1{Xn = b}|X0 = a]
lim 1{Xn = b} = (3)
t→∞ t
n=1 ExTa+

1
so that PTa+ −1
1 Ex[ k=0 1{Xn = b}|X0 = a]
+
= (4)
ExTa ExTa+
then +
a −1
TX
ExTa+
Ex[ 1{Xn = b}|x0 = a] = (5)
n=0 ExTb+

Claim: Whenver f = (fi )i∈Ω is a bounded vector and where P is the tran-
sition matrix of the chain

hπa , P f i = hπa , f i

holds.
Proof. Start with the definition
X
hπa , P f i = πa (x)(P f )(x).
x∈Ω

We note that X
(P f )(x) = hPx∗ , f i = f (y).
y∈Ω

Then owing to πa = πa P we arrive at

X X X X X
hπa , P f i = πa (x) Pxy f (y) = f (y) πa (x)Pxy = f (y)πa (y) = hf, πa i.
x∈Ω y∈Ω y∈Ω y∈Ω y∈Ω

Claim: In a finite irreducible chain ExTyx < ∞ for ∀x, y ∈ Ω .

Proof. Finiteness and irreducibility of the chain implies that there is an n
j
and an  > 0 s.t for any two state z, w there is a j ≤ n such that Pzw > .
Then for every value of Xt the probability of hitting x within the time frame
[t, t + n] is at least . We have that

Pr[Tyx > kn] = Pr[X0 = y, X1 6= x, ..., Xn 6= x]×

Pr[Reaching x from the state reached at Xn requires > (k − 1)n steps|X0 = y, ..., Xn 6= x]
By our first statement we know that

Pr[X0 6= x, ..., Xn 6= x] ≤ 1 − 

2
The second probability is simply

Pr[Tx+ > (k − 1)n|X0 = y] = Pr[Tyx > (k − 1)n]

Hence,
Pr[Tyx > kn] ≤ (1 − )Pr[Tyx > (k − 1)n] ≤ (1 − )k
Holds for every k ≥ 0 Now trough the fact that
X
ExTyx = Pr[Tyx > t]
t≥0

owing the fact that Pr[Tyx > t] decreases with t we may write
X X
ExTyx = Pr[Tyx > t] ≤ nPr[Tyx > kn]
t≥0 k≥0

We have assigned each t ∈ [kn, (k + 1)n] the weight Pr[Tyx ≥ kn] which is
larger.Owing to summetion of geometric series we have that

(1 − )k < ∞
X X
nPr[Tyx > kn] ≤ n
k≥0 k≥0