MARKOV CHAINS

Syllabus Chapter-2 (Second

Part)
Markov Chains & Simulation
Techniques:
Markov
chains:
Applications related to management
functional areas, Implications of
Steady state Probabilities, Decision
making based on the inferences
Monte Carlo Simulation, scope and
limitations.

Outline
Introduction to Markov Chains
Applications Related to Management
Functional Areas.
Implications of Steady state
Probabilities
Introduction to Simulation Techniques:
Decision
making
based
on
the
inferences Monte Carlo Simulation,
scope and limitations.

MARKOV CHAINS
a. Markov Chains: A sequence of events(outcomes) in which , an event
depends upon the immediate preceding event (only) ,but not on the other
prior events is called as Markov Chain or Markov Process. Eg. The
market shares for a product during a month or year , condition of
machines to be used for production each week etc.
b. States: Each of these systems consist of several possible states ,Eg the
various brands of the product represent the states for the market share
problem, similarly working ,fairly working and non-working conditions
represent the states of the machines etc.
The states are assumed to be finite in number and are collectively
exhaustive and mutually exclusive .A state is called as an absorbing state
if there is no tendency to leave that state , otherwise it is an nonabsorbing state. Eg If a customer once uses a brand ,does not change it
at all, then it is an absorbing state.
Markovian analysis is used to study the probabilities corresponding to
the states at any given time period, considering the movements from one
state to another.

Continued.
c. Transition Probabilities: The probabilities of the system to change
from a state (i) to the state (j) is called as a transition probability
and it represents the likelihood of the system to change the states,
from one period to the next. These transition probabilities are
assumed to remain constant over a period of time.
Eg: Probabilities that customers change their brand of a product from
A to other say B and C etc.
d. Transition Matrix(P) : A matrix representing the states in one
period (rows) and the state in the next period (columns) ,along
with the transition probabilities between them is called as a
transition matrix(P).

Example:1.
If over a time ,it is found that 70% of the customers using brand A
continue to use it next year while 20% shift to brand B and 10% to C.
Similarly ,60% of customers using B continue to use it while 25%
change it to A and 15% shift to C and for C, 75% are retained while
20% are lost to A and 5% to B .
Solution: Transition Probability from A to A ie retention probability
of A =70% ie 0.7
Transition Probability from A to B =20% ie 0.2
Transition probability from A to C =10% ie 0.1
Similarly, Transition Probability from B to A =25% ie 0.25
Transition probability from B to B ie retention probability of B =60%
ie 0.6 and Transition probability from B to C =15% ie 0.15.
Similarly , we can write the transition probabilities from C to all the
brands and thus ,we construct the transition matrix as follows:

Continued..
Next State (n=1)
A
B
C
P= Present State A 0.70
0.20
B 0.25

0.60

0.10
0.15

C 0.20
0.05
0.75
-----------Retention & Loss---------------

We can represent the states of a system and transition probability

by two types of diagrams as follows.

1) Transition Diagram:

For the previous example , we can

draw the diagram as follows:

Diagram????

2) Probability Tree Diagram: This is drawn as follows If the customer

is buying brand A in period(n=0) then
Diagram????

Initial Condition
e. Initial Condition:

It represents the probabilities for the

various states(called state probabilities), for the initial period of time.
Eg If initially ,the market share of brands A,B and C are 50%, 30% and
20% respectively ,then the initial condition(for period n=0) is given by
R0 = [ 0.5 0.3 0.2]
f. Note:
i) If the state probabilities at a given period of time depend only upon
those of the preceding immediate period ,it is a first order Markov
Chain. If they depend upon the state probabilities of the last two
periods , it is a second order Markov Chain and so on.
ii) It is assumed that the transitions between the states take place only
once in one period of time eg customers shift the brand only once in
a year.

A. To find the State probabilities for the Kth period of Time

Markov Chains can be used to predict the future. Thus if a1, a2 ,a3
etc represent the probabilities for the various state (state
probabilities) in the initial period (n=0) we can represent them by
a row matrix as
R0 = [a1 a2 a3 .]
Hence ,the state probabilities for the next period (n=1) are given
by the matrix product of R0 and the transition matrix P. Thus ,
R1=R0 x P
Similarly , the state probabilities for the period (n=2) are given by
R2 =R1xP = R0xP2 Where =P2 =P x P
Thus ,in general the state probabilities for the kth period are
given by the row matrix
Rk = Rk-1 x P = R0 x P k

B. Steady State Probabilities

B. Steady State Condition: If the transitions from one state to the others continue
indefinitely ,we reach a stage where the system becomes stable and the state probabilities
tend to remain constant. This is the steady state (equilibrium) condition:
This is symbolically given by
R k = R k-1
and we have Rk = Rk-1xP
R k = R k x P [ Rk-1 = Rk]
Thus, if SA, SB, SC etc are the steady state probabilities then
[a

c] = [a

c]x P

Which gives us simultaneous equations a, b,c etc.

Also we always have ,a+b+c=1, Which is another equation.
Solving all these equations simultaneously , we get the values of steady state probabilities
a ,b, and c..etc.

Problems
Q1) Consider the case where the present market shares of three
brands of soft drinks A,B and C are 60%, 30% and 10%. Also ,let
their transition probability matrix (on the basis of the shifting
pattern for a year) be
A
A 0.7

B
0.2

C
0.1

B 0.2

0.6

0.2

C 0.1

0.1

0.8

P=

Continued.
Q2) Market share of Brand A,B and C are 50%, 30% and 20%
.Customers shifts the brands . Brands switching matrix every
quarter is given below.
TO
From

50%

30%

20%

20%

70%

10%

20%

20%

60%

Find market share at the end of quarter.

Continued..
Q3) Market survey is made on two brands of breakfast foods A and B
.Every time a customer purchases ,he may buy the same brand or
switch to another brand. The transition matrix is given below.
TO

From

0.8

0.2

0.6

0.4

At present 60% of people buy brand A and 40% buy brand B.

Determine market shares of brands A and B in the steady state.

Continued

Q4) A student tries to be punctual for the classes. If he is 90 percent

sure to be on time next day. Similarly ,if he is on time ,then there
is a 30 percent chance that he will be late on the next day. How
often ,in the long run ,is expected to be late for the class?
on time

late

on time

70%

30%

late

90%

10%

P=

Continued
Q5. Suppose that new razar blades were introduced in the market by
three companies at the same time. When they were introduced ,each
company had equal shares of the number, but during the first year
some changes took place which are shown by the following transition
matrix.
A

0.9

0.03

0.07

0.1

0.7

0.2

0.1

0.1

0.8

Assuming that no changes in the buying habits of the customers occur,

i) What is the market share of the three companies at the end of the
first year and second year?
ii) What are the long run market shares of the companies?

Continued.
Q6.) A manufacturing company has a certain piece of equipment that is inspected at the end of
each day and is classified as just overhauled ,good, fair or inoperative. If the piece is
inoperative, it is overhauled, a procedure that takes one day. Assume that the working condition
of the equipment follows as a Markov Chain with the transition matrix given below .

It cost Rs 125 to overhaul a machine (including lost time) on an average and Rs

75 as production lost if a machine is found inoperative. Using the steady state
probabilities ,compute the expected per day cost of maintenance.