writing down markov chain transition matrix

Question

Question:

An experimental animal can stay in room-A until 1 minute,and it can stay in room-B until 2 minutes. There exist deadly gases in room-C. One room among these three rooms is being randomly chosen per a minute. And The animal goes to this chosen room.

How to write down transition matrix?

---

Answer

State space is

S ={ 1minute in room-A, 1minute in room-B, 2minutes in room-B, room-C}={0, 1, 2, 3} (respevtively)

Transition matrix has been written like that;

$$\mathcal P = \begin{bmatrix} 1/3 & 1/3 & 0 & 1/3 \\ 0 & 0 & 1 & 0 \\ 1/3 & 1/3 & 0 & 1/3 \\ 0 & 0 & 0 & 1 \end{bmatrix}$$

---

But, I cannot understand how to write down this transition matrix?

Why $P_{00}^{(1)}=1/3$ or $P_{12}^{(1)}=1$ or ...? Please how to write these probability values in the matrix? Explain it with reasons. thank you.

Answer 1

If I understand your question correctly...

There are four possible states:

• 0: In Room A
• 1: In Room B, first of two minutes
• 2: In Room B, second of two minutes
• 3: In Room C

The system changes state each minute. Consider each of the four starting positions:

State 0: If the animal is currently in Room A, then it either stays in Room A, or moves to Room B, or moves to Room C, each with probability 1/3. If the animal moves to Room B, then it starts the first minute of a 2-minute stay. Hence the shift in state is from State 0 to State 1. There is no possibility to shift directly from State 0 to State 2: the animal can't shift from not being in Room B, directly into the second of two minutes in Room B. Hence P_{02} = 0.

State 1: If the animal is currently in Room B and in the first minute of a two-minute stay, then the animal is in State 1. In this case, the animal must remain in Room B for an additional minute. In so doing, the animal shifts from State 1 to State 2. Hence P_{12}=1.

State 2: If the animal is currently in Room B and in the second minute of a two-minute stay, then the animal either shifts to Room A (State 0), starts a new 2-minute stay in Room B (goes back to State 1), or shifts to Room C (State 3), each with Prob = 1/3.

State 3 is an absorbing state: P_{33}=1.