# Calculation of transition probabilities of Markov Chain problem

In the step of learning Markov chains, I came across few questions from assignments on UTDallas website. Finding the transition probabilities seems a bit hard for me in this one particular question only. The question is

(Sec 7.3, page 355, #1) Consider a system with two components. We observe the state of the system every hour. A given component operating at time $n$ has probability $p$ of failing before the next observation at time $n + 1$. A component that was in a failed condition at time $n$ has a probability $r$ of being repaired by time $n + 1$, independent of how long the component has been in a failed state. The component failures and repairs are mutually independent events. Let $X_n$ be the number of components in operation at time $n$. The process $\{X_n, n = 0, 1, . . .\}$ is a discrete time homogeneous Markov chain with state space $I = \{0, 1, 2\}$.

a) Determine its transition probability matrix, and draw the state diagram.

b) Obtain the steady state probability vector, if it exists

Although the answers are given, but I cannot understand that on what basis the transition probabilities are calculated. Can someone help me in this...

I had the following guesses my ownself (which mostly proved to be wrong)

\begin{bmatrix} 1-r-r^2 & r & r^2\\ ??& ?? & r(1-p) \\ p^2 & ?? & ?? \end{bmatrix}

The actual solution that the website pose is following...

\begin{bmatrix} (1-r)^2 & 2r(1-r) & r^2\\ p(1-r)& pr + (1-p)(1-r) & r(1-p) \\ p^2 & 2p(1-p) & (1-p)^2 \end{bmatrix}

• Thanks Yes it is self-study question. What I am asking is not the solution since I am not taking this exam, I am trying to learn the probability, in general, and markov chains, in specific. Appreciate your comment @Tavrock. – Kashan Feb 25 '17 at 4:11
• I am looking if someone can help me to learn HERE how transition probabilities are calculated...Since to my understanding, transitional probabilities are different than normal probabilities... – Kashan Feb 25 '17 at 10:59
• Can you explain the reasoning behind your guesses? (Also, based on the last comment, do you know what a transition probability is?) – Juho Kokkala Feb 25 '17 at 14:27

Note that, there are two components and each of them individually could be either in working or failure state. Then the states of the Markov chain be designated as $ff$ both components failed, $wf$ one component working and one component failed, and $ww$ both components working. Given that, the components function independently of each other.

• If the chain is in state $ff$ at time $n$, at the next time point it could be in the state

• $ff$ with probability $(1-r)(1-r)=(1-r)^2$, as the probability of not being repaired by next time point is $(1-r)$;

• $wf$ with probability $2r(1-r)$, as it may be the first component
that was repaired or the the second component that was repaired, so
that only one of the two components will be working;

• $ww$ with probability $r^2$, if both the components got repaired.

• If the chain is in the state $wf$ at time $n$, then at the next time point it could be in the state

• $ff$ with probability $p(1-r)$, as the working component failed and the component requiring repair was not yet repaired;

• $wf$ with probability $pr+(1-p)(1-r)$, as the working component was
not failed and the component requiring repair is not repaired, so
that only one component is working, which has a probability
$(1-p)(1-r)$; or the component working at the previous time has
failed and the component requiring repair has got repaired, which has a probability $pr$; mutually exclusiveness of the events results in the required probability;

• $ww$ with probability $r(1-p)$, as the working component does not
require repair and the failed component got repaired.

• If the chain is in the state $ww$ at time $n$, then at the next time point it could be in the state

• $ff$ with probability $p^2$, as both components have failed;
• $wf$ with probability $2p(1-p)$, which is the sum of probabilities
of two mutually exclusive events, viz., the first component failed
and the second working or the first component working and the
second component failed;

• $ww$ with probability $(1-p)^2$, as both components are still
working.

Hence, the transition matrix:

\begin{equation*} P=\begin{array}{c|ccc} &ff & wf & ww\\ \hline ff & (1-r)^2& 2r(1-r) & r^2\\ wf & p(1-r) & pr+(1-p)(1-r) & r(1-p) \\ ww & p^2 & 2p(1-p) & (1-p)^2 \end{array} \end{equation*}

• Thanks.... Helped me a lot in developing understanding and brushing off some dust over my mind – Kashan Feb 26 '17 at 2:20