0
$\begingroup$

I have the following question. It's about transient and recurrent states in Markov Chains. I know when a state is one or the other, but there is one thing I can't figure out or understand. We have learned that the sum of the First Return Probabilites (FRP) must be = 1 for a state to be recurrent, otherwise it is transient (see Figure 1).

In my example below both states should be recurrent, but if I calculate the First Return Probability for state 1 (a time step) I get 0.5. The others are irrelevant because we already had our First Return. So the sum of the FRP = 0.5 so < 1. But with that it would not be recurrent anymore. What am I doing wrong?

Figure 1:

Example:

$\endgroup$
1
  • $\begingroup$ Why $f_{11}^{(2)} = 0$? It is clearly greater than $0$. $\endgroup$
    – Zhanxiong
    Commented Feb 22, 2023 at 21:31

1 Answer 1

1
$\begingroup$

You made a mistake on evaluating $f_{11}^{(2)}$, which should be \begin{align} P[X_2 = 1, X_1 = 2 | X_0 = 1] = p_{12}p_{21} = \frac{1}{2} \times 1 = \frac{1}{2}. \end{align}

For $n \geq 3$, it indeed holds that $f_{11}^{(n)} = 0$, as once the chain visits the state $2$, it for sure will return to the state $1$ in the next step. Therefore, \begin{align} f_{11} = \sum_{n = 1}^\infty f_{11}^{(n)} = f_{11}^{(1)} + f_{11}^{(2)} = \frac{1}{2} + \frac{1}{2} = 1, \end{align} showing that the state $1$ is recurrent.

P.S., The "Figure 1" in your post seems irrelevant and should be deleted.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.