1
$\begingroup$

I'm having a difficult time with discrete Markov Chains. In particular, I need help with part c, d, and e on this problem. I do not understand how you can directly calculate the expected first return time to state 6 in part c, how to calculate the probability that the first return time to 6 is finite given you start in state 5, and the probability that the first return time to 6 is finite given you start in state 3. Any help would be great, thanks!

Here is a picture to the problem. By the way, P is the transition probability matrix for X where X is a Markov Chain with state space S={1,2,3,4,5,6}.

http://i.imgur.com/Zk8X9.jpg?1

$\endgroup$
2
$\begingroup$

Hint: If you start in $\lbrace 4,5,6\rbrace$, you stay in $\lbrace 4,5,6\rbrace$, and you have a $1/3$ chance to move to $6$ at each step.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Ok so for part d), would I condition on the first step? So my probability becomes the sum of P[T6<inf|X1=k,X0=4]P[X1=k|X0=4] from k=4 to 6? $\endgroup$ – user17601 Dec 8 '12 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.