# Homework Help on Simple Markov Chain

I have the following problem and questions provided:

Assume the state of the weather in Sydney on any particular day can be modelled using a state space S={1≡sunny,2≡overcast,3≡wet}. Also assume that the likelihood of any state depends only on the state on the previous day. A sunny day is followed by a sunny day 60% of the time and otherwise it is overcast. An overcast day is equally likely to be followed by either a sunny or a wet day. 20% of days after a wet day are also wet, but 20% are sunny.

a) Write down the transition probability matrix for this Markov chain.

#            Sunny   Overcast  Wet
# Sunny       .6      .4        0
# Overcast    .5      0         .5
# Wet         .2      .6        .2

P <- matrix(c(.6, .5, .2, .4, 0, .6, 0, .5, .2), nrow = 3)


b) Draw the state transition diagram for this chain and check if it is irre-ducible or reducible. If reducible, identify the equivalence classes.

So when I draw out the diagram it looks like it is an irreducible chain insofar as you can get to any state from any other state. But I'm also confused about the meaning of irreducible because when I take P to the nth power, it does reduce like so:

#make the power function for matrix powers first

'%^%' <- function(P,n){
if(n > 1){
M <- P
for(i in 2:n){
P <- P%*%M
}
}
return(P)
}

(reducedP <- (P%^%100))


c) If it is sunny all weekend, calculate the probability it is also sunny on Monday.

pi0 <- matrix(c(1, 0, 0), nrow = 3)
(t(pi0)%*%(P%^%2))[1]


But, because the conditional probability of being sunny given the previous day being sunny is necessarily .6, shouldn't this basically be (.6^2)*1 = .36?

d) Calculate the probability that a Monday is wet if the previous Monday was wet.

pi0 <- matrix(c(0, 0, 1), nrow = 3)
(t(pi0)%*%(P%^%7))[3]


e) Calculate the probability that Wednesday through Friday are all sunny,if the previous Sunday was sunny.

For this problem, it seems like i should see what probability the chain gives me for it being sunny on wednesday and then calculate two conditional probabilities, but I don't understand how to get the intersection. Disclaimer: I haven't had a proper probability course before.

I'm hoping someone can help me and let me know if I'm on the right track/right or what I'm missing/need to be doing because at this point I'm totally lost (my professor won't help much but to say I'm on the right track and he doesn't know why the book example does what it does as it's seemingly incorrect.

edit: so here is basically where I'm at, what my general problem is using the easy question from part c):

If any state pi(n) does not depend on the initial state, then, for example in part c, isn't the probability that it will be sunny on monday given it was sunny all weekend only actually depend on what the state was on sunday? So we don't even care about saturday... That means the probability should be .6 because you can only get to a sunny monday in one transition step if sunday was sunny, and going from sunny to sunny is a .6 probability. Otherwise, wouldn't the answer be 1*.6*.6, that is, the initial state which is known, the probability of transitioning to a sunny day at t + 1, and then the probability of transitioning to a sunny day again at t + 2 | t + 1? That gives different answers. And the very confusing bit is that if I use the P matrix, I get .56 (as in my original email), but if I use the P* matrix, I get 0.377 which is close to the 0.36 when you do 1*.6*.6 .

So I have two different paths to take, but no idea which one is the right path and why.

A Markov chain is said to be irreducible if it is possible to get to any state from any state. In fact, when you compute $$P^{100}$$, you have shown explicitly that each entries are positive. It is indeed irreducible.

For part (c), you should not compute $$P^2$$, in fact, whether Monday rains just depends on whether it rains on Sunday.

For part $$(e)$$, let $$Q$$ be the event that it rains on previous Sunday, let $$W$$ be the event that it is sunny on Wednesday, $$T$$ be the event that it rains on Thursday and $$F$$ be the event that it rains on $$Friday$$.

\begin{align} P(WTF|Q) &= P(W|Q)P(T|W)P(F|T) \end{align}

Try to compute this quantity.

• Thanks for finishing this (+1). – BruceET Nov 23 '19 at 6:01
• So for c it asked what the probability is it will be sunny on monday given it was sunny on saturday and sunday. It sounds like you're saying I should ignore the transitions from any time < t-1 to get the probability, following the 1st order markov principle. So the probability that it is sunny on monday given it was sunny on saturday AND sunday should still just be .6? Not 1*.6*.6? That seems wrong. Also for e, that is what i was trying to do using the P matrix to get P(W|Q). But I don't understand whether I should be using the matrix P or the limiting matrix. – Michael Nov 23 '19 at 8:01
• If you compute $0.6(0.6)$, is the probability that you know that Saturday is sunny, what is the probability that both Sunday and Monday are sunny (the setting where you are unsure if Sunday is sunny). The essence of Markov is that given the present, the future and the past are independent. For part e, you should use $P^k$ as you are computing $k$ steps probability. – Siong Thye Goh Nov 23 '19 at 8:23
• Right, so for part c, isn't it P^2? So the probability of the initial state is 1 because it's given. So we're computing 1*(P^2), no? Then for e, we want 1*(P^3) to get the probability it will be sunny on wednesday if sunday was wet. Then we want to multiply that answer by P(T|W)*P(F|T) as you suggested above. In R this would be: {r} pi0 <- matrix(c(0, 0, 1), nrow = 3) ((t(pi0)%*%(P%^%3))[1])*.6*.6 #the first element is the probability it is sunny after three steps (monday, tuesday, wednesday) and then we go from sunny to sunny, and again from sunny to sunny.  – Michael Nov 23 '19 at 8:33
• (c) The Markov property implies that you condition on the last observed or know realisation, which is Sunday. Saturday has no relevance once Sunday is observed. – Xi'an Nov 23 '19 at 8:37

By taking $$\mathbf{P}^{100}$$ you get (correct to to seven places) the limiting matirix $$\lim_{n\rightarrow\infty}\mathbf{P}^{n},$$ of which all three rows are $$\lambda = (0.4901961,\, 0.3137255,\, 0.1960784).$$

In an ergodic chain such a limiting distribution always exists. For some finite power $$m$$ the matrix $$\mathbf{P}^m$$ has all positive elements. Your chain is ergodic because $$\mathbf{P}^2$$ has all positive elements.

P %*% P
[,1] [,2] [,3]
[1,] 0.56 0.24 0.20
[2,] 0.40 0.50 0.10
[3,] 0.46 0.20 0.34


The limiting distribution $$\lambda$$ is also the steady-state distribution vector $$\sigma,$$ such that $$\sigma\mathbf{P}= \sigma.$$

For ergodic $$\mathbf{P},$$ you can use the R function eigen to find $$\sigma,$$ as shown below. It is necessary to use the transpose t(P) because eigen finds right eigenvectors and we want a left eigenvector. It is convenient to use as.numeric to avoid complex number notation in case some eigenvectors are not real. For an ergodic chain, the steady-state vector is the eigenvector listed first (having smallest modulus) and it is always a real vector proportional to $$\sigma.$$ The last step ensures that the elements of $$\sigma$$ sum to unity.

eigen(t(P))$vectors[,1] [1] -0.7981886 -0.5108407 -0.3192754 g = eigen(t(P))$vectors[,1]
sgm = as.numeric(g)/sum(g);  sgm
[1] 0.4901961 0.3137255 0.1960784

• Does this mean the matrix is reducible? I'm unclear about that because I thought it was irreducible considering you can get to any state from any state you start, which seems to basically be the definition of ergodicity (that over time the state of a process is independent of the initial state). Also, what does this mean for the solutions? Seems like I should be using the reduced P matrix for the transition matrix. In that case, I do arrive at a reasonably close answer to c) that I calculate manually (.6*.6*1). – Michael Nov 23 '19 at 5:50
• Irreducible means it has no subclasses. Terminology differs from one author to another. The condition I mentioned that some finite power of the transition matrix has all positive elements implies there are no subclasses and that the chain is aperiodic (not periodic). Irreducible and aperiodic implies ergodic (having a limiting dist'n). [I hope none of that terminology conflicts with what you are using.] // If you have transition probabilities estimated from data, make sure that all rows of $\mathbf{P}$ sum exactly to $1.$ Otherwise, your clever R function %^% will lead to a mess. – BruceET Nov 23 '19 at 5:58
• Irreducibility, meaning there is a positive probability that the chain visits an arbitrary state when starting from an arbitrary initial state, is weaker than strong irreducibility where the probability that $X_{t+1}$ visits an arbitrary state when $X_t$ is an arbitrary initial state is positive. – Xi'an Nov 23 '19 at 8:41