In my notes on Markov processes, I am presented with two related examples:

Example 1:

Classify daily weather for some region as Sunny (state $1$), Cloudy (state $2$), or rainy (state $3$).

Let $X_n$ be the weather state on day $n = 0, 1, 2, \dots$, numbering from some arbitrary point in time.

If you assume that $(X_n)$ is a Markov chain, then $p_{ij}$ must be estimated from weather records.

Suppose you have $t$ days of records. Let $t_{ij}$ denote the number of $i \to j$ transitions in your record, and $t_i = t_{i1} + t_{i2} + t_{i3}$ be the number of transitions out of state $i$.

Then $\hat{p}_{ij} = t_{ij}/t_i \ \ \ (i, j = 1, 2, 3)$ is the observed proportion of transitions in your record out of the fixed state $i$ into $j = 1, 2, 3$.

It can be shown that $\hat{p}_{ij}$ is a maximum likelihood estimate of $p_{ij}$.

The matrix of estimates $[\hat{p}_{ij}]$ is stochastic.

For a numerical example, suppose that

$$\mathcal{P} = \begin{bmatrix} 0.5 & 0.5 & 0.4 \\ 0.3 & 0.2 & 0.5 \\ 0.2 & 0.3 & 0.1 \end{bmatrix}.$$

Therefore, $p_{11} = 0.5$ is the probability of being sunny tomorrow, given that it is sunny today.

You would want to use your record to test the MC assumption.

Example 2:

Recall the weather pattern MC in example 1. You are planning a two-day holiday to begin in seven days, i.e., you are away on day $7$ and $8$. A travel insurance deal will pay you \$2500 if it rains on both days, nothing if not, and the premium is \$100. Should you buy this insurance if it is sunny today?

One way to make a decision would be to compare the expected pay-out with the premium. The actual return is

$$R = \begin{cases} 2500 & \text{if} \ X_7 = X_8 = 3 \\ 0 & \text{otherwise}, \end{cases}$$

where $X_n$ is the weather state on day $n$. Counting today as day $0$, the expected return is

$$\begin{align} E(R) &= 2500 \times P(X_7 = X_8 = 3 \vert X_0 = 1) \\ &= 2500P(X_8 = 3 \vert X_7 = 3, X_0 = 1)P(X_7 = 3 \vert X_0 = 1) \\ &= 2500P(X_8 = 3 \vert X_7 = 3) p^{(7)}_{13} \\ &= 2500p_{33}p^{(7)}_{13} \end{align}$$

Evaluation of the one number $p^{(7)}_{13}$ requires evaluation of $\mathcal{P}^7$, where $\mathcal{P}^{(n)}$ is the $n$-step transition matrix.

You will find that $p^{(7)}_{13} = 0.2101$, and $p_{33} = 0.1$, so $E(R) = 52.52$.

This is much less than the premium, so if your decision is based on expected returns, you should not buy this insurance.

Now, in trying to recreate the results for example 2, I used the following R code:


P <- matrix(c(0.5,0.3,0.2,0.5,0.2,0.3,0.4,0.5,0.1), nrow=3) ## 1-step transition probability matrix

P7 <- P%^%7 ## 7-step transition probability matrix

> P7

[,1] [,2] [,3] [1,] 0.4789915 0.4789985 0.4789816 [2,] 0.3109247 0.3109008 0.3109585 [3,] 0.2100838 0.2101007 0.2100599

According to my result, $P^{(7)}_{13} = 0.4789816$. For the given one-step transition matrix $\mathcal{P}$ (above), we have that $p_{33} = 0.1$, which agrees with what was said in example 2.

Does anyone know what's going on here? Have I made an error? Or is there an error in the notes?

I would greatly appreciate it if people would please take the time to clarify this.


1 Answer 1


I think it's a notational issue. In your source, transition matrix sum up to $1$ columnwise. So, transitioning from a sunny day to rainy day, i.e. $P(X_k=3|X_{k-1}=1)$ is the entry at the third row and the first column of the seven times exponentiated transition matrix, i.e. $p_{13}^7=(P^7)_{31}$.

  • $\begingroup$ Thanks for the answer. So are you saying that my work is correct? After all, both results cannot be correct. $\endgroup$ Commented Mar 15, 2020 at 22:21
  • $\begingroup$ No, you should be taking P7[3,1] instead of P7[1,3] $\endgroup$
    – gunes
    Commented Mar 15, 2020 at 22:22
  • $\begingroup$ Ok, thanks for the clarification. $\endgroup$ Commented Mar 15, 2020 at 22:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.