In my notes on Markov processes, I am presented with two related examples:
Example 1:
Classify daily weather for some region as Sunny (state $1$), Cloudy (state $2$), or rainy (state $3$).
Let $X_n$ be the weather state on day $n = 0, 1, 2, \dots$, numbering from some arbitrary point in time.
If you assume that $(X_n)$ is a Markov chain, then $p_{ij}$ must be estimated from weather records.
Suppose you have $t$ days of records. Let $t_{ij}$ denote the number of $i \to j$ transitions in your record, and $t_i = t_{i1} + t_{i2} + t_{i3}$ be the number of transitions out of state $i$.
Then $\hat{p}_{ij} = t_{ij}/t_i \ \ \ (i, j = 1, 2, 3)$ is the observed proportion of transitions in your record out of the fixed state $i$ into $j = 1, 2, 3$.
It can be shown that $\hat{p}_{ij}$ is a maximum likelihood estimate of $p_{ij}$.
The matrix of estimates $[\hat{p}_{ij}]$ is stochastic.
For a numerical example, suppose that
$$\mathcal{P} = \begin{bmatrix} 0.5 & 0.5 & 0.4 \\ 0.3 & 0.2 & 0.5 \\ 0.2 & 0.3 & 0.1 \end{bmatrix}.$$
Therefore, $p_{11} = 0.5$ is the probability of being sunny tomorrow, given that it is sunny today.
You would want to use your record to test the MC assumption.
Example 2:
Recall the weather pattern MC in example 1. You are planning a two-day holiday to begin in seven days, i.e., you are away on day $7$ and $8$. A travel insurance deal will pay you \$2500 if it rains on both days, nothing if not, and the premium is \$100. Should you buy this insurance if it is sunny today?
One way to make a decision would be to compare the expected pay-out with the premium. The actual return is
$$R = \begin{cases} 2500 & \text{if} \ X_7 = X_8 = 3 \\ 0 & \text{otherwise}, \end{cases}$$
where $X_n$ is the weather state on day $n$. Counting today as day $0$, the expected return is
$$\begin{align} E(R) &= 2500 \times P(X_7 = X_8 = 3 \vert X_0 = 1) \\ &= 2500P(X_8 = 3 \vert X_7 = 3, X_0 = 1)P(X_7 = 3 \vert X_0 = 1) \\ &= 2500P(X_8 = 3 \vert X_7 = 3) p^{(7)}_{13} \\ &= 2500p_{33}p^{(7)}_{13} \end{align}$$
Evaluation of the one number $p^{(7)}_{13}$ requires evaluation of $\mathcal{P}^7$, where $\mathcal{P}^{(n)}$ is the $n$-step transition matrix.
You will find that $p^{(7)}_{13} = 0.2101$, and $p_{33} = 0.1$, so $E(R) = 52.52$.
This is much less than the premium, so if your decision is based on expected returns, you should not buy this insurance.
Now, in trying to recreate the results for example 2, I used the following R code:
library(expm)
P <- matrix(c(0.5,0.3,0.2,0.5,0.2,0.3,0.4,0.5,0.1), nrow=3) ## 1-step transition probability matrix
P7 <- P%^%7 ## 7-step transition probability matrix
> P7
[,1] [,2] [,3]
[1,] 0.4789915 0.4789985 0.4789816
[2,] 0.3109247 0.3109008 0.3109585
[3,] 0.2100838 0.2101007 0.2100599
According to my result, $P^{(7)}_{13} = 0.4789816$. For the given one-step transition matrix $\mathcal{P}$ (above), we have that $p_{33} = 0.1$, which agrees with what was said in example 2.
Does anyone know what's going on here? Have I made an error? Or is there an error in the notes?
I would greatly appreciate it if people would please take the time to clarify this.
