# Estimating Markov chain probabilities

What would be the common way of estimating MC transition matrix given the timeseries?

Is there R function for doing that?

• Is this a discrete or continuous state markov chain? – Macro Aug 16 '11 at 16:37
• Discrete I think. I have 5 possible states S1 to S5 – user333 Aug 16 '11 at 16:41
• Building on the previous nice answers: yes, there is a way that is position-aware. I think it is possible by means of nth-order Markov models. – user9439 Feb 26 '12 at 7:12

Since the time series is discrete valued, you can estimate the transition probabilities by the sample proportions. Let $Y_{t}$ be the state of the process at time $t$, ${\bf P}$ be the transition matrix then

$${\bf P}_{ij} = P(Y_{t} = j | Y_{t-1} = i)$$

Since this is a markov chain, this probability depends only on $Y_{t-1}$, so it can be estimated by the sample proportion. Let $n_{ik}$ be the number of times that the process moved from state $i$ to $k$. Then,

$$\hat{{\bf P}}_{ij} = \frac{ n_{ij} }{ \sum_{k=1}^{m} n_{ik} }$$

where $m$ is the number of possible states ($m=5$ in your case). The denominator, $\sum_{k=1}^{m} n_{ik}$, is the total number of movements out of state $i$. Estimating the entries in this way actually corresponds to the maximum likelihood estimator of the transition matrix, viewing the outcomes as multinomial, conditioned on $Y_{t-1}$.

Edit: This does assume that you have the time series observed at evenly spaced intervals. Otherwise, the transition probabilities would also depend on the time lag (even if they are still markovian).

• I hear what you are saying. Basically observed frequencies will be my matrix... In simle words! – user333 Aug 16 '11 at 18:36
• How about continuous state space ? Althou I'm struggling a bit to understand the concept? – user333 Aug 16 '11 at 19:58
• For a continuous state space the problem becomes much more complicated, as you then need to estimate a transition function rather than a matrix. In that case, since the marginal probability of being in any particular state is 0 (similarly to how the probability of taking any particular point in the sample space is 0 for any continuous distribution) what I've described above doesn't make sense. In the continuous case I believe the estimate of the transition function is the solution to a set of differential equations (I'm not very familiar with this so someone please correct me if I'm wrong) – Macro Aug 17 '11 at 1:03
• Does this method not assume 1 continuous observation, rather than many as per the post beneath? For example imagine E was an absorbing state... Then this would not be revealed here surely? – HCAI Sep 12 '12 at 23:52

It is very, with the hypothesis that your time series is stationary :

To simplify the excellent answer of Macro

Here you have your time series with 5 states : A, B, C, D, E

• Great example, thank you. So Markov Chains concern themselves only with the number of transitions, not their placement, correct? For example, would AAABBBA have a a same matrix as ABBBAAA? – Marcin Aug 17 '11 at 13:25