I have a data that is being modelled through the continuous time Markov chain with discrete state space. The model is simple: only 4 states. However, I have an additional condition imposed on the model: if the subject spends more than 2 years in the second state, then he/she cannot go back to the state 1, only to the states 3 or 4, from which there is no way back to state 1.

I could take a very "rude" way and simulate this Markov chain numerically with this additional condition, but I would rather have it solved mathematically.

In other words I'm looking for some refereces on the Markov chain case, when after some deterministic time interval, spent in one state, at least on transition rate becomes 0 (in my case transition rate from state 2 to state 1 becomes 0).

The main objective is to obtain transition probabilities for various time periods.

  • 1
    $\begingroup$ Is it possible to go from state 1 -> 3 or 4 (i.e. bypassing state 2?) $\endgroup$ Commented Jul 19, 2015 at 16:16
  • 1
    $\begingroup$ No, it is not possible to bypass state 2 while going from 1 to any other state. $\endgroup$
    – Tomas
    Commented Jul 19, 2015 at 17:13

1 Answer 1


Technically speaking, what you've described is not a Markov Chain since its transitioning behaviour depends on more than the previous state (i.e. it also depends on whether it has previously spent 2+ years in state 2). I believe this can be 'fixed' using a technique similar to the way that time inhomogeneous Markov chains may be altered to be homogeneous (see also).

Basically, by artificially increasing the state space, an augmented Markov chain that contains the information of the original process, yet has certain 'nice' properties, can be constructed.

In your case, what this means is creating an auxiliary process that depends on the the process you've described, but keeps track of whether the original process was ever in the second state for two years. Lets call it $Y_t$. For simplicity, I will assume that the original process (call it $X_t$) was discrete in time with each step corresponding to one year. More formally, define:

$Y_t = \begin{cases} 0 & Y_{t-1} \neq 2 \text{ and } X_t \neq 2\\ 1 & Y_{t-1} = 0 \text{ and } X_{t-1} = 2\\ 2 & Y_{t-1} = 2 \text{ or (} Y_{t-1} = 1 \text{ and } X_{t-1} = 2\text{)}\end{cases}.$

This looks a bit messy, but essentially if $X$ is in state 2 $Y$ records how long it has been there and if it ever reaches 2 years the process stays in the state $Y = 2$ permanently.

Okay, now we can make out new Markov chain which is just the joint process $Z_t = (X_t, Y_t)$. By design, $Y_t$ only depends on $X_{t-1}$ and $Y_{t-1}$, and $X_t$ uses the initial transition rule when $Y \neq 2$ and the altered transition rule (no going to state 1) when $Y = 2$. By definition $Z$ is a Markov chain with $4\times 3 = 12$ possible states. For your purposes, you can group these states by their $X$ value and use traditional Markov chain analysis to measure the quantities you're interested in.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.