# Markov chain with conditions

I have a data that is being modelled through the continuous time Markov chain with discrete state space. The model is simple: only 4 states. However, I have an additional condition imposed on the model: if the subject spends more than 2 years in the second state, then he/she cannot go back to the state 1, only to the states 3 or 4, from which there is no way back to state 1.

I could take a very "rude" way and simulate this Markov chain numerically with this additional condition, but I would rather have it solved mathematically.

In other words I'm looking for some refereces on the Markov chain case, when after some deterministic time interval, spent in one state, at least on transition rate becomes 0 (in my case transition rate from state 2 to state 1 becomes 0).

The main objective is to obtain transition probabilities for various time periods.

• Is it possible to go from state 1 -> 3 or 4 (i.e. bypassing state 2?) – P.Windridge Jul 19 '15 at 16:16
• No, it is not possible to bypass state 2 while going from 1 to any other state. – Tomas Jul 19 '15 at 17:13

In your case, what this means is creating an auxiliary process that depends on the the process you've described, but keeps track of whether the original process was ever in the second state for two years. Lets call it $Y_t$. For simplicity, I will assume that the original process (call it $X_t$) was discrete in time with each step corresponding to one year. More formally, define:
$Y_t = \begin{cases} 0 & Y_{t-1} \neq 2 \text{ and } X_t \neq 2\\ 1 & Y_{t-1} = 0 \text{ and } X_{t-1} = 2\\ 2 & Y_{t-1} = 2 \text{ or (} Y_{t-1} = 1 \text{ and } X_{t-1} = 2\text{)}\end{cases}.$
This looks a bit messy, but essentially if $X$ is in state 2 $Y$ records how long it has been there and if it ever reaches 2 years the process stays in the state $Y = 2$ permanently.
Okay, now we can make out new Markov chain which is just the joint process $Z_t = (X_t, Y_t)$. By design, $Y_t$ only depends on $X_{t-1}$ and $Y_{t-1}$, and $X_t$ uses the initial transition rule when $Y \neq 2$ and the altered transition rule (no going to state 1) when $Y = 2$. By definition $Z$ is a Markov chain with $4\times 3 = 12$ possible states. For your purposes, you can group these states by their $X$ value and use traditional Markov chain analysis to measure the quantities you're interested in.