0
$\begingroup$

My textbook, Modeling and Analysis of Stochastic Systems, third edition, by Kulkarni, introduces continuous-time Markov chains (CTMCs) as follows:

In Chapters 2, 3, and 4 we studied DTMCs. They arose as stochastic models of systems with countable state-space that change their state at time $n = 1, 2, \dots$, and have Markov property at those times. Thus, the probabilistic nature of the future behavior of these systems after time $n$ depends on the past only through their state at time $n$.

In this chapter we study a system with a countable state-space that can change its state at any point in time. Let $S_n, n \ge 1$, be time of the $n$th change of state or transition. $Y_n = S_n - S_{n - 1}$, (with $S_0 = 0$), be the $n$th sojourn time, and $X_n$ be the state of the system immediately after the $n$th transition. Define

$$N(t) = \sup\{ n \ge 0 : S_n \le t \}, \ \ \ t \ge 0.$$

Thus $N(t)$ is the number of transitions the system undergoes over $(0, t]$, and $\{ N(t), t \ge 0 \}$ is a counting process generated by $\{ Y_n, n \ge 1 \}$. it has piecewise constant sample paths that start with $N(0) = 0$ and jump up by $+1$ at time $S_n, n \ge 1$.

Under the typical mathematical definition of "continuous", I do not understand how CTMCs are "continuous" -- it seems to me that the time is still discrete, just as for DTMCs. I would greatly appreciate it if people would please take the time to explain the difference here.

$\endgroup$
1
  • $\begingroup$ @Xi'an "The Markov chain can thus move to another location on a countable state space at any time in the future." I don't understand exactly what this means. Why is this notable? How is it relevant to continuity? DTMCs can also move to other locations on a countable state space at any time in the future, assuming the DTMC is irreducible (that is, all classes communicate); or, maybe they can't, depending on what you mean by your statement, since DTMCs can't just jump from any state to any other, but, rather, must take some path. Like I said, I'm not sure exactly what your statement means. $\endgroup$ Commented May 25, 2020 at 5:58

1 Answer 1

1
$\begingroup$

Time is not discrete. $N(t)$ is a continuous-time process. Its sample paths $N(\omega, t)$ are cadlag functions $$ N(\omega, t): [0, \infty) \rightarrow \mathbb{Z}_+\,\, $$ $\omega$-almost surely, and $$ N(t) - N(t_-) \leq 1. $$ At each given $t \in [0, \infty)$, $N(\omega, t)$ is the number of times the process has switched states up to time $t$. Each jump of $N(t)$, i.e. when $N(t) - N(t_-) = 1$, the CTMC undergoes a state transition.

The sample paths of the corresponding CTMC $M(\omega, t)$ are piecewise constant cadlag functions $$ M(\omega, t): [0, \infty) \rightarrow \mathcal{S} $$ where $\mathcal{S}$ is the state space. By definition, $N(t)$ has the same discontinuities as $M(t)$.

In contrast, the sample path of a DTMC $X(\omega, t)$ is a sequence $$ X(\omega, t): \mathbb{N} \rightarrow \mathcal{S}. $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.