# Struggling to understand the difference between a DTMC and a CTMC

My textbook, Modeling and Analysis of Stochastic Systems, third edition, by Kulkarni, introduces continuous-time Markov chains (CTMCs) as follows:

In Chapters 2, 3, and 4 we studied DTMCs. They arose as stochastic models of systems with countable state-space that change their state at time $$n = 1, 2, \dots$$, and have Markov property at those times. Thus, the probabilistic nature of the future behavior of these systems after time $$n$$ depends on the past only through their state at time $$n$$.

In this chapter we study a system with a countable state-space that can change its state at any point in time. Let $$S_n, n \ge 1$$, be time of the $$n$$th change of state or transition. $$Y_n = S_n - S_{n - 1}$$, (with $$S_0 = 0$$), be the $$n$$th sojourn time, and $$X_n$$ be the state of the system immediately after the $$n$$th transition. Define

$$N(t) = \sup\{ n \ge 0 : S_n \le t \}, \ \ \ t \ge 0.$$

Thus $$N(t)$$ is the number of transitions the system undergoes over $$(0, t]$$, and $$\{ N(t), t \ge 0 \}$$ is a counting process generated by $$\{ Y_n, n \ge 1 \}$$. it has piecewise constant sample paths that start with $$N(0) = 0$$ and jump up by $$+1$$ at time $$S_n, n \ge 1$$.

Under the typical mathematical definition of "continuous", I do not understand how CTMCs are "continuous" -- it seems to me that the time is still discrete, just as for DTMCs. I would greatly appreciate it if people would please take the time to explain the difference here.

• @Xi'an "The Markov chain can thus move to another location on a countable state space at any time in the future." I don't understand exactly what this means. Why is this notable? How is it relevant to continuity? DTMCs can also move to other locations on a countable state space at any time in the future, assuming the DTMC is irreducible (that is, all classes communicate); or, maybe they can't, depending on what you mean by your statement, since DTMCs can't just jump from any state to any other, but, rather, must take some path. Like I said, I'm not sure exactly what your statement means. Commented May 25, 2020 at 5:58

Time is not discrete. $$N(t)$$ is a continuous-time process. Its sample paths $$N(\omega, t)$$ are cadlag functions $$N(\omega, t): [0, \infty) \rightarrow \mathbb{Z}_+\,\,$$ $$\omega$$-almost surely, and $$N(t) - N(t_-) \leq 1.$$ At each given $$t \in [0, \infty)$$, $$N(\omega, t)$$ is the number of times the process has switched states up to time $$t$$. Each jump of $$N(t)$$, i.e. when $$N(t) - N(t_-) = 1$$, the CTMC undergoes a state transition.
The sample paths of the corresponding CTMC $$M(\omega, t)$$ are piecewise constant cadlag functions $$M(\omega, t): [0, \infty) \rightarrow \mathcal{S}$$ where $$\mathcal{S}$$ is the state space. By definition, $$N(t)$$ has the same discontinuities as $$M(t)$$.
In contrast, the sample path of a DTMC $$X(\omega, t)$$ is a sequence $$X(\omega, t): \mathbb{N} \rightarrow \mathcal{S}.$$