I have a finite state and time-homogeneous continuous-time Markov chain (CTMC) which is not irreducible. Will steady state probabilities exist for this CTMC? How to prove this?

  • $\begingroup$ "CTMC" = Continuous Time Markov Chain, presumably. (Using unexplained acronyms is a good way to lose many potential replies.) $\endgroup$ – whuber Dec 6 '11 at 15:41
  • $\begingroup$ Please update our question and be more specific, in particular, about what you mean by "steady-state probabilities". Also, if you can include any other known structure about the chain you are interested in, it might help. $\endgroup$ – cardinal Dec 7 '11 at 1:24

The short answer is "No."

First, it would be helpful to know if your underlying discrete-time Markov chain is aperiodic, unless you are using the phrase "steady state probabilities" loosely to mean "long-run proportion of the time the CTMC is in the various states" or something else other than "stationary distribution." Aperiodicity in combination with irreducibility is sufficient to guarantee a unique stationary distribution in the case of finite state-spaces, which you are assuming.

Second, the lack of irreducibility means that you will have either some transient states and/or more than 1 closed communicating class. Each class will have its own steady state probabilities (given aperiodicity) and there may be stationary distributions that span multiple communicating classes. Which class you wind up in depends upon what happened during the transient part of the chain's operation, and perhaps upon the initial state.

Consider a two-state discrete-time MC with transition matrix $P(1,1) = P(2,2)=1, P(1,2)=P(2,1) = 0$. Clearly it is aperiodic but not irreducible. Any steady-state distribution $\pi$ satisfies $\pi = \pi P$, is nonnegative, and sums to one. Obviously this is satisfied for any $\pi$ that is nonnegative and sums to one. So every distribution on $\{1,2\}$ is a steady-state distribution for this example. (Hence, clearly, a unique one does not exist.)

| cite | improve this answer | |
  • $\begingroup$ I think this post could be improved in the following ways: (1) Lack of irreducibility is not the same as having more than one closed communicating class. For example, there may be none. (2) It is not true that each class will have its own invariant measure but the whole chain will not. In fact, the whole chain can have an infinite number of unique invariant measures. (Example?) (3) The last paragraph would profit from some rewording. You've given an example where every distribution is invariant. This proves nonuniqueness, but certainly not nonexistence. $\endgroup$ – cardinal Dec 6 '11 at 19:35
  • $\begingroup$ Hmm, maybe I shouldn't have answered, since I may have forgotten more than I realized. Nonetheless, doesn't the finiteness of the state space mean that lack of irreducibility does imply more than one closed communicating class, and that each class, given its own finiteness (and aperiodicity), will have its own invariant measure? $\endgroup$ – jbowman Dec 6 '11 at 20:28
  • $\begingroup$ ...and I have no idea how the word unique snuck into my point (2) above. Yikes. $\endgroup$ – cardinal Dec 6 '11 at 21:46
  • 2
    $\begingroup$ (+1) I have deleted a previous comment and instead made direct edits to the answer. I hope you don't mind. Please feel free to edit more to your liking and style. You have provided some nice, concise intuition. $\endgroup$ – cardinal Dec 7 '11 at 2:23
  • $\begingroup$ I don't mind at all; it helps me improve my answers to other questions too (and possibly learn something as well)! $\endgroup$ – jbowman Dec 7 '11 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.