From all the explanations I have had, I only get what the Markov part of MDP is, not the process part of it. Is it same as the process as used in stochastic processes, gaussian processes or poisson processes?

I can't get a Wikipedia link to process and the definitions of MDP or any of the aforementioned processes do not go as "Stochastic process is a process which...". Maybe examples of generic processes and defining these as special cases would help me understand better.

  • 1
    $\begingroup$ 1. Yes, it is the same. 2. A Markov Chain has a discrete state space and/or a discrete time index, so it's a subset of Markov processes. 3. A Markov decision process is a Markov process in which, at certain epochs and certain states, an external process, for example, you, makes a decision that affects the evolution of the MDP. If you have a fixed decision rule, the MDP given the decision rule is a Markov process. If you change the decision rule, it's a different Markov process but the same MDP. $\endgroup$ – jbowman Apr 26 '18 at 21:08
  • $\begingroup$ I think the Wikipedia link might help: en.wikipedia.org/wiki/Markov_chain $\endgroup$ – jbowman Apr 27 '18 at 1:53
  • $\begingroup$ In general, a stochastic "process" is a collection of random variables $\{X_t : t\in \mathbb T\}$ where $\mathbb T$ is an indexing set. Often $\mathbb T=\{0,1,2,\ldots\}$ (which we consider a "discrete-time" process) or $\mathbb T=[0,\infty)$ (which we consider a "continuous-time" process). $\endgroup$ – Math1000 May 17 '18 at 20:10
  • $\begingroup$ @Math1000 So, MDP = {??? : t ∈ T}, what goes in there? $\endgroup$ – Saravanabalagi Ramachandran May 17 '18 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.