Markov Process : A stochastic process has Markov property if conditional probability distribution of future states of process depends only upon present state and not on the sequence of events that preceded.

Markov Decision Process: A Markov decision process (MDP) is a discrete time stochastic control process. It provides a mathematical framework for modeling decision making in situations where outcomes are partly random and partly under the control of a decision maker.

Asper my understanding Markov Decision Process is just a framework for Markov Process or there is something else I am missing. One more question is it says it as Stochastic control process meaning it is not completely random and Markov Process is completely random . Can someone help me with this


1 Answer 1


MDP is an expansion of MP, and the difference is written in your question, but maybe an illustration will help.

MP is a stone rolling down the road and has a probability in each fork to go left or right.

MDP is when you put some on the stone that tries to can affect it, and steer a bit to the left or the right.

This answer is weird, but I hope it helps you understand.

  • $\begingroup$ You mean to say Markov Process : The way environment is designed and Markov Decision Process is a framework or a sort of guide for an agent in that environment $\endgroup$
    – nithin
    Oct 31, 2018 at 9:23
  • $\begingroup$ no no! I mean MP is still the process of the stone rolling (both are processes!) the difference is that MDP takes into account the action of an agent in at each point $\endgroup$
    – Cherny
    Oct 31, 2018 at 9:27
  • $\begingroup$ For example, just to be clear, if you have an MDP which only has 1 action (meaning nothing to choose), it will be precisely an MP $\endgroup$
    – Cherny
    Oct 31, 2018 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.