# Questions tagged [markov-decision-process]

The tag has no usage guidance.

35 questions
Filter by
Sorted by
Tagged with
16 views

### In reinforcement learning, after an action, the state does not change

After conducting an action, the state of the agent may change into a new state, also may not change. So is it still reasonable to formulate the problem by MDP and solved it by reinforcement learning ...
7 views

### MDP in Predictive Maintenance sample implementation

I am searching for a sample python implementation of Reinforcement Learning, Markov Decision Process in the domain of predictive maintenance. I have tried on my own, but either found sample related to ...
23 views

### Python library and supporting tutorial for markov decision process policy optimization?

I'm interested in a framework for finding RL solutions to user defined Markov Decision Processes. Notionally, MDPToolkit, a python library, looked to answer the "ask." However, the homepage ...
33 views

### Bellman Optimality Operator fixed point

I'm reading Szepesvári's book on RL. My question is concerning the proof of Theorem A.10 (p. 71). Theorem Let $V$ be the fixed point of $T^∗$ and assume that there is policy $π$ which is greedy w.r.t ...
11 views

### MDP optimal policy inverse problem

Given a map $\pi: S \to A$, is there an MDP with state ans action spaces $S,A$ such that it has $\pi$ as an optimal policy if we suppose the MDP is over an infinite time horizon and the optimality ...
17 views

### Is random policy a stochastic policy?

I'm a student to start to study RL. When I studied MDP and watched the gridworld example, I had one question. In the gridworld, we usually assume that we can have four actions in any states, e.g. up, ...
62 views

### Q-learning convergence with stochastic reward function

Every proof for convergence of Q-learning I can find assumes that the reward is a function $r(s, a, s')$ i.e. deterministic. However, MDPs are often defined with a stochastic reward, as exemplified in ...
14 views

### Local independence vs global independence in markov network

I am having a hard time understanding the basic differences between the local independence and global independence of a markov network. Please help me illustrate with a graph or any example
11 views

### Total Cost Constrained MDP Python implementation?

I am looking for any Python code that could help me solving Constrained MDP with infinite-horizon. In short I have a problem with two types of costs: A and B, and I want to solve "class" MDP ...
85 views

### Open AI Gym for TSP problem?

In a previous question I asked about use of Open AI Gym as a vehicle for modeling business problems as MDPs. A comment suggested that I start a new question with more refined scope. In general, I'm ...
84 views

### How DynaQ behaves in stochastic world in comparison with other reinforcement learning algorithms?

I came across of implementations of a bunch of algorithms on stochastic windy gridworld. This is the graph comparing their performance: So clearly, it seems that DynaQ performs better than all other ...
26 views

### What kind of model to optimize the allocation a ressource in the context of time to event outcome?

I have a list of N patients that are competing for one treatment at each time. A treatment becomes available at times t=1,...,T. I want to build a model that can take the time-varying characteristics ...
203 views

### Whats exactly deterministic and non deterministic in deterministic and nondeterministic MDP policies?

Consider below Markov Decision Process: Blue hexagons are states and orange circles are actions. I have rather simple confusion. What will be nature of deterministic and non deterministic MDPs? This ...
23 views

### Finite MDPs, Pole-Balancing and Q-Learning (RL by Sutton, Barto)

Currently I am reading Reinforcement Learning by S. Sutton and A. Barto. In chapter 3 the authors introduce the concept of Finite Markov Decision Processes. A feature of finite MDPs is that the state, ...
53 views

### Optimal action-value as function of optimal value. Proof

Currently reading through Algorithms for Reinforcement Learning, I think these notes are good, but there're bits that are a bit unclear, and I have few questions that I think are quite basic: ...
96 views

### Equivalent definitions of Markov Decision Process

I'm currently reading through Sutton's Reinforcement Learning where in Chapter 3 the notion of MDP is defined. What it seems to me the author is saying is that an MDP is completely defined by means of ...
82 views

### Convergence of policy iteration with discount $\gamma=1$

This question is related to my question (and my comment of RobPratt's answer) at https://math.stackexchange.com/questions/3860303/markov-decision-process-with-target-states-and-shortest-path-as-only-...
79 views

### Is a policy $\pi(s)$ on Markov decision process a random variable?

Citing Wikipedia: The goal in a Markov decision process is to find a good "policy" for the decision maker: a function $\pi$ that specifies the action $\pi(s)$ that the decision maker will ...
50 views

### How to solve a Markov Decision Problem with State Transition Matrix and Reward Matrix

I'm stuck in solving a simple dynamic probabilistic model. I have Three states {Sunny, Cloudy, Rainy}. I have the ...
33 views

### does it make sense to define average reward in finite horizon

I am new to reinforcement learning but there is a situation I am considering using average reward instead of sum reward as objective for a finite horizon application problem. Specifically, there are ...
73 views

### How to calculate the probability Matrix (Alpha) for Regular Markov chains

Pardon me for being a novice here. In the image attached, eq 3.1 represents the transition matrix (it's pretty clear). I am not able to comprehend the eq 3.2, alpha*P = alpha, as well as the further ...
147 views

### Bellmans equation and existence of optimal policy for MDPs

I'm trying to understand the proof of existence of an optimal policy from this question Why is there always at least one policy that is better than or equal to all other policies? by Lovelris. - ...
1k views

### What is the difference between Reinforcement Learning(RL) and Markov Decision Process(MDP)?

What is the difference between a Reinforcement Learning(RL) and a Markov Decision Process(MDP)? I believed I understood the principles of both, but now when I need to compare the two I feel lost. ...
64 views

### States in Bandit Problems

I am wondering if there is an interpretation of the Bandit Problem with more than one states. I know that there are versions which views each slot machine as an independent Markovian machines and as ...
284 views

### Calculating Value State matrix for a finite MDP without limit condition

In the book "Reinforcement Learning" From Andrew Sutton and Barto there is an example given for the Bellman equations: Figure 3.2 (left) shows a rectangular gridworld representation of a simple ...
83 views

### Large state space and Markov decision process

I am working on a project where I have an MDP but with a very large state space (each state is described by a tuple (a,b,c,d) where a,b,c,d are integers in the range [0, 1000]). My goal is to compute ...
48 views

### Does an optimal value function exist for an MDP with continous state and action spaces?

If the state set $\mathcal{S}$ and action set $\mathcal{A}$ of a Markov Decision Process are infinite does an optimal value function $v_\pi(s)$ exist?
34 views

### Can someone explain how an action reward function is calculated in markov decision process

In his second lecture on Reinforcement Learning, David Silver, writes the expression for reward function(for MDP) as: Why do we need to calculate the expected value? Because if we are in state s1 and ...
57 views

### How to model transition probability if action does not lead to a state change (in MDP)?

An MDP (markov decision process) is defined as a set of states $S$, actions space $A$, Transition Probabilities $T$ and Rewards $R$. An action $a$ in a state $s$ usually result in a change of state ...
4k views

### Dyna-Q Algorithm Reinforcement Learning

In step(f) of the Dyna-Q algorithm we plan by taking random samples from the experience/model for some steps. Wouldn't it be more efficient if we construct an MDP from experience by computing the ...
661 views

### UCB Exploration in Reinforcement Learning

I have two questions regarding the upper confidence bounds (UCB) exploration in reinforcement learning: UCB exploration is derived from Hoeffding's inequality which assumes that the reward is bounded ...
36 views

### Model or State Uncertainty in Queueing Model due to uncertain arrival rate

$\textbf{Introduction}$ I am currently modelling a scenario where two queues need to be served by a single server in a non preemptive discipline. I am quite sorted on generating the optimal policy ...
44 views

### What is the point of doing simulation on Markov Chain?

I am studying Markov Chain and I am currently reading about simulation on Markov Chain but I can't see the point of simulation on Markov Chain. What does simulation mean in Markov Chain and what can ...
Suppose we have a Markov decision process with a finite state set and a finite action set. We calculate the expected reward with a discount of $\gamma \in [0,1]$. In chapter 3.8 of the book "...