Questions tagged [markov-decision-process]

Filter by
Sorted by
Tagged with
0
votes
2answers
16 views

How to calculate the probability Matrix (Alpha) for Regular Markov chains

Pardon me for being a novice here. In the image attached, eq 3.1 represents the transition matrix (it's pretty clear). I am not able to comprehend the eq 3.2, alpha*P = alpha, as well as the further ...
0
votes
0answers
16 views

Bellmans equation and existence of optimal policy for MDPs

I'm trying to understand the proof of existence of an optimal policy from this question Why is there always at least one policy that is better than or equal to all other policies? by Lovelris. - ...
1
vote
1answer
38 views

What is the difference between Reinforcement Learning(RL) and Markov Decision Process(MDP)?

What is the difference between a Reinforcement Learning(RL) and a Markov Decision Process(MDP)? I believed I understood the principles of both, but now when I need to compare the two I feel lost. ...
2
votes
1answer
42 views

States in Bandit Problems

I am wondering if there is an interpretation of the Bandit Problem with more than one states. I know that there are versions which views each slot machine as an independent Markovian machines and as ...
0
votes
0answers
18 views

Subset selection with reinforcement learning

I have a set of $N$ items of which a subset of arbitrary size can be chosen. I want a reinforcement learning (RL) agent to perform the subset selection and am unsure how to best design the action ...
0
votes
1answer
25 views

Calculating Value State matrix for a finite MDP without limit condition

In the book "Reinforcement Learning" From Andrew Sutton and Barto there is an example given for the Bellman equations: Figure 3.2 (left) shows a rectangular gridworld representation of a simple ...
0
votes
0answers
7 views

Markovian Decision Process and inventory problem

I am interested in an inventory problem modelized by an MDP: -my state space is very large (each state is described by a tuple (u_1,u_2,u_3,u_4) where u_1,u_2,u_3,u_4 are integers in the range [0, ...
1
vote
0answers
25 views

Large state space and Markov decision process

I am working on a project where I have an MDP but with a very large state space (each state is described by a tuple (a,b,c,d) where a,b,c,d are integers in the range [0, 1000]). My goal is to compute ...
0
votes
0answers
48 views

Empirical Bernstein Inequalities in Reinforcement Learning

Recently, I have studied the following paper: "Near-optimal Regret Bounds for Optimistic Reinforcement Learning using Empirical Bernstein Inequalities" https://arxiv.org/abs/1905.12425 I was ...
0
votes
1answer
19 views

Does an optimal value function exist for an MDP with continous state and action spaces?

If the state set $\mathcal{S}$ and action set $\mathcal{A}$ of a Markov Decision Process are infinite does an optimal value function $v_\pi(s)$ exist?
0
votes
0answers
23 views

Can someone explain how an action reward function is calculated in markov decision process

In his second lecture on Reinforcement Learning, David Silver, writes the expression for reward function(for MDP) as: Why do we need to calculate the expected value? Because if we are in state s1 and ...
1
vote
1answer
17 views

How to model transition probability if action does not lead to a state change (in MDP)?

An MDP (markov decision process) is defined as a set of states $S$, actions space $A$, Transition Probabilities $T$ and Rewards $R$. An action $a$ in a state $s$ usually result in a change of state ...
0
votes
0answers
16 views

Quantiles of the Q values of an MDP

Cross-posted from Math StackExchange: Consider an MDP with $n$ states, $k$ actions, and discount factor $\gamma \in [0,1)$. We are uncertain of its reward function $R \in \mathbb{R}^{n \times k}$ and ...
1
vote
1answer
751 views

Dyna-Q Algorithm Reinforcement Learning

In step(f) of the Dyna-Q algorithm we plan by taking random samples from the experience/model for some steps. Wouldn't it be more efficient if we construct an MDP from experience by computing the ...
2
votes
1answer
340 views

UCB Exploration in Reinforcement Learning

I have two questions regarding the upper confidence bounds (UCB) exploration in reinforcement learning: UCB exploration is derived from Hoeffding's inequality which assumes that the reward is bounded ...
3
votes
0answers
26 views

Model or State Uncertainty in Queueing Model due to uncertain arrival rate

$\textbf{Introduction}$ I am currently modelling a scenario where two queues need to be served by a single server in a non preemptive discipline. I am quite sorted on generating the optimal policy ...
0
votes
0answers
43 views

What is the point of doing simulation on Markov Chain?

I am studying Markov Chain and I am currently reading about simulation on Markov Chain but I can't see the point of simulation on Markov Chain. What does simulation mean in Markov Chain and what can ...
1
vote
0answers
28 views

How to increase the total number of iterations it takes to converge a MDP?

I was reading about Policy Iteration. What are the factors that influence the total number of iterations the algorithm takes to converge? For a given MDP which converges in 3 iterations, what setting ...
2
votes
2answers
2k views

Uniqueness of the optimal value function for an MDP

Suppose we have a Markov decision process with a finite state set and a finite action set. We calculate the expected reward with a discount of $\gamma \in [0,1]$. In chapter 3.8 of the book "...