Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [markov-decision-process]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

Quantiles of the Q values of an MDP

Cross-posted from Math StackExchange: Consider an MDP with $n$ states, $k$ actions, and discount factor $\gamma \in [0,1)$. We are uncertain of its reward function $R \in \mathbb{R}^{n \times k}$ and ...
1
vote
1answer
29 views

Dyna-Q Algorithm Reinforcement Learning

In step(f) of the Dyna-Q algorithm we plan by taking random samples from the experience/model for some steps. Wouldn't it be more efficient if we construct an MDP from experience by computing the ...
1
vote
1answer
23 views

UCB Exploration in Reinforcement Learning

I have two questions regarding the upper confidence bounds (UCB) exploration in reinforcement learning: UCB exploration is derived from Hoeffding's inequality which assumes that the reward is bounded ...
2
votes
0answers
19 views

Model or State Uncertainty in Queueing Model due to uncertain arrival rate

$\textbf{Introduction}$ I am currently modelling a scenario where two queues need to be served by a single server in a non preemptive discipline. I am quite sorted on generating the optimal policy ...
0
votes
0answers
37 views

What is the point of doing simulation on Markov Chain?

I am studying Markov Chain and I am currently reading about simulation on Markov Chain but I can't see the point of simulation on Markov Chain. What does simulation mean in Markov Chain and what can ...
0
votes
0answers
9 views

Relation between optimal bellman operator and expected bellman operator of the optimal policy

In the classical context of infinite-horizon discounted Markov decision process. Denote the optimal bellman operator as $T$ and the expected bellman operator as $T_\pi$, where $\pi$ is the ...
1
vote
0answers
25 views

How to increase the total number of iterations it takes to converge a MDP?

I was reading about Policy Iteration. What are the factors that influence the total number of iterations the algorithm takes to converge? For a given MDP which converges in 3 iterations, what setting ...
0
votes
0answers
16 views

Validity of the argument in Puterman's MDP literature

I first posted this question on math stackexchange, but I think stats stackexchange would be more appropriate for the question. I'm reading Chapter 6 of Puterman's MDP :Discrete Stocastic Dynamic ...
0
votes
0answers
12 views

Zero Sum problem with MDP formulation and the difference with minimax approach

Suppose that we have formulate a zero sum game with MDP and Ua(s) and Ub(s) are the utilities of A and B in the s state. Suppose that all rewards and utilities are calculated from the A's point of ...
0
votes
0answers
13 views

How to model this as a POMDP?

I would like to fit a DLM to a dataset in R but I don't know the underlying transition matrices between states nor do I have a guess for the emission matrix (given a state, what responses should I see)...
1
vote
2answers
1k views

Uniqueness of the optimal value function for an MDP

Suppose we have a Markov decision process with a finite state set and a finite action set. We calculate the expected reward with a discount of $\gamma \in [0,1]$. In chapter 3.8 of the book "...