Skip to main content

Questions tagged [markov-decision-process]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Markov Decision Process and Message Passing

I am reading the book “Bayesian Reasoning and Machine Learning”. I am reading Chapter 7.5 on Markov Decision process. I come from an optimal stopping / optimal control background and am familiar with ...
Lost1's user avatar
  • 708
0 votes
0 answers
39 views

Reward function definition in MRP/MDP, reinforcement learning different notations

I started to self-taught reinforcement learning a few weeks ago. These days I've encountered a problem with the definition of the reward function. The reward function, defines and quantifies the ...
SuperSlow's user avatar
1 vote
1 answer
72 views

What studied statistical model (if any) fits this application?

I'm having trouble identifying what statistical model or methodology is suited for my application. My situation is as follows: I want to create a stock trading agent that trades a single stock-cash ...
QMath's user avatar
  • 451
0 votes
0 answers
18 views

Learning Markov Decision Processes from Puterman's text with limited mathematical background

I am studying Markov Decision Processes (MDPs) using Puterman's classic text, but am finding it challenging due to my limited background in measure theory and Borel subsets. Additionally, the lack of ...
user1351336's user avatar
5 votes
1 answer
123 views

Is reinforcement learning conceptually equivalent to time-series with a latent dependent variable?

In reinforcement learning, there is a state $s_t$, an action $a_t$, and a policy $\pi(a|s)$ that maps states to the Probability Distribution Function (PDF) of actions. The goal is to choose the ...
Colin T Bowers's user avatar
0 votes
0 answers
25 views

What is state transition probability in an MDP? A matrix or a 3-D tensor?

I find the following written in places: $P$ is a state transition probability matrix, $P_{ss'}^{a} = P[S_{t+1} = s' | S_t = s, A_t = a]$ (notes by david silver - slide 24 and, sutton and barto) How is ...
figs_and_nuts's user avatar
1 vote
1 answer
85 views

Can two states have different actions in a deterministic policy? How to specify states which have probability linked with them in the policy?

The agent has two actions, a0 and a1, whose effects in each state σ0; . . . ; σ3 are described in Figure 1. The edges from actions are labeled with the probability that this transition occurs. For ...
Trileo Stark's user avatar
0 votes
1 answer
101 views

Can the observation function in a POMDP be a function of the previous state?

I would like to model my problem with a Partially Observable Markov Decision Process (POMDP) but I have as an observation the previous state $o_t = s_{t-1}$. However, I see in all formal definitions ...
Bennox75's user avatar
0 votes
1 answer
98 views

Existence of the optimal control in finite horizon MDP

For infinite horizon MDP with finite state and action space, there exists an optimal (stationary) policy. For finite horizon MDP with finite state and action space, does there exist an optimal policy? ...
Stephanie's user avatar
0 votes
0 answers
47 views

How to estimate the order of a controlled Markov process from data?

Consider a non-stationary controlled Markov process represented by a sequence of states and actions $(s_0,a_0,\ldots,s_{T-1},a_{T-1},s_T)$ over a finite number of discrete time steps. If the process ...
jonem's user avatar
  • 151
1 vote
0 answers
24 views

What theoretical guarantees are lost in modeling reward as a function of next state?

There are a couple of threads (1, 2) which address the dependency of reward on the next state in addition to the current state and action. Clearly, modeling the transition probability as a joint ...
lerner's user avatar
  • 11
1 vote
0 answers
22 views

Can we do Deep Reinforcement Learning with Disjoint Action Sets?

I'm defining a construction you can apply to a Markov decision process*, and it involves extending an equivalence relation from the the state space of the MDP to an equivalence relation on the action ...
Principal Pigeonhole's user avatar
1 vote
1 answer
23 views

Why $A_{t-1}$ in reinforcement learning history $H_t = O_1, R_1, A_1, ..., A_{t-1}, O_t, R_t$? [closed]

I learn with David Silver's slides reinforcement learning. His definition of the history $H_t$ is: $H_t = O_1, R_1, A_1, ..., A_{t-1}, O_t, R_t$ $O =$ observations $R =$ rewards $A =$ actions Why do ...
yemy's user avatar
  • 129
0 votes
1 answer
317 views

Estimated Optimal Policy vs True Optimal Policy

In an MDP / in Reinforcement Learning, the optimal policy is often defined as something similar to: a policy that maximizes the value of all states at the same time. But I am not sure whether this ...
John Rowlay's user avatar
1 vote
0 answers
32 views

Fixed point of the Bellman operator for suboptimal policies

Consider an MDP and let the Bellman operator be defined as follows, $$ (T^\pi_\gamma V)(s) = \sum_{a\in A}\pi(s)\big(r(s,a) + \gamma \sum_{s' \in S} p(s'\mid s,a) V(s')\big) $$ where, $\pi:S\to \...
Erik M's user avatar
  • 141
0 votes
1 answer
136 views

In reinforcement learning, after an action, the state does not change

After conducting an action, the state of the agent may change into a new state, also may not change. So is it still reasonable to formulate the problem by MDP and solved it by reinforcement learning ...
Youzhen_Frank's user avatar
3 votes
1 answer
334 views

Bellman Optimality Operator fixed point

I'm reading Szepesvári's book on RL. My question is concerning the proof of Theorem A.10 (p. 71). Theorem Let $V$ be the fixed point of $T^∗$ and assume that there is policy $π$ which is greedy w.r.t ...
Nick Halden's user avatar
2 votes
0 answers
245 views

Is random policy a stochastic policy?

I'm a student to start to study RL. When I studied MDP and watched the gridworld example, I had one question. In the gridworld, we usually assume that we can have four actions in any states, e.g. up, ...
beef stew's user avatar
1 vote
0 answers
638 views

Q-learning convergence with stochastic reward function

Every proof for convergence of Q-learning I can find assumes that the reward is a function $r(s, a, s')$ i.e. deterministic. However, MDPs are often defined with a stochastic reward, as exemplified in ...
corazza's user avatar
  • 131
1 vote
0 answers
755 views

Open AI Gym for TSP problem?

In a previous question I asked about use of Open AI Gym as a vehicle for modeling business problems as MDPs. A comment suggested that I start a new question with more refined scope. In general, I'm ...
jbuddy_13's user avatar
  • 3,392
1 vote
1 answer
28 views

What kind of model to optimize the allocation a ressource in the context of time to event outcome?

I have a list of N patients that are competing for one treatment at each time. A treatment becomes available at times t=1,...,T. I want to build a model that can take the time-varying characteristics ...
Mery's user avatar
  • 11
1 vote
1 answer
2k views

Whats exactly deterministic and non deterministic in deterministic and nondeterministic MDP policies?

Consider below Markov Decision Process: Blue hexagons are states and orange circles are actions. I have rather simple confusion. What will be nature of deterministic and non deterministic MDPs? This ...
Rnj's user avatar
  • 205
1 vote
0 answers
176 views

Optimal action-value as function of optimal value. Proof

Currently reading through Algorithms for Reinforcement Learning, I think these notes are good, but there're bits that are a bit unclear, and I have few questions that I think are quite basic: ...
user8469759's user avatar
1 vote
1 answer
403 views

Equivalent definitions of Markov Decision Process

I'm currently reading through Sutton's Reinforcement Learning where in Chapter 3 the notion of MDP is defined. What it seems to me the author is saying is that an MDP is completely defined by means of ...
user8469759's user avatar
2 votes
1 answer
281 views

Is a policy $\pi(s)$ on Markov decision process a random variable?

Citing Wikipedia: The goal in a Markov decision process is to find a good "policy" for the decision maker: a function $\pi$ that specifies the action $\pi(s)$ that the decision maker will ...
Multivac's user avatar
  • 242
2 votes
1 answer
282 views

How to solve a Markov Decision Problem with State Transition Matrix and Reward Matrix

I'm stuck in solving a simple dynamic probabilistic model. I have Three states {Sunny, Cloudy, Rainy}. I have the ...
Sammy's user avatar
  • 75
1 vote
0 answers
66 views

does it make sense to define average reward in finite horizon

I am new to reinforcement learning but there is a situation I am considering using average reward instead of sum reward as objective for a finite horizon application problem. Specifically, there are ...
lll's user avatar
  • 211
-1 votes
1 answer
543 views

How to calculate the probability Matrix (Alpha) for Regular Markov chains

Pardon me for being a novice here. In the image attached, eq 3.1 represents the transition matrix (it's pretty clear). I am not able to comprehend the eq 3.2, alpha*P = alpha, as well as the further ...
Mritunjay Kumar's user avatar
1 vote
0 answers
612 views

Bellmans equation and existence of optimal policy for MDPs

I'm trying to understand the proof of existence of an optimal policy from this question Why is there always at least one policy that is better than or equal to all other policies? by Lovelris. - ...
John P's user avatar
  • 121
6 votes
1 answer
7k views

What is the difference between Reinforcement Learning(RL) and Markov Decision Process(MDP)?

What is the difference between a Reinforcement Learning(RL) and a Markov Decision Process(MDP)? I believed I understood the principles of both, but now when I need to compare the two I feel lost. ...
Thalassophile's user avatar
3 votes
1 answer
431 views

States in Bandit Problems

I am wondering if there is an interpretation of the Bandit Problem with more than one states. I know that there are versions which views each slot machine as an independent Markovian machines and as ...
dezdichado's user avatar
1 vote
1 answer
1k views

Calculating Value State matrix for a finite MDP without limit condition

In the book "Reinforcement Learning" From Andrew Sutton and Barto there is an example given for the Bellman equations: Figure 3.2 (left) shows a rectangular gridworld representation of a simple ...
SirPeople's user avatar
  • 113
2 votes
1 answer
661 views

Large state space and Markov decision process

I am working on a project where I have an MDP but with a very large state space (each state is described by a tuple $(a,b,c,d)$ where $a,b,c,d$ are integers in the range $[0, 1000]$). My goal is to ...
Rory's user avatar
  • 21
0 votes
1 answer
219 views

Does an optimal value function exist for an MDP with continous state and action spaces?

If the state set $\mathcal{S}$ and action set $\mathcal{A}$ of a Markov Decision Process are infinite does an optimal value function $v_\pi(s)$ exist?
KaneM's user avatar
  • 149
0 votes
0 answers
94 views

Can someone explain how an action reward function is calculated in markov decision process

In his second lecture on Reinforcement Learning, David Silver, writes the expression for reward function(for MDP) as: Why do we need to calculate the expected value? Because if we are in state s1 and ...
Bhuwan Bhatt's user avatar
1 vote
1 answer
297 views

How to model transition probability if action does not lead to a state change (in MDP)?

An MDP (markov decision process) is defined as a set of states $S$, actions space $A$, Transition Probabilities $T$ and Rewards $R$. An action $a$ in a state $s$ usually result in a change of state ...
Khalid Ibrahim's user avatar
3 votes
1 answer
8k views

Dyna-Q Algorithm Reinforcement Learning

In step(f) of the Dyna-Q algorithm we plan by taking random samples from the experience/model for some steps. Wouldn't it be more efficient if we construct an MDP from experience by computing the ...
gnikol's user avatar
  • 751
3 votes
1 answer
1k views

UCB Exploration in Reinforcement Learning

I have two questions regarding the upper confidence bounds (UCB) exploration in reinforcement learning: UCB exploration is derived from Hoeffding's inequality which assumes that the reward is bounded ...
gnikol's user avatar
  • 751
3 votes
0 answers
47 views

Model or State Uncertainty in Queueing Model due to uncertain arrival rate

$\textbf{Introduction}$ I am currently modelling a scenario where two queues need to be served by a single server in a non preemptive discipline. I am quite sorted on generating the optimal policy ...
Dylan Solms's user avatar
0 votes
0 answers
80 views

What is the point of doing simulation on Markov Chain?

I am studying Markov Chain and I am currently reading about simulation on Markov Chain but I can't see the point of simulation on Markov Chain. What does simulation mean in Markov Chain and what can ...
xabzakabecd's user avatar
  • 3,535
1 vote
0 answers
86 views

How to increase the total number of iterations it takes to converge a MDP?

I was reading about Policy Iteration. What are the factors that influence the total number of iterations the algorithm takes to converge? For a given MDP which converges in 3 iterations, what setting ...
Amanda's user avatar
  • 111
5 votes
2 answers
5k views

Uniqueness of the optimal value function for an MDP

Suppose we have a Markov decision process with a finite state set and a finite action set. We calculate the expected reward with a discount of $\gamma \in [0,1]$. In chapter 3.8 of the book "...
jakab922's user avatar
  • 211