# Questions tagged [markov-decision-process]

The tag has no usage guidance.

42 questions
Filter by
Sorted by
Tagged with
10 views

### Markov Decision Process and Message Passing

I am reading the book “Bayesian Reasoning and Machine Learning”. I am reading Chapter 7.5 on Markov Decision process. I come from an optimal stopping / optimal control background and am familiar with ...
• 708
39 views

### Reward function definition in MRP/MDP, reinforcement learning different notations

I started to self-taught reinforcement learning a few weeks ago. These days I've encountered a problem with the definition of the reward function. The reward function, defines and quantifies the ...
1 vote
72 views

### What studied statistical model (if any) fits this application?

I'm having trouble identifying what statistical model or methodology is suited for my application. My situation is as follows: I want to create a stock trading agent that trades a single stock-cash ...
• 451
18 views

### Learning Markov Decision Processes from Puterman's text with limited mathematical background

I am studying Markov Decision Processes (MDPs) using Puterman's classic text, but am finding it challenging due to my limited background in measure theory and Borel subsets. Additionally, the lack of ...
• 101
123 views

### Is reinforcement learning conceptually equivalent to time-series with a latent dependent variable?

In reinforcement learning, there is a state $s_t$, an action $a_t$, and a policy $\pi(a|s)$ that maps states to the Probability Distribution Function (PDF) of actions. The goal is to choose the ...
25 views

### What is state transition probability in an MDP? A matrix or a 3-D tensor?

I find the following written in places: $P$ is a state transition probability matrix, $P_{ss'}^{a} = P[S_{t+1} = s' | S_t = s, A_t = a]$ (notes by david silver - slide 24 and, sutton and barto) How is ...
• 2,613
1 vote
85 views

### Can two states have different actions in a deterministic policy? How to specify states which have probability linked with them in the policy?

The agent has two actions, a0 and a1, whose effects in each state σ0; . . . ; σ3 are described in Figure 1. The edges from actions are labeled with the probability that this transition occurs. For ...
101 views

### Can the observation function in a POMDP be a function of the previous state?

I would like to model my problem with a Partially Observable Markov Decision Process (POMDP) but I have as an observation the previous state $o_t = s_{t-1}$. However, I see in all formal definitions ...
98 views

### Existence of the optimal control in finite horizon MDP

For infinite horizon MDP with finite state and action space, there exists an optimal (stationary) policy. For finite horizon MDP with finite state and action space, does there exist an optimal policy? ...
47 views

### How to estimate the order of a controlled Markov process from data?

Consider a non-stationary controlled Markov process represented by a sequence of states and actions $(s_0,a_0,\ldots,s_{T-1},a_{T-1},s_T)$ over a finite number of discrete time steps. If the process ...
• 151
1 vote
24 views

### What theoretical guarantees are lost in modeling reward as a function of next state?

There are a couple of threads (1, 2) which address the dependency of reward on the next state in addition to the current state and action. Clearly, modeling the transition probability as a joint ...
• 11
1 vote
22 views

### Can we do Deep Reinforcement Learning with Disjoint Action Sets?

I'm defining a construction you can apply to a Markov decision process*, and it involves extending an equivalence relation from the the state space of the MDP to an equivalence relation on the action ...
1 vote
23 views

### Why $A_{t-1}$ in reinforcement learning history $H_t = O_1, R_1, A_1, ..., A_{t-1}, O_t, R_t$? [closed]

I learn with David Silver's slides reinforcement learning. His definition of the history $H_t$ is: $H_t = O_1, R_1, A_1, ..., A_{t-1}, O_t, R_t$ $O =$ observations $R =$ rewards $A =$ actions Why do ...
• 129