Questions tagged [reinforcement-learning]

A set of dynamic strategies by which an algorithm can learn the structure of an environment online by adaptively taking actions associated with different rewards so as to maximize the rewards earned.

679 questions
10 views

Reinforcement Learning: A2C agent does not learn

I am trying to implement an A2C algorithm, but for some reasons, my agent does not learn very well. I build a custom environment using Unity ML Agents. The environment is very simple: an agent can ...
7 views

Natural actor critic with nonlinear function approximation

I was wondering, if there was a way to implement natural actor critic without having to deal with choosing the right features for the action-state-space. Basically, in Sutton,Barto - Reinforcement ...
16 views

Convergence Proof of First Visit Monte Carlo Control

I am currently trying to find a formal proof of convergence for the Monte Carlo Reinforcement Learning Methods described in Sutton,Barto's Book "Reinforcement Learning - An Introduction" , Section 5. ...
29 views

Is there any way to make the notion of a “policy” in reinforcement learning less abstract?

In many reinforcement learning related literature, I see the author suddenly introduces an abstract function called policy $\pi$ which maps from the state to actions. In other words, $\pi$ is a ...
29 views

Multi-armed bandit epsilon greedy

This is the code from a lecture from the Artificial Intelligence Reinforcement Learning in Python course on Udemy to implement the multi-armed bandit epsilon greedy. ...
21 views

How to increase the total number of iterations it takes to converge a MDP?

I was reading about Policy Iteration. What are the factors that influence the total number of iterations the algorithm takes to converge? For a given MDP which converges in 3 iterations, what setting ...
21 views

Reinforcement Learning: Afterstate and Afterstate value functions

While reading the book: Reinforcement Learning, An Introduction by Rich Sutton, I came across a doubt regarding afterstates and afterstates value functions and I am afraid I don't understand the ...
18 views

About the time differences in the Bellman equation

I am trying to grasp fundamental mathematics behind the Reinforcement Learning and so far I have unterstood how the Value Iteration and Policy algorithms do converge (contractions, etc.) I have still ...
10 views

Reward does not increase for a maze escaping problem with DQN

I am using deep reinforcement learning to solve a classic maze escaping task, similar to the implementation provided here, except the following three key differences: instead of using a ...
20 views

Why not sample action from Q values?

When collecting experience from which to estimate a Q(s,a) function, one common technique in the literature is to follow an epsilon greedy-strategy. In this strategy, the agent selects a random action ...
16 views

Understanding replay memory and epsilon in deep reinforcement learning

I am tentatively reusing a codebase of pacman to train my own deep reinforcement learning model. While most of the components seem reasonable and understandable to me, there are two things that seem ...
21 views

How does the Dyna Q algorithm works?

I'm having a hard time trying to understand how the dyna Q algorithm works. I put the picture which helps me to understand. My questions are: What planning really means? (it's the (f) in this picture)...
18 views

Is it interesting to do several updates using the same batch in Stochastic Gradient Descent

I am working on a reinforcement learning problem. I was given a code where people used to train their neural-network as a Q-function estimator. During the training process, they sample $m$ (m small) ...
15 views

Reinforcement Learning in Real Life/Practical Terms

In every day life, it seems that we all have various habits and actions that we perform. For example, we wake up and check our email/facebook etc. on our phones. We don't look at are current state ...
20 views

In RL, why using a behavior policy instead of the target policy for an episode is interesting?

I heard about off-policy methods in RL some days ago. While I understand the idea, the algorithms and the maths behind it, I'm not too sure why it's interesting to use a different policy in order to ...
15 views

Validity of the argument in Puterman's MDP literature

I first posted this question on math stackexchange, but I think stats stackexchange would be more appropriate for the question. I'm reading Chapter 6 of Puterman's MDP :Discrete Stocastic Dynamic ...
31 views

I got confused after reviewing several Q/A on this topic. As in "how to make a reward function in reinforcement learning", the answer states "For the case of a continuous state space, if you want an ...
19 views

What is the relation between expected average reward and single step mean reward for a non-stationary MDP policy?

The expected average reward for a policy $\pi$ is: $$\rho_\pi = \lim_{T \rightarrow \infty } \frac{1}{T} \sum_{t=1}^{T} r_t$$ where $r_t$ is the reward obtained at time $t$ following policy $\pi$. ...
27 views

Reinforcement Learning: what does “γ-just” mean in advantage function estimation? [closed]

In "HIGH-DIMENSIONAL CONTINUOUS CONTROL USING GENERALIZED ADVANTAGE ESTIMATION" by Schulman et al. (https://arxiv.org/abs/1506.02438) the advantage estimator, under certain conditions, is claimed to ...
20 views

Deriving Value Function of a Markov Reward Process

I am looking through the coursework for a Reinforcement Learning course (I am not enrolled in it, this is for my own study). In the lecture notes, on page 6, equation 11, they provide the following ...
28 views

Difference between eligibility traces and momentum?

Eligibility traces and function approximators. I'm looking at Sutton & Barto's use of eligibility traces combined with function approximation (e.g. sections 13.5, 13.6) and I noticed that it ...
21 views

How to add user input to the reinforcement learning?

I have a problem, I want to build an reinforcement learning network so as, based on the emotion a user has while listening a music to suggest another music such as he feels better. For the environment ...
54 views

What is the difference between policy-based, on-policy, value-based, off-policy, model-free and model-based?

I'm trying to clear things out for myself, there are a lot of different categorizations within RL. Some people talk about: On-policy & Off-Policy Model-based & Model-free Model-based, Policy-...
18 views

Gradually increasing reinforcement learning environment complexity

The problem of robotic arm control has different levels of complexity ranging from simulation to real-world application. For example, a simulation may not model friction of the joints, which becomes ...
29 views

Understanding Approximate Dynamic Programming

I am trying to write a paper for my optimization class about Approximate Dynamic Programming. I found a few good papers but they all seem to dive straight into the material without talking about the ...
26 views

44 views

how “Continuous control with Gaussian policies” are compute?

in blog about Reinforcement learning in part, they discuss "Continuous control with Gaussian policies" Hui define the values for actions as Gaussian distributed . and the policy is defined using a ...
21 views

How to deal with observations that are only partially available?

Consider a reinforcement learning setup where some parts of the observation space are only partially available, i.e. the corresponding information is sometimes not available. For example consider a ...
25 views

How to shape the reward when the goal is to get as close as possible?

I'm curious how one should define the reward for problems where it is not clear whether a target goal can be reached or not but getting as close as possible is desired. For example in Sutton & ...
32 views

Why is there no Target Value function in PPO?

I just implemented the PPO algorithm in tensorflow and strictly followed the algorithm provided in the original PPO paper by Schulman et. al. 2017 Previously I did some experiments with the DDPG ...
17 views

34 views

Reinforcement Learning - When to stop training?

I have built a deep reinforcement learning based portfolio optimisation agent. At a high level it is using macro economic data, valuations of the assets and a few technical indicators as the features. ...
9 views

Automating Feature Selection

Currently trying to use Monte-Carlo Tree Search to automate the process of feature selection for SVM (which I'm using to evaluate my features). Although successful so far, It tightly depends on the ...
20 views

How do Tile Coding offsets still cover full state space / affect edge cases?

Reading Sutton & Barto I’m having a hard time visualizing the implementation of the tile coding discretization of states. Specifically, if tilings are offset, how does this effect edge cases? For ...
6 views

Parallelizing Rollout/Simulation phase of Monte Carlo Tree Search

I have a Monte Carlo Tree Search implementation that I need to optimize. So I thought about parallelizing the rollout phase. How to do that? Are there any python modules etc that you would recommend?