Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [reinforcement-learning]

A set of dynamic strategies by which an algorithm can learn the structure of an environment online by adaptively taking actions associated with different rewards so as to maximize the rewards earned.

1
vote
0answers
10 views

Reinforcement Learning: A2C agent does not learn

I am trying to implement an A2C algorithm, but for some reasons, my agent does not learn very well. I build a custom environment using Unity ML Agents. The environment is very simple: an agent can ...
0
votes
0answers
7 views

Natural actor critic with nonlinear function approximation

I was wondering, if there was a way to implement natural actor critic without having to deal with choosing the right features for the action-state-space. Basically, in Sutton,Barto - Reinforcement ...
1
vote
0answers
16 views

Convergence Proof of First Visit Monte Carlo Control

I am currently trying to find a formal proof of convergence for the Monte Carlo Reinforcement Learning Methods described in Sutton,Barto's Book "Reinforcement Learning - An Introduction" , Section 5. ...
0
votes
2answers
29 views

Is there any way to make the notion of a “policy” in reinforcement learning less abstract?

In many reinforcement learning related literature, I see the author suddenly introduces an abstract function called policy $\pi$ which maps from the state to actions. In other words, $\pi$ is a ...
0
votes
1answer
29 views

Multi-armed bandit epsilon greedy

This is the code from a lecture from the Artificial Intelligence Reinforcement Learning in Python course on Udemy to implement the multi-armed bandit epsilon greedy. ...
1
vote
0answers
21 views

How to increase the total number of iterations it takes to converge a MDP?

I was reading about Policy Iteration. What are the factors that influence the total number of iterations the algorithm takes to converge? For a given MDP which converges in 3 iterations, what setting ...
2
votes
1answer
21 views

Reinforcement Learning: Afterstate and Afterstate value functions

While reading the book: Reinforcement Learning, An Introduction by Rich Sutton, I came across a doubt regarding afterstates and afterstates value functions and I am afraid I don't understand the ...
0
votes
0answers
18 views

About the time differences in the Bellman equation

I am trying to grasp fundamental mathematics behind the Reinforcement Learning and so far I have unterstood how the Value Iteration and Policy algorithms do converge (contractions, etc.) I have still ...
0
votes
0answers
10 views

Reward does not increase for a maze escaping problem with DQN

I am using deep reinforcement learning to solve a classic maze escaping task, similar to the implementation provided here, except the following three key differences: instead of using a ...
1
vote
1answer
20 views

Why not sample action from Q values?

When collecting experience from which to estimate a Q(s,a) function, one common technique in the literature is to follow an epsilon greedy-strategy. In this strategy, the agent selects a random action ...
0
votes
0answers
16 views

Understanding replay memory and epsilon in deep reinforcement learning

I am tentatively reusing a codebase of pacman to train my own deep reinforcement learning model. While most of the components seem reasonable and understandable to me, there are two things that seem ...
0
votes
1answer
21 views

How does the Dyna Q algorithm works?

I'm having a hard time trying to understand how the dyna Q algorithm works. I put the picture which helps me to understand. My questions are: What planning really means? (it's the (f) in this picture)...
1
vote
0answers
18 views

Is it interesting to do several updates using the same batch in Stochastic Gradient Descent

I am working on a reinforcement learning problem. I was given a code where people used to train their neural-network as a Q-function estimator. During the training process, they sample $m$ (m small) ...
0
votes
0answers
15 views

Reinforcement Learning in Real Life/Practical Terms

In every day life, it seems that we all have various habits and actions that we perform. For example, we wake up and check our email/facebook etc. on our phones. We don't look at are current state ...
1
vote
1answer
20 views

In RL, why using a behavior policy instead of the target policy for an episode is interesting?

I heard about off-policy methods in RL some days ago. While I understand the idea, the algorithms and the maths behind it, I'm not too sure why it's interesting to use a different policy in order to ...
0
votes
0answers
15 views

Validity of the argument in Puterman's MDP literature

I first posted this question on math stackexchange, but I think stats stackexchange would be more appropriate for the question. I'm reading Chapter 6 of Puterman's MDP :Discrete Stocastic Dynamic ...
1
vote
0answers
31 views

reward function design in reinforcement learning

I got confused after reviewing several Q/A on this topic. As in "how to make a reward function in reinforcement learning", the answer states "For the case of a continuous state space, if you want an ...
2
votes
0answers
19 views

What is the relation between expected average reward and single step mean reward for a non-stationary MDP policy?

The expected average reward for a policy $\pi$ is: $$ \rho_\pi = \lim_{T \rightarrow \infty } \frac{1}{T} \sum_{t=1}^{T} r_t$$ where $r_t$ is the reward obtained at time $t$ following policy $\pi$. ...
1
vote
0answers
27 views

Reinforcement Learning: what does “γ-just” mean in advantage function estimation? [closed]

In "HIGH-DIMENSIONAL CONTINUOUS CONTROL USING GENERALIZED ADVANTAGE ESTIMATION" by Schulman et al. (https://arxiv.org/abs/1506.02438) the advantage estimator, under certain conditions, is claimed to ...
1
vote
0answers
20 views

Deriving Value Function of a Markov Reward Process

I am looking through the coursework for a Reinforcement Learning course (I am not enrolled in it, this is for my own study). In the lecture notes, on page 6, equation 11, they provide the following ...
0
votes
0answers
28 views

Difference between eligibility traces and momentum?

Eligibility traces and function approximators. I'm looking at Sutton & Barto's use of eligibility traces combined with function approximation (e.g. sections 13.5, 13.6) and I noticed that it ...
0
votes
0answers
21 views

How to add user input to the reinforcement learning?

I have a problem, I want to build an reinforcement learning network so as, based on the emotion a user has while listening a music to suggest another music such as he feels better. For the environment ...
0
votes
1answer
54 views

What is the difference between policy-based, on-policy, value-based, off-policy, model-free and model-based?

I'm trying to clear things out for myself, there are a lot of different categorizations within RL. Some people talk about: On-policy & Off-Policy Model-based & Model-free Model-based, Policy-...
1
vote
1answer
18 views

Gradually increasing reinforcement learning environment complexity

The problem of robotic arm control has different levels of complexity ranging from simulation to real-world application. For example, a simulation may not model friction of the joints, which becomes ...
2
votes
0answers
29 views

Understanding Approximate Dynamic Programming

I am trying to write a paper for my optimization class about Approximate Dynamic Programming. I found a few good papers but they all seem to dive straight into the material without talking about the ...
0
votes
0answers
26 views

ACER: optimization using the KKT conditions

In Page 5 Sample Efficient Actor-Critic with Experience Replay, the authors define an optimization problem with a linearized KL divergence constraint (Eq.11)as follow $$ \min_z{1\over 2}\Vert \hat g_t^...
0
votes
0answers
13 views

How does Generalized Policy Iteration stabilize to the optimal policy and value function?

I've seen this question answered here Why does the policy iteration algorithm converge to optimal policy and value function? and here The proof for policy iteration algorithm's optimality however ...
1
vote
1answer
22 views

DQN - How to feed the input of 4 still frames from a game as one single state input

I was reading this blog about Deep Q-Learning. 1- In the The input section of the blog, I wanted to know how do we feed the 4 still-frames/screenshots from the game, that represent the input state, ...
0
votes
1answer
27 views

DQN - breaking correlation between consecutive samples and random sampling

I was reading through some blogs about Deep Q-Learning (DQN), and I have 2 questions: 1- I didn't understand how breaking the correlation between consecutive samples (i.e. train the network with ...
0
votes
0answers
19 views

Is it okay to calculate all the gradients for an LSTM at once?

I'm trying to use the AC method reported in "Asynchronous Methods for Deep Reinforcement Learning" for a project. The relevant algorithm is shown in pseudocode at the bottom, Algorithm S3. I'm using ...
0
votes
0answers
18 views

Q-learning python implementation

I have tried to recreate the model in this post. Essentially, the agent (denoted player) starts at the beginning of a 1x5 matrix and can move either forward or backward (but only foward if at position ...
0
votes
1answer
19 views

how the natural policy gradient simplifies?

in David Silver RL course page 37 for "natural policy gradient" Using compatible function approximation we have: So the natural policy gradient simplifies: my question is how the above equation ...
0
votes
1answer
21 views

Inference can be the goal of an unsupervised learning method or a semi-supervised learning method or even more of a reinforcement learning method?

I am new to machine learning, and I am reading a pair of machine learning books. These references talk about 2 different learning approaches: Prediction and inference, I understand the difference ...
0
votes
0answers
13 views

Why is the reward fluctuating for Double Q-Learning?

I am trying to implement Double Q-Learning using neural networks from the Keras library. When I first tried Simple DQN, the graph of the reward was fluctuating a lot so, I implemented a Double DQN. ...
0
votes
0answers
22 views

Is there a branch of machine learning that can deal with near infinite state spaces

So I have a game type problem defined as follows; Up to 10 players Each player has: 64 tiles 200 piece types Up to 20 pieces in play at any time There's a random ...
1
vote
1answer
23 views

Question about representation hierarchical reinforcement learning

I'm reading near-optimal representation learning for hierarchical reinforcement learning. At page 5, they obtain Equation 13 as follows $$ \begin{align} J(\theta,s_t,\pi)&=\sum_{k=1}^c\gamma^{k-1}...
0
votes
0answers
44 views

how “Continuous control with Gaussian policies” are compute?

in blog about Reinforcement learning in part, they discuss "Continuous control with Gaussian policies" Hui define the values for actions as Gaussian distributed . and the policy is defined using a ...
1
vote
0answers
21 views

How to deal with observations that are only partially available?

Consider a reinforcement learning setup where some parts of the observation space are only partially available, i.e. the corresponding information is sometimes not available. For example consider a ...
2
votes
0answers
25 views

How to shape the reward when the goal is to get as close as possible?

I'm curious how one should define the reward for problems where it is not clear whether a target goal can be reached or not but getting as close as possible is desired. For example in Sutton & ...
0
votes
1answer
32 views

Why is there no Target Value function in PPO?

I just implemented the PPO algorithm in tensorflow and strictly followed the algorithm provided in the original PPO paper by Schulman et. al. 2017 Previously I did some experiments with the DDPG ...
0
votes
0answers
17 views

How does Dueling DQN prevent overestimate

From what I have read 1 2, it seems that the noise during training and the max function leads to overestimate in DQN. $$Q(s, a) = r + \gamma \text{max}_{a'}[Q(s', ...
2
votes
1answer
23 views

Calculating the Policy Gradient for a Monte Carlo REINFORCE Algorithm

I am currently trying to implement the Monte Carlo REINFORCE algorithm, as described in Sutton and Barto's book Reinforcement Learning (p. 328, Second Edition). If $\theta$ denotes the parameter for ...
-1
votes
2answers
94 views

Looking for a proper reinforcement learning solution

I am looking for a proper reinforcement learning solution for the following problem: Suppose I have a pool of candidate functions f \in Pool(it's like ...
0
votes
0answers
11 views

IQN: Confusion about distortion risk measure

In the paper "Implicit Quantile Networks for Distributional Reinforcement Learning", they define $$ \begin{align} Z_\tau&:=F_Z^{-1}(\tau)\tag 1\\ Q&:=\mathbb E_{\tau\sim U([0,1])}[Z_\tau]\tag ...
2
votes
0answers
34 views

Reinforcement Learning - When to stop training?

I have built a deep reinforcement learning based portfolio optimisation agent. At a high level it is using macro economic data, valuations of the assets and a few technical indicators as the features. ...
0
votes
0answers
9 views

Automating Feature Selection

Currently trying to use Monte-Carlo Tree Search to automate the process of feature selection for SVM (which I'm using to evaluate my features). Although successful so far, It tightly depends on the ...
0
votes
1answer
20 views

How do Tile Coding offsets still cover full state space / affect edge cases?

Reading Sutton & Barto I’m having a hard time visualizing the implementation of the tile coding discretization of states. Specifically, if tilings are offset, how does this effect edge cases? For ...
0
votes
0answers
6 views

Parallelizing Rollout/Simulation phase of Monte Carlo Tree Search

I have a Monte Carlo Tree Search implementation that I need to optimize. So I thought about parallelizing the rollout phase. How to do that? Are there any python modules etc that you would recommend?
0
votes
0answers
35 views

Feature Selection using Monte Carlo Tree Search

Trying to tackle the problem of feature selection as an RL problem inspired by this paper here: https://hal.inria.fr/inria-00484049/document So I used Monte-Carlo Tree search for this problem, where ...
0
votes
0answers
27 views

Different algorithms categorized in reinforcement learning

For some time I am going through reinforcement learning, and have found a lot of diverse information specially in area of Policies (algorithms). I figured out that policies can be classified in On ...