Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [reinforcement-learning]

A set of dynamic strategies by which an algorithm can learn the structure of an environment online by adaptively taking actions associated with different rewards so as to maximize the rewards earned.

0
votes
0answers
3 views

Why is the reward fluctuating for Double Q-Learning?

I am trying to implement Double Q-Learning using neural networks from the Keras library. When I first tried Simple DQN, the graph of the reward was fluctuating a lot so, I implemented a Double DQN. ...
0
votes
0answers
16 views

is backpropagation appropriate for training actor-critic when using Neural networks? [on hold]

I'm confused whether the backpropagation is appropriate to train the actor as well as the critic. If it possible I would like to know what is the update part for both. I used already to rain the ...
0
votes
0answers
20 views

Is there a branch of machine learning that can deal with near infinite state spaces

So I have a game type problem defined as follows; Up to 10 players Each player has: 64 tiles 200 piece types Up to 20 pieces in play at any time There's a random ...
1
vote
1answer
16 views

Question about representation hierarchical reinforcement learning

I'm reading near-optimal representation learning for hierarchical reinforcement learning. At page 5, they obtain Equation 13 as follows $$ \begin{align} J(\theta,s_t,\pi)&=\sum_{k=1}^c\gamma^{k-1}...
0
votes
0answers
32 views

how “Continuous control with Gaussian policies” are compute?

in blog about Reinforcement learning in part, they discuss "Continuous control with Gaussian policies" Hui define the values for actions as Gaussian distributed . and the policy is defined using a ...
1
vote
0answers
14 views

How to deal with observations that are only partially available?

Consider a reinforcement learning setup where some parts of the observation space are only partially available, i.e. the corresponding information is sometimes not available. For example consider a ...
2
votes
0answers
18 views

How to shape the reward when the goal is to get as close as possible?

I'm curious how one should define the reward for problems where it is not clear whether a target goal can be reached or not but getting as close as possible is desired. For example in Sutton & ...
0
votes
1answer
11 views

Why is there no Target Value function in PPO?

I just implemented the PPO algorithm in tensorflow and strictly followed the algorithm provided in the original PPO paper by Schulman et. al. 2017 Previously I did some experiments with the DDPG ...
2
votes
1answer
17 views

Calculating the Policy Gradient for a Monte Carlo REINFORCE Algorithm

I am currently trying to implement the Monte Carlo REINFORCE algorithm, as described in Sutton and Barto's book Reinforcement Learning (p. 328, Second Edition). If $\theta$ denotes the parameter for ...
0
votes
0answers
9 views

Looking for a proper reinforcement learning solution

I am looking for a proper reinforcement learning solution for the following problem: Suppose I have a pool of candidate functions f \in Pool(it's like ...
0
votes
0answers
28 views

Solving Mountain-Car problem with policy gradient [on hold]

I'm having hard times trying to solving Open AI Gym's Mountain Car problem. I'm approaching it using Reinforcement Learning and Policy gradient optimization (please forgive me if terms are not ...
0
votes
0answers
10 views

IQN: Confusion about distortion risk measure

In the paper "Implicit Quantile Networks for Distributional Reinforcement Learning", they define $$ \begin{align} Z_\tau&:=F_Z^{-1}(\tau)\tag 1\\ Q&:=\mathbb E_{\tau\sim U([0,1])}[Z_\tau]\tag ...
2
votes
0answers
23 views

Reinforcement Learning - When to stop training?

I have built a deep reinforcement learning based portfolio optimisation agent. At a high level it is using macro economic data, valuations of the assets and a few technical indicators as the features. ...
0
votes
0answers
9 views

Automating Feature Selection

Currently trying to use Monte-Carlo Tree Search to automate the process of feature selection for SVM (which I'm using to evaluate my features). Although successful so far, It tightly depends on the ...
0
votes
1answer
14 views

How do Tile Coding offsets still cover full state space / affect edge cases?

Reading Sutton & Barto I’m having a hard time visualizing the implementation of the tile coding discretization of states. Specifically, if tilings are offset, how does this effect edge cases? For ...
0
votes
0answers
6 views

Parallelizing Rollout/Simulation phase of Monte Carlo Tree Search

I have a Monte Carlo Tree Search implementation that I need to optimize. So I thought about parallelizing the rollout phase. How to do that? Are there any python modules etc that you would recommend?
0
votes
0answers
30 views

Feature Selection using Monte Carlo Tree Search

Trying to tackle the problem of feature selection as an RL problem inspired by this paper here: https://hal.inria.fr/inria-00484049/document So I used Monte-Carlo Tree search for this problem, where ...
0
votes
0answers
24 views

Different algorithms categorized in reinforcement learning

For some time I am going through reinforcement learning, and have found a lot of diverse information specially in area of Policies (algorithms). I figured out that policies can be classified in On ...
0
votes
0answers
43 views

Reinforcement Learning: Bellman Equations and relations for optimal (state)/value function?

This question is about (stationary) Markov Decision Processes in the case of discounted reward. My questions are: Why is $v^*(s) = \sup_{a \in A} Q^*(s,a)$? Why is $Q^*(s,a) = E[R_t|S_t=s, A_t=a] + ...
2
votes
0answers
18 views

Input/ouput design for deep reinforcement learning for imperfect information game

I'm working on a bot that plays an imperfect information game similar to chess, where each move you are effectively moving a piece from one location to another. I'm trying to decide what the best way ...
1
vote
1answer
40 views

Why is the value function obtained from a greedy policy different from its original value function (i.e. $ V_k \neq V_{\pi_k } $)?

Consider a vector of values $V_k$ and consider the related value $V_{\pi_k}$ obtained by coming the policy $\pi_k$ by acting greedily according to it. i.e. $$ \pi_k(i) := arg \min_{a \in A} \{ R(i,a) ...
-1
votes
0answers
23 views

what is the parameterized policy for Example 13.1 Short corridor with switched actions?

in Sutton's book Reinforcement Learning: An Introduction (http://incompleteideas.net/book/bookdraft2017nov5.pdf) Chapter 13. for Example 13.1 Short corridor with switched actions, for calculating the ...
1
vote
1answer
28 views

how Deriving the formula for “The on-policy distribution in episodic tasks”?

in Sutton's book Reinforcement Learning: An Introduction Chapter 9, how to drive the formula for "The on-policy distribution in episodic tasks" as flow? that h(s) denotes the probability that an ...
0
votes
1answer
20 views

Value function and action value function

In an MDP, For any state s, what is the difference between the action value function and value function? I guess they are same because they both are defined w.r.t a policy (say pi). Now if policy is ...
0
votes
0answers
19 views

About using a neural network to train another neural network (on the same problem)

While experimenting with OpenAI gym to play a bit with reinforcement learning, I ended with using an approach that I can't exactly say what type of paradigm of training is, and I'd like to get your ...
1
vote
0answers
33 views

Examples of features construction for linear methods in Reinforcement Learning

I am referring to page 210 of of Sutton and Barto book on Reinforcement Learning available here: book Linear function approximation for state-value functions are of the form $$\hat{v}(s,\mathbf{w}) = ...
0
votes
0answers
12 views

How to obtain the reward in Inverse reinforcement learning?

How to obtain the reward in Inverse reinforcement learning? Can anyone given step by step instructions of the algorithm for IRL? I have the states, the action, Transition probability, policy(I believe ...
0
votes
1answer
14 views

Product Price Prediction - using online scrapped data [closed]

AIM: To Predict Price of products based on data that I have taken from other online stores. e.g Predict price of Samsung Galaxy S10, data will be from multiple online stores. Problem: Which Machine ...
0
votes
1answer
39 views

Interpreting gradient descent as a constrained optimization problem- Reinforcement learning

I' m studying the lectures of Sergey Levine in reinforcement learning, specifically the TRPO algorithm, during his explanation we claims that gradient descent is the same as doing this. He does ...
0
votes
1answer
20 views

Limits and constraints for Q-learning

I have simple implementation of Q-learning algorithm and I'm trying to run it on States space size = 36865 Actions space size = 25 So my resulting Q-table is ...
0
votes
1answer
18 views

Sample complexity of deep reinforcement learning agents on smaller state spaces versus zero-padded state spaces

If I train two agents, one on environment A and one on environment A', where A' is just environment A padded with 10 rows of zeros, what can I predict will happen in terms of relative sample ...
0
votes
1answer
42 views

Can most optimization problems be framed and tackled as reinforcement learning problems?

There is a clear overlap between both. Which characteristics can help us identify problems that could be tackled as classic optimization and rf also?
1
vote
1answer
29 views

Q-learning shows worse results than value iteration

I'm trying to solve the same problem with different algorithms (Travel max possible distance with a car). While using value iteration and policy iteration I was able to get the best results possible ...
1
vote
0answers
18 views

Train model on “bootstrapped” target?

Question I'd like to train a model in scikit-learn with the following input. Instead of having (X, y), I have (X, dy) where <...
1
vote
0answers
40 views

Which optimization algorithm applies better for this production problem? Cost minimization

This is a cost minimization problem, where I have to plan the development of a field and the installation of some machines there. Rectangles A,B,C,D represent production areas, that can overlap. I ...
0
votes
1answer
26 views

Reinforcement learning based Q-learning for wireless routing

In the Q-learning method to get the optimal strategy, the update method is like the following: \begin{equation} Q(S,A) \leftarrow \ Q(S,A) + \alpha [R+\gamma~max_a(Q(s',a)) -Q(S,A)] \end{equation} If ...
4
votes
1answer
50 views

Incremental solution for matrix inverse using Shermann-Morrison in $O(n^2)$

I have been reading a presentation on Value Function Approximation by David Silver (Introduction to Reinforcement Learning Course). On page 43 he finds a solution for linear least squares for an ...
0
votes
1answer
41 views

Computing Empirical Fisher Information matrix for natural gradient

I would like to implement the natural gradient for reinforcement learning as described in the following paper: https://arxiv.org/pdf/1703.02660.pdf However, I do not know how to compute the empirical ...
1
vote
2answers
65 views

in reinforcement learning off policy mc may not work

I noticed off-policy mc prediction(or control) will not work, as being descripted by boxed algorithm in page 110 of the book "reinforcement learning an introduction". The weight W should before C's(...
0
votes
0answers
39 views

Why is the objective of Multi-Armed Bandits (MAB) not the same as the one for Reinforcement Learning (RL)?

I was learning about Multi-Armed Bandits (MAB) and came across the so called regret: $$ R_T(\pi) = \max_{i \in [n]} G_T(i) - G_T(\pi) = \max_{i \in [n] } \sum^T_{t=1} r_{i,t} - \sum^T_{t=1} r_{\pi(t),...
0
votes
0answers
27 views

Monte Carlo actor critic algorithm

Has anyone implemented Monte Carlo on policy actor-critic algorithm or know a codebase online and can share it? P.S: I am not sure if this is the right place to post this, sorry for that.
0
votes
2answers
38 views

Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

I am looking for a little clarity on what the policy gradient theorem means. My confusion lies in the fact that the reward $R$ in reinforcement learning is non-differentiable in the policy parameters. ...
1
vote
0answers
56 views

Reinforcement Learning partial derivative of loss function w.r.t. input of softmax

In the paper "Self-critical sequence training for image captioning" (link) on page 3 they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected ...
0
votes
0answers
12 views

Does training a VAE online from a nonstationary distribution affect convergence?

For example, using data being sampled from reinforcement learning as the policy improves. If there is an issue, how would we address the issue?
3
votes
1answer
137 views

Why are rewards scaled when using Reinforcement Learning (RL) algorithms in practice?

I was going through this tutorial in pytorch and saw the following code: ...
2
votes
1answer
46 views

The proof for policy iteration algorithm's optimality

I am trying to understand why the policy iteration algorithm in Reinforcement Learning always improves the value function until it converges. Let's assume we have the policy $\pi_0(s)$ and our value ...
2
votes
0answers
54 views

REINFORCE calculating the log policy gradient for a continuous action space

I've noticed that when modelling a continuous action space, the default thing to do is to estimate a mean and a variance where each is parameterized by a neural network or some other model. I also ...
2
votes
1answer
73 views

Understanding the temporal difference prediction error formula which uses a derivative

I'm very new to understanding the concept of prediction error underlying the output of the critic in the critic-actor method (RL learning), so bear with me, please. For the temporal difference ...
2
votes
0answers
37 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor", they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{...
1
vote
0answers
31 views

Best approach for learning Reinforcement Learning coming from economics?

I have an economics background so I have have Calculus, Linear Algebra, Diff. Eq., 2 semesters of Stats and Prob. and some Python Knowledge. My school offers a 2 months postgraduate course in ...