Questions tagged [policy-gradient]
The policy-gradient tag has no usage guidance.
48
questions
1
vote
0
answers
72
views
Batches in policy gradient methods – theory vs practice
I am currently trying to understand the implementation of batching in policy gradient / actor-critic methods. My understanding is that these methods in principal work as follows: collect a batch of $N$...
- 11
0
votes
0
answers
11
views
In reinforce algorithm (policy gradient), why is the reward not dependent upon the policy parameters?
The following is the precursor to the policy gradient algorithm
Why is the return $r(\tau)$ over the trajectory taken out of the gradient as if it is constant w.r.t the policy parameters $\theta$. Is ...
- 2,543
0
votes
0
answers
42
views
Policy Gradients Therom for Episodic Cases: How are different formulas related?
While I am generally aware of how the Policy Gradients algorithms work theoretically, I was recently a bit confused between two definitions of the Policy Gradient and later the derivation of the ...
- 2,298
1
vote
0
answers
35
views
Policy gradient theorem proofs
I'm looking at two proofs for the policy gradient theorem. van heeswijk starts of with expressing the policy gradient as a gradient over expected reward trajectories:
and the rest is easy. However ...
- 3,300
1
vote
1
answer
34
views
REINFORCE implementation - should we multiply $G_t$ by $\gamma^t$?
I think I see a contradiction between two blog posts, please help me decide if there's an error somewhere or how do these align:
lilian weng claims this:
while wouter van heeswijk claims this: [
who ...
- 3,300
2
votes
1
answer
98
views
Is TRPO just a "safe" version of off-policy policy iteration
So, the constrained TRPO objective is the following:
$$
J(\theta) = E_t\left[
\frac{\pi_\theta(a_t|s_t)}{\pi_{old}(a_t|s_t)}\cdot A_t
\right]\\
st. D_{KL}[\pi_{old}(\cdot|s_t)||\pi(\cdot|s_t)] \le \...
- 1,139
3
votes
1
answer
194
views
REINFORCE: how to sample the derivative
When implementing the REINFORCE algorithm, it requires the derivative of the log probability. As highlighted in the image below (from David Silver's 7th lecture).
In many implementations I've seen ...
- 443
1
vote
1
answer
63
views
Reinforcement learning policy gradient derivation
I was reading a document about Reinforcement Learning policy gradient http://web.stanford.edu/class/cs234/CS234Win2019/slides/lnotes8.pdf when I encountered this expression
$ \nabla_{\theta} \mathbb{...
- 183
1
vote
0
answers
91
views
Reinforcement learning: is a softmax policy actor-critic expected to work on mountain car?
I am following David Silver's RL course and I'm struggling to apply the Actor Critic concept to the Mountain Car environment.
I am using a softmax policy with linear function approximation. I am also ...
- 11
3
votes
1
answer
331
views
Why the $\gamma^t$ is needed here in REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for $\pi_{*}$?
While reading PG method in Prof Sutton's RL book again, I found there is $\gamma^t$ in the last row (as shown below) in pseudo code. The book said
The second difference between the pseudocode update ...
- 1,361
1
vote
0
answers
27
views
Notation in Trust Region Policy Optimization by John Schulman et al
I am quite new to the area of reinforcement learning and find it hard to convice myself that the different notations used for reward function, state/action value function etc. coincide.
Apparently I ...
- 86
1
vote
1
answer
232
views
Derivation of expected value in REINFORCE policy gradient
This is a derivation in the book "Reinforcement Learning, an Introduction, 2ed" for the REINFORCE algorithm.
By definition $q_\pi(s,a)=\mathbb{E}[G_t|S_t=s,A_t=a]$. I don't understand how ...
- 151
2
votes
1
answer
218
views
Reinforcement Learning with Oracle Policy
I'm working on a reinforcement learning problem. The simulation environment is pretty simple (like those maze problems) so I can manually work out its solution. The idea I have is: since I can work ...
- 2,043
1
vote
1
answer
547
views
How to choose the baseline for policy gradient
I learnt that the baseline could be V or Q (could it be advantage?), but how to choose among these different options? What is the rationale behind making these choices? Thanks.
- 11
0
votes
0
answers
71
views
Bellman equations and reinforcement learning
We need to define just a few things first:
$(a_t, s_t) \in \mathcal{A} \times \mathcal{S}$ are the action and state at time $t$;
$r(s_t, a_t)$ is the reward for taking action $a_t$ in state $s_t$;
...
- 258
1
vote
0
answers
461
views
Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm
In the A3C algorithm from the original paper:
the gradient with respect to log policy involves the term
$$\log \pi(a_i|s_i;\theta')$$
where $s_i$ is the state of the environment at time step $i$, and ...
- 175
1
vote
0
answers
206
views
Calculating the Fisher information matrix for implementing a policy-gradient algorithm
I am having difficulty implementing a reinforcement learning algorithm using a policy gradient approach, where the agent is a neural network. More specifically, I am having trouble calculating the '...
- 11
1
vote
0
answers
14
views
Policy gradient theorem for a finite number of steps
Consider a continuous state-action space $\mathcal{S} \times \mathcal{A}$. For the timing convention, suppose that we start in period $t=0$ by drawing a state $s_0 \in \mathcal{S}$ from some ...
- 258
2
votes
1
answer
260
views
policy gradient sampling stationary state distribution
Currently learning about the policy gradient theorem for reinforcement learning. The final derivation for the policy gradient simplifies to
$$E_{\pi}[Q^{\pi}(s,a)\nabla_{\theta}ln\,\pi_{\theta}(a|s)]$$...
- 1,126
1
vote
1
answer
75
views
A question about policy gradient with function approximation
I've just read a paper about policy gradient: Sutton, R. S., McAllester, D. A., Singh, S. P., & Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. ...
- 123
1
vote
2
answers
149
views
Is matrix $H$ positive definite in TRPO algorithm?
TRPO Taylor expands the objective and constraint to
$$
\theta=\mathop{\arg\max}_\theta g^T(\theta-\theta_{\text{old}})\quad\text{s.t.}\quad \frac{1}{2}(\theta-\theta_{\text{old}})^TH(\theta-\theta_{\...
- 123
1
vote
1
answer
635
views
REINFORCE algorithm, help for the proof of the variance reduction by subtracting a baseline
I'm trying to find a proof or an approximate argument justifying that, in the REINFORCE algorithm, subtracting a baseline to the episode reward reduces the variance. I believe this proof can be done ...
- 168
0
votes
0
answers
50
views
Non authorized actions with Policy gradient algorithm
I am using Reinforce algorithm to learn the optimal policy in problem where the action space is dynamic. It depends on the state, in each state, the agent is allowed to choose among a subset of the ...
- 21
2
votes
3
answers
435
views
Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?
In Maxim Lapan's book Deep Reinforcement Learning Hands-on, section Continuous A2C, it says
By definition, the probability density function of the Gaussian
Distribution is $$f(x | \mu, \sigma^2) =...
- 141
2
votes
0
answers
375
views
PPO: Reinforcement learning algorithm with multiple, seperate outputs
I'm implementing the PPO algorithm on a problem where initially I only had a single discrete action. For this I have a shared neural network base, with an output head for the discrete output and an ...
2
votes
1
answer
118
views
Reinforcement learning using the gradient of expected value doesn't lead to the optimal policy
I'm trying to learn more about reinforcement learning, and I've devised a very simple game as a thought experiment. The game consists of a single turn where the agent plays one of three possible cards....
- 23
2
votes
0
answers
236
views
Can't use replay memory with policy gradient, why?
One of the approaches to improving the stability of the Policy
Gradient family of methods is to use multiple environments in
parallel. The reason behind this is the fundamental problem we
...
- 141
0
votes
1
answer
36
views
Isn't a simulation a great model for model-based reinforcement learning?
Most reinforcement learning agents are trained in simulated environments. And the goal is often to maximize performance in this same environment.
Why is the simulation not used for planning in these ...
- 123
1
vote
1
answer
79
views
Does maximizing the average reward also maximizes the expected return in the initial state?
Suppose I have an episodic Markov Decision Process where all episodes start in the same state, $s_0$. I also have a parameterized policy $\pi_\theta$, and I'm trying to find a $\theta$ such that the ...
- 306
1
vote
1
answer
251
views
policy gradient for non-differentiable policy
Is it possible to apply policy gradient if the parameters of policy are not differentiable? If not, is there any other algorithm for optimizing such type of policies?
One example I'm thinking about ...
- 2,043
0
votes
0
answers
57
views
Reinforcement Learning on quantum circuit
I am trying to teach an agent to make any random 1-qubit state reach uniform superposition. So basically, the full circuit will be ...
- 113
1
vote
1
answer
651
views
Policy Gradient Methods advantages over value-based methods
In the RL bible by Sutton and Barto it says on page 322 regarding the advantages of policy gradient methods:
If the action space is discrete and not too large, then a natural and common kind of ...
- 21
0
votes
0
answers
322
views
How can I coumpute Policy Gradient LOSS in tensorflow
I am self-studying RL and currently doing hw2 from Berkeley CS294-112. The thing I cannot figure out is how to compute loss in policy gradients. Basically, REINFORCE algorithm has the following update ...
5
votes
4
answers
7k
views
Is it possible to use DDPG for discrete action space?
In Deep Deterministic Policy Gradients(DDPG) method, we use two neural networks, one is Actor and the other is Critic.
From actor-network, we can directly map states to actions (the output of the ...
- 71
1
vote
3
answers
82
views
Is there any way to make the notion of a "policy" in reinforcement learning less abstract?
In many reinforcement learning related literature, I see the author suddenly introduces an abstract function called policy $\pi$ which maps from the state to actions.
In other words, $\pi$ is a ...
- 129
0
votes
1
answer
55
views
how the natural policy gradient simplifies?
in David Silver RL course page 37 for "natural policy gradient" Using compatible function approximation we have:
So the natural policy gradient simplifies:
my question is how the above equation ...
- 43
0
votes
0
answers
112
views
how "Continuous control with Gaussian policies" are compute?
in blog about Reinforcement learning in part, they discuss "Continuous control with Gaussian policies" Hui define the values for actions as Gaussian distributed . and the policy is defined using a ...
- 43
3
votes
1
answer
1k
views
Why is there no Target Value function in PPO?
I just implemented the PPO algorithm in tensorflow and strictly followed the algorithm provided in the original PPO paper by Schulman et. al. 2017
Previously I did some experiments with the DDPG ...
- 247
2
votes
0
answers
31
views
Train model on "bootstrapped" target?
Question
I'd like to train a model in scikit-learn with the following input. Instead of having (X, y), I have (X, dy) where <...
- 271
1
vote
1
answer
2k
views
Computing Empirical Fisher Information matrix for natural gradient
I would like to implement the natural gradient for reinforcement learning as described in the following paper: https://arxiv.org/pdf/1703.02660.pdf
However, I do not know how to compute the empirical ...
- 213
2
votes
2
answers
298
views
Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?
I am looking for a little clarity on what the policy gradient theorem means. My confusion lies in the fact that the reward $R$ in reinforcement learning is non-differentiable in the policy parameters. ...
- 2,543
3
votes
0
answers
247
views
REINFORCE calculating the log policy gradient for a continuous action space
I've noticed that when modelling a continuous action space, the default thing to do is to estimate a mean and a variance where each is parameterized by a neural network or some other model.
I also ...
- 539
4
votes
0
answers
111
views
Why does the Policy Gradient Theorem generalize to continuous action spaces
The policy gradient is generally in the shape of the following:
$$
L^{PG}(\theta) = \mathbb{E}_t \left[ \log \pi_\theta(a_t \mid s_t) A_t \right]
$$
Where $\pi$ represents the probability of taking ...
- 497
4
votes
1
answer
2k
views
Variance of reparameterization trick and score function
For a function $\mathbf E_{z\sim q_\phi(z|x)}[f(z)]$(assuming $f$ is continuous), where $q_\phi$ is a Gaussian distribution, if we want to compute the gradient w.r.t. $\phi$, we have two way to do ...
- 1,075
1
vote
2
answers
214
views
How can policy parameterization be simpler than action-value parameterization in function approximation?
In the second edition of the book "Reinforcement Learning: an introduction" by Sutton and Bato page 323 (Policy gradient chapter) it says that:
"Perhaps the simplest advantage that policy ...
- 33
0
votes
2
answers
115
views
The effect of policy parameter on the action and the state distribution in policy gradient method for episodic tasks
In the second edition of the book "Reinforcement Learning: an introduction" by Sutton and Bato page 324 (Policy gradient chapter):
It says that:
Given a state, the effect of the policy parameter on ...
- 33
6
votes
1
answer
2k
views
Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?
Following David Silver's course, I came across the actor-critic policy improvement algorithm family.
It holds For one-step Markov decision processes that
$$\nabla_{\theta}J(\theta) = \mathbb{E}_{\...
- 553
2
votes
1
answer
839
views
Can Q-learning or SARSA be used to find an stochastic policy?
If the optimal policy is known to be stochastic (e.g. like in the stone, paper, scissors game), can this stochastic policy be found using SARSA or Q-learning, or is it only possible with policy ...
- 33