Skip to main content

Questions tagged [policy-gradient]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
75 views

Batches in policy gradient methods – theory vs practice

I am currently trying to understand the implementation of batching in policy gradient / actor-critic methods. My understanding is that these methods in principal work as follows: collect a batch of $N$...
mathiasj's user avatar
0 votes
0 answers
12 views

In reinforce algorithm (policy gradient), why is the reward not dependent upon the policy parameters?

The following is the precursor to the policy gradient algorithm Why is the return $r(\tau)$ over the trajectory taken out of the gradient as if it is constant w.r.t the policy parameters $\theta$. Is ...
figs_and_nuts's user avatar
0 votes
0 answers
44 views

Policy Gradients Therom for Episodic Cases: How are different formulas related?

While I am generally aware of how the Policy Gradients algorithms work theoretically, I was recently a bit confused between two definitions of the Policy Gradient and later the derivation of the ...
Ufuk Can Bicici's user avatar
1 vote
0 answers
36 views

Policy gradient theorem proofs

I'm looking at two proofs for the policy gradient theorem. van heeswijk starts of with expressing the policy gradient as a gradient over expected reward trajectories: and the rest is easy. However ...
ihadanny's user avatar
  • 3,340
1 vote
1 answer
35 views

REINFORCE implementation - should we multiply $G_t$ by $\gamma^t$?

I think I see a contradiction between two blog posts, please help me decide if there's an error somewhere or how do these align: lilian weng claims this: while wouter van heeswijk claims this: [ who ...
ihadanny's user avatar
  • 3,340
2 votes
1 answer
106 views

Is TRPO just a "safe" version of off-policy policy iteration

So, the constrained TRPO objective is the following: $$ J(\theta) = E_t\left[ \frac{\pi_\theta(a_t|s_t)}{\pi_{old}(a_t|s_t)}\cdot A_t \right]\\ st. D_{KL}[\pi_{old}(\cdot|s_t)||\pi(\cdot|s_t)] \le \...
Alberto's user avatar
  • 1,207
3 votes
1 answer
207 views

REINFORCE: how to sample the derivative

When implementing the REINFORCE algorithm, it requires the derivative of the log probability. As highlighted in the image below (from David Silver's 7th lecture). In many implementations I've seen ...
alwayscurious's user avatar
1 vote
1 answer
63 views

Reinforcement learning policy gradient derivation

I was reading a document about Reinforcement Learning policy gradient http://web.stanford.edu/class/cs234/CS234Win2019/slides/lnotes8.pdf when I encountered this expression $ \nabla_{\theta} \mathbb{...
endeavor's user avatar
  • 183
1 vote
0 answers
94 views

Reinforcement learning: is a softmax policy actor-critic expected to work on mountain car?

I am following David Silver's RL course and I'm struggling to apply the Actor Critic concept to the Mountain Car environment. I am using a softmax policy with linear function approximation. I am also ...
pabsan-0's user avatar
3 votes
1 answer
338 views

Why the $\gamma^t$ is needed here in REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for $\pi_{*}$?

While reading PG method in Prof Sutton's RL book again, I found there is $\gamma^t$ in the last row (as shown below) in pseudo code. The book said The second difference between the pseudocode update ...
GoingMyWay's user avatar
  • 1,381
1 vote
0 answers
27 views

Notation in Trust Region Policy Optimization by John Schulman et al

I am quite new to the area of reinforcement learning and find it hard to convice myself that the different notations used for reward function, state/action value function etc. coincide. Apparently I ...
Jannis H.'s user avatar
1 vote
1 answer
234 views

Derivation of expected value in REINFORCE policy gradient

This is a derivation in the book "Reinforcement Learning, an Introduction, 2ed" for the REINFORCE algorithm. By definition $q_\pi(s,a)=\mathbb{E}[G_t|S_t=s,A_t=a]$. I don't understand how ...
João Pedro's user avatar
2 votes
1 answer
226 views

Reinforcement Learning with Oracle Policy

I'm working on a reinforcement learning problem. The simulation environment is pretty simple (like those maze problems) so I can manually work out its solution. The idea I have is: since I can work ...
DiveIntoML's user avatar
  • 2,043
1 vote
1 answer
563 views

How to choose the baseline for policy gradient

I learnt that the baseline could be V or Q (could it be advantage?), but how to choose among these different options? What is the rationale behind making these choices? Thanks.
veydan's user avatar
  • 11
0 votes
0 answers
71 views

Bellman equations and reinforcement learning

We need to define just a few things first: $(a_t, s_t) \in \mathcal{A} \times \mathcal{S}$ are the action and state at time $t$; $r(s_t, a_t)$ is the reward for taking action $a_t$ in state $s_t$; ...
Stéphane's user avatar
  • 258
1 vote
0 answers
464 views

Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm

In the A3C algorithm from the original paper: the gradient with respect to log policy involves the term $$\log \pi(a_i|s_i;\theta')$$ where $s_i$ is the state of the environment at time step $i$, and ...
Kagaratsch's user avatar
1 vote
0 answers
216 views

Calculating the Fisher information matrix for implementing a policy-gradient algorithm

I am having difficulty implementing a reinforcement learning algorithm using a policy gradient approach, where the agent is a neural network. More specifically, I am having trouble calculating the '...
Vim154's user avatar
  • 11
1 vote
0 answers
14 views

Policy gradient theorem for a finite number of steps

Consider a continuous state-action space $\mathcal{S} \times \mathcal{A}$. For the timing convention, suppose that we start in period $t=0$ by drawing a state $s_0 \in \mathcal{S}$ from some ...
Stéphane's user avatar
  • 258
2 votes
1 answer
270 views

policy gradient sampling stationary state distribution

Currently learning about the policy gradient theorem for reinforcement learning. The final derivation for the policy gradient simplifies to $$E_{\pi}[Q^{\pi}(s,a)\nabla_{\theta}ln\,\pi_{\theta}(a|s)]$$...
calveeen's user avatar
  • 1,116
1 vote
1 answer
75 views

A question about policy gradient with function approximation

I've just read a paper about policy gradient: Sutton, R. S., McAllester, D. A., Singh, S. P., & Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. ...
Wheel's user avatar
  • 123
1 vote
2 answers
153 views

Is matrix $H$ positive definite in TRPO algorithm?

TRPO Taylor expands the objective and constraint to $$ \theta=\mathop{\arg\max}_\theta g^T(\theta-\theta_{\text{old}})\quad\text{s.t.}\quad \frac{1}{2}(\theta-\theta_{\text{old}})^TH(\theta-\theta_{\...
Wheel's user avatar
  • 123
1 vote
1 answer
676 views

REINFORCE algorithm, help for the proof of the variance reduction by subtracting a baseline

I'm trying to find a proof or an approximate argument justifying that, in the REINFORCE algorithm, subtracting a baseline to the episode reward reduces the variance. I believe this proof can be done ...
nathan raynal's user avatar
0 votes
0 answers
50 views

Non authorized actions with Policy gradient algorithm

I am using Reinforce algorithm to learn the optimal policy in problem where the action space is dynamic. It depends on the state, in each state, the agent is allowed to choose among a subset of the ...
AR795's user avatar
  • 21
2 votes
3 answers
443 views

Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?

In Maxim Lapan's book Deep Reinforcement Learning Hands-on, section Continuous A2C, it says By definition, the probability density function of the Gaussian Distribution is $$f(x | \mu, \sigma^2) =...
jgauth's user avatar
  • 141
2 votes
0 answers
385 views

PPO: Reinforcement learning algorithm with multiple, seperate outputs

I'm implementing the PPO algorithm on a problem where initially I only had a single discrete action. For this I have a shared neural network base, with an output head for the discrete output and an ...
Marcin Kozłowski's user avatar
2 votes
1 answer
119 views

Reinforcement learning using the gradient of expected value doesn't lead to the optimal policy

I'm trying to learn more about reinforcement learning, and I've devised a very simple game as a thought experiment. The game consists of a single turn where the agent plays one of three possible cards....
MingYun's user avatar
  • 23
2 votes
0 answers
237 views

Can't use replay memory with policy gradient, why?

One of the approaches to improving the stability of the Policy Gradient family of methods is to use multiple environments in parallel. The reason behind this is the fundamental problem we ...
jgauth's user avatar
  • 141
0 votes
1 answer
36 views

Isn't a simulation a great model for model-based reinforcement learning?

Most reinforcement learning agents are trained in simulated environments. And the goal is often to maximize performance in this same environment. Why is the simulation not used for planning in these ...
Ray Walker's user avatar
1 vote
1 answer
80 views

Does maximizing the average reward also maximizes the expected return in the initial state?

Suppose I have an episodic Markov Decision Process where all episodes start in the same state, $s_0$. I also have a parameterized policy $\pi_\theta$, and I'm trying to find a $\theta$ such that the ...
JLagana's user avatar
  • 306
1 vote
1 answer
256 views

policy gradient for non-differentiable policy

Is it possible to apply policy gradient if the parameters of policy are not differentiable? If not, is there any other algorithm for optimizing such type of policies? One example I'm thinking about ...
DiveIntoML's user avatar
  • 2,043
0 votes
0 answers
57 views

Reinforcement Learning on quantum circuit

I am trying to teach an agent to make any random 1-qubit state reach uniform superposition. So basically, the full circuit will be ...
Sarvagya Gupta's user avatar
1 vote
1 answer
664 views

Policy Gradient Methods advantages over value-based methods

In the RL bible by Sutton and Barto it says on page 322 regarding the advantages of policy gradient methods: If the action space is discrete and not too large, then a natural and common kind of ...
frederik's user avatar
0 votes
0 answers
322 views

How can I coumpute Policy Gradient LOSS in tensorflow

I am self-studying RL and currently doing hw2 from Berkeley CS294-112. The thing I cannot figure out is how to compute loss in policy gradients. Basically, REINFORCE algorithm has the following update ...
Yong Jin Shin's user avatar
5 votes
4 answers
8k views

Is it possible to use DDPG for discrete action space?

In Deep Deterministic Policy Gradients(DDPG) method, we use two neural networks, one is Actor and the other is Critic. From actor-network, we can directly map states to actions (the output of the ...
Rachel's user avatar
  • 71
1 vote
3 answers
82 views

Is there any way to make the notion of a "policy" in reinforcement learning less abstract?

In many reinforcement learning related literature, I see the author suddenly introduces an abstract function called policy $\pi$ which maps from the state to actions. In other words, $\pi$ is a ...
Curaçao Hajek's user avatar
0 votes
1 answer
56 views

how the natural policy gradient simplifies?

in David Silver RL course page 37 for "natural policy gradient" Using compatible function approximation we have: So the natural policy gradient simplifies: my question is how the above equation ...
arash moradi's user avatar
0 votes
0 answers
112 views

how "Continuous control with Gaussian policies" are compute?

in blog about Reinforcement learning in part, they discuss "Continuous control with Gaussian policies" Hui define the values for actions as Gaussian distributed . and the policy is defined using a ...
arash moradi's user avatar
3 votes
1 answer
1k views

Why is there no Target Value function in PPO?

I just implemented the PPO algorithm in tensorflow and strictly followed the algorithm provided in the original PPO paper by Schulman et. al. 2017 Previously I did some experiments with the DDPG ...
flxh's user avatar
  • 257
2 votes
0 answers
31 views

Train model on "bootstrapped" target?

Question I'd like to train a model in scikit-learn with the following input. Instead of having (X, y), I have (X, dy) where <...
Kris's user avatar
  • 271
1 vote
1 answer
2k views

Computing Empirical Fisher Information matrix for natural gradient

I would like to implement the natural gradient for reinforcement learning as described in the following paper: https://arxiv.org/pdf/1703.02660.pdf However, I do not know how to compute the empirical ...
cookiedealer's user avatar
2 votes
2 answers
306 views

Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

I am looking for a little clarity on what the policy gradient theorem means. My confusion lies in the fact that the reward $R$ in reinforcement learning is non-differentiable in the policy parameters. ...
figs_and_nuts's user avatar
3 votes
0 answers
249 views

REINFORCE calculating the log policy gradient for a continuous action space

I've noticed that when modelling a continuous action space, the default thing to do is to estimate a mean and a variance where each is parameterized by a neural network or some other model. I also ...
tryingtolearn's user avatar
4 votes
0 answers
111 views

Why does the Policy Gradient Theorem generalize to continuous action spaces

The policy gradient is generally in the shape of the following: $$ L^{PG}(\theta) = \mathbb{E}_t \left[ \log \pi_\theta(a_t \mid s_t) A_t \right] $$ Where $\pi$ represents the probability of taking ...
BadProgrammer's user avatar
4 votes
1 answer
2k views

Variance of reparameterization trick and score function

For a function $\mathbf E_{z\sim q_\phi(z|x)}[f(z)]$(assuming $f$ is continuous), where $q_\phi$ is a Gaussian distribution, if we want to compute the gradient w.r.t. $\phi$, we have two way to do ...
Maybe's user avatar
  • 1,085
1 vote
2 answers
216 views

How can policy parameterization be simpler than action-value parameterization in function approximation?

In the second edition of the book "Reinforcement Learning: an introduction" by Sutton and Bato page 323 (Policy gradient chapter) it says that: "Perhaps the simplest advantage that policy ...
Ali Ghghgh's user avatar
0 votes
2 answers
115 views

The effect of policy parameter on the action and the state distribution in policy gradient method for episodic tasks

In the second edition of the book "Reinforcement Learning: an introduction" by Sutton and Bato page 324 (Policy gradient chapter): It says that: Given a state, the effect of the policy parameter on ...
Ali Ghghgh's user avatar
6 votes
1 answer
2k views

Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?

Following David Silver's course, I came across the actor-critic policy improvement algorithm family. It holds For one-step Markov decision processes that $$\nabla_{\theta}J(\theta) = \mathbb{E}_{\...
Gulzar's user avatar
  • 583
2 votes
1 answer
848 views

Can Q-learning or SARSA be used to find an stochastic policy?

If the optimal policy is known to be stochastic (e.g. like in the stone, paper, scissors game), can this stochastic policy be found using SARSA or Q-learning, or is it only possible with policy ...
andporji's user avatar