# Questions tagged [policy-gradient]

The tag has no usage guidance.

13 questions
1answer
19 views

### how the natural policy gradient simplifies?

in David Silver RL course page 37 for "natural policy gradient" Using compatible function approximation we have: So the natural policy gradient simplifies: my question is how the above equation ...
0answers
38 views

### how “Continuous control with Gaussian policies” are compute?

in blog about Reinforcement learning in part, they discuss "Continuous control with Gaussian policies" Hui define the values for actions as Gaussian distributed . and the policy is defined using a ...
1answer
20 views

### Why is there no Target Value function in PPO?

I just implemented the PPO algorithm in tensorflow and strictly followed the algorithm provided in the original PPO paper by Schulman et. al. 2017 Previously I did some experiments with the DDPG ...
0answers
21 views

### Train model on “bootstrapped” target?

Question I'd like to train a model in scikit-learn with the following input. Instead of having (X, y), I have (X, dy) where <...
1answer
58 views

### Computing Empirical Fisher Information matrix for natural gradient

I would like to implement the natural gradient for reinforcement learning as described in the following paper: https://arxiv.org/pdf/1703.02660.pdf However, I do not know how to compute the empirical ...
2answers
42 views

### Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

I am looking for a little clarity on what the policy gradient theorem means. My confusion lies in the fact that the reward $R$ in reinforcement learning is non-differentiable in the policy parameters. ...
0answers
57 views

### REINFORCE calculating the log policy gradient for a continuous action space

I've noticed that when modelling a continuous action space, the default thing to do is to estimate a mean and a variance where each is parameterized by a neural network or some other model. I also ...
0answers
59 views

### Why does the Policy Gradient Theorem generalize to continuous action spaces

The policy gradient is generally in the shape of the following: $$L^{PG}(\theta) = \mathbb{E}_t \left[ \log \pi_\theta(a_t \mid s_t) A_t \right]$$ Where $\pi$ represents the probability of taking ...
1answer
181 views

### Variance of reparameterization trick and score function

For a function $\mathbf E_{z\sim q_\phi(z|x)}[f(z)]$(assuming $f$ is continuous), where $q_\phi$ is a Gaussian distribution, if we want to compute the gradient w.r.t. $\phi$, we have two way to do ...
2answers
34 views

### How can policy parameterization be simpler than action-value parameterization in function approximation?

In the second edition of the book "Reinforcement Learning: an introduction" by Sutton and Bato page 323 (Policy gradient chapter) it says that: "Perhaps the simplest advantage that policy ...
2answers
31 views

### The effect of policy parameter on the action and the state distribution in policy gradient method for episodic tasks

In the second edition of the book "Reinforcement Learning: an introduction" by Sutton and Bato page 324 (Policy gradient chapter): It says that: Given a state, the effect of the policy parameter on ...
1answer
139 views

### Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?

Following David Silver's course, I came across the actor-critic policy improvement algorithm family. It holds For one-step Markov decision processes that \nabla_{\theta}J(\theta) = \mathbb{E}_{\...
1answer
282 views

### Can Q-learning or SARSA be used to find an stochastic policy?

If the optimal policy is known to be stochastic (e.g. like in the stone, paper, scissors game), can this stochastic policy be found using SARSA or Q-learning, or is it only possible with policy ...