Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [policy-gradient]

The tag has no usage guidance.

0
votes
1answer
19 views

how the natural policy gradient simplifies?

in David Silver RL course page 37 for "natural policy gradient" Using compatible function approximation we have: So the natural policy gradient simplifies: my question is how the above equation ...
0
votes
0answers
38 views

how “Continuous control with Gaussian policies” are compute?

in blog about Reinforcement learning in part, they discuss "Continuous control with Gaussian policies" Hui define the values for actions as Gaussian distributed . and the policy is defined using a ...
0
votes
1answer
20 views

Why is there no Target Value function in PPO?

I just implemented the PPO algorithm in tensorflow and strictly followed the algorithm provided in the original PPO paper by Schulman et. al. 2017 Previously I did some experiments with the DDPG ...
1
vote
0answers
21 views

Train model on “bootstrapped” target?

Question I'd like to train a model in scikit-learn with the following input. Instead of having (X, y), I have (X, dy) where <...
0
votes
1answer
58 views

Computing Empirical Fisher Information matrix for natural gradient

I would like to implement the natural gradient for reinforcement learning as described in the following paper: https://arxiv.org/pdf/1703.02660.pdf However, I do not know how to compute the empirical ...
0
votes
2answers
42 views

Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

I am looking for a little clarity on what the policy gradient theorem means. My confusion lies in the fact that the reward $R$ in reinforcement learning is non-differentiable in the policy parameters. ...
2
votes
0answers
57 views

REINFORCE calculating the log policy gradient for a continuous action space

I've noticed that when modelling a continuous action space, the default thing to do is to estimate a mean and a variance where each is parameterized by a neural network or some other model. I also ...
3
votes
0answers
59 views

Why does the Policy Gradient Theorem generalize to continuous action spaces

The policy gradient is generally in the shape of the following: $$ L^{PG}(\theta) = \mathbb{E}_t \left[ \log \pi_\theta(a_t \mid s_t) A_t \right] $$ Where $\pi$ represents the probability of taking ...
2
votes
1answer
181 views

Variance of reparameterization trick and score function

For a function $\mathbf E_{z\sim q_\phi(z|x)}[f(z)]$(assuming $f$ is continuous), where $q_\phi$ is a Gaussian distribution, if we want to compute the gradient w.r.t. $\phi$, we have two way to do ...
1
vote
2answers
34 views

How can policy parameterization be simpler than action-value parameterization in function approximation?

In the second edition of the book "Reinforcement Learning: an introduction" by Sutton and Bato page 323 (Policy gradient chapter) it says that: "Perhaps the simplest advantage that policy ...
0
votes
2answers
31 views

The effect of policy parameter on the action and the state distribution in policy gradient method for episodic tasks

In the second edition of the book "Reinforcement Learning: an introduction" by Sutton and Bato page 324 (Policy gradient chapter): It says that: Given a state, the effect of the policy parameter on ...
3
votes
1answer
139 views

Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?

Following David Silver's course, I came across the actor-critic policy improvement algorithm family. It holds For one-step Markov decision processes that $$\nabla_{\theta}J(\theta) = \mathbb{E}_{\...
3
votes
1answer
282 views

Can Q-learning or SARSA be used to find an stochastic policy?

If the optimal policy is known to be stochastic (e.g. like in the stone, paper, scissors game), can this stochastic policy be found using SARSA or Q-learning, or is it only possible with policy ...