6
votes
Accepted
Variance of reparameterization trick and score function
The notation here is far more complicated than it needs to be, and I suspect this is contributing to the issue of understanding this method. To clarify the problem, I'm going to re-frame this in ...
- 100k
5
votes
Accepted
Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?
Why do we need a critic at all?
I just can't see where the critic suddenly came from and what it
solves.
The critic solves the problem of high variance in the reward signal. If you run the ...
- 23.3k
4
votes
Accepted
Can Q-learning or SARSA be used to find an stochastic policy?
Neither Q-learning nor SARSA define strictly how the policy should be derived from action-values, so you might be able to use them to learn some approximation to stochastic policies, if you used a ...
- 6,284
3
votes
Accepted
Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?
Well first, you're trying to take the gradient of the return, not the reward. Also unless both the environment and the policy is deterministic, you'd be taking the gradient of the expected return. Now ...
- 23.3k
3
votes
Accepted
Why is there no Target Value function in PPO?
A target Q-function is employed because of the "moving target" problem: the target is dependent on the Q-network, but so is the prediction, which causes problems for convergence.
In actor-critic ...
- 23.3k
2
votes
Is it possible to use DDPG for discrete action space?
DDPG extends actor-critic methods from the discrete action-space environments they were originally developed on to continous action-space environments.
With that in mind -- sure, you can use actor-...
- 23.3k
2
votes
Accepted
Is there any way to make the notion of a "policy" in reinforcement learning less abstract?
As Brale mentions, it's actually a function from states to a distribution over the set of actions.
Because the domain and range of this function are usually discrete,
The state is very often ...
- 23.3k
2
votes
policy gradient sampling stationary state distribution
If you just do a "roll out" (technical terminology meaning that you just play the game / run the MDP forward) according to your policy $\pi_\theta$ for long enough, you'll be sampling states ...
- 23.3k
2
votes
policy gradient for non-differentiable policy
You could try a straight-through estimator of the gradient, $\frac{\partial \ \text{sign}(x)}{\partial x} = 1$. You could also try to train a stochastic policy $\pi(a_0) = \sigma(\frac{w^tx}{\tau})$ ...
- 23.3k
1
vote
REINFORCE: how to sample the derivative
Policy gradient methods (like REINFORCE) optimise a policy function (which I've called $P$). [Note a function in Maths is very similar to a function (aka a method) in most programming languages.] The ...
- 150
1
vote
Accepted
Why the $\gamma^t$ is needed here in REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for $\pi_{*}$?
The intuitive, short version is that policy gradient methods directly try to find a policy that maximizes the agent's rewards conditioned on the starting state $S_0$ (or in general, the starting state ...
- 83
1
vote
Derivation of expected value in REINFORCE policy gradient
I came across this very interesting website with lots of properties for conditional expected values: https://randomservices.org/random/expect/Conditional.html
One of the formulas is this:
$$r(X)\...
- 151
1
vote
Policy Gradient Methods advantages over value-based methods
I think I just figured it out. The advantage of policy gradients is that approaching a deterministic policy is dependent on the experiences as only certain experiences will push the score of some ...
- 21
1
vote
Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?
This computational principle applies in a wide range of probability problems involving density and mass functions. The reason for this advice is that probability density values can be very small ...
- 100k
1
vote
Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?
Look at the standard normal PDF and the range of values within $x=[0,9]$:
The Gaussian PDF values go from 0.4 to 1e-18, while their log varies from -0.9 to -41. Numerical algorithms are less stable ...
- 56.8k
1
vote
Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?
If you take the log of $f(x\vert \mu, \sigma^2)$, you get the expression you want. I'm not sure why the author wants to change from $f$ to $\pi_\theta$, but perhaps he explains the notation elsewhere ...
- 27.1k
1
vote
Accepted
Reinforcement learning using the gradient of expected value doesn't lead to the optimal policy
The idea of taking a gradient step and then adjusting the parameter values to respect the constraint is called "projected gradient descent", and it is theoretically sound. The only catch is that the ...
- 23.3k
1
vote
Is there any way to make the notion of a "policy" in reinforcement learning less abstract?
A real world example of policy are the house rules that Black Jack dealers adhere to. The dealer is the agent. The rules, which describe when the dealer can and cannot play cards, are the policy. The ...
- 11
1
vote
Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?
R is a constant used to scale the gradient. Instead of reward it could be returns, advantage, etc.
The gradient with respect to the parameters is found from the log probability of taking a specific ...
- 487
1
vote
Accepted
How can policy parameterization be simpler than action-value parameterization in function approximation?
Consider a game on the positive half of the number line, where you start at some integer $k$, and can move down by 1 or up by 1 each turn. The reward function is $f(x)$ for some monotonically ...
- 23.3k
1
vote
The effect of policy parameter on the action and the state distribution in policy gradient method for episodic tasks
@AliGh What the author means by "the effect of the policy on the state distribution is a function of the environment and is typically unknown" is that to compute the state distribution besides the ...
- 11
Only top scored, non community-wiki answers of a minimum length are eligible