6 votes
Accepted

Variance of reparameterization trick and score function

The notation here is far more complicated than it needs to be, and I suspect this is contributing to the issue of understanding this method. To clarify the problem, I'm going to re-frame this in ...
user avatar
  • 100k
5 votes
Accepted

Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?

Why do we need a critic at all? I just can't see where the critic suddenly came from and what it solves. The critic solves the problem of high variance in the reward signal. If you run the ...
user avatar
  • 23.3k
4 votes
Accepted

Can Q-learning or SARSA be used to find an stochastic policy?

Neither Q-learning nor SARSA define strictly how the policy should be derived from action-values, so you might be able to use them to learn some approximation to stochastic policies, if you used a ...
user avatar
  • 6,284
3 votes
Accepted

Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

Well first, you're trying to take the gradient of the return, not the reward. Also unless both the environment and the policy is deterministic, you'd be taking the gradient of the expected return. Now ...
user avatar
  • 23.3k
3 votes
Accepted

Why is there no Target Value function in PPO?

A target Q-function is employed because of the "moving target" problem: the target is dependent on the Q-network, but so is the prediction, which causes problems for convergence. In actor-critic ...
user avatar
  • 23.3k
2 votes

Is it possible to use DDPG for discrete action space?

DDPG extends actor-critic methods from the discrete action-space environments they were originally developed on to continous action-space environments. With that in mind -- sure, you can use actor-...
user avatar
  • 23.3k
2 votes
Accepted

Is there any way to make the notion of a "policy" in reinforcement learning less abstract?

As Brale mentions, it's actually a function from states to a distribution over the set of actions. Because the domain and range of this function are usually discrete, The state is very often ...
user avatar
  • 23.3k
2 votes

policy gradient sampling stationary state distribution

If you just do a "roll out" (technical terminology meaning that you just play the game / run the MDP forward) according to your policy $\pi_\theta$ for long enough, you'll be sampling states ...
user avatar
  • 23.3k
2 votes

policy gradient for non-differentiable policy

You could try a straight-through estimator of the gradient, $\frac{\partial \ \text{sign}(x)}{\partial x} = 1$. You could also try to train a stochastic policy $\pi(a_0) = \sigma(\frac{w^tx}{\tau})$ ...
user avatar
  • 23.3k
1 vote

REINFORCE: how to sample the derivative

Policy gradient methods (like REINFORCE) optimise a policy function (which I've called $P$). [Note a function in Maths is very similar to a function (aka a method) in most programming languages.] The ...
user avatar
  • 150
1 vote
Accepted

Why the $\gamma^t$ is needed here in REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for $\pi_{*}$?

The intuitive, short version is that policy gradient methods directly try to find a policy that maximizes the agent's rewards conditioned on the starting state $S_0$ (or in general, the starting state ...
user avatar
1 vote

Derivation of expected value in REINFORCE policy gradient

I came across this very interesting website with lots of properties for conditional expected values: https://randomservices.org/random/expect/Conditional.html One of the formulas is this: $$r(X)\...
user avatar
1 vote

Policy Gradient Methods advantages over value-based methods

I think I just figured it out. The advantage of policy gradients is that approaching a deterministic policy is dependent on the experiences as only certain experiences will push the score of some ...
user avatar
1 vote

Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?

This computational principle applies in a wide range of probability problems involving density and mass functions. The reason for this advice is that probability density values can be very small ...
user avatar
  • 100k
1 vote

Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?

Look at the standard normal PDF and the range of values within $x=[0,9]$: The Gaussian PDF values go from 0.4 to 1e-18, while their log varies from -0.9 to -41. Numerical algorithms are less stable ...
user avatar
  • 56.8k
1 vote

Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?

If you take the log of $f(x\vert \mu, \sigma^2)$, you get the expression you want. I'm not sure why the author wants to change from $f$ to $\pi_\theta$, but perhaps he explains the notation elsewhere ...
user avatar
1 vote
Accepted

Reinforcement learning using the gradient of expected value doesn't lead to the optimal policy

The idea of taking a gradient step and then adjusting the parameter values to respect the constraint is called "projected gradient descent", and it is theoretically sound. The only catch is that the ...
user avatar
  • 23.3k
1 vote

Is there any way to make the notion of a "policy" in reinforcement learning less abstract?

A real world example of policy are the house rules that Black Jack dealers adhere to. The dealer is the agent. The rules, which describe when the dealer can and cannot play cards, are the policy. The ...
user avatar
  • 11
1 vote

Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

R is a constant used to scale the gradient. Instead of reward it could be returns, advantage, etc. The gradient with respect to the parameters is found from the log probability of taking a specific ...
user avatar
1 vote
Accepted

How can policy parameterization be simpler than action-value parameterization in function approximation?

Consider a game on the positive half of the number line, where you start at some integer $k$, and can move down by 1 or up by 1 each turn. The reward function is $f(x)$ for some monotonically ...
user avatar
  • 23.3k
1 vote

The effect of policy parameter on the action and the state distribution in policy gradient method for episodic tasks

@AliGh What the author means by "the effect of the policy on the state distribution is a function of the environment and is typically unknown" is that to compute the state distribution besides the ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible