Tag Info

Accepted

Variance of reparameterization trick and score function

The notation here is far more complicated than it needs to be, and I suspect this is contributing to the issue of understanding this method. To clarify the problem, I'm going to re-frame this in ...
• 100k
Accepted

Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?

Why do we need a critic at all? I just can't see where the critic suddenly came from and what it solves. The critic solves the problem of high variance in the reward signal. If you run the ...
• 23.3k
Accepted

Can Q-learning or SARSA be used to find an stochastic policy?

Neither Q-learning nor SARSA define strictly how the policy should be derived from action-values, so you might be able to use them to learn some approximation to stochastic policies, if you used a ...
• 6,284
Accepted

Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

Well first, you're trying to take the gradient of the return, not the reward. Also unless both the environment and the policy is deterministic, you'd be taking the gradient of the expected return. Now ...
• 23.3k
Accepted

Why is there no Target Value function in PPO?

A target Q-function is employed because of the "moving target" problem: the target is dependent on the Q-network, but so is the prediction, which causes problems for convergence. In actor-critic ...
• 23.3k

Is it possible to use DDPG for discrete action space?

DDPG extends actor-critic methods from the discrete action-space environments they were originally developed on to continous action-space environments. With that in mind -- sure, you can use actor-...
• 23.3k
Accepted

Is there any way to make the notion of a "policy" in reinforcement learning less abstract?

As Brale mentions, it's actually a function from states to a distribution over the set of actions. Because the domain and range of this function are usually discrete, The state is very often ...
• 23.3k

policy gradient sampling stationary state distribution

If you just do a "roll out" (technical terminology meaning that you just play the game / run the MDP forward) according to your policy $\pi_\theta$ for long enough, you'll be sampling states ...
• 23.3k

You could try a straight-through estimator of the gradient, $\frac{\partial \ \text{sign}(x)}{\partial x} = 1$. You could also try to train a stochastic policy $\pi(a_0) = \sigma(\frac{w^tx}{\tau})$ ...
• 23.3k
1 vote

REINFORCE: how to sample the derivative

Policy gradient methods (like REINFORCE) optimise a policy function (which I've called $P$). [Note a function in Maths is very similar to a function (aka a method) in most programming languages.] The ...
• 150
1 vote
Accepted

Why the $\gamma^t$ is needed here in REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for $\pi_{*}$?

The intuitive, short version is that policy gradient methods directly try to find a policy that maximizes the agent's rewards conditioned on the starting state $S_0$ (or in general, the starting state ...
1 vote

Derivation of expected value in REINFORCE policy gradient

I came across this very interesting website with lots of properties for conditional expected values: https://randomservices.org/random/expect/Conditional.html One of the formulas is this: r(X)\...
• 151
1 vote

I think I just figured it out. The advantage of policy gradients is that approaching a deterministic policy is dependent on the experiences as only certain experiences will push the score of some ...
• 21
1 vote

Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?

This computational principle applies in a wide range of probability problems involving density and mass functions. The reason for this advice is that probability density values can be very small ...
• 100k
1 vote

Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?

Look at the standard normal PDF and the range of values within $x=[0,9]$: The Gaussian PDF values go from 0.4 to 1e-18, while their log varies from -0.9 to -41. Numerical algorithms are less stable ...
• 56.8k
1 vote

Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?

If you take the log of $f(x\vert \mu, \sigma^2)$, you get the expression you want. I'm not sure why the author wants to change from $f$ to $\pi_\theta$, but perhaps he explains the notation elsewhere ...
• 27.1k
1 vote
Accepted

Reinforcement learning using the gradient of expected value doesn't lead to the optimal policy

The idea of taking a gradient step and then adjusting the parameter values to respect the constraint is called "projected gradient descent", and it is theoretically sound. The only catch is that the ...
• 23.3k
1 vote

Is there any way to make the notion of a "policy" in reinforcement learning less abstract?

A real world example of policy are the house rules that Black Jack dealers adhere to. The dealer is the agent. The rules, which describe when the dealer can and cannot play cards, are the policy. The ...
• 11
1 vote

Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

R is a constant used to scale the gradient. Instead of reward it could be returns, advantage, etc. The gradient with respect to the parameters is found from the log probability of taking a specific ...
1 vote
Accepted

How can policy parameterization be simpler than action-value parameterization in function approximation?

Consider a game on the positive half of the number line, where you start at some integer $k$, and can move down by 1 or up by 1 each turn. The reward function is $f(x)$ for some monotonically ...
• 23.3k
1 vote

The effect of policy parameter on the action and the state distribution in policy gradient method for episodic tasks

@AliGh What the author means by "the effect of the policy on the state distribution is a function of the environment and is typically unknown" is that to compute the state distribution besides the ...
• 11

Only top scored, non community-wiki answers of a minimum length are eligible