# Tag Info

Accepted

### Variance of reparameterization trick and score function

The notation here is far more complicated than it needs to be, and I suspect this is contributing to the issue of understanding this method. To clarify the problem, I'm going to re-frame this in ...
• 100k
Accepted

### Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?

Why do we need a critic at all? I just can't see where the critic suddenly came from and what it solves. The critic solves the problem of high variance in the reward signal. If you run the ...
• 23.3k
Accepted

### Can Q-learning or SARSA be used to find an stochastic policy?

Neither Q-learning nor SARSA define strictly how the policy should be derived from action-values, so you might be able to use them to learn some approximation to stochastic policies, if you used a ...
• 6,284
Accepted

### Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

Well first, you're trying to take the gradient of the return, not the reward. Also unless both the environment and the policy is deterministic, you'd be taking the gradient of the expected return. Now ...
• 23.3k
Accepted

### Why is there no Target Value function in PPO?

A target Q-function is employed because of the "moving target" problem: the target is dependent on the Q-network, but so is the prediction, which causes problems for convergence. In actor-critic ...
• 23.3k

### Is it possible to use DDPG for discrete action space?

DDPG extends actor-critic methods from the discrete action-space environments they were originally developed on to continous action-space environments. With that in mind -- sure, you can use actor-...
• 23.3k
Accepted

### Is there any way to make the notion of a "policy" in reinforcement learning less abstract?

As Brale mentions, it's actually a function from states to a distribution over the set of actions. Because the domain and range of this function are usually discrete, The state is very often ...
• 23.3k

### policy gradient sampling stationary state distribution

If you just do a "roll out" (technical terminology meaning that you just play the game / run the MDP forward) according to your policy $\pi_\theta$ for long enough, you'll be sampling states ...
• 23.3k

### policy gradient for non-differentiable policy

You could try a straight-through estimator of the gradient, $\frac{\partial \ \text{sign}(x)}{\partial x} = 1$. You could also try to train a stochastic policy $\pi(a_0) = \sigma(\frac{w^tx}{\tau})$ ...
• 23.3k
1 vote

### REINFORCE: how to sample the derivative

Policy gradient methods (like REINFORCE) optimise a policy function (which I've called $P$). [Note a function in Maths is very similar to a function (aka a method) in most programming languages.] The ...
• 150
1 vote
Accepted

### Why the $\gamma^t$ is needed here in REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for $\pi_{*}$?

The intuitive, short version is that policy gradient methods directly try to find a policy that maximizes the agent's rewards conditioned on the starting state $S_0$ (or in general, the starting state ...
1 vote

### Derivation of expected value in REINFORCE policy gradient

I came across this very interesting website with lots of properties for conditional expected values: https://randomservices.org/random/expect/Conditional.html One of the formulas is this: r(X)\...
• 151
1 vote

I think I just figured it out. The advantage of policy gradients is that approaching a deterministic policy is dependent on the experiences as only certain experiences will push the score of some ...
• 21
1 vote

### Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?

This computational principle applies in a wide range of probability problems involving density and mass functions. The reason for this advice is that probability density values can be very small ...
• 100k
1 vote

### Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?

Look at the standard normal PDF and the range of values within $x=[0,9]$: The Gaussian PDF values go from 0.4 to 1e-18, while their log varies from -0.9 to -41. Numerical algorithms are less stable ...
• 56.8k
1 vote

### Why the approximation of $\log \pi_{\theta}(a|s)$ improves numerical stability?

If you take the log of $f(x\vert \mu, \sigma^2)$, you get the expression you want. I'm not sure why the author wants to change from $f$ to $\pi_\theta$, but perhaps he explains the notation elsewhere ...
• 27.1k
1 vote
Accepted

### Reinforcement learning using the gradient of expected value doesn't lead to the optimal policy

The idea of taking a gradient step and then adjusting the parameter values to respect the constraint is called "projected gradient descent", and it is theoretically sound. The only catch is that the ...
• 23.3k
1 vote

### Is there any way to make the notion of a "policy" in reinforcement learning less abstract?

A real world example of policy are the house rules that Black Jack dealers adhere to. The dealer is the agent. The rules, which describe when the dealer can and cannot play cards, are the policy. The ...
• 11
1 vote

### Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

R is a constant used to scale the gradient. Instead of reward it could be returns, advantage, etc. The gradient with respect to the parameters is found from the log probability of taking a specific ...
1 vote
Accepted

### How can policy parameterization be simpler than action-value parameterization in function approximation?

Consider a game on the positive half of the number line, where you start at some integer $k$, and can move down by 1 or up by 1 each turn. The reward function is $f(x)$ for some monotonically ...
• 23.3k
1 vote

### The effect of policy parameter on the action and the state distribution in policy gradient method for episodic tasks

@AliGh What the author means by "the effect of the policy on the state distribution is a function of the environment and is typically unknown" is that to compute the state distribution besides the ...
• 11

Only top scored, non community-wiki answers of a minimum length are eligible