3
$\begingroup$

I'm trying to understand the justification behind clipping in Proximal Policy Optimization (PPO).

In the paper "Proximal Policy Optimization Algorithms" (by John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford and Oleg Klimov), on page 3, equation 7 is written the following objective function

$$L^\text{CLIP}(\theta) = \mathbb{E}[\min(r_t(\theta)\hat{A}_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_t)]$$

where $r_t(\theta) = \frac{\pi_{\theta}}{\pi_{\theta_\text{old}}}$, $\pi$ refers to a policy and $\hat{A}_t$ is an advantage estimator.

The paper justifies the clipping as follows:

removes the incentive for moving $r_t$ outside of the interval $[1 - \epsilon, 1 + \epsilon]$.

It seems to me that clipping would cause gradients to be $0$. Isn't this effectively throwing the samples away?

$\endgroup$
4
  • 1
    $\begingroup$ The clipping changes the objective function and its gradient, or so it appears to me. Why do you say otherwise? $\endgroup$ Nov 21, 2017 at 22:27
  • $\begingroup$ but if the slope is zero outside the clipped interval wouldn't that effectively be doing nothing in the parameter update? $\endgroup$ Nov 22, 2017 at 1:31
  • 1
    $\begingroup$ Look at the entire objective function. It's different. The gradient is different. Therefore the algorithm can take a different path. The authors claim it is better, but that might not be obvious without seeing results. $\endgroup$ Nov 22, 2017 at 2:16
  • $\begingroup$ So moving outside of this region when evaluated to be less than the surrogate objective would cause the gradient to be zero correct? And this would cause there to be zero update to the policy's parameters? $\endgroup$ Feb 3, 2018 at 5:38

1 Answer 1

2
$\begingroup$

This doesn't set the gradient to zero. We just bound it. Its like setting the loss of an objective function we minimize to a smaller value so that the gradient updates are smaller.

Here, say that by clipping we make sure that the increase in the action probability at a state $\big(\pi(action|state)\big)$ of a "good" action is limited so that the change is not more that $\epsilon$ if the change is towards a positive direction. Hence the limiting $r(\theta)$ at $1+\epsilon$. If it is towards negative direction (meaning even though it is a good action with A>0, the probability is reducing), then we let it move.

Similarly if the ratio of a "bad" action (A<0) is falling too much, we say we don't want it to fall that much and clip it at $1-\epsilon$. We let it be any value higher than that.

Basically the update of the objective will try to increase the probability of good action hence increasing the ratio, a bad action's probability will be reduced. We are saying we don't want to change the probability of any action more than $\epsilon$

$\endgroup$
1
  • 1
    $\begingroup$ This doesn't set the gradient to zero pretty sure that when it's clipped, the gradient is zero $\endgroup$
    – Alberto
    Jun 9, 2023 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.