# Why do we clip the surrogate objective in PPO?

I'm trying to understand the justification behind clipping in Proximal Policy Optimization (PPO).

In the paper "Proximal Policy Optimization Algorithms" (by John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford and Oleg Klimov), on page 3, equation 7 is written the following objective function

$$L^\text{CLIP}(\theta) = \mathbb{E}[\min(r_t(\theta)\hat{A}_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_t)]$$

where $$r_t(\theta) = \frac{\pi_{\theta}}{\pi_{\theta_\text{old}}}$$, $$\pi$$ refers to a policy and $$\hat{A}_t$$ is an advantage estimator.

The paper justifies the clipping as follows:

removes the incentive for moving $$r_t$$ outside of the interval $$[1 - \epsilon, 1 + \epsilon]$$.

It seems to me that clipping would cause gradients to be $$0$$. Isn't this effectively throwing the samples away?

• The clipping changes the objective function and its gradient, or so it appears to me. Why do you say otherwise? Commented Nov 21, 2017 at 22:27
• but if the slope is zero outside the clipped interval wouldn't that effectively be doing nothing in the parameter update? Commented Nov 22, 2017 at 1:31
• Look at the entire objective function. It's different. The gradient is different. Therefore the algorithm can take a different path. The authors claim it is better, but that might not be obvious without seeing results. Commented Nov 22, 2017 at 2:16
• So moving outside of this region when evaluated to be less than the surrogate objective would cause the gradient to be zero correct? And this would cause there to be zero update to the policy's parameters? Commented Feb 3, 2018 at 5:38

Here, say that by clipping we make sure that the increase in the action probability at a state $$\big(\pi(action|state)\big)$$ of a "good" action is limited so that the change is not more that $$\epsilon$$ if the change is towards a positive direction. Hence the limiting $$r(\theta)$$ at $$1+\epsilon$$. If it is towards negative direction (meaning even though it is a good action with A>0, the probability is reducing), then we let it move.
Similarly if the ratio of a "bad" action (A<0) is falling too much, we say we don't want it to fall that much and clip it at $$1-\epsilon$$. We let it be any value higher than that.
Basically the update of the objective will try to increase the probability of good action hence increasing the ratio, a bad action's probability will be reduced. We are saying we don't want to change the probability of any action more than $$\epsilon$$