# RL - Policy Proximal Optimization and clipping

As I've understood, PPO utilizes clipping to stabilize the training such that the updates are smooth. I'm not sure though that I understand how this clipping mechanism works.

Essentially, we look to increase the likelihood of an action, $a_t$, if the advantage function, $A_t > 0$ and we clip the value of the ratio at $1+\epsilon$.

If $A_t < 0$, then we want to move in a direction which decreases the likelihood of $a_t$ and we clip our value of the ratio, $r(t)$ at $1-\epsilon$.

They describe this as being a lower bound for policy optimization. If what I've stated above is correct then I think that this algorithm is saying basically that we prioritize focusing on moving away from "bad" actions and then updates that move us closer to "good" actions take over as the policy improves.

Because of this I don't really follow how this incentivizes the agent to have $r(t)$ between $1-\epsilon$ and $1+\epsilon$ in general since it's dependent on whether or not the action is good or not?

Thanks

It's not the case that we want to incentivize the agent to have $r$ within $1 \pm \epsilon$. Instead, we are saying that the surrogate loss function $E[rA]$, which is first order approximation to the true function we wish to optimize, is only a reasonable approximation in some narrow region.
TRPO handles this by penalizing the KL divergence between the current and the last policy. However this is difficult to do in practice. PPO offers an simpler alternative: limit the maximum change in the ratio $r$. So if the agent would've done an action with probability $0.5$ before, now it is constrained to do that same action with some probability between $0.4$ and $0.6$ -- it is limited in its changes and can't exit the region where the surrogate is good.
• I'm not entirely sure I follow. In the paper it states that "The second term, clip(..) modifies the surrogate objective by clipping the probability ratio, which removes the incentive for moving $r(t)$ outside of the interval $[1 − \epsilon, 1 + \epsilon]$". And the graph in figure 1 seems to indicate that if $A_t$ is +ve then there is a cap on the size of the update. However, if it's negative, then the ratio can be > $1+\epsilon$ which means a very large update which seems to be outside the region? Hopefully you can clear up my misunderstanding! – tryingtolearn Aug 6 '18 at 15:55