1
$\begingroup$

As I've understood, PPO utilizes clipping to stabilize the training such that the updates are smooth. I'm not sure though that I understand how this clipping mechanism works.

Essentially, we look to increase the likelihood of an action, $a_t$, if the advantage function, $A_t > 0$ and we clip the value of the ratio at $1+\epsilon$.

If $A_t < 0$, then we want to move in a direction which decreases the likelihood of $a_t$ and we clip our value of the ratio, $r(t)$ at $1-\epsilon$.

They describe this as being a lower bound for policy optimization. If what I've stated above is correct then I think that this algorithm is saying basically that we prioritize focusing on moving away from "bad" actions and then updates that move us closer to "good" actions take over as the policy improves.

Because of this I don't really follow how this incentivizes the agent to have $r(t)$ between $1-\epsilon$ and $1+\epsilon$ in general since it's dependent on whether or not the action is good or not?

Thanks

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It's not the case that we want to incentivize the agent to have $r$ within $1 \pm \epsilon$. Instead, we are saying that the surrogate loss function $E[rA]$, which is first order approximation to the true function we wish to optimize, is only a reasonable approximation in some narrow region.

If we wish to optimize this surrogate loss, we need to take into account the fact that changing the policy too much will result in exiting the region in which the surrogate is a reasonable approximation.

TRPO handles this by penalizing the KL divergence between the current and the last policy. However this is difficult to do in practice. PPO offers an simpler alternative: limit the maximum change in the ratio $r$. So if the agent would've done an action with probability $0.5$ before, now it is constrained to do that same action with some probability between $0.4$ and $0.6$ -- it is limited in its changes and can't exit the region where the surrogate is good.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I'm not entirely sure I follow. In the paper it states that "The second term, clip(..) modifies the surrogate objective by clipping the probability ratio, which removes the incentive for moving $r(t)$ outside of the interval $ [1 − \epsilon, 1 + \epsilon] $". And the graph in figure 1 seems to indicate that if $A_t$ is +ve then there is a cap on the size of the update. However, if it's negative, then the ratio can be > $1+\epsilon$ which means a very large update which seems to be outside the region? Hopefully you can clear up my misunderstanding! $\endgroup$
    – tryingtolearn
    Commented Aug 6, 2018 at 15:55
  • $\begingroup$ @tryingtolearn Figure 1 depicts the combined clipped and unclipped surrogate, where we take the more pessimal of the two surrogate functions. Clearly, the optimization process won't make a very large update to increase the ratio when the advantage is negative because that would decrease the objective function. $\endgroup$
    – shimao
    Commented Aug 6, 2018 at 16:07
  • $\begingroup$ Are you looking at figure 2? $\endgroup$
    – tryingtolearn
    Commented Aug 6, 2018 at 16:12
  • $\begingroup$ @tryingtolearn no, i was talking about figure 1. $\endgroup$
    – shimao
    Commented Aug 6, 2018 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.