Skip to main content
2 of 2
deleted 3 characters in body
shimao
  • 26.5k
  • 2
  • 62
  • 107

It's not the case that we want to incentivize the agent to have $r$ within $1 \pm \epsilon$. Instead, we are saying that the surrogate loss function $E[rA]$, which is first order approximation to the true function we wish to optimize, is only a reasonable approximation in some narrow region.

If we wish to optimize this surrogate loss, we need to take into account the fact that changing the policy too much will result in exiting the region in which the surrogate is a reasonable approximation.

TRPO handles this by penalizing the KL divergence between the current and the last policy. However this is difficult to do in practice. PPO offers an simpler alternative: limit the maximum change in the ratio $r$. So if the agent would've done an action with probability $0.5$ before, now it is constrained to do that same action with some probability between $0.4$ and $0.6$ -- it is limited in its changes and can't exit the region where the surrogate is good.

shimao
  • 26.5k
  • 2
  • 62
  • 107