4
$\begingroup$

In PPO the authors say this of the objective function:

a novel objective function that enables multiple epochs of minibatch updates

But what is it about PPO's objective that allows this?

The PPO objective is as follows:

ratio $r = \frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)}$

loss $L = \min(r * advantage, clipped(r, 1 - epsilon, 1 + epsilon) * advantage)$

After the first minibatch update is applied we are using rollouts collected from a different policy to optimize on and this seems like it could cause problems (deadly triad).

$\endgroup$

1 Answer 1

5
$\begingroup$

The raw objective which is being optimized in both TRPO and PPO (meaning, the objective less the penalty terms / clipping), is a first order approximation of the policy gradient.

If we simply optimized the policy for multiple iterations without sampling new data, then the optimization would fail, since 1. the first order approximation would break down and 2. the expectation in the objective would be over a significantly different distribution than the one we are sampling from.

However, since TRPO penalizes the KL-divergence between the old and new policy, and PPO clips the policy ratio to disincentivize going too far from the original policy, we avoid both issues 1 and 2.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.