1
$\begingroup$

How should I understand the clipping function in the loss function?

Usually, clipping is done on the gradient directly, making the model be updated in restricted manner if the gradient is too big.

However, in PPO, the clipping is done on the probability ratio. I can hardly understand the mechanism of it. Also, I am curious if the clipped part can be differentiated to calculate the gradient.

enter image description here

$\endgroup$

2 Answers 2

1
$\begingroup$

The accepted answer of the question mentioned by user3059627 already explains PPO very well. As for the question "if the clipped part can be differentiated", the answer is NO and yes. As discussed in this comment, the gradient of the clipped samples are zero, since the default clip values $1\pm\epsilon$ do not depend on policy's parameters $\theta$. The too large/small $r_t(\theta)$ values are not used in the gradient calculation to avoid stepping too far in the local policy optimization.

$\endgroup$
0
$\begingroup$

About the understanding:

A similar question in Stackoverflow

The intuition at 24:12.

There are lots of open source projects had it implemented and worked, such as RLlib and stablebaselines. So "clipped part can be differentiated to calculate the gradient". Without code reading, guess it worked like: suppose the partial gradient function is g(r), then in calculation, the value of r would be replaced by the clipped value.

Besides, there is a paper declaring different idea about this algorithm: the coding improvement may weight more than clip trick.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.