3
$\begingroup$

I just implemented the PPO algorithm in tensorflow and strictly followed the algorithm provided in the original PPO paper by Schulman et. al. 2017

Previously I did some experiments with the DDPG algorithm by Lillicrap et. al. 2016, in which they employ a target q-function in order to stabilize the training. However in the PPO paper they do not seem to use a target v-function.

Why no target value function is needed in the PPO algorithm? And would there by any benefits in using a target v-function with soft updates in the PPO algorithm?

$\endgroup$
1

1 Answer 1

3
$\begingroup$

A target Q-function is employed because of the "moving target" problem: the target is dependent on the Q-network, but so is the prediction, which causes problems for convergence.

In actor-critic methods, the target is dependent on the actor, and the prediction is output by the critic, so there is no longer any moving target problem, and no reason to have a separate target network.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.