2
$\begingroup$

So, the constrained TRPO objective is the following: $$ J(\theta) = E_t\left[ \frac{\pi_\theta(a_t|s_t)}{\pi_{old}(a_t|s_t)}\cdot A_t \right]\\ st. D_{KL}[\pi_{old}(\cdot|s_t)||\pi(\cdot|s_t)] \le \epsilon $$ The off policy actor critic objective instead is: $$ \nabla J(\theta) = E_t\left[ \frac{\pi_\theta(a_t|s_t)}{b(a_t|s_t)} \nabla \log \pi_\theta(a_t|s_t)\cdot A_t \right] $$ Which is nothing else than the on-policy with the importance sampling correction. However, that $\log$ appears in the derivations considering it's derivative being $f'/f$, thus we can bring it back to: $$ \begin{align} J(\theta) & = E_t\left[ \frac{\pi_\theta(a_t|s_t)}{b(a_t|s_t)} \frac{\nabla \pi_\theta(a_t|s_t)}{\pi_\theta(a_t|s_t)}\cdot A_t \right]\\ & = E_t\left[ \frac{\nabla \pi_\theta(a_t|s_t)}{b(a_t|s_t)}\cdot A_t \right] \end{align} $$ Now, if we take the gradient for the TRPO update, we also have: $$ \nabla J(\theta) = E_t\left[ \frac{\nabla\pi_\theta(a_t|s_t)}{\pi_{old}(a_t|s_t)}\cdot A_t \right]\\ st. D_{KL}[\pi_{old}(\cdot|s_t)||\pi(\cdot|s_t)] \le \epsilon $$

So my question is then: why is TRPO considered on-policy, if the update is literally the off policy update, just with an additional constraint?
I get that the KL is there not to deviate too much, so in a off-policy setting this might lead to learn the same/very close policy as the behavioral one, however I don't see why we could not use TRPO also in the fully off policy setting (I'm referring to it's unconstrained version cited in the PPO paper, introducing the KL as a penalty, which makes the optimization easily implementable in the off policy setting)

$\endgroup$

1 Answer 1

1
$\begingroup$

Although you've rightly observed structural similarity between the problem formulations of the constrained TRPO and off-policy actor critic (OFF-PAC) in terms of importance sampling, their implementation algos are completely different.

OFF-PAC uses a same behavior policy $b$ to generate action and observes its resultant experiences during each incremental iterative step to form your above advantage function $A_t$, and more importantly update the parameter of the same single target policy $\pi$ thus it's off-policy learning. Implementation details can be found in Degris et al's (2012) paper "Off-Policy Actor-Critic".

On the other hand, TRPO as referenced in Schulman's original 2015 paper uses Single Path or Vine method to sample average as stochastic approximation of its maximization target $\mathbb{E}_{s \sim \rho_{\theta_{old}}, a \sim {\pi_{\theta_{old}}}}[\frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)}\cdot A_{\theta_{old}}]$ and further replace above advantage function simply by Q value estimates under current (old) policy $\pi_{\theta_{old}}$ during each current policy evaluation and improvement step until final convergence to arrive at an optimal target policy hopefully. Since TRPO learns from the experiences generated by the current policy and also improves the same current policy at every iterative step therefore it's on-policy similar to the general policy iteration method of classic dynamic programming and SARSA.

$\endgroup$
3
  • $\begingroup$ the TRPO algorithm has an inner loop, that is the "advantage" wrt using A2C, which is that you can do more than one update to your actor given that you dont go too far, at that point is better to start again... so to me, it seems just a pessimistic version of offpac, since offpac lets you go also very far away from the behavioral policy, instead TRPO no $\endgroup$
    – Alberto
    Mar 22, 2023 at 22:14
  • $\begingroup$ Are you talking about the (unwritten) inner loop supposedly implementing the $argmax_{\pi}$ line in Schulman's TRPO paper's Algorithm 1? This is similar to on-policy Q-learning algo, though we have a max operator we can still make it on-policy. The only criterion for on/off policy is simple without actor parameters update details, insofar as the actor updates the current (behavior) policy not a predefined fixed target policy it's on-policy such as A2C/TRPO/PPO which is more stable while not sample efficient. $\endgroup$
    – cinch
    Mar 22, 2023 at 22:58
  • 1
    $\begingroup$ no nevermind, i've confused the PPO pseudocode with the TRPO one, where the PPO psudocode has "perform some steps on the collected trajectories", instead TRPO seem to perform only one step for the actor, at least considering OpenAI doc spinningup.openai.com/en/latest/algorithms/trpo.html $\endgroup$
    – Alberto
    Mar 23, 2023 at 11:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.