Is TRPO just a "safe" version of off-policy policy iteration

Question

So, the constrained TRPO objective is the following: $$ J(\theta) = E_t\left[ \frac{\pi_\theta(a_t|s_t)}{\pi_{old}(a_t|s_t)}\cdot A_t \right]\\ st. D_{KL}[\pi_{old}(\cdot|s_t)||\pi(\cdot|s_t)] \le \epsilon $$ The off policy actor critic objective instead is: $$ \nabla J(\theta) = E_t\left[ \frac{\pi_\theta(a_t|s_t)}{b(a_t|s_t)} \nabla \log \pi_\theta(a_t|s_t)\cdot A_t \right] $$ Which is nothing else than the on-policy with the importance sampling correction. However, that $\log$ appears in the derivations considering it's derivative being $f'/f$, thus we can bring it back to: $$ \begin{align} J(\theta) & = E_t\left[ \frac{\pi_\theta(a_t|s_t)}{b(a_t|s_t)} \frac{\nabla \pi_\theta(a_t|s_t)}{\pi_\theta(a_t|s_t)}\cdot A_t \right]\\ & = E_t\left[ \frac{\nabla \pi_\theta(a_t|s_t)}{b(a_t|s_t)}\cdot A_t \right] \end{align} $$ Now, if we take the gradient for the TRPO update, we also have: $$ \nabla J(\theta) = E_t\left[ \frac{\nabla\pi_\theta(a_t|s_t)}{\pi_{old}(a_t|s_t)}\cdot A_t \right]\\ st. D_{KL}[\pi_{old}(\cdot|s_t)||\pi(\cdot|s_t)] \le \epsilon $$

So my question is then: why is TRPO considered on-policy, if the update is literally the off policy update, just with an additional constraint?
I get that the KL is there not to deviate too much, so in a off-policy setting this might lead to learn the same/very close policy as the behavioral one, however I don't see why we could not use TRPO also in the fully off policy setting (I'm referring to it's unconstrained version cited in the PPO paper, introducing the KL as a penalty, which makes the optimization easily implementable in the off policy setting)

Answer 1

Although you've rightly observed structural similarity between the problem formulations of the constrained TRPO and off-policy actor critic (OFF-PAC) in terms of importance sampling, their implementation algos are completely different.

OFF-PAC uses a same behavior policy $b$ to generate action and observes its resultant experiences during each incremental iterative step to form your above advantage function $A_t$, and more importantly update the parameter of the same single target policy $\pi$ thus it's off-policy learning. Implementation details can be found in Degris et al's (2012) paper "Off-Policy Actor-Critic".

On the other hand, TRPO as referenced in Schulman's original 2015 paper uses Single Path or Vine method to sample average as stochastic approximation of its maximization target $\mathbb{E}_{s \sim \rho_{\theta_{old}}, a \sim {\pi_{\theta_{old}}}}[\frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)}\cdot A_{\theta_{old}}]$ and further replace above advantage function simply by Q value estimates under current (old) policy $\pi_{\theta_{old}}$ during each current policy evaluation and improvement step until final convergence to arrive at an optimal target policy hopefully. Since TRPO learns from the experiences generated by the current policy and also improves the same current policy at every iterative step therefore it's on-policy similar to the general policy iteration method of classic dynamic programming and SARSA.