# Is TRPO just a "safe" version of off-policy policy iteration

So, the constrained TRPO objective is the following: $$J(\theta) = E_t\left[ \frac{\pi_\theta(a_t|s_t)}{\pi_{old}(a_t|s_t)}\cdot A_t \right]\\ st. D_{KL}[\pi_{old}(\cdot|s_t)||\pi(\cdot|s_t)] \le \epsilon$$ The off policy actor critic objective instead is: $$\nabla J(\theta) = E_t\left[ \frac{\pi_\theta(a_t|s_t)}{b(a_t|s_t)} \nabla \log \pi_\theta(a_t|s_t)\cdot A_t \right]$$ Which is nothing else than the on-policy with the importance sampling correction. However, that $$\log$$ appears in the derivations considering it's derivative being $$f'/f$$, thus we can bring it back to: \begin{align} J(\theta) & = E_t\left[ \frac{\pi_\theta(a_t|s_t)}{b(a_t|s_t)} \frac{\nabla \pi_\theta(a_t|s_t)}{\pi_\theta(a_t|s_t)}\cdot A_t \right]\\ & = E_t\left[ \frac{\nabla \pi_\theta(a_t|s_t)}{b(a_t|s_t)}\cdot A_t \right] \end{align} Now, if we take the gradient for the TRPO update, we also have: $$\nabla J(\theta) = E_t\left[ \frac{\nabla\pi_\theta(a_t|s_t)}{\pi_{old}(a_t|s_t)}\cdot A_t \right]\\ st. D_{KL}[\pi_{old}(\cdot|s_t)||\pi(\cdot|s_t)] \le \epsilon$$

So my question is then: why is TRPO considered on-policy, if the update is literally the off policy update, just with an additional constraint?
I get that the KL is there not to deviate too much, so in a off-policy setting this might lead to learn the same/very close policy as the behavioral one, however I don't see why we could not use TRPO also in the fully off policy setting (I'm referring to it's unconstrained version cited in the PPO paper, introducing the KL as a penalty, which makes the optimization easily implementable in the off policy setting)

OFF-PAC uses a same behavior policy $$b$$ to generate action and observes its resultant experiences during each incremental iterative step to form your above advantage function $$A_t$$, and more importantly update the parameter of the same single target policy $$\pi$$ thus it's off-policy learning. Implementation details can be found in Degris et al's (2012) paper "Off-Policy Actor-Critic".
On the other hand, TRPO as referenced in Schulman's original 2015 paper uses Single Path or Vine method to sample average as stochastic approximation of its maximization target $$\mathbb{E}_{s \sim \rho_{\theta_{old}}, a \sim {\pi_{\theta_{old}}}}[\frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)}\cdot A_{\theta_{old}}]$$ and further replace above advantage function simply by Q value estimates under current (old) policy $$\pi_{\theta_{old}}$$ during each current policy evaluation and improvement step until final convergence to arrive at an optimal target policy hopefully. Since TRPO learns from the experiences generated by the current policy and also improves the same current policy at every iterative step therefore it's on-policy similar to the general policy iteration method of classic dynamic programming and SARSA.
• Are you talking about the (unwritten) inner loop supposedly implementing the $argmax_{\pi}$ line in Schulman's TRPO paper's Algorithm 1? This is similar to on-policy Q-learning algo, though we have a max operator we can still make it on-policy. The only criterion for on/off policy is simple without actor parameters update details, insofar as the actor updates the current (behavior) policy not a predefined fixed target policy it's on-policy such as A2C/TRPO/PPO which is more stable while not sample efficient. Commented Mar 22, 2023 at 22:58