# Trust Region Policy Optimization - Difference between the Expected Discounted Reward and its Approximation

This question relates to this paper on Trust Region Policy Optimization.

I understand the reasoning behind the proof of Theorem 1 presented in Appendix A, but fail to see how the absolute value was removed in going from equation (45) to Theorem 1:

$$|\eta(\tilde{\pi}) - L_{\pi}(\tilde{\pi})| \le \frac{4\alpha^2 \gamma \epsilon}{(1 - \gamma)^2} \tag{45}$$

$$\eta(\tilde{\pi}) \ge L_{\pi}(\tilde{\pi}) - \frac{4\alpha^2 \gamma \epsilon}{(1 - \gamma)^2} \tag{Theorem 1}$$

This suggests that the approximation is an overestimate ($\eta(\tilde{\pi}) \le L_{\pi}(\tilde{\pi})$), but I don't see why that must be true.

If the approximation is an underestimate: $\eta(\tilde \pi) > L_\pi(\tilde \pi)$ and the RHS of (45) is non-negative, then the theorem is trivially true.
• I see. It was a simple leap of logic that I missed. Specifically, if $\eta(\tilde{\pi}) > L_{\pi}(\tilde{\pi})$, then $\eta(\tilde{\pi}) - L_{\pi}(\tilde{\pi}) > 0 \ge -\frac{4\alpha^2 \gamma \epsilon}{(1 - \gamma)^2}$. – rish987 Jul 10 '18 at 14:02