# Is the objective function in policy gradient methods exactly the expected value function?

I was reading Spinning Up in DRL. I was wondering if the objective in policy gradient algorithms, the $$J_\theta$$, is exactly the expected value function $$E_{S_0}[V^\pi(S_0)]$$. I've never seen people write the objective as V but I feel like they're the same. Could anyone confirm this?

## 1 Answer

There're different ways to specify $$J(\pi_{\theta})$$ or $$J(\theta)$$ in policy gradient method. One way just uses the fixed start-state value function without any expectation such as in Sutton & Button's RL book:

we define the performance measure as the value of the start state of the episode. We can simplify the notation without losing any meaningful generality by assuming that every episode starts in some particular (non-random) state $$s_0$$. Then, in the episodic case we define performance as $$J(\theta) = v_{\pi_{\theta}}(s_0)$$... From here on in our discussion we will assume no discounting ($$\gamma=1$$) for the episodic case, although for completeness we do include the possibility of discounting in the boxed algorithms.

Other ways like in your reference, according to its definition of on-policy value function, $$V^{\pi}(s) = E_{\tau \sim \pi}[R(\tau)|s_0 = s]$$, obviously to expand $$J(\pi_{\theta}) = E_{\tau \sim \pi_{\theta}}[R(\tau)]$$ you just need to take expectation of the value function $$V^{\pi}(S_0)$$ w.r.t. $$S_0 \sim \rho_0(s_0)$$ which is the distribution of the random initial state $$S_0$$ and just appears after your quoted section.