What does an expectation with respect to a policy mean in the reinforcement learning value function I would like to know what the formal definition of the following expression is
$$
V_\pi(s) = \mathbb{E}_{\pi}(G_{t+1} | S_t =s)
$$
What does it mean to have the policy in the subscript? How would I write this out as a summation over states, rewards, actions? Is it equivalent to:
$$
\mathbb{E}_{A_t,R_t,S_{t+1}, A_{t+1}, R_{t+1}, S_{t+2}, \dots}(G_{t+1} | S_t =s)
$$
 A: Your understanding is heading a good direction.
The best way how to understand $V_\pi(s) = \mathbb{E}_{\pi}(G_{t+1} | S_t =s)$ is to inspect the probability distribution of the whole trajectory that starts at $S_t=s$:
$$
p(S_{T},R_{T},A_{T-1},S_{T-1},R_{T-1}\dots,S_{t+1},R_{t+1},A_t|S_t=s)
$$
using the chain rule and Markov property, we obtain:
$$
= p(S_{T},R_{T}|A_{T-1},S_{T-1})\pi(A_{T-1}|S_{T-1})p(S_{T-1},R_{T-1}|A_{T-2},S_{T-2})\pi(A_{T-2}|S_{T-2})\dots p(S_{t+1},R_{t+1}|A_{t},S_{t}=s)\pi(A_{t}|S_{t}=s)
$$
Using this probability distribution, we calculate $V_\pi(s) = \mathbb{E}_{\pi}(G_{t+1} | S_t =s)$ with $G_{t+1} = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3}\dots$ with $T\to \infty$.
By a reasonable rearranging, we can decompose the expectation as:
$$
\mathbb{E}_{\pi}[G_{t+1}|S_t=s] = \int_{\textrm{trajectory starting in } s}G_{t+1}(\textrm{trajectory starting in } s)p(\textrm{trajectory starting in  s})$$
$$
= \int_{\textrm{trajectory from } s}G_{t+1}(\textrm{trajectory starting in } s)p(\textrm{trajectory from }  s')p(s',r|s,a)\pi(a|s)$$
$$
= \int_{\textrm{trajectory from } s}[R_{t+1} +\gamma G_{t+2}(\textrm{trajectory from } s')]p(\textrm{trajectory from }  s')p(s',r|s,a)\pi(a|s)$$
$$
= \int_{\textrm{trajectory from } s}[r +\gamma G_{t+2}(\textrm{trajectory from } s')]p(\textrm{trajectory from }  s')p(s',r|s,a)\pi(a|s)$$
$$
= \int_{s',r,a}\int_{\textrm{trajectory from } s'}[r +\gamma G_{t+2}(\textrm{trajectory from } s')]p(\textrm{trajectory from }  s')p(s',r|s,a)\pi(a|s)$$
$$
= \int_{s',r,a}p(s',r|s,a)\pi(a|s)\left[r +\gamma\int_{\textrm{trajectory from } s'} G_{t+2}(\textrm{trajectory from } s')p(\textrm{trajectory from }  s')\right]$$
$$
= \int_{s',r,a}p(s',r|s,a)\pi(a|s)\left[r +\gamma \mathbb{E}_{\pi}[G_{t+2}|S_{t+1}=s'] \right]$$
