In the draft for Sutton's latest RL book (see https://webdocs.cs.ualberta.ca/~sutton/book/bookdraft2016sep.pdf, page 270), he derives the REINFORCE algorithm from the policy gradient theorem. The first part is the equivalence

$$\sum_{s}d_\pi(s)\sum_{a}{q_\pi(s,a)\nabla\pi_(a|s,\theta)} = \mathbf{E}[\gamma^t\sum_{a}{q_\pi(S_t,a)\nabla\pi_(a|S_t,\theta)}] $$

where $$d_\pi(s) = \sum_{k=0}^{\infty}{\gamma^kP(S_k = s | S_0, \pi)}$$

This makes intuitive sense (we just sample our trajectory, and we expect that we will average the long term trajectory), but I'm having trouble deriving this analytically. The discount factor $\gamma^t$ stops us from treating $d_\pi$ as simply a probability distribution, so we aren't just taking the expected value over the states. Can anyone point me in the right direction please?

In the draft for Sutton's latest RL book, page 270, he derives the REINFORCE algorithm from the policy gradient theorem. The first part is the equivalence

$$\sum_{s}d_\pi(s)\sum_{a}{q_\pi(s,a)\nabla\pi (a|s,\theta)} = \mathbf{E}[\gamma^t\sum_{a}{q_\pi(S_t,a)\nabla\pi (a|S_t,\theta)}] $$

where $$d_\pi(s) = \sum_{k=0}^{\infty}{\gamma^kP(S_k = s | S_0, \pi)}$$

This makes intuitive sense (we just sample our trajectory, and we expect that we will average the long term trajectory), but I'm having trouble deriving this analytically. The discount factor $\gamma^t$ stops us from treating $d_\pi$ as simply a probability distribution, so we aren't just taking the expected value over the states. Can anyone point me in the right direction please?