policy gradient sampling stationary state distribution

Question

Currently learning about the policy gradient theorem for reinforcement learning. The final derivation for the policy gradient simplifies to $$E_{\pi}[Q^{\pi}(s,a)\nabla_{\theta}ln\,\pi_{\theta}(a|s)]$$ where $E_{\pi}$ is equivalent to $E_{s \sim d^{\pi},a \sim \pi_{\theta}}$. Sampling $a \sim \pi$ is straightforward. My question is how do we sample from the stationary state distribution $s \sim d^{\pi}$ for the current policy $\pi_{\theta}$ ?

Answer 1

If you just do a "roll out" (technical terminology meaning that you just play the game / run the MDP forward) according to your policy $\pi_\theta$ for long enough, you'll be sampling states from $d^\pi$. There are some mild technical conditions, but basically you'll always converge to the stationary distribution after enough time.

However, usually people play games of finite episode length, and in that case there's a finite horizon policy gradients which is almost exactly the same thing but instead of sampling from $d^\pi$, people just sample over the distribution of states from rollouts of finite length. And you can sample from that distribution the same way -- just play the game a bunch of times.