In Sutton and Barto's book, the REINFORCE algorithm derived from the policy gradient theorem is shown to be:
$\theta_{t+1} = \theta_{t} + \alpha\gamma^{t}G_{t}\nabla_{\theta}\log\pi(A_t|S_t, \theta)$
I'm confused about the $\gamma^t$ term. My understanding is that it's there because in the previous step we're averaging over all states weighted by how frequently we come across them weighted by the discount factor up until that point. Since the frequency is implicitly obtained by sampling from our policy, all we have to do is multiply by the discount factor to keep the expectation unchanged.
Every other explanation I've ever seen of REINFORCE, however, says we simply increase the policy's log-likelihood of getting $A_t$ given $S_t$, in proportion to the reward $G_t$ — with no mention of $\gamma^t$. Every implementation I've seen neglects it too.
Do we need $\gamma^t$? How come no one else seems to use it?
