In Sutton and Barto's book, the REINFORCE algorithm derived from the policy gradient theorem is shown to be:

$\theta_{t+1} = \theta_{t} + \alpha\gamma^{t}G_{t}\nabla_{\theta}\log\pi(A_t|S_t, \theta)$

I'm confused about the $\gamma^t$ term. My understanding is that it's there because in the previous step we're averaging over all states weighted by how frequently we come across them weighted by the discount factor up until that point. Since the frequency is implicitly obtained by sampling from our policy, all we have to do is multiply by the discount factor to keep the expectation unchanged.

Every other explanation I've ever seen of REINFORCE, however, says we simply increase the policy's log-likelihood of getting $A_t$ given $S_t$, in proportion to the reward $G_t$ — with no mention of $\gamma^t$. Every implementation I've seen neglects it too.

Do we need $\gamma^t$? How come no one else seems to use it?


REINFORCE assumes a finite horizon. For a finite horizon problem you don't necessarily need a discount factor. You can have one, but it can essentially be part of the reward depending on what your definitions are.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.