While reading PG method in Prof Sutton's RL book again, I found there is $\gamma^t$ in the last row (as shown below) in pseudo code. The book said

The second difference between the pseudocode update and the REINFORCE update equation (13.8) is that the former includes a factor of $t$. This is because, as mentioned earlier, in the text we are treating the non-discounted case ($\gamma=1$) while in the boxed algorithms we are giving the algorithms for the general discounted case. All of the ideas go through in the discounted case with appropriate adjustments (including to the box on page 199) but involve additional complexity that distracts from the main ideas.

However, I still cannot figure out the exact reason. Can you help me?

enter image description here


1 Answer 1


The intuitive, short version is that policy gradient methods directly try to find a policy that maximizes the agent's rewards conditioned on the starting state $S_0$ (or in general, the starting state distribution). Thus, when you are computing the returns from a state $S_t$, you add the discounting factor $\gamma^t$ to the return from that state since it is $t$ time-steps away from $S_0$.

For a full derivation, you can refer to page 56 of the solutions manual, which also admits that the book is a bit unclear on this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.