# Reinforcement learning, question from Sutton's new book

In Sutton's new RL book, in chapter 3, there is an exercise

Exercise 3.6 Suppose you treated pole-balancing as an episodic task but also used discounting, with all rewards zero except for −1 upon failure. What then would the return be at each time? How does this return differ from that in the discounted, continuing formulation of this task?

My answer is if it is episodic with discounting, the return at each time step is $-\gamma^{K_1} -\gamma^{K_2} - \cdots -\gamma^{K_n}$ where $K_i$ is the number of total time steps until it fails at episode $i$.

And for continuous tasks with discounting, the return at each time step is $-\gamma^K$ where $K$ is the number of time steps until it fails.

I am not sure whether my answer is right or not. Could you please help me?

• If you complete a chapter's exercises that you can email your attempts to the author, and he will provide some sample answers - see incompleteideas.net/book/solutions-2nd.html - I have done this, and he really does send answers (it may take a few weeks). – Neil Slater Aug 9 '18 at 8:29

## 1 Answer

I don't think so, rewriting book's eq. (3.7) with discounts, we would get

$$G_t = R_{t+1} + \gamma \cdot R_{t+2} + \gamma^2 \cdot R_{t+3} + ... + \gamma^{T-1} \cdot R_{T}$$

where $$T$$ is the terminal state and $$\gamma$$ is the discount rate.

Given that all rewards zero ($$R_{t+1}, R_{t+2}, R_{t+3}$$) except for −1 upon failure ($$R_T$$), you can extract the expected return at each time, which is different than what you propose.