Reinforcement Learning: definition of expected discounted return in Sutton and Barto's book

Question

I am going through Sutton and Barto's book on reinforcement learning http://incompleteideas.net/book/bookdraft2017nov5.pdf

In the book pg 44 equation 3.8, the authors define expected discounted return as

$G_t = R_{t+1} + \gamma R_{t+2} + \ldots$

1. Why is it called "expected" return? Does it have anything to do with expectation? i.e., can I rewrite $G_t$ in terms of expectation?

   Is $R_{t+1}$ a random variable? (the authors defined on page 38, top of the page near the diagram, that $R_{t+1}$ is a real number, but a random variable is a function, no?)

2. Why does the expected discounted return start with $t+1$?

In my opinion, this causes a problem later on when the authors define the optimal Q value in terms of the "Bellman optimality equation", pg 51, equation 3.20

$q^\star(s,a) = \mathbb{E}[R_{t+1} + r \max_{a^\prime} q^\star(S_{t+1}, a^\prime)| S_t = s, A_t = a]$

The optimal q value should have the intuitive definition as the reward at the CURRENT time step $t$ plus the optimal expected discounted reward. So that $R_{t+1}$ should be re-written as $R_{t}$ should it not?

Answer 1

1. Why is it called "expected" return? Does it have anything to do with expectation? i.e., can I rewrite $G_t$ in terms of expectation?

No. This is just a problem with parsing the English, where it is ambiguous.

"The expected return $G_t$" can mean two things in normal English:

• "Expected return" is represented by $G_t$

• "Return" is represented by $G_t$ and we are interested in its expected value

The sentence in Sutton and Barto and equation 3.8 are using the second meaning.

$G_t$ is a random variable, and does not itself include any expected value. There is a difference between $G_t$, which is a random variable with a distribution and $\mathbb{E}[G_t]$. In reinforcement learning, discovering the value of $\mathbb{E}[G_t]$, under some initial conditions and constraints, is often of interest. For instance the state value function $v_{\pi}(s)$ can be written $\mathbb{E}_{\pi}[G_t | S_t = s]$ i.e. the expected return given starting in state $s$ and following policy $\pi$.

Is $R_{t+1}$ a random variable?

Yes.

but a random variable is a function, no?

No. The random variable is a placeholder for general case of "the actual value can/will be sampled at some point", and is used to write formulae that would apply regardless of outcome. Or it can be viewed more formally as a (potentially unknown) distribution of values, and all equations as manipulations of those distributions.

2. Why does the expected discounted return start with $t+1$?

This is just a convention. You will find other sources that use the convention that reward $r$ from taking action $a$ in state $s$ at time $t$ is noted $r_{t}$ (random variable placeholder $R_{t}$). This changes all the related equations as you would expect, but makes no difference to any results or behaviour of algorithms.

The convention is based on the feedback loop - the reward and next state are received after taking an action, and often at the same time, so it is natural to assign them to the same time step index.

The optimal q value should have the intuitive definition as the reward at the CURRENT time step t plus the optimal expected discounted reward.

For the Q function to be useful, it has to predict consequences of an action in a specific state. Regardless of the indexing convention in use, that means ignoring rewards that have already been received (including any for arriving in a particular state), and only counting the rewards received after taking the action.