# Is experience replay taking individual memory samples out of context?

I understand that in reinforcement learning experience is collected in a memory buffer, which then contains state0, reward, done and state1.

When the deep q network is trained, random elements from the memory are being sampled and the network is then trained to predict rewards based on certain states.

My question is:

1. What elements does state1 play in training?
2. If individual actions are trained, wouldn't they be taken out of context? I.e. most of them are intermediary steps that may not even have any reward. How would training on them without knowing the end state (end of episode) have any benefits?
3. Are rewards adjusted for each intermediary steps before training happens?

I understand that in reinforcement learning experience is collected in a memory buffer, which then contains state0, reward, done and state1.

This list should also include the action taken in $$s_0$$, so your list would be something like $$s_0, a_0, r_1, s_1, done$$

What elements does state1 play in training?

In Q learning (and DQN, where you are most likely to come across the need for experience replay), it helps form the single-step TD Target:

$$G_{t:t+1} = r_{t+1} + \gamma \text{max}_{a'} \hat{q}(s_{t+1},a')$$

Where $$\hat{q}$$ is your neural network approximation to the action value function.

You use this TD target $$G_{t:t+1}$$, which is a (biased - but bias will reduce over time) sample of the expected return, to update the neural network action value estimate of $$\hat{q}(s_t, a_t)$$ - i.e. use it as the training data for that memory example.

If individual actions are trained, wouldn't they be taken out of context?

Yes, the single step is used as-is, ignoring the rest of the trajectory. This is a good thing for training a neural network, as feeding a NN with lots of correlated inputs - because they involve the same data just one time step on - can cause it to learn badly or not at all.

How would training on them without knowing the end state (end of episode) have any benefits?

The "outer" part of Q learning is designed so that it effectively stitches back together these single steps taken out of context. This can even be better than knowing the end result. Q learning can be shown just one successful route and several failed ones that came close, and still figure out which parts of the failed routes were more optimal to take than the successful example. The Q learning algorithm will calculate action values based on all the data it has seen, effectively filling in the whole graph of potential routes that it could reach based on the examples.

Are rewards adjusted for each intermediary steps before training happens?

No. Leave the rewards as-is. They are the raw data that Q learning consumes, and do not usually need special processing.

However, you do need to recalculate the TD target each time you use a sample from memory.

Note that if you were performing Monte Carlo control, you would take a whole trajectory at a time, and calculate the full return from each time step:

$$G_t = \sum_{k=0}^{T-t} \gamma^k r_{t+k+1}$$

(where $$T$$ is the end of the episode) and use that as your target value for updating $$\hat{q}(s_t,a_t)$$

Even with Monte Carlo though, you don't modify the reward values. Instead, you use the sum over the trajectory to calculate a return $$G_t$$.