Deep Q-Learning: Experience replay overriding old Memories?

This is my first question on SE in general. So if I make any mistakes - please feel free to point them out to me.

My Question is about Deep Q-Learning. I've been working into some code examples and some tutorials. I also understand the concept of experience replay, but it brings a question to my mind.

As we progress with the Learning and the NN further converges, we pick random samples from our experience memory to train the NN on and we also further explore the environment. Which means that, theoretically, we could experience the same state twice but with different Q-Values.

This may lead to us training the Network on the "older Memory" with the lower Q-Value instead of the newer, more correct memory.

Is this usually a problem, that is worth adressing one way or the other? Am I maybe missing something here?

We know that

$$q_\pi(s,a) = E_{r|s,a}[r] + E_{s'|s,a}[ \max_{a'} q_\pi(s',a')]$$

This is where the Q-learning loss comes from:

$$\left( q_\pi(s,a) - r - \max_{a'} q_\pi(s',a') \right)^2$$

Note that we approximate the expectation by sampling.

Crucially, the expectation does not depend on the policy $\pi$! It only depends on the dynamics of the environment $P(r|s,a)$ and $P(s'|s,a)$. Therefore it is perfectly fine to sample $s,a,r,s'$ from any policy rollout, including an "old" version of a policy. This is why Q-learning is called "off-policy".

Contrast this to the on-policy method REINFORCE:

$$\nabla_\theta J = E_{\tau \sim \pi_\theta}[\nabla_\theta \log p_\theta(\tau) r(\tau)]$$ where $J$ is the expected reward of the policy and $\tau$ are "trajectories" sampled from that policy. Since the expectation draws $\tau$ from $\pi_\theta$, a replay memory could not be used in this case, since we would be incorrectly sampling $\tau$ from $\pi_{\theta_\text{old}}$.

Is this usually a problem, that is worth adressing one way or the other? Am I maybe missing something here?

No, it is not a problem, because the Q value is not stored in the replay memory.

You store $s, a, r, s'$

When you read from the memory, you re-calculate the target Q value in the update step, as a maximum over possible $a'$ using your current Q-value estimator (in DQN this is typically a snapshot of an earlier learning network).

So all the replay memory is giving you is a sample of rewards and next states in order to calculate the new estimates.

There is an issue with keeping very old replay memories: They won't necessarily have the same distribution of states and actions as a later policy generates. This is important for neural networks, because they learn to predict on statistics of input and output data. So it is a good idea to limit the amount of replay memory, although the "best" balance between keeping old transition data around and forgetting it is not 100% clear.