This is my first question on SE in general. So if I make any mistakes - please feel free to point them out to me.
My Question is about Deep Q-Learning. I've been working into some code examples and some tutorials. I also understand the concept of experience replay, but it brings a question to my mind.
As we progress with the Learning and the NN further converges, we pick random samples from our experience memory to train the NN on and we also further explore the environment. Which means that, theoretically, we could experience the same state twice but with different Q-Values.
This may lead to us training the Network on the "older Memory" with the lower Q-Value instead of the newer, more correct memory.
Is this usually a problem, that is worth adressing one way or the other? Am I maybe missing something here?
Many thanks in advance!