I think I have a reasonable understanding of how neural networks and Q-learning work as separate concepts. But when combining the two to form deep Q-learning I struggle with understanding what exactly is used as a target when training the NN.
As I understand it the neural network takes a state/action pair as input and produces an estimation of the Q-value. So far so good. But in order to improve the estimation the network needs to compare its estimation with the ‘true’ value and then backpropagate on the error between them. But where does the true Q-value come from?!
I have read several blogs/papers about deep RL and they all swoosh past the details of the backpropagation as if the concept is obvious to everyone who’s reading which is greatly annoying. So if someone could explain that specific part to me I would be grateful.