1
$\begingroup$

I think I have a reasonable understanding of how neural networks and Q-learning work as separate concepts. But when combining the two to form deep Q-learning I struggle with understanding what exactly is used as a target when training the NN.

As I understand it the neural network takes a state/action pair as input and produces an estimation of the Q-value. So far so good. But in order to improve the estimation the network needs to compare its estimation with the ‘true’ value and then backpropagate on the error between them. But where does the true Q-value come from?!

I have read several blogs/papers about deep RL and they all swoosh past the details of the backpropagation as if the concept is obvious to everyone who’s reading which is greatly annoying. So if someone could explain that specific part to me I would be grateful.

$\endgroup$
1
$\begingroup$

If I'm correct, you are struggling with the concept of the 'true' Q-value? I'm not an expert on back propagation, but I might be able to shine some light on the whereabouts of this 'oracle' Q-value!

The Q(s,a) describes the total expected reward starting in state s, performing action a, receiving reward r and subsequently following the current policy until termination.

In TD learning, one calculates Q(s,a), and subsequently (using the same approximator, like the same NN), the value of Q(s',a'), in which (s',a') are one time-step later than (s,a). Using a perfect Q-function, the difference between Q(s,a) and Q(s',a') should be exactly the direct reward received in going from s to s' by action a: $Q(s,a) = Q(s',a') +r$. Therefore, $\delta$ is computed as $\delta = Q(s',a')+r-Q(s,a)$, neglecting discount rate $\gamma$.

The trick here is that the 'real' $Q$ was never known, but the real reward $r$ is. Using $\delta$ as a measure for the error, the weights of your approximator can be adjusted, usually using a gradient descent type approach.

Did this help you at all?

$\endgroup$
2
  • $\begingroup$ Just came across this blog post, which addresses your issue directly: danieltakeshi.github.io/2016/12/01/… $\endgroup$ Jul 3 '17 at 11:08
  • $\begingroup$ Thank you very much. With this together with some other reading I think I’m slowly starting to get how deep RL works now! $\endgroup$
    – Petahanks
    Jul 4 '17 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.