In DeepMind's paper titled Human-level control through deep reinforcement learning authors present Algorithm 1 (available in the paper reachable with mentioned link) for training of deep Q-learning network. In the algorithm, value of y_j, representing a target predicted value for given episode sampled from replay memory, is assigned. As far as I know y_j is supposed to be a vector of numbers representing values of actions for given state which the deep Q-network should output. However in the algorithm single scalar value is assigned into y_j. How is this possible?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Is there a way to put the crux of the matter into your post? I'm not sure people are going to want to navigate elsewhere & read a paper to answer your question for you. In short, I'm not sure if this will be answerable here. $\endgroup$– gung - Reinstate MonicaCommented May 25, 2016 at 0:45
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Note that before the line in question, it says:
Sample random minibatch of transitions $(\phi_j,a_j,r_j,\phi_{j+1})$ from D
Then it performs gradient descent on
$(y_j - Q(\phi_j,a_j;\theta))^2$
It means that you will update the weights related only to the selected action $j$ from the experience replay memory. As you noted, there is one output for each action, but you can't update all values at once since only 1 action is taken at each step. This comes from the Q-learning update formula:
$Q(s,a) = Q(s,a) + \alpha[r + \gamma max_aQ(s',a) - Q(s,a)]$
Note that it updates the value for only 1 action, which is the action taken in state $s$.
$\begingroup$
$\endgroup$
I believe $y_j$ in this context is the maximum of the the values taken over all of the possible actions. In other words, take the vector that you were thinking of and use the maximum entry in that vector.
