9
$\begingroup$

I have a doubt about how exactly the loss function of a Deep Q-Learning Network is trained. I am using a 2 layer feedforward network with linear output layer and relu hidden layers.

  1. Let's suppose I have 4 possible actions. Thus, the output of my network for the current state $s_t$ is $Q(s_t) \in \mathbb{R}^4$. To make it more concrete let's assume $Q(s_t) = [1.3, 0.4, 4.3, 1.5]$
  2. Now I take the action $a_t = 2$ corresponding to the value $4.3$ i.e the 3rd action, and reach a new state $s_{t+1}$.
  3. Next, I compute the forward pass with state $s_{t+1}$ and lets say I obtain the following values at the output layer $Q(s_{t+1}) = [9.1, 2.4, 0.1, 0.3]$. Also let's say the reward $r_t = 2$, and $\gamma = 1.0$.
  4. Is the loss given by:

    $\mathcal{L} = (11.1- 4.3)^2$

    OR

    $\mathcal{L} = \frac{1}{4}\sum_{i=0}^3 ([11.1, 11.1, 11.1, 11.1] - [1.3, 0.4, 4.3, 1.5])^2$

    OR

    $\mathcal{L} = \frac{1}{4}\sum_{i=0}^3 ([11.1, 4.4, 2.1, 2.3] - [1.3, 0.4, 4.3, 1.5])^2$

Thank you, sorry I had to write this out in a very basic way... I am confused by all the notation. ( I think the correct answer is the second one...)

$\endgroup$
  • 1
    $\begingroup$ This question with the clear example made me understand deep q learning more than any other medium article I've read in the past week. $\endgroup$ – dhruvm Jul 5 '18 at 20:42
4
$\begingroup$

After reviewing the equations a few more times. I think the correct loss is the following:

$$\mathcal{L} = (11.1 - 4.3)^2$$

My reasoning is that the q-learning update rule for the general case is only updating the q-value for a specific $state,action$ pair.

$$Q(s,a) = r + \gamma \max_{a*}Q(s',a*)$$

This equation means that the update happens only for one specific $state,action$ pair and for the neural q-network that means the loss is calculated only for one specific output unit which corresponds to a specific $action$.

In the example provided $Q(s,a) = 4.3$ and the $target$ is $r + \gamma \max_{a*}Q(s',a*) = 11.1$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.