# How exactly to compute Deep Q-Learning Loss Function?

I have a doubt about how exactly the loss function of a Deep Q-Learning Network is trained. I am using a 2 layer feedforward network with linear output layer and relu hidden layers.

1. Let's suppose I have 4 possible actions. Thus, the output of my network for the current state $s_t$ is $Q(s_t) \in \mathbb{R}^4$. To make it more concrete let's assume $Q(s_t) = [1.3, 0.4, 4.3, 1.5]$
2. Now I take the action $a_t = 2$ corresponding to the value $4.3$ i.e the 3rd action, and reach a new state $s_{t+1}$.
3. Next, I compute the forward pass with state $s_{t+1}$ and lets say I obtain the following values at the output layer $Q(s_{t+1}) = [9.1, 2.4, 0.1, 0.3]$. Also let's say the reward $r_t = 2$, and $\gamma = 1.0$.
4. Is the loss given by:

$\mathcal{L} = (11.1- 4.3)^2$

OR

$\mathcal{L} = \frac{1}{4}\sum_{i=0}^3 ([11.1, 11.1, 11.1, 11.1] - [1.3, 0.4, 4.3, 1.5])^2$

OR

$\mathcal{L} = \frac{1}{4}\sum_{i=0}^3 ([11.1, 4.4, 2.1, 2.3] - [1.3, 0.4, 4.3, 1.5])^2$

Thank you, sorry I had to write this out in a very basic way... I am confused by all the notation. ( I think the correct answer is the second one...)

• This question with the clear example made me understand deep q learning more than any other medium article I've read in the past week. – dhruvm Jul 5 '18 at 20:42

$$\mathcal{L} = (11.1 - 4.3)^2$$
My reasoning is that the q-learning update rule for the general case is only updating the q-value for a specific $state,action$ pair.
$$Q(s,a) = r + \gamma \max_{a*}Q(s',a*)$$
This equation means that the update happens only for one specific $state,action$ pair and for the neural q-network that means the loss is calculated only for one specific output unit which corresponds to a specific $action$.
In the example provided $Q(s,a) = 4.3$ and the $target$ is $r + \gamma \max_{a*}Q(s',a*) = 11.1$.