0
$\begingroup$

In the Deepmind DQN paper, the authors mention that familiar Q Learning can be recovered by updating weights of target network at every step

enter image description here enter image description here

so if $\theta_i^-=\theta_i$ in the first loss equation, does the gradient still hold and will the update still be called stochastic gradient descent? I am very confused because don't we need to differentiate $\theta$ with max part too?

$\endgroup$

1 Answer 1

1
$\begingroup$

The first loss equation is meant to update the parameters of the Q-network $\theta_i $, in which the target network of parameters $\theta_i^-$ is used to approximate, according to the Bellman equation, the optimal Q-function $Q^*$.

So if $\theta_i^-=θ_i$ in the first loss equation, does the gradient still hold and will the update still be called stochastic gradient descent?

Yes, it still holds. Please notice that the target term deriving from the Bell Equation $r_i + \gamma \max\limits_{a'}Q(s',a',\theta_i^-) $ is computed utilising the next action and state, while $Q(s,a,\theta_i)$ uses the current action and state. This temporal difference error defines your Loss function (first equation), which gradient can be explained by the second equation.

I am very confused because don't we need to differentiate θ with max part too?

We don't. The "max part" is used only in the target term to select the action which returns the best Q-value given the next state $s'$, while the second term is used to compute the Q-value given the current state $s$ and action $a$, that are both sample from the replay buffer $\mathcal{D}$.

$\endgroup$
1
  • $\begingroup$ Thanks. Makes sense now :) $\endgroup$
    – Shimano
    Feb 12, 2020 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.