0
$\begingroup$

I'm reading Humman-level control through deep reinforcement learning , at the beginning of Page 7, it defines a loss function (I omit some parameters to make it cleaner) $$ L(\theta)=\mathbf E\left[\left(r+\gamma \max_{a'}\hat Q(s', a';\theta^-)-Q(s, a;\theta)\right)^2\right] $$ Then I differentiate it with respect to the weights, I get $$ \nabla_\theta L(\theta)=\mathbf E\left[-2\left(r+\gamma \max_{a'}Q(s', a'; \theta^-)-Q(s, a;\theta)\right)\nabla_\theta Q(s, a;\theta)\right] $$ It is not consistent with the gradient in the paper. (my version has $-2$). What's wrong with my inference? Thanks in advance.

$\endgroup$
  • 3
    $\begingroup$ Perhaps the paper is only interested in finding the zeros of the gradient? In that case it would be unconcerned about any constant nonzero multiple. $\endgroup$ – whuber Apr 16 '18 at 22:25
  • $\begingroup$ @whuber Thanks for commenting. And sorry that I'm not so sure about what you meant, could you be more specific? Why is the paper only interested in finding the zeros of the gradient? IMHO, $\nabla_\theta L(\theta)$ is going to be used to update $\theta$, and zero seems meaningless in that case. $\endgroup$ – Maybe Apr 16 '18 at 23:42
  • 1
    $\begingroup$ @SherwinChen It's quite common in statistics that we ignore the constant for differentiation. The paper is right. whuber is correct. $\endgroup$ – SmallChess Apr 17 '18 at 2:25
  • $\begingroup$ @SmallChess Thanks, so that's just a convention? $\endgroup$ – Maybe Apr 17 '18 at 3:09
  • $\begingroup$ I changed your title to something more descriptive, feel free to edit it $\endgroup$ – Tim Apr 17 '18 at 5:37
2
$\begingroup$

As already said by others in the comments, such constants are often ignored, as they don't influence the result of minimizing the loss. So squared loss is often defined as $\tfrac{1}{2} (x-y)^2$, so that $\tfrac{1}{2} $ cancels out with $2$ and you don't have to bother with it. I'd guess that's also the case in here. The sign is only for the optimization algorithm, since some of them specialize in minimizing, while other in maximizing. You minimize the loss, or maximize negative loss (i.e. utility).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I agree with you about the $2$ part, but what confuses me is the negative sign, I think it becomes important when doing gradient ascent/descent. so why should we ignore that? $\endgroup$ – Maybe Apr 18 '18 at 5:20
  • 1
    $\begingroup$ @SherwinChen the sign is only for your optimization algorithm, since some of them specialize in minimizing things, while other in maximizing. You minimize the loss, or maximize negative loss (or utility). $\endgroup$ – Tim Apr 18 '18 at 5:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.