7
$\begingroup$

Suppose we have the absolute difference as an error function:

$\mathit{loss}(w) = |m_x(w) - t|$

where $m_x$ is simply some model with input $x$ and weight setting $w$, and $t$ is the target value.

In gradient-descent optimisation, the initial idea is to take the gradient of the loss function, and update $w$ as below:

$w = w - \alpha\cdot\nabla \mathit{loss}(w)$

where the $\alpha$ is the learning rate. Wouldn't the gradient of the loss function in our case be:

$\nabla \mathit{loss}(w) = \nabla m_x(w)$

where $t$ is dropped because it is a constant? I feel I am missing a huge crucial point here.

$\endgroup$
2
  • $\begingroup$ This question doesn't really have anything to do with backpropagation. $\endgroup$ Nov 12, 2021 at 13:17
  • $\begingroup$ @leftaroundabout feel free to edit it. $\endgroup$
    – mesllo
    Nov 12, 2021 at 14:03

2 Answers 2

11
$\begingroup$

No. A proper Norm will not allow it to be.

Even the simplest absolute value function as a loss will depend on $t$: $|m(w)-t|’=\pm m’(w)$, here the sign depends on $t$.

TL;DR; Generally, your loss function will be $L(w|t,X)$, so the first derivative is $\partial L(w|t,X)/\partial w$, and there's no reason for $t$ to disappear from the expression unless you construct $L$ for this purpose only, for instance you make $L$ strictly linear on $w$. However, $L$ can't be just any function in a problem that you imply, i.e. where you have a target to hit.

Clearly, loss can't be negative, because the best you could do in this kind of a problem is to hit a target then there's no loss, i.e. $L(w^*)=0$. This means that no matter what loss function you chose, it has to be nonlinear around the optimal $w^*$. The example of the absolute value norm above shows you that even a loss function that is totally linear on $w$ everywhere but in one point will still depend on $t$.

$\endgroup$
3
  • $\begingroup$ Unless I'm mistaken, I've come to understand that the derivative of the absolute value (not any other norm), is not what it should be on the original question. It does indeed depend on $t$. $\endgroup$
    – mesllo
    Nov 11, 2021 at 1:42
  • 5
    $\begingroup$ Any norm will depend on $t$. Think logically, it's some kind of distance from a target, how could the derivative not depend on where the target is? $\endgroup$
    – Aksakal
    Nov 11, 2021 at 1:46
  • 1
    $\begingroup$ This is exactly what I asked myself. I guess I misinterpreted the derivative of an absolute value! $\endgroup$
    – mesllo
    Nov 11, 2021 at 1:52
14
$\begingroup$

If we are considering the absolute difference as a norm, that is:

$loss(w) = |m_x(w) - t|$

then $\nabla loss(w)$ is far from simply being equivalent to $\nabla m_x(w)$.

By definition of the derivative for an absolute value (and using the chain rule), we actually get:

$\nabla loss(w) = \frac{m_x(w) - t}{|m_x(w) - t|}. m_x'(w)$

This is similar Aksakal's answer but I wanted to show exactly why we get $\pm m_x'(w)$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.