In GD-optimisation, if the gradient of the error function is w.r.t to the weights, isn't the target value dropped since it's a lone constant?

Question

Suppose we have the absolute difference as an error function:

$\mathit{loss}(w) = |m_x(w) - t|$

where $m_x$ is simply some model with input $x$ and weight setting $w$, and $t$ is the target value.

In gradient-descent optimisation, the initial idea is to take the gradient of the loss function, and update $w$ as below:

$w = w - \alpha\cdot\nabla \mathit{loss}(w)$

where the $\alpha$ is the learning rate. Wouldn't the gradient of the loss function in our case be:

$\nabla \mathit{loss}(w) = \nabla m_x(w)$

where $t$ is dropped because it is a constant? I feel I am missing a huge crucial point here.

Answer 1

No. A proper Norm will not allow it to be.

Even the simplest absolute value function as a loss will depend on $t$: $|m(w)-t|’=\pm m’(w)$, here the sign depends on $t$.

TL;DR; Generally, your loss function will be $L(w|t,X)$, so the first derivative is $\partial L(w|t,X)/\partial w$, and there's no reason for $t$ to disappear from the expression unless you construct $L$ for this purpose only, for instance you make $L$ strictly linear on $w$. However, $L$ can't be just any function in a problem that you imply, i.e. where you have a target to hit.

Clearly, loss can't be negative, because the best you could do in this kind of a problem is to hit a target then there's no loss, i.e. $L(w^*)=0$. This means that no matter what loss function you chose, it has to be nonlinear around the optimal $w^*$. The example of the absolute value norm above shows you that even a loss function that is totally linear on $w$ everywhere but in one point will still depend on $t$.

Answer 2

If we are considering the absolute difference as a norm, that is:

$loss(w) = |m_x(w) - t|$

then $\nabla loss(w)$ is far from simply being equivalent to $\nabla m_x(w)$.

By definition of the derivative for an absolute value (and using the chain rule), we actually get:

$\nabla loss(w) = \frac{m_x(w) - t}{|m_x(w) - t|}. m_x'(w)$

This is similar Aksakal's answer but I wanted to show exactly why we get $\pm m_x'(w)$