In GD-optimisation, if the gradient of the error function is w.r.t to the weights, isn't the target value dropped since it's a lone constant? Suppose we have the absolute difference as an error function:
$\mathit{loss}(w) = |m_x(w) - t|$
where $m_x$ is simply some model with input $x$ and weight setting $w$, and $t$ is the target value.
In gradient-descent optimisation, the initial idea is to take the gradient of the loss function, and update $w$ as below:
$w = w - \alpha\cdot\nabla \mathit{loss}(w)$
where the $\alpha$ is the learning rate. Wouldn't the gradient of the loss function in our case be:
$\nabla \mathit{loss}(w) = \nabla m_x(w)$
where $t$ is dropped because it is a constant? I feel I am missing a huge crucial point here.
 A: If we are considering the absolute difference as a norm, that is:
$loss(w) = |m_x(w) - t|$
then $\nabla loss(w)$ is far from simply being equivalent to $\nabla m_x(w)$.
By definition of the derivative for an absolute value (and using the chain rule), we  actually get:
$\nabla loss(w) = \frac{m_x(w) - t}{|m_x(w) - t|}. m_x'(w)$
This is similar Aksakal's answer but I wanted to show exactly why we get $\pm m_x'(w)$
A: No. A proper Norm will not allow it to be.
Even the simplest absolute value function as a loss will depend on $t$: $|m(w)-t|’=\pm m’(w)$, here the sign depends on $t$.
TL;DR;
Generally, your loss function will be $L(w|t,X)$, so the first derivative is $\partial L(w|t,X)/\partial w$, and there's no reason for $t$ to disappear from the expression unless you construct $L$ for this purpose only, for instance you make $L$ strictly linear on $w$. However, $L$ can't be just any function in a problem that you imply, i.e. where you have a target to hit.
Clearly, loss can't be negative, because the best you could do in this kind of a problem is to hit a target then there's no loss, i.e. $L(w^*)=0$. This means that no matter what loss function you chose, it has to be nonlinear around the optimal $w^*$. The example of the absolute value norm above shows you that even a loss function that is totally linear on $w$ everywhere but in one point will still depend on $t$.
