0
$\begingroup$

I was looking through this lecture https://davidrosenberg.github.io/ml2015/docs/3a.loss-functions.pdf

Slide 3: Absolute or Laplace or L1 loss not differentiable

What does it mean L1 loss not differentiable? I understand that derivative not exist at x=0, but what practical problems can arise from this fact?

What does it mean gives median regression?

Update:

Looks like answer to the second question:

https://stats.stackexchange.com/a/363369/16843

Update 2:

Here is related question: https://stackoverflow.com/questions/41518869/how-does-tensorflow-handle-the-differentials-for-l1-regularization It doesn't describe how tensorflow implement it, but example shows that at x = 0.0 the gradient = 0.0

$\endgroup$
  • $\begingroup$ Your last question is answered at stats.stackexchange.com/questions/251600. The somewhat delicate analyses in the answers there, which avoid differentiating the loss, show one reason why lack of differentiability may be an issue. $\endgroup$ – whuber Oct 2 at 18:40
4
$\begingroup$

$L_1$ loss uses the absolute value of the difference between the predicted and the actual value to measure the loss (or the error) made by the model. The absolute value (or the modulus function), i.e. $f(x) = |x|$ is not differentiable is the way of saying that its derivative is not defined for its whole domain. For modulus function the derivative at $x=0$ is undefined, i.e. we have:

$$ \frac{d|x|}{dx} = \begin{cases} -1, & x < 0 \\ 1, & x > 0 \end{cases} $$

$\endgroup$
  • 2
    $\begingroup$ Explicitly, the derivative is undefined at $x=0$. $\endgroup$ – Dave Oct 2 at 18:38
  • $\begingroup$ I understand that derivative not exist at x=0, but what practical problems can arise from this fact? $\endgroup$ – mrgloom Oct 2 at 21:16
  • $\begingroup$ Apart from making the theory based on differentiable loss functions inapplicable, getting the wrong answers, or having your calculations fail, you mean? $\endgroup$ – whuber Oct 2 at 22:01
  • $\begingroup$ Having Fact 1: L1 loss used in practice in regression, Fact 2: L1 loss not differentiable at x=0 what conclusions can we make? Option 1: L1 loss not differentiable at x=0 is not a problem Option 2: In practice people somehow overcome this problem while minimizing L1 loss, i.e. adding epsilon to x, when x is 0? $\endgroup$ – mrgloom Oct 3 at 9:30
  • 1
    $\begingroup$ @mrgloom, researchers back-door this problem by adding a small value to x when it is zero, I have seen such solutions working in practice. $\endgroup$ – Alexey says Reinstate Monica Oct 4 at 11:39
2
$\begingroup$

I understand that derivative not exist at x=0, but what practical problems can arise from this fact?

$$ L = |x*a - y|; $$

$$ \frac{\partial L}{\partial a} = \dfrac{x\left(xa-y\right)}{\left|xa-y\right|} $$

When faced with loss equals zero for any sample you train your model with, the gradient calculator will need to divide expression by zero, which will cause error.

This is usually mitigated by, for example, adding a small value to denominator when $L$ is zero.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.