5
$\begingroup$

I was looking through this lecture https://davidrosenberg.github.io/ml2015/docs/3a.loss-functions.pdf

Slide 3: Absolute or Laplace or L1 loss not differentiable

What does it mean L1 loss not differentiable? I understand that derivative not exist at x=0, but what practical problems can arise from this fact?

What does it mean gives median regression?

Update:

Looks like answer to the second question:

https://stats.stackexchange.com/a/363369/16843

Update 2:

Here is related question: https://stackoverflow.com/questions/41518869/how-does-tensorflow-handle-the-differentials-for-l1-regularization It doesn't describe how tensorflow implement it, but example shows that at x = 0.0 the gradient = 0.0

Improve this question
$\endgroup$
1
  • $\begingroup$ Your last question is answered at stats.stackexchange.com/questions/251600. The somewhat delicate analyses in the answers there, which avoid differentiating the loss, show one reason why lack of differentiability may be an issue. $\endgroup$
    – whuber ♦
    Commented Oct 2, 2019 at 18:40

3 Answers 3

Reset to default
6
$\begingroup$

$L_1$ loss uses the absolute value of the difference between the predicted and the actual value to measure the loss (or the error) made by the model. The absolute value (or the modulus function), i.e. $f(x) = |x|$ is not differentiable is the way of saying that its derivative is not defined for its whole domain. For modulus function the derivative at $x=0$ is undefined, i.e. we have:

$$ \frac{d|x|}{dx} = \begin{cases} -1, & x < 0 \\ 1, & x > 0 \end{cases} $$

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ Explicitly, the derivative is undefined at $x=0$. $\endgroup$
    – Dave
    Commented Oct 2, 2019 at 18:38
  • $\begingroup$ I understand that derivative not exist at x=0, but what practical problems can arise from this fact? $\endgroup$
    – mrgloom
    Commented Oct 2, 2019 at 21:16
  • 1
    $\begingroup$ Apart from making the theory based on differentiable loss functions inapplicable, getting the wrong answers, or having your calculations fail, you mean? $\endgroup$
    – whuber ♦
    Commented Oct 2, 2019 at 22:01
  • 1
    $\begingroup$ Having Fact 1: L1 loss used in practice in regression, Fact 2: L1 loss not differentiable at x=0 what conclusions can we make? Option 1: L1 loss not differentiable at x=0 is not a problem Option 2: In practice people somehow overcome this problem while minimizing L1 loss, i.e. adding epsilon to x, when x is 0? $\endgroup$
    – mrgloom
    Commented Oct 3, 2019 at 9:30
  • 2
    $\begingroup$ @mrgloom, researchers back-door this problem by adding a small value to x when it is zero, I have seen such solutions working in practice. $\endgroup$
    – alexeymosco
    Commented Oct 4, 2019 at 11:39
6
$\begingroup$

I understand that derivative not exist at x=0, but what practical problems can arise from this fact?

$$ L = |x*a - y|; $$

$$ \frac{\partial L}{\partial a} = \dfrac{x\left(xa-y\right)}{\left|xa-y\right|} $$

When faced with loss equals zero for any sample you train your model with, the gradient calculator will need to divide expression by zero, which will cause error.

This is usually mitigated by, for example, adding a small value to denominator when $L$ is zero.

Improve this answer
$\endgroup$
1
4
$\begingroup$

+1 to both Tomasz and Alexey posts.

I would add that a good surrogate for the $L_1$ loss is the Pseudo-Huber loss function: $ L_{\delta }(x) =$ $\delta ^{2}\left({\sqrt {1+(x/\delta )^{2}}}-1\right)$ with $\delta = 1$. It allows us to approximate the $L_1$ rather faithfully away from $x=1$ and within $[-1,1]$ behaves like a quadratic. It has first (and second) derivatives everywhere.

enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.