24
$\begingroup$

I was hoping to understand what the smooth $l_1$ loss does, but I'm not able to find any good explanation of online, I know $l_1$ loss calculates the absolute error, but what is the use of smooth $L_1$, any answers would be helpful.

$\endgroup$
1
  • $\begingroup$ It tries to mimic the l1-loss (look at the graph), while being smooth. The smoothness property allows for treatment as smooth continuous optimization, which is in general easier than non-smooth opt. $\endgroup$
    – sascha
    Commented Jun 17, 2018 at 12:07

1 Answer 1

30
$\begingroup$

Smooth L1-loss can be interpreted as a combination of L1-loss and L2-loss. It behaves as L1-loss when the absolute value of the argument is high, and it behaves like L2-loss when the absolute value of the argument is close to zero. The equation is:

$L_{1;smooth} = \begin{cases}|x| & \text{if $|x|>\alpha$;} \\ \frac{1}{|\alpha|}x^2 & \text{if $|x| \leq \alpha$}\end{cases}$

$\alpha$ is a hyper-parameter here and is usually taken as 1. $\frac{1}{\alpha}$ appears near $x^2$ term to make it continuous.

Smooth L1-loss combines the advantages of L1-loss (steady gradients for large values of $x$) and L2-loss (less oscillations during updates when $x$ is small).

Another form of smooth L1-loss is Huber loss. They achieve the same thing. Taken from Wikipedia, Huber loss is

$ L_\delta (a) = \begin{cases} \frac{1}{2}{a^2} & \text{for } |a| \le \delta, \\ \delta (|a| - \frac{1}{2}\delta), & \text{otherwise.} \end{cases} $

$\endgroup$
4
  • $\begingroup$ Should be $|x|-0.5$ $\endgroup$
    – Tim
    Commented Sep 30, 2018 at 12:25
  • $\begingroup$ @Tim, can you please be more specific about the correction? $\endgroup$ Commented Oct 1, 2018 at 8:00
  • $\begingroup$ Try plotting it, the appropriate form is en.wikipedia.org/wiki/Huber_loss $\endgroup$
    – Tim
    Commented Oct 1, 2018 at 8:16
  • 1
    $\begingroup$ Oh yes, it needs to be continuous. I will include two varieties then $\endgroup$ Commented Oct 1, 2018 at 10:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.