6
$\begingroup$

I was hoping to understand what the smooth $l_1$ loss does, but I'm not able to find any good explanation of online, I know $l_1$ loss calculates the absolute error, but what is the use of smooth $L_1$, any answers would be helpful.

$\endgroup$

migrated from stackoverflow.com Jun 18 '18 at 6:28

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ It tries to mimic the l1-loss (look at the graph), while being smooth. The smoothness property allows for treatment as smooth continuous optimization, which is in general easier than non-smooth opt. $\endgroup$ – sascha Jun 17 '18 at 12:07
7
$\begingroup$

Smooth L1-loss can be interpreted as a combination of L1-loss and L2-loss. It behaves as L1-loss when the absolute value of the argument is high, and it behaves like L2-loss when the absolute value of the argument is close to zero. The equation is:

$L_{1;smooth} = \begin{cases}|x| & \text{if $|x|>\alpha$;} \\ \frac{1}{|\alpha|}x^2 & \text{if $|x| \leq \alpha$}\end{cases}$

$\alpha$ is a hyper-parameter here and is usually taken as 1. $\frac{1}{\alpha}$ appears near $x^2$ term to make it continuous.

Smooth L1-loss combines the advantages of L1-loss (steady gradients for large values of $x$) and L2-loss (less oscillations during updates when $x$ is small).

Another form of smooth L1-loss is Huber loss. They achieve the same thing. Taken from Wikipedia, Huber loss is

$ L_\delta (a) = \begin{cases} \frac{1}{2}{a^2} & \text{for } |a| \le \delta, \\ \delta (|a| - \frac{1}{2}\delta), & \text{otherwise.} \end{cases} $

$\endgroup$
  • $\begingroup$ Should be $|x|-0.5$ $\endgroup$ – Tim Sep 30 '18 at 12:25
  • $\begingroup$ @Tim, can you please be more specific about the correction? $\endgroup$ – Gautam Oct 1 '18 at 8:00
  • $\begingroup$ Try plotting it, the appropriate form is en.wikipedia.org/wiki/Huber_loss $\endgroup$ – Tim Oct 1 '18 at 8:16
  • 1
    $\begingroup$ Oh yes, it needs to be continuous. I will include two varieties then $\endgroup$ – Gautam Oct 1 '18 at 10:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.