1
$\begingroup$

I was reading a paper "Real-Time User-Guided Image Colorization with Learned Deep Priors" https://arxiv.org/abs/1705.02999

Here a quote from it:

the smooth-L1 is also a robust estimator (Huber 1964), which can help avoid the averaging problem.

Why is smooth-L1 more robust than L1/L2 losses? What is the averaging problem?

$\endgroup$
2
  • 1
    $\begingroup$ The words in parenthesis are a citation supporting the claim made in the main text. What does Huber 1964 say? Do you have a specific question about what Huber 1964 says? $\endgroup$
    – Sycorax
    Aug 10, 2021 at 13:56
  • 1
    $\begingroup$ I don't see a claim in what's quoted that it's more robust than L1, only that it is also robust and '"avoids the averaging problem"', whatever that is. Where does the premise in the question arise (that it is more robust)? $\endgroup$
    – Glen_b
    Aug 10, 2021 at 15:46

1 Answer 1

3
$\begingroup$

Without reading the linked paper: Huber's loss was introduced by Huber in 1964 in the context of estimating a one-dimensional location of a distribution. In this context, the mean (average) is the estimator optimising L2-loss, and the median is the estimator optimising L1-loss.

The mean is very vulnerable to extreme outliers. This has to do with its connection to L2-loss, as in L2-loss all values are squared; meaning that all values are weighted by themselves and outlying values have the largest weight. Chances are that this is meant by the "averaging problem".

The estimator optimising L1-loss is the median. The median is very robust against outliers, but doesn't make a very efficient use of the information in the observations. If you change observations but don't move them from one side of the median to the other, the median remains the same, it will not react on such changes, which may be meaningful. ("Efficient use of the information in the observations" can be made precise but I won't do that here. Huber does it, but he may use the terminology in a different way.) Also it is not smooth at zero, which may or may not be a problem, depending on what it is used for.

Huber's loss (probably in the paper called "smooth-L1") is a compromise and uses L2-loss around zero and L1-loss further away. It is therefore not dominated by extreme outliers (this is not "more robust" than L1 but rather using L1's robustness characteristic), however still uses more of the information in the data and is smooth at zero, so it is meant to share the positive features of both L1 and L2, and remove their problems. It has a disadvantage though, which is that it depends on the scaling of the data, which L1 and L2 do not. (Whoever uses Huber's loss needs to do something about scaling. It depends on scaling which data are in the L1- and which in the L2-part of the loss.) The so-called Huber M-estimator is the location estimator optimising Huber's loss, and as such a compromise between median and mean.

L1- and L2-loss are used in many other problems, and their issues (the robustness issue of L2 and the lack of smoothness of L1, sometimes also the efficiency issue) are relevant in all kinds of setups, so people have started using Huber's loss as a compromise also far beyond the use in the original paper, sometimes with good theoretical founding (Huber himself has defined M-estimators based on it for a number of other statistical problems), sometimes rather ad hoc.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.