2
$\begingroup$

The Huber Loss is: $$ huber = \begin{cases} \frac{1}{2} t^2 & \quad\text{if}\quad |t|\le \beta \\ \beta |t| &\quad\text{else} \end{cases} $$ The pseudo huber is: $$ pseudo = \delta^2\left(\sqrt{1+\left(\frac{t}{\delta}\right)^2}-1\right)$$

What are the pros and cons of using pseudo huber over huber? I don't really see much research using pseudo huber, so I wonder why?

For me, pseudo huber loss allows you to control the smoothness and therefore you can specifically decide how much you penalise outliers by, whereas huber loss is either MSE or MAE. Also, the huber loss does not have a continuous second derivative.

So, what exactly are the cons of pseudo if any?

$\endgroup$
1
$\begingroup$

Advantages of the Huber loss:

  1. You don't have to choose a $\delta$. (Of course you may like the freedom to "control" that comes with such a choice, but some would like to avoid choices without having some clear information and guidance how to make it.)

  2. The M-estimator with Huber loss function has been proved to have a number of optimality features. It is the estimator of the mean with minimax asymptotic variance in a symmetric contamination neighbourhood of the normal distribution (as shown by Huber in his famous 1964 paper), and it is the estimator of the mean with minimum asymptotic variance and a given bound on the influence function, assuming a normal distribution, see Frank R. Hampel, Elvezio M. Ronchetti, Peter J. Rousseeuw and Werner A. Stahel, Robust Statistics. The Approach Based on Influence Functions. Hampel has written somewhere that Huber's M-estimator (based on Huber's loss) is optimal in four respects, but I've forgotten the other two. Note that these properties also hold for other distributions than the normal for a general Huber-estimator with a loss function based on the likelihood of the distribution of interest, of which what you wrote down is the special case applying to the normal distribution.

  3. I'm not saying that the Huber loss is generally better; one may want to have smoothness and be able to tune it, however this means that one deviates from optimality in the sense above.

$\endgroup$
2
  • $\begingroup$ Thanks, although i would say that 1 and 3 are not really advantages, i.e. we can make $\delta$ so it is the same curvature as MSE. And for point 2, is this applicable for loss functions in neural networks? I'm not sure $\endgroup$
    – CCZ23
    Feb 14 at 16:59
  • 1
    $\begingroup$ I'm not telling you what to do, I'm just telling you why some prefer the Huber loss function. If you don't find these reasons convincing, that's fine by me. Indeed you're right suspecting that 2 actually has nothing to do with neural networks and may therefore for this use not be relevant. I'm not sure whether any optimality theory exists there, but I suspect that the community has nicked the original Huber loss from robustness theory and people thought it will be good because Huber showed that it's optimal in some - not necessarily relevant - sense. $\endgroup$ Feb 14 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.