# What's the pros and cons between Huber and Pseudo Huber Loss Functions?

The Huber Loss is: $$huber = \begin{cases} \frac{1}{2} t^2 & \quad\text{if}\quad |t|\le \beta \\ \beta |t| &\quad\text{else} \end{cases}$$ The pseudo huber is: $$pseudo = \delta^2\left(\sqrt{1+\left(\frac{t}{\delta}\right)^2}-1\right)$$

What are the pros and cons of using pseudo huber over huber? I don't really see much research using pseudo huber, so I wonder why?

For me, pseudo huber loss allows you to control the smoothness and therefore you can specifically decide how much you penalise outliers by, whereas huber loss is either MSE or MAE. Also, the huber loss does not have a continuous second derivative.

So, what exactly are the cons of pseudo if any?

1. You don't have to choose a $$\delta$$. (Of course you may like the freedom to "control" that comes with such a choice, but some would like to avoid choices without having some clear information and guidance how to make it.)

2. The M-estimator with Huber loss function has been proved to have a number of optimality features. It is the estimator of the mean with minimax asymptotic variance in a symmetric contamination neighbourhood of the normal distribution (as shown by Huber in his famous 1964 paper), and it is the estimator of the mean with minimum asymptotic variance and a given bound on the influence function, assuming a normal distribution, see Frank R. Hampel, Elvezio M. Ronchetti, Peter J. Rousseeuw and Werner A. Stahel, Robust Statistics. The Approach Based on Influence Functions. Hampel has written somewhere that Huber's M-estimator (based on Huber's loss) is optimal in four respects, but I've forgotten the other two. Note that these properties also hold for other distributions than the normal for a general Huber-estimator with a loss function based on the likelihood of the distribution of interest, of which what you wrote down is the special case applying to the normal distribution.

3. I'm not saying that the Huber loss is generally better; one may want to have smoothness and be able to tune it, however this means that one deviates from optimality in the sense above.

• Thanks, although i would say that 1 and 3 are not really advantages, i.e. we can make $\delta$ so it is the same curvature as MSE. And for point 2, is this applicable for loss functions in neural networks? I'm not sure Feb 14, 2021 at 16:59
• I'm not telling you what to do, I'm just telling you why some prefer the Huber loss function. If you don't find these reasons convincing, that's fine by me. Indeed you're right suspecting that 2 actually has nothing to do with neural networks and may therefore for this use not be relevant. I'm not sure whether any optimality theory exists there, but I suspect that the community has nicked the original Huber loss from robustness theory and people thought it will be good because Huber showed that it's optimal in some - not necessarily relevant - sense. Feb 14, 2021 at 17:11

Both the Huber and the Pseudo-Huber allow control over the treatment of "inliers" vs. "outliers" via one required user-specified parameter. The interpretation of the Huber's parameter is more straightforward, as it is just the max distance between the predicted value and the output value that you would still consider as an inlier (and thus use quadratic loss instead of the L1 loss for outliers). So it is quite easy to assign a decent Huber parameter value with only a cursory look at your input data. For the Pseudo-Huber, the transition between inlier and outlier regions is much more gradual and therefore the interpretation of the Pseudo-Huber parameter is less clear.

The main advantage of the Pseudo-Huber is that it is twice differentiable (i.e. "smooth"), whereas the Huber is only differentiable once. While a twice differentiable loss function is not required for vanilla SGD, other solvers and SGD variants may require the second derivative's existence to converge properly.

Below I have plotted an example of both functions and their derivatives, fixing the parameters at 1.0:

Lastly, I found the image below online which illustrates changing the Pseudo-Huber parameter, as compared to L1 and L2 loss functions: