6
$\begingroup$

Suppose we make predictions for a continuous outcome $Y$ conditional on a vector of covariates $X$. If we use mean squared error loss (MSE), the population minimizer is $\mathbf{E}[Y \;|\; X]$. If instead we use mean absolute error as our loss function (MAE), the population minimizer is the conditional median.

What is the population minimizer for Huber loss? For example, if I fit a gradient boosting machine (GBM) with Huber loss, what optimal prediction am I attempting to learn?


See The Elements of Statistical Learning (Second Edition), 2.4 Statistical Decision Theory for the population minimizers under MSE and MAE, and section 10.6 Loss Functions and Robustness for a definition of Huber loss:

$\begin{equation*}L(y, \,f(x)) = \begin{cases}(y - f(x))^2 \text{ if } \lvert y - f(x) \rvert \leq \delta \\ 2\delta \, \lvert y - f(x) \rvert - \delta^2 \text{ otherwise}\end{cases}\end{equation*}$

$\endgroup$
3
  • 2
    $\begingroup$ It doesn't have closed form -- the estimates are obtained iteratively; however it's sort of analogous to a trimmed mean, at least in a particular sense. $\endgroup$
    – Glen_b
    Commented Aug 17, 2017 at 0:56
  • $\begingroup$ Do you have a reference you could point me to? I'd happily accept your comment plus a reference as an answer. $\endgroup$
    – Adrian
    Commented Aug 17, 2017 at 1:40
  • 1
    $\begingroup$ Not really any reference that would state it explicitly (that I can think of). More generally, outside of the classic works by Huber and Hampel et al and some of the papers around that era, which should give you some of the tools you need to see why I said that (like looking at influence functions), I can't think of a reference off the top of my head. I could perhaps write something up about empirical influence functions later. $\endgroup$
    – Glen_b
    Commented Aug 17, 2017 at 1:52

2 Answers 2

4
$\begingroup$

The sample Huber estimator doesn't have closed form -- the estimates are obtained iteratively. However it's sort of analogous to a trimmed mean, at least in a particular sense.

If it were applied to the whole population it's essentially going to come down to a mean of values between two quantiles ($x_{\alpha_1}=F_X^{-1}(\alpha_1)$ and $x_{1-\alpha_2}$) but which quantiles those are depends on the particulars of the distribution and the specifics of the Huber spread estimation and value of $k$. (In the general case those quantiles won't be symmetric -- i.e. typically $\alpha_1\neq \alpha_2$).

Specifically, consider the influence function. In this case we'll look at something closely related to (and similar in appearance to) the empirical influence function, which in this example is simply the estimate itself but where the sample is taken to be a set of expected normal order statistics (or rather, approximations to them, though it hardly matters), plus an additional observation that's allowed to vary across the real line. I'll call this the influence function, but strictly speaking it would need to be adjusted (by things that won't change the general appearance). The population influence functions will be similar in general appearance.

One advantage of using the empirical function on a sample made to be about as much like we'd expect the population to look (expected quantiles) is that we can give the flavor of what's going on with something that will behave very like the population equivalent while avoiding introducing Gateaux derivatives. For details on those see the references.

With them we see how the two estimates respond to a changing observation:

Plot of empirical influence function for Trimmed mean and Huber, for particular choices of trimming fraction and Huber parameter k; both are continuous, linear in the middle but flat beyond a lower and upper threshold

These are quantitatively very similar - indicating that the two respond very similarly to a small proportion of outliers (below $\alpha$ for the trimmed mean) for a fairly symmetric sample. However, there are differences in how they respond -- if you get more extreme outliers, the Huber will effectively "trim" a greater percentage, e.g. for a very skewed sample it would act more like an asymmetric trim for example, in effect "trimming" little or nothing from the "light-tailed" side but "trimming" heavily on the heavy-tailed side and if both tails got very heavy it would act as if it was trimming more.

Here's some R code, because someone will want it:

library(MASS)

opar = par()
par(mfrow=c(1,2),cex.main=1)

xn=seq(-4,4,l=201)
x=qnorm(ppoints(19,a=3/8))
f=function(xc,x) mean(c(x,xc),trim=0.05)
f2=function(xc,x) huber(c(x,xc),k=1.95)$mu
infl=sapply(xn,f,x=x)
plot(infl~xn,type="l",xlim=c(-4,4),col="blue3",main="Trimmed mean empirical IF")
inflh=sapply(xn,f2,x=x)
plot(inflh~xn,type="l",xlim=c(-4,4),col="blue3",main="Huber empirical IF")

par(opar)

As for references, the classic ones are the books by Huber [1] and Hampel et al. [2]. There's a little on M-estimation in the first 4 pages here. The wikipedia page is a bit sparse but may help.

A caveat: a number of references claim that the influence for trimmed means redescend. As we see by actually doing it, this is not so (and it's easy to see why -- trimmed means don't completely ignore observations that are trimmed, since we count how many are each side and that continues to pull the resulting estimator no matter how far away the observation may get).

[1] Huber, Peter J. (1981), Robust statistics, New York: John Wiley & Sons, Inc., ISBN 0-471-41805-6, MR 606374.
(Republished in paperback, 2004. 2nd ed., Wiley)

[2] Hampel, Frank R.; Ronchetti, Elvezio M.; Rousseeuw, Peter J.; Stahel, Werner A. (1986), Robust statistics, Wiley

$\endgroup$
2
$\begingroup$

Brilliant answer by Glen_b, the empirical influence curve (aka sensitivity curve, or SC) is much easier for non-specialists to understand than the true influence curve (IC) based on Gateaux derivatives, as Glen says. May I recommend a more modern book than the two classics Glen cited (both of which I own and have read over-and-over, but now reach for the following instead): "Robust Statistics, Theory and Methods", by Maronna, Martin, & Yohai (Wiley, 2006). There is a proof in the book that the IC is indeed the correct asymptotic form of the SC. You can trust this book -- the authors are very well-respected, accomplished experts in this field. If instead you really want all the rigorous mathematical details, read: http://j-kahn.com/files/influencefunctions.pdf . I know the guidelines say "don't respond to other answers", but I really had to tip-the-hat to Glen_b.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.