Sensitivity of the mean to outliers

Question

Is the mean sensitive to the presence of outliers? I initially thought it wasn't, because a small amount of observations shouldn't have much impact, but was told that since those observations have very different values from the rest, they have a considerable impact. Thoughts?

Answer 1

Consider what would happen if you wanted to take the mean of some numbers, but you dragged one of them off toward infinity. Sure, at first it wouldn't have a huge impact on the mean, but the farther you drag it off, the more your mean changes.

Every number has a (proportionally) small contribution to the mean, but they do all contribute. So if one number is really different than the others, it can still have a big influence.

This idea of dragging values off toward infinity and seeing how the estimator behaves is formalized by the breakdown point: the proportion of data that can get arbitrarily large before the estimator also becomes arbitrarily large.

The mean has a breakdown point of 0, because it only takes 1 bad data point to make the whole estimator bad (this is actually the asymptotic breakdown point, the finite sample breakdown point is 1/N).

On the other hand, the median has breakdown point 0.5 because it doesn't care about how strange data gets, as long as the middle point doesn't change. You can take half of the data and make it arbitrarily large and the median shrugs it off.

You can even construct an estimator with whatever breakdown point you want (between 0 and 0.5) by 'trimming' the mean by that percentage--throwing away some of the data before computing the mean.

So, what does this mean for actually doing work? Is the mean just a terrible idea? Well, like everything else in life, it depends. If you desperately need to protect yourself against outliers, yeah, the mean probably isn't for you. But the median pays a price of losing a lot of potentially helpful information to get that high breakdown point.

If you're interested in reading more about it, here's a set of lecture notes that really helped me when I was learning about robust statistics. http://www.stat.umn.edu/geyer/5601/notes/break.pdf

Answer 2

In some sense, the mean depends equally on all the items on data — it is perfectly democratic. We can see this by considering the arithmetic mean as a special case of weighted mean,

$$\bar x = \sum_{i=1}^n \alpha_i x_i \tag{1}$$

where the weights $\alpha_i$ are all set equal at $\frac{1}{n}$.

However, when we talk about sensitivity to inputs (of a function, of a model, etc), we are generally interested in "how much of an effect would it have if our inputs were different." And on that count, outliers can be very influential in our result for the arithmetic mean.

One way to see this is to take a data set containing an outlier, and consider, for each piece of data, what would be the effect on the mean if this data point were deleted (or more counterfactually, imagine that we had never sampled and recorded it).

Take the data set $\{1,3,4,5,7,100\}$ for instance. This has a mean of $120/6 = 20$. If the $1$ were deleted, the mean rises to $119/5 = 23.8$, a change of $+3.8$. Deleting the $3$, $4$, $5$ or $7$ would have had even small effects, producing changes of $+3.4$, $+3.2$, $+3$ and $+2.6$ respectively. But deleting the $100$ would reduce the mean to $20/5=4$, a change of $-16$. In this sense, the mean is very sensitive to the inclusion of the $100$ in the data set: its value would have been very different without it. The impact of removing the outlier is noticeably larger than for any of the other data points. Here's a visual representation: the dot plot at the top shows the full distribution and its mean (the black bar); the following plots show the effect on the mean of deleting successive points.

In contrast, the median is a less sensitive measure of central tendency. The median of our entire data set is $4.5$, and deletions give the following changes:

\begin{array}{lll} &\text{Delete} &\text{New median} &\text{Change} \\ \hline &1 &5 &+0.5 \\ &3 &5 &+0.5 \\ &4 &5 &+0.5 \\ &5 &4 &-0.5 \\ &7 &4 &-0.5 \\ &100 &4 &-0.5 \\ \end{array}

Not only is the median less sensitive to changes overall, but removing the outlier had no more effect than deleting any of the other data points. It could even cope with several outliers. In fact, many outliers are okay, so long as they constitute less than half of the data set — see the the answer by @Sullysarus.

Note that the sensitivity of the mean is not necessarily a bad thing; it's often the case that we use the mean because we like its sensitivity, i.e. the way it makes full use of all the data. See the thread "If mean is so sensitive, why use it in the first place?"

---

Note that the same principles apply to outliers on the left tail, i.e. values that are "unusually small" for the data set: consider e.g. $\{-1, -3, -4, -5, -7, -100 \}$ where it is clear that the results will come out similarly to before, but with the sign (i.e. direction) of changes reversed.

Also, to head off a possible misunderstanding, looking at $\frac{1}{n} x_i$ as the "contribution" of $x_i$ to the mean may not be the most helpful approach to considering "sensitivity", even though it's true that $\bar x$ is the sum of these contributions (as written in equation $(1)$). Consider the translation of our original data set to $\{10001, 10003, 10004, 10005, 10007, 10100 \}$. Now each contribution looks fairly similar: proportionately there's not much difference between $1666\frac{5}{6}$ (the smallest, from the $10001$) and $1683\frac{1}{3}$ (the largest, from the $10100$). Each contribution is very close to one sixth of the mean, so it doesn't look like the outlier is making much more difference than the other numbers did.

But this is misleading, since the "sensitivity to deletion" argument will give the same results as before. It's the relative position of the number from the mean that matters, not the relative proportion it contributes towards the sum. (Another issue with the "relative proportion" or "contribution" approach is that it doesn't make much sense when you have a mixture of positive and negative data.)

Consider the new mean after the deletion of $x_j$: we would divide the new total, which is the previous total, $\sum_{i=1}^n x_i = n \bar x$, divided by number of items of data remaining, $n-1$. This can be manipulated to give

$$\frac{n \bar x - x_j}{n-1} = \frac{n \bar x - \bar x + \bar x - x_j}{n-1} = \frac{(n - 1) \bar x + (\bar x - x_j)}{n-1} = \bar x + \frac{\bar x - x_j}{n-1}$$

This shows that deleting a data point $x_j$ causes the mean to increase by $\frac{\bar x - x_j}{n-1}$. It should now be clear why deleting data that lies far from the mean, in either direction, should have a greater effect than removing data that lies close to the mean.

R code for plot

library(grDevices)
fulldata <- c(1,3,4,5,7,100)
n <- length(fulldata)
par(bg="grey98")
plot(NULL, xlim=c(min(fulldata)-5, max(fulldata)+5), ylim=c(0,n+2), axes=FALSE,
     ylab="", xlab="")
cols <- rainbow(n)
points(rep(fulldata,n+1), rep(1:(n+1),each=n), col="black", bg=cols, pch=21, cex=1)
points(fulldata[1:n], n:1, pch=4, cex=2)
abline(v=mean(fulldata), col="grey", lty=3)
segments(mean(fulldata),n+1-0.2,
         mean(fulldata),n+1+0.2, col="black", lwd=2)
segments((sum(fulldata)-fulldata[1:n])/(n-1),n:1-0.2,
         (sum(fulldata)-fulldata[1:n])/(n-1),n:1+0.2,col=cols, lwd=2)
segments((sum(fulldata)-fulldata[1:n])/(n-1),n:1,rep(mean(fulldata),n),n:1,col=cols,lwd=2)

Answer 3

Arithmetic mean is a sum of all the values divided by their count

$$ \frac{x_1 + x_2 + \dots + x_n}{N} $$

so each of the values have the same impact on the final estimate. Let me say it once again: each of the values has impact on the estimate. Means' "sensibility" to data is actually one of the reasons why we choose it as a estimator of location. Obviously, if one or more of the values deviate from the other, they influence the mean. If they deviate by large, their influence is larger, if deviation is smaller, than their influence is smaller. It is true that "small amount of observations shouldn't have much impact" on it's result, but that doesn't mean that they have no impact.

As correctly suggested by whuber, such robustness can be shown by measures such as breakdown point. Breakdown point is basically the proportion of observations that are needed to influence the estimate. In case of mean it is zero, because changing a single value is enough to influence the final result. More robust measures, like median, are less sensible and a greater fraction of outlying cases may be needed to influence their estimates.