What do you call an average that does not include outliers?

What do you call an average that does not include outliers?

For example if you have a set:

{90,89,92,91,5} avg = 73.4


but excluding the outlier (5) we have

{90,89,92,91(,5)} avg = 90.5


How do you describe this average in statistics?

-

migrated from stackoverflow.comSep 29 '11 at 11:04

This question came from our site for professional and enthusiast programmers.

It's called the trimmed mean. Basically what you do is compute the mean of the middle 80% of your data, ignoring the top and bottom 10%. Of course, these numbers can vary, but that's the general idea.

-
Using a rule like "biggest 10%" doesn't make sense. What if there are no outliers? The 10% rule would eliminate some data anyway. Unacceptable. – Jason Cohen Feb 2 '09 at 14:45
See my answer for a statistically-significant way to decide which data qualify as an "outlier." – Jason Cohen Feb 2 '09 at 14:46
Well, there's no rigorous definition of outlier. As for your response, if there are outliers they will affect your estimate of the standard deviation. Furthermore, standard deviation can be a bad measure of dispersion for non-normally distributed data. – dsimcha Feb 2 '09 at 14:47
The trimmed mean also enjoys the benefit of including the median as a limiting case, ie, when you trim 50% of data on both sides. – Andrew M Dec 3 '14 at 16:55
@whuber I agree; personally I wouldn't use trimming to describe what is in effect an outlier removal approach based on some other criterion, including visceral guesses. But the distinction is in the mind of the beholder: there is a difference between "for data like this, trimming 5% in each tail seems a good idea" and "I've looked at the data and the top 5% are probably best ignored", etc. The formulas don't know the analyst's attitudes, but the latter are the researcher's justification for what is done. – Nick Cox Jan 8 at 14:32

A statistically sensible approach is to use a standard deviation cut-off.

For example, remove any results +/-3 standard deviations.

Using a rule like "biggest 10%" doesn't make sense. What if there are no outliers? The 10% rule would eliminate some data anyway. Unacceptable.

-
I was going to say this approach doesn't work (pathological case = 1000 numbers between -1 and +1, and then a single outlier of value +10000) because an outlier can bias the mean so that none of the results are within 3 stddev of the mean, but it looks like mathematically it does work. – Jason S Feb 2 '09 at 15:21
en.wikipedia.org/wiki/Chebychev%27s_inequality This applies regardless of the distribution. – dsimcha Feb 2 '09 at 20:49
The problem is that "outlier" isn't post-hoc conclusion about a particular realized data set. It's hard to know what people mean by outlier without knowing what the purpose of their proposed mean statistic is. – Gregg Lind Mar 3 '09 at 20:11
So your categorial statement of "unacceptable" is non-sense, and not really very helpful. The trimmed mean has some useful properties, and some less useful, like any statistic. – Gregg Lind Mar 3 '09 at 20:12
Note that contrary to comments elsewhere in this thread, such a procedure is not associated with statistical significance. – Nick Cox Dec 3 '14 at 16:51

Another standard test for identifying outliers is to use,LQ-(1.5xIQR) and UQ+(1.5xIQR). This is somewhat easier than computing the standard deviation and more general since it doesn't make any assumptions about the underlying data being from a normal distribution.

-
But if it doesn't make any assumption, what is it then based on ? It must at least something like a definition of an outlier ? – fcop Mar 26 at 10:15

The "average" you're talking about is actually called the "mean".

It's not exactly answering your question, but a different statistic which is not affected by outliers is the median, that is, the middle number.

{90,89,92,91,5} mean: 73.4
{90,89,92,91,5} median: 90


This might be useful to you, I dunno.

-
You are all missing the point. It has nothing to do with the mean, median, mode, stdev etc. Consider this: you have {1,1,2,3,2,400} avg = 68.17 but what we want is: {1,1,2,3,2,400} avg = 1.8 //minus the [400] value What do you call that? – Tawani Feb 2 '09 at 15:41
@Tawani - they are not all missing the point. What you say needs to be defined using generic terms. You cannot go with a single example. Without general definitions, if 400 is 30 is it still an outlier? And if it is 14? And 9? Where do you stop? You need stddev's, ranges, quartiles, to do that. – Daniel Daranas Feb 2 '09 at 17:05

For a very specific name, you'll need to specify the mechanism for outlier rejection. One general term is "robust".

dsimcha mentions one approach: trimming. Another is clipping: all values outside a known-good range are discarded.

-

There is no official name because of the various mechanisms, such as Q test, used to get rid of outliers.

Removing outliers is called trimming.

No program I have ever used has average() with an integrated trim()

-
mean() in R has a trim argument stat.ethz.ch/R-manual/R-devel/library/base/html/mean.html – Jeromy Anglim Sep 29 '11 at 11:55
In trimming you don't remove outliers; you just don't include them in the calculation. "Remove" might suggest that points are no longer in the dataset. And you don't remove (or ignore) them because they are outliers; the criterion is (usually) just that they are in some extreme fraction of the data. A value not included in a trimmed mean often is only slightly more (or less) than the highest (lowest) value included. – Nick Cox Dec 3 '14 at 16:48

I don't know if it has a name, but you could easily come up with a number of algorithms to reject outliers:

1. Find all numbers between the 10th and 90th percentiles (do this by sorting then rejecting the first $N/10$ and last $N/10$ numbers) and take the mean value of the remaining values.

2. Sort values, reject high and low values as long as by doing so, the mean/standard deviation change more than $X\%$.

3. Sort values, reject high and low values as long as by doing so, the values in question are more than $K$ standard deviations from the mean.

-

... {90,89,92,91(,5)} avg = 90.5

How do you describe this average in statistics? ...

There's no special designation for that method. Call it any name you want, provided that you always tell the audience how you arrived at your result, and you have the outliers in hand to show them if they request (and believe me: they will request).

-

The most common way of having a Robust (the usual word meaning resistant to bad data) average is to use the median. This is just the middle value in the sorted list (of half way between the middle two values), so for your example it would be 90.5 = half way between 90 and 91.

If you want to get really into robust statistics (such as robust estimates of standard deviation etc) I would recommend a lost of the code at The AGORAS group but this may be too advanced for your purposes.

-

If all you have is one variable (as you imply) I think some of the respondents above are being over critical of your approach. Certainly other methods that look at things like leverage are more statistically sound; however that implies you are doing modeling of some sort. If you just have for example scores on a test or age of senior citizens (plausible cases of your example) I think it is practical and reasonable to be suspicious of the outlier you bring up. You could look at the overall mean and the trimmed mean and see how much it changes, but that will be a function of your sample size and the deviation from the mean for your outliers.

With egregious outliers like that, you would certainly want to look into te data generating process to figure out why that's the case. Is it a data entry or administrative fluke? If so and it is likely unrelated to actual true value (that is unobserved) it seems to me perfectly fine to trim. If it is a true value as far as you can tell you may not be able to remove unless you are explicit in your analysis about it.

-

My statistics textbook refers to this as a Sample Mean as opposed to a Population Mean. Sample implies there was a restriction applied to the full dataset, though no modification (removal) to the dataset was made.

-
0. Welcome to the site. 1. Which book? Please give a reference. 2. "Sample mean" does not typically refer to a mean obtained after removing outliers. – Juho Kokkala Mar 26 at 8:06

It can be the median. Not always, but sometimes. I have no idea what it is called in other occasions. Hope this helped. (At least a little.)

-