Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

What do you call an average that does not include outliers?

For example if you have a set:

{90,89,92,91,5} avg = 73.4

but excluding the outlier (5) we have

{90,89,92,91(,5)} avg = 90.5

How do you describe this average in statistics?

share|improve this question

migrated from stackoverflow.com Sep 29 '11 at 11:04

This question came from our site for professional and enthusiast programmers.

11 Answers 11

up vote 15 down vote accepted

Another standard test for identifying outliers is to use 1.5 times the interquartile range. This is somewhat easier than computing the standard deviation and more general since it doesn't make any assumptions about the underlying data being from a normal distribution.

share|improve this answer

It's called the trimmed mean. Basically what you do is compute the mean of the middle 80% of your data, ignoring the top and bottom 10%. Of course, these numbers can vary, but that's the general idea.

share|improve this answer
4  
Using a rule like "biggest 10%" doesn't make sense. What if there are no outliers? The 10% rule would eliminate some data anyway. Unacceptable. –  Jason Cohen Feb 2 '09 at 14:45
1  
See my answer for a statistically-significant way to decide which data qualify as an "outlier." –  Jason Cohen Feb 2 '09 at 14:46
3  
Well, there's no rigorous definition of outlier. As for your response, if there are outliers they will affect your estimate of the standard deviation. Furthermore, standard deviation can be a bad measure of dispersion for non-normally distributed data. –  dsimcha Feb 2 '09 at 14:47
    
True there's no rigorous definition, but eliminating based on percentile is certainly wrong in many common cases, including the example given in the question. –  Jason Cohen Feb 2 '09 at 14:50
    
Also, outliers will not affect standard deviation much. Unless there are many of them, in which case they aren't outliers! You might for example have a bi-modal or linearly random distribution, but then throwing out data is wrong, and indeed the notion of "average" might be wrong. –  Jason Cohen Feb 2 '09 at 14:51

The "average" you're talking about is actually called the "mean".

It's not exactly answering your question, but a different statistic which is not affected by outliers is the median, that is, the middle number.

{90,89,92,91,5} mean: 73.4
{90,89,92,91,5} median: 90

This might be useful to you, I dunno.

share|improve this answer
2  
You are all missing the point. It has nothing to do with the mean, median, mode, stdev etc. Consider this: you have {1,1,2,3,2,400} avg = 68.17 but what we want is: {1,1,2,3,2,400} avg = 1.8 //minus the [400] value What do you call that? –  Tawani Feb 2 '09 at 15:41
2  
@Tawani - they are not all missing the point. What you say needs to be defined using generic terms. You cannot go with a single example. Without general definitions, if 400 is 30 is it still an outlier? And if it is 14? And 9? Where do you stop? You need stddev's, ranges, quartiles, to do that. –  Daniel Daranas Feb 2 '09 at 17:05

A statistically sensible approach is to use a standard deviation cut-off.

For example, remove any results +/-3 standard deviations.

Using a rule like "biggest 10%" doesn't make sense. What if there are no outliers? The 10% rule would eliminate some data anyway. Unacceptable.

share|improve this answer
    
I was going to say this approach doesn't work (pathological case = 1000 numbers between -1 and +1, and then a single outlier of value +10000) because an outlier can bias the mean so that none of the results are within 3 stddev of the mean, but it looks like mathematically it does work. –  Jason S Feb 2 '09 at 15:21
3  
en.wikipedia.org/wiki/Chebychev%27s_inequality This applies regardless of the distribution. –  dsimcha Feb 2 '09 at 20:49
4  
The problem is that "outlier" isn't post-hoc conclusion about a particular realized data set. It's hard to know what people mean by outlier without knowing what the purpose of their proposed mean statistic is. –  Gregg Lind Mar 3 '09 at 20:11
2  
So your categorial statement of "unacceptable" is non-sense, and not really very helpful. The trimmed mean has some useful properties, and some less useful, like any statistic. –  Gregg Lind Mar 3 '09 at 20:12
1  
Note that contrary to comments elsewhere in this thread, such a procedure is not associated with statistical significance. –  Nick Cox Dec 3 at 16:51

For a very specific name, you'll need to specify the mechanism for outlier rejection. One general term is "robust".

dsimcha mentions one approach: trimming. Another is clipping: all values outside a known-good range are discarded.

share|improve this answer

There is no official name because of the various mechanisms, such as Q test, used to get rid of outliers.

Removing outliers is called trimming.

No program I have ever used has average() with an integrated trim()

share|improve this answer
3  
mean() in R has a trim argument stat.ethz.ch/R-manual/R-devel/library/base/html/mean.html –  Jeromy Anglim Sep 29 '11 at 11:55
    
In trimming you don't remove outliers; you just don't include them in the calculation. "Remove" might suggest that points are no longer in the dataset. And you don't remove (or ignore) them because they are outliers; the criterion is (usually) just that they are in some extreme fraction of the data. A value not included in a trimmed mean often is only slightly more (or less) than the highest (lowest) value included. –  Nick Cox Dec 3 at 16:48

I don't know if it has a name, but you could easily come up with a number of algorithms to reject outliers:

  1. Find all numbers between the 10th and 90th percentiles (do this by sorting then rejecting the first $N/10$ and last $N/10$ numbers) and take the mean value of the remaining values.

  2. Sort values, reject high and low values as long as by doing so, the mean/standard deviation change more than $X\%$.

  3. Sort values, reject high and low values as long as by doing so, the values in question are more than $K$ standard deviations from the mean.

share|improve this answer

The most common way of having a Robust (the usual word meaning resistant to bad data) average is to use the median. This is just the middle value in the sorted list (of half way between the middle two values), so for your example it would be 90.5 = half way between 90 and 91.

If you want to get really into robust statistics (such as robust estimates of standard deviation etc) I would recommend a lost of the code at The AGORAS group but this may be too advanced for your purposes.

share|improve this answer

... {90,89,92,91(,5)} avg = 90.5

How do you describe this average in statistics? ...

There's no special designation for that method. Call it any name you want, provided that you always tell the audience how you arrived at your result, and you have the outliers in hand to show them if they request (and believe me: they will request).

share|improve this answer

If all you have is one variable (as you imply) I think some of the respondents above are being over critical of your approach. Certainly other methods that look at things like leverage are more statistically sound; however that implies you are doing modeling of some sort. If you just have for example scores on a test or age of senior citizens (plausible cases of your example) I think it is practical and reasonable to be suspicious of the outlier you bring up. You could look at the overall mean and the trimmed mean and see how much it changes, but that will be a function of your sample size and the deviation from the mean for your outliers.

With egregious outliers like that, you would certainly want to look into te data generating process to figure out why that's the case. Is it a data entry or administrative fluke? If so and it is likely unrelated to actual true value (that is unobserved) it seems to me perfectly fine to trim. If it is a true value as far as you can tell you may not be able to remove unless you are explicit in your analysis about it.

share|improve this answer

It can be the median. Not always, but sometimes. I have no idea what it is called in other occasions. Hope this helped. (At least a little.)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.