99
$\begingroup$

What do you call an average that does not include outliers?

For example if you have a set:

{90,89,92,91,5} avg = 73.4

but excluding the outlier (5) we have

{90,89,92,91(,5)} avg = 90.5

How do you describe this average in statistics?

$\endgroup$
2
  • $\begingroup$ sciencing.com/calculate-outliers-5201412.html I felt the above link surely has answered the question. $\endgroup$
    – Sam
    Commented Feb 20, 2018 at 8:20
  • 3
    $\begingroup$ This depends how the assumed outliers are defined. It could be a trimmed mean or a Winsorized mean or some other form of robust estimate of location. $\endgroup$ Commented Feb 20, 2018 at 8:47

15 Answers 15

83
$\begingroup$

It's called the trimmed mean. Basically what you do is compute the mean of the middle 80% of your data, ignoring the top and bottom 10%. Of course, these numbers can vary, but that's the general idea.

$\endgroup$
11
  • 16
    $\begingroup$ Using a rule like "biggest 10%" doesn't make sense. What if there are no outliers? The 10% rule would eliminate some data anyway. Unacceptable. $\endgroup$
    – Jason Cohen
    Commented Feb 2, 2009 at 14:45
  • 4
    $\begingroup$ See my answer for a statistically-significant way to decide which data qualify as an "outlier." $\endgroup$
    – Jason Cohen
    Commented Feb 2, 2009 at 14:46
  • 8
    $\begingroup$ Well, there's no rigorous definition of outlier. As for your response, if there are outliers they will affect your estimate of the standard deviation. Furthermore, standard deviation can be a bad measure of dispersion for non-normally distributed data. $\endgroup$
    – dsimcha
    Commented Feb 2, 2009 at 14:47
  • 2
    $\begingroup$ @Nick Thank you for the clarification. I would go further, though, and suggest that unless that one "outlier" was excluded due to considerations that (a) were independent of the observed distribution of the data and (b) a priori suggested 20% trimming of the low end, then it would be misleading to characterize the process in the question as a "trimming" procedure. It looks like outlier detection and rejection, pure and simple. Although the result may look the same, as statistical procedures the two processes of trimming and outlier removal are very different. $\endgroup$
    – whuber
    Commented Jan 8, 2016 at 14:24
  • 2
    $\begingroup$ @whuber I agree; personally I wouldn't use trimming to describe what is in effect an outlier removal approach based on some other criterion, including visceral guesses. But the distinction is in the mind of the beholder: there is a difference between "for data like this, trimming 5% in each tail seems a good idea" and "I've looked at the data and the top 5% are probably best ignored", etc. The formulas don't know the analyst's attitudes, but the latter are the researcher's justification for what is done. $\endgroup$
    – Nick Cox
    Commented Jan 8, 2016 at 14:32
46
$\begingroup$

A statistically sensible approach is to use a standard deviation cut-off.

For example, remove any results +/-3 standard deviations.

Using a rule like "biggest 10%" doesn't make sense. What if there are no outliers? The 10% rule would eliminate some data anyway. Unacceptable.

$\endgroup$
9
  • 2
    $\begingroup$ I was going to say this approach doesn't work (pathological case = 1000 numbers between -1 and +1, and then a single outlier of value +10000) because an outlier can bias the mean so that none of the results are within 3 stddev of the mean, but it looks like mathematically it does work. $\endgroup$
    – Jason S
    Commented Feb 2, 2009 at 15:21
  • 4
    $\begingroup$ en.wikipedia.org/wiki/Chebychev%27s_inequality This applies regardless of the distribution. $\endgroup$
    – dsimcha
    Commented Feb 2, 2009 at 20:49
  • 8
    $\begingroup$ The problem is that "outlier" isn't post-hoc conclusion about a particular realized data set. It's hard to know what people mean by outlier without knowing what the purpose of their proposed mean statistic is. $\endgroup$
    – Gregg Lind
    Commented Mar 3, 2009 at 20:11
  • 7
    $\begingroup$ So your categorial statement of "unacceptable" is non-sense, and not really very helpful. The trimmed mean has some useful properties, and some less useful, like any statistic. $\endgroup$
    – Gregg Lind
    Commented Mar 3, 2009 at 20:12
  • 3
    $\begingroup$ Note that contrary to comments elsewhere in this thread, such a procedure is not associated with statistical significance. $\endgroup$
    – Nick Cox
    Commented Dec 3, 2014 at 16:51
23
$\begingroup$

Another standard test for identifying outliers is to use LQ $-$ (1.5$\times$IQR) and UQ $+$ (1.5$\times$ IQR). This is somewhat easier than computing the standard deviation and more general since it doesn't make any assumptions about the underlying data being from a normal distribution.

$\endgroup$
3
  • 1
    $\begingroup$ But if it doesn't make any assumption, what is it then based on ? It must at least something like a definition of an outlier ? $\endgroup$
    – user83346
    Commented Mar 26, 2016 at 10:15
  • $\begingroup$ the formula is quartile based, so it's dependent on median rather than mean $\endgroup$
    – arahant
    Commented Jan 13, 2019 at 5:28
  • 1
    $\begingroup$ The 1.5 multiplier raises a question, why 1.5? And apparently it is somewhat based on normal distribution. If you apply this, directly on a guassian distribution, you get: 0.675σ + 1.5 * (0.675 - [-0.675])σ = 0.675σ + 1.5 * 1.35σ = 2.7σ which is an acceptable range to mark as "outliers". reference: medium.com/mytake/… $\endgroup$
    – Munawwar
    Commented Apr 26, 2020 at 19:13
20
$\begingroup$

For a very specific name, you'll need to specify the mechanism for outlier rejection. One general term is "robust".

dsimcha mentions one approach: trimming. Another is clipping: all values outside a known-good range are discarded.

$\endgroup$
20
$\begingroup$

The "average" you're talking about is actually called the "mean".

It's not exactly answering your question, but a different statistic which is not affected by outliers is the median, that is, the middle number.

{90,89,92,91,5} mean: 73.4
{90,89,92,91,5} median: 90

This might be useful to you, I dunno.

$\endgroup$
2
  • 3
    $\begingroup$ You are all missing the point. It has nothing to do with the mean, median, mode, stdev etc. Consider this: you have {1,1,2,3,2,400} avg = 68.17 but what we want is: {1,1,2,3,2,400} avg = 1.8 //minus the [400] value What do you call that? $\endgroup$
    – Tawani
    Commented Feb 2, 2009 at 15:41
  • 8
    $\begingroup$ @Tawani - they are not all missing the point. What you say needs to be defined using generic terms. You cannot go with a single example. Without general definitions, if 400 is 30 is it still an outlier? And if it is 14? And 9? Where do you stop? You need stddev's, ranges, quartiles, to do that. $\endgroup$
    – Daniel Daranas
    Commented Feb 2, 2009 at 17:05
9
$\begingroup$

There is no official name because of the various mechanisms, such as Q test, used to get rid of outliers.

Removing outliers is called trimming.

No program I have ever used has average() with an integrated trim()

$\endgroup$
3
  • 5
    $\begingroup$ mean() in R has a trim argument stat.ethz.ch/R-manual/R-devel/library/base/html/mean.html $\endgroup$ Commented Sep 29, 2011 at 11:55
  • 2
    $\begingroup$ In trimming you don't remove outliers; you just don't include them in the calculation. "Remove" might suggest that points are no longer in the dataset. And you don't remove (or ignore) them because they are outliers; the criterion is (usually) just that they are in some extreme fraction of the data. A value not included in a trimmed mean often is only slightly more (or less) than the highest (lowest) value included. $\endgroup$
    – Nick Cox
    Commented Dec 3, 2014 at 16:48
  • $\begingroup$ Excel has a TRIMMEAN() function which removes the top and bottom X% before calculating the mean. Isn't that an average with an integrated trim? $\endgroup$
    – Simon East
    Commented Sep 11, 2022 at 1:01
7
$\begingroup$

I don't know if it has a name, but you could easily come up with a number of algorithms to reject outliers:

  1. Find all numbers between the 10th and 90th percentiles (do this by sorting then rejecting the first $N/10$ and last $N/10$ numbers) and take the mean value of the remaining values.

  2. Sort values, reject high and low values as long as by doing so, the mean/standard deviation change more than $X\%$.

  3. Sort values, reject high and low values as long as by doing so, the values in question are more than $K$ standard deviations from the mean.

$\endgroup$
5
$\begingroup$

The most common way of having a Robust (the usual word meaning resistant to bad data) average is to use the median. This is just the middle value in the sorted list (of half way between the middle two values), so for your example it would be 90.5 = half way between 90 and 91.

If you want to get really into robust statistics (such as robust estimates of standard deviation etc) I would recommend a lost of the code at The AGORAS group but this may be too advanced for your purposes.

$\endgroup$
4
$\begingroup$

... {90,89,92,91(,5)} avg = 90.5

How do you describe this average in statistics? ...

There's no special designation for that method. Call it any name you want, provided that you always tell the audience how you arrived at your result, and you have the outliers in hand to show them if they request (and believe me: they will request).

$\endgroup$
3
$\begingroup$

If all you have is one variable (as you imply) I think some of the respondents above are being over critical of your approach. Certainly other methods that look at things like leverage are more statistically sound; however that implies you are doing modeling of some sort. If you just have for example scores on a test or age of senior citizens (plausible cases of your example) I think it is practical and reasonable to be suspicious of the outlier you bring up. You could look at the overall mean and the trimmed mean and see how much it changes, but that will be a function of your sample size and the deviation from the mean for your outliers.

With egregious outliers like that, you would certainly want to look into te data generating process to figure out why that's the case. Is it a data entry or administrative fluke? If so and it is likely unrelated to actual true value (that is unobserved) it seems to me perfectly fine to trim. If it is a true value as far as you can tell you may not be able to remove unless you are explicit in your analysis about it.

$\endgroup$
2
$\begingroup$

I love the discussion here - the trimmed mean is a powerful tool to get a central tendency estimate concentrated around the middle of the data.

The one thing I would add is that there is a choice to be made about which "metric" to use in the cases of small and large sample sizes. In some cases we talk about

  • means in the context of large samples because of central-limit theorem,
  • medians as robust small-sample alternatives
  • and trimmed means as robust to outliers.

Obviously the above is a gross generalization, but there are interesting papers that talk about the families and classes of estimators in large and small sample settings and their properties. I work in bioinformatics aand usually you deal with small samples (3-10s) usually in mice models, and what not, and this paper gives a good technical overview of what alternatives exist and what properties these estimators have.

Robust estimation in very small samples

This is off-course one paper, but there are plenty others that discuss these types of estimators. Hope this helps.

$\endgroup$
0
$\begingroup$

There are superior methods to the IQR or SD based methods. Due to outliers being present, the distribution likely has issues with normality already (unless ouliers are evenly distributed at both ends of the distribution). This inflates the SD a lot, making the SDs use less than desirable, however the SD method has some desirable aspects over the IQR method, namely 1.5 times the IQR is a relatively subjective cutoff. While subjectivity in these matters is unavoidable it is preferable to reduce it.

A Hampel Identifier on the other hand uses robust methods to estimate outliers. Essentially its the same as the SD method, but you would replace means with medians and SD with Median Absolute Deviations (MAD). MADs are just the median distance from the media. This MAD is multiplied by a scaling constant .675. The formula comes out to (X - Median)/(.675*MAD). The resulting statistic is treated identically to a Z-score. This bypasses the issue of the likely non-normality that if you have outliers may be present.

As for what to call it. Trimmed means are normally reserved for the method of trimming the bottom and top ten percent mentioned by @dsimcha. If it has been completely cleaned you may refer to it as the cleaned mean, or just the mean. Just be sure to be clear what you did to it in your write-up.

Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., & Stahel, W. A. (1986). Robust Statistics. John Wiley & Sons, New York.

$\endgroup$
0
$\begingroup$

disclaimer - this method is ad hoc and without rigorous study. Use at your own risk :)

What I found to be quite good was to reduce the relevancy of a points contribution to the mean by the square of its number of standard deviations from the mean but only if the point is more than one standard deviation from the mean.

Steps:

  1. Calculate the mean and standard deviation as usual.
  2. Recalculate the mean, but this time, for each value, if it is more than one standard deviation from the mean reduce its contribution to the mean. To do reduce its contribution, divide its value by the square of its number of deviations before adding to the total. Also because it's contributing less, we need to Reduce N, so subtract 1-1/(square of values deviation) from N.
  3. Recalculate the standard deviation, but use this new mean rather than the old mean.

example: stddev = 0.5 mean = 10 value = 11

then, deviations = distance from mean / stddev = |10-11|/0.5 = 2

so value changes from 11 to 11/(2)^2 = 11/4

also N changes, it is reduced to N-3/4.

code:

def mean(data):
    """Return the sample arithmetic mean of data."""
    n = len(data)
    if n < 1:
        raise ValueError('mean requires at least one data point')
    return 1.0*sum(data)/n # in Python 2 use sum(data)/float(n)

def _ss(data):
    """Return sum of square deviations of sequence data."""
    c = mean(data)
    ss = sum((x-c)**2 for x in data)
    return ss, c

def stddev(data, ddof=0):
    """Calculates the population standard deviation
    by default; specify ddof=1 to compute the sample
    standard deviation."""
    n = len(data)
    if n < 2:
        raise ValueError('variance requires at least two data points')
    ss, c = _ss(data)
    pvar = ss/(n-ddof)
    return pvar**0.5, c

def rob_adjusted_mean(values, s, m):
    n = 0.0
    tot = 0.0
    for v in values:
        diff = abs(v - m)
        deviations = diff / s
        if deviations > 1:
            #it's an outlier, so reduce its relevancy / weighting by square of its number of deviations
            n += 1.0/deviations**2
            tot += v/deviations**2
        else:
            n += 1
            tot += v
    return tot/n

def rob_adjusted_ss(values, s, m):
    """Return sum of square deviations of sequence data."""
    c = rob_adjusted_mean(values, s, m)
    ss = sum((x-c)**2 for x in values)
    return ss, c

def rob_adjusted_stddev(data, s, m, ddof=0):
    """Calculates the population standard deviation
    by default; specify ddof=1 to compute the sample
    standard deviation."""
    n = len(data)
    if n < 2:
        raise ValueError('variance requires at least two data points')
    ss, c = rob_adjusted_ss(data, s, m)
    pvar = ss/(n-ddof)
    return pvar**0.5, c

s, m = stddev(values,ddof=1)
print s, m
s, m = rob_adjusted_stddev(values, s, m, ddof=1)
print s, m

output before and after adjustment of my 50 measurements:

0.0409789841609 139.04222
0.0425867309757 139.030745443

enter image description here

$\endgroup$
10
  • 4
    $\begingroup$ Why might this be better than traditional methods? $\endgroup$ Commented Mar 8, 2018 at 19:23
  • 3
    $\begingroup$ Thanks, I'm not familiar w/ this approach. Dividing by the square of a deviation might produce unusual results when the deviations are <|1|. Is there a theoretical basis for this method? $\endgroup$ Commented Mar 8, 2018 at 20:32
  • 6
    $\begingroup$ Although this procedure is ad hoc, in spirit it is much like an M-estimator. One reason for the comments you are getting is that the properties of procedures like this can be analyzed and studied and that, in general, the lack of such study shows the procedure is not well understood and likely is inferior to others. Thus, it is incumbent on anyone proposing a new procedure to characterize its properties sufficiently to enable intelligent, correct application of it. Absent such characterization, readers ought to be reluctant to adopt it. $\endgroup$
    – whuber
    Commented Mar 9, 2018 at 15:13
  • 1
    $\begingroup$ I admire your attitude (seriously!). Do note, however, that the burden of proof is on you. It's your job to demonstrate the correctness or usefulness of your recommendation (either through citation or a legitimate argument). It's not incumbent on us to perform that analysis. I have pointed to a theory that gives you some hope this procedure has good properties, but it's a general--yet extremely effective--meta-law of statistics that ad hoc procedures are inadmissible until proven otherwise (which simply means there is some other procedure that works better). $\endgroup$
    – whuber
    Commented Mar 9, 2018 at 22:30
  • 1
    $\begingroup$ Thanks for the clarification, that makes a lot more sense. $\endgroup$ Commented Mar 10, 2018 at 1:46
-4
$\begingroup$

It can be the median. Not always, but sometimes. I have no idea what it is called in other occasions. Hope this helped. (At least a little.)

$\endgroup$
-4
$\begingroup$

My statistics textbook refers to this as a Sample Mean as opposed to a Population Mean. Sample implies there was a restriction applied to the full dataset, though no modification (removal) to the dataset was made.

$\endgroup$
2
  • 2
    $\begingroup$ 0. Welcome to the site. 1. Which book? Please give a reference. 2. "Sample mean" does not typically refer to a mean obtained after removing outliers. $\endgroup$ Commented Mar 26, 2016 at 8:06
  • 1
    $\begingroup$ This is not correct. $\endgroup$ Commented Feb 16, 2019 at 16:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.