Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a data set $\mathcal{S}$, which is univariate. I would like to discover the data points which can be thought of as outliers of $\mathcal{S}$. I use $\bar{\mathcal{S}}+2*\sigma(\mathcal{S})$ and $\bar{\mathcal{S}}-2*\sigma(\mathcal{S})$ to compute the upper and lower threshold in terms of 95% confidence interval. $\bar{\mathcal{S}}$ is the mean of $\mathcal{S}$ and $\sigma(\mathcal{S})$ is the standard deviation. I know it assumes the normal distribution. I would like to know are there any loopholes for this approach from the statistical analysis perspective. Moreover, are there any other better statistical approaches to give this kind of threshold.

share|improve this question
    
To me, this approach sounds fine, provided you do assume your data are normally distributed (surely you ought to check for approximate normality first). Another concern would be that your sample be large enough (to be able to establish the distribution of your data, among other things). –  Patrick Coulombe Feb 11 at 23:34
1  
have you made any graphs or done any tests to check for normality? There are Kolomogrof-Smirnoff and Shapiro-Wilk tests in R. qqplot in the car package tests quantiles of a variable against a distribution you can compare. Normality might be a really poor assumption for your data (and then so might your confidence interval) –  Eric Peterson Feb 11 at 23:50
    
You don't necessarily need to assume normality. Normality will guarantee that 95% of the distribution will lie w/i +/- 1.96 SDs (although that assumes sigma isn't estimated from the same sample). Nonetheless, you can still form confidence intervals. You might want to read up on Chebyshev's inequality. –  gung Feb 12 at 0:36
    
Why do you need to identify outliers? What does this have to do w/ ANOVA? –  gung Feb 12 at 0:37

3 Answers 3

up vote 4 down vote accepted

A few comments:

  1. Your title "95% confidence interval for a given data set" is misleading. Confidence intervals are calculated for computed values like means. It doesn't mean anything really to have a confidence interval for a data set.

  2. Your proposed method will work with huge data sets, but will fail with small data sets. Why? Because the outlier itself increases the SD (assuming you are computing the SD from your actual values). With small data sets (say n=4), it is impossible for any outlier to be more than 2 SD from the mean. Try it! The most SD that a value can be from the mean is (n-1)/sqrt(n), where n is sample size.

  3. There are established methods for determining outliers assuming the other values are sampled from a Gaussian distribution. Look up Grubbs test, for one of the most widely used tests.

  4. Deciding what to do with outliers is tricky.

share|improve this answer

An outlier is a surprising value.

Therefore, your method is flawed.

If the data set is large, then, if it is normally distributed, you expect some values beyond 2 sd from the mean. If the data set is small, and normally distributed, you don't expect values even 1 sd from the mean. e.g. it is not surprising that there are some 7 foot tall people in the USA.

In addition, you have calculated the mean and sd with the possible outliers IN the data. If the outliers are truly not part of the data (e.g. they are data entry errors; they got picked for the sample by mistake) then you should not have done this.

I would say that there are no really good statistical tests for outliers, even univariately. At best, there are ways to identify points that might be worth looking at. A better way than the one you propose is to graph the data. If you suppose it is normal, you could do a normal quantile plot. Even if you don't suppose it is normal, you could do a density plot.

In addition, you should always think of the substantive nature of the values. Is it *possible? E.g., among human adults, a weight of 500 pounds is possible (albeit very large). A weight of 5000 pounds is not.

share|improve this answer

Another approach you might consider is Tukey’s Outlier Filter. It is robust as well, since it is based on quantiles.

The idea behind is to classify as outlier data points that lies:

  • above 3rd quartile + 1.5 times the Inter Quartile Range (IQR, distance between the 1st and 3rd quartiles), and
  • below 1st quartile - 1.5 times the IQR.

The 1.5 range is arbitrary and was suggest empirically. You might use 3 instead to be more strict. Take care with asymmetric data, which would need an adjustment in the classification criterion.

Boxplot is a good plot to evaluate the points that may be considered outliers through this filter. See R example below applied to data posted by user603:

x <- c(-2.21,-1.84,-.95,-.91,-.36,-.19,-.11,-.1,.18,.3,.31,.43,.51,.64,.67,.72,1.22,1.35,8.1,17.6)
boxplot(x, range=3)

boxplot using range = 3

Reading suggestions:

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.