12
$\begingroup$

I have small datasets of size 40-50 points. Without assuming that the data is normally distributed I wanted to find out the outliers with 90% confidence at least. I thought boxplot could be a good way to do that but I am not sure.

Any help appreciated.

Also with boxplot implementations I could not find a implementation which besides drawing the plot explicitly spits out the outliers.

Improve this question
$\endgroup$
3
  • 7
    $\begingroup$ 90% confidence of what? $\endgroup$
    – Henry
    Commented Apr 19, 2012 at 23:54
  • $\begingroup$ What I also see sometimes is that researchers drop the top and bottom X % of their observations to reduce the influence of extreme cases. But I'm unsure whether I agree with it, it's quite arbitrary isn't it? $\endgroup$
    – Raynor
    Commented May 10, 2012 at 19:54
  • $\begingroup$ You don't have to assume that your data are normally distributed, but since you know what data you're dealing with, you may be able to use another parametric distribution. For example, waiting times are often Poisson-distributed. Then it makes sense to say whether one Poisson data point likely to be generated by a given distribution of them. $\endgroup$
    – Jack Tanner
    Commented May 10, 2012 at 21:24

4 Answers 4

Reset to default
24
$\begingroup$

That's because such an algorithm can't exist. You require an assumed distribution in order to be able to classify something as lying outside the range of expected values.

Even if you do assume a normal distribution, declaring data points as outliers is a fraught business. In general, you not only need a good estimate of the true distribution, which is often unavailable, but also a good theoretically supported reason for making your decision (i.e. the subject broke the experimental setup somehow). Such a judgement is usually impossible to codify in an algorithm.

Improve this answer
$\endgroup$
2
  • 11
    $\begingroup$ +1. Also, the use of "with 90% confidence" reveals a misunderstanding of the way the concept of confidence could apply in this case. Without a basis for a degree of confidence, there's no systematic way to quantify the level of confidence one might have. It would come down to an arbitrary thing, as if one were to say "I'm x% confident that this soup is too salty." $\endgroup$
    – rolando2
    Commented Apr 20, 2012 at 0:40
  • 6
    $\begingroup$ @rolando2, that is as it may be, but nonetheless, I'm 90% confident that's a good comment. $\endgroup$
    – gung - Reinstate Monica
    Commented May 10, 2012 at 15:13
6
$\begingroup$

This does not directly answer your question, but you may learn something from looking at the outliers dataset in the TeachingDemos package for R and working through the examples on the help page. This may give you a better understanding of some of the issues with automatic outlier detection.

Improve this answer
$\endgroup$
3
$\begingroup$

R will spit out the outliers as in

dat <- c(6,8.5,-12,1,rnorm(40),-1,10,0)
boxplot(dat)$out

which will draw the boxplot and give

[1]   6.0   8.5 -12.0  10.0
Improve this answer
$\endgroup$
11
  • 2
    $\begingroup$ No - the default definition of "outlier" for a boxplot is anything more than $1.5 \, IQR$ below the lower hinge or quartile or above $1.5 \, IQR$ above the higher hinge or quartile, where $IQR$ is the interquartile range. Since asymmetry will usually affect the relative position of the quartiles and median, you cannot say this assumes a symmetrical distribution. For something like an exponential distribution you will typically only see outliers at the high end, but this is what you would expect anyway. $\endgroup$
    – Henry
    Commented Apr 20, 2012 at 6:37
  • 4
    $\begingroup$ It's worth noting that finding points $>|1.5IQR|$ is something that should be expected to happen fairly often, and doesn't necessarily indicate any problems. $\endgroup$
    – gung - Reinstate Monica
    Commented May 10, 2012 at 15:27
  • 5
    $\begingroup$ @gung: This is $1.5 IQR$ beyond the quartile, so about $2 IQR$ from the median for a symmetric distribution. It also depends on what you mean by "fairly often" and the distribution: almost never for a sample from a uniform distribution; about 0.7% of a sample from a normal distribution; about 5% for a sample from an exponential distribution; about 16% for a sample from a Cauchy distribution. $\endgroup$
    – Henry
    Commented May 10, 2012 at 20:24
  • 1
    $\begingroup$ I remember having seen a brief paper on this a while ago, of course I can't find it now, but here's my thinking: I start w/ 2*(1-pnorm(4*qnorm(.75))), which returns [1] 0.006976603, the value you report above, but then I simulate as follows: Set.seed(1); out = c(); for(i in 1:100) x = rnorm(50) y = boxplot(x, plot=F) out[i] = length(y$out)>=1} sum(out)/100 which returns [1] 0.3. Ie, 30% of samples w/ $n=50$ will show as having outliers by this method, even though there actually aren't any. $\endgroup$
    – gung - Reinstate Monica
    Commented May 11, 2012 at 1:20
  • 1
    $\begingroup$ @gung: set.seed(1); out = c(); for(i in 1:100) {x = rnorm(500); y = boxplot(x, plot=F); out[i] = length(y$out)}; sum(out)/50000 gives 0.00738 which is closer to what I was describing $\endgroup$
    – Henry
    Commented May 11, 2012 at 22:45
1
$\begingroup$

As others have said you have stated the question poorly in terms of confidence. There are statistical tests for outlier's like Grubbs' test and Dixon's ratio test that I have referred to on another post. They assume the population distribution is normal although Dixon's test is robust to the normality assumption in small samples. A boxplot is a nice informal way to spot outliers in your data. Usually the whiskers are set at the 5th and 95th percentile and obsevations plotted beyond the whiskers are usually considered to be possible outliers. However this does not involve formal statistical testing.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ Setting the whiskers at these fixed percentiles seems strange to me. Do you have a reference for this? (Tukey, who originated the boxplot, did not use this method: he set the whiskers either at the extremes, if they are sufficiently close to the quartiles, but no further than 1.5 "steps" (equal approximately to 1.5 times the IQR) out from the quartiles.) This is much more robust for outlier detection than using an extreme percentile, which--by definition--would always identify 10% of the data as "outliers," which wouldn't be a very useful procedure. $\endgroup$
    – whuber ♦
    Commented May 10, 2012 at 22:28
  • $\begingroup$ I don't know if I should have said usually. I think a lot of different points have been used for the whiskers. I think the 1st percentile and 99th have also been used and the min and max. But if you use min and max you can't find outliers beyond the whiskers. I have no specific reference that come to mind at the moment. I did not mean that anything outside the whiskers would be an outlier when the 5th and 95th percentiles are used. I just meant that visually you can see them because they will be far above or below the whiskers. $\endgroup$
    – Michael R. Chernick
    Commented May 10, 2012 at 22:32
  • $\begingroup$ @whuber there is an assumption of normality when you set the limit to 1.5xIQR. So if your distribution isnt normal, that limit isnt really a good choice. towardsdatascience.com/…. $\endgroup$
    – Shervin Rad
    Commented Aug 27, 2021 at 13:02
  • 1
    $\begingroup$ @Shervin Tukey explicitly did not assume Normality, but he did use it as a reference for justifying that choice. Tukey also presumed that a preliminary re-expression of the data would be applied to make the distribution approximately symmetric. The spirit is one of EDA, which emphasizes robustness and making no distributional assumptions. As far as setting the limits in narrower situations goes, we have addressed this question in particular circumstances: see, for instance, stats.stackexchange.com/a/13101/919. $\endgroup$
    – whuber ♦
    Commented Aug 27, 2021 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.