8
$\begingroup$

I have small datasets of size 40-50 points. Without assuming that the data is normally distributed I wanted to find out the outliers with 90% confidence at least. I thought boxplot could be a good way to do that but I am not sure.

Any help appreciated.

Also with boxplot implementations I could not find a implementation which besides drawing the plot explicitly spits out the outliers.

$\endgroup$
  • 7
    $\begingroup$ 90% confidence of what? $\endgroup$ – Henry Apr 19 '12 at 23:54
  • $\begingroup$ What I also see sometimes is that researchers drop the top and bottom X % of their observations to reduce the influence of extreme cases. But I'm unsure whether I agree with it, it's quite arbitrary isn't it? $\endgroup$ – C. Pieters May 10 '12 at 19:54
  • $\begingroup$ You don't have to assume that your data are normally distributed, but since you know what data you're dealing with, you may be able to use another parametric distribution. For example, waiting times are often Poisson-distributed. Then it makes sense to say whether one Poisson data point likely to be generated by a given distribution of them. $\endgroup$ – Jack Tanner May 10 '12 at 21:24
23
$\begingroup$

That's because such an algorithm can't exist. You require an assumed distribution in order to be able to classify something as lying outside the range of expected values.

Even if you do assume a normal distribution, declaring data points as outliers is a fraught business. In general, you not only need a good estimate of the true distribution, which is often unavailable, but also a good theoretically supported reason for making your decision (i.e. the subject broke the experimental setup somehow). Such a judgement is usually impossible to codify in an algorithm.

| cite | improve this answer | |
$\endgroup$
  • 11
    $\begingroup$ +1. Also, the use of "with 90% confidence" reveals a misunderstanding of the way the concept of confidence could apply in this case. Without a basis for a degree of confidence, there's no systematic way to quantify the level of confidence one might have. It would come down to an arbitrary thing, as if one were to say "I'm x% confident that this soup is too salty." $\endgroup$ – rolando2 Apr 20 '12 at 0:40
  • 6
    $\begingroup$ @rolando2, that is as it may be, but nonetheless, I'm 90% confident that's a good comment. $\endgroup$ – gung - Reinstate Monica May 10 '12 at 15:13
6
$\begingroup$

This does not directly answer your question, but you may learn something from looking at the outliers dataset in the TeachingDemos package for R and working through the examples on the help page. This may give you a better understanding of some of the issues with automatic outlier detection.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

R will spit out the outliers as in

dat <- c(6,8.5,-12,1,rnorm(40),-1,10,0)
boxplot(dat)$out

which will draw the boxplot and give

[1]   6.0   8.5 -12.0  10.0
| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ No - the default definition of "outlier" for a boxplot is anything more than $1.5 \, IQR$ below the lower hinge or quartile or above $1.5 \, IQR$ above the higher hinge or quartile, where $IQR$ is the interquartile range. Since asymmetry will usually affect the relative position of the quartiles and median, you cannot say this assumes a symmetrical distribution. For something like an exponential distribution you will typically only see outliers at the high end, but this is what you would expect anyway. $\endgroup$ – Henry Apr 20 '12 at 6:37
  • 4
    $\begingroup$ It's worth noting that finding points $>|1.5IQR|$ is something that should be expected to happen fairly often, and doesn't necessarily indicate any problems. $\endgroup$ – gung - Reinstate Monica May 10 '12 at 15:27
  • 5
    $\begingroup$ @gung: This is $1.5 IQR$ beyond the quartile, so about $2 IQR$ from the median for a symmetric distribution. It also depends on what you mean by "fairly often" and the distribution: almost never for a sample from a uniform distribution; about 0.7% of a sample from a normal distribution; about 5% for a sample from an exponential distribution; about 16% for a sample from a Cauchy distribution. $\endgroup$ – Henry May 10 '12 at 20:24
  • 1
    $\begingroup$ I remember having seen a brief paper on this a while ago, of course I can't find it now, but here's my thinking: I start w/ 2*(1-pnorm(4*qnorm(.75))), which returns [1] 0.006976603, the value you report above, but then I simulate as follows: Set.seed(1); out = c(); for(i in 1:100) x = rnorm(50) y = boxplot(x, plot=F) out[i] = length(y$out)>=1} sum(out)/100 which returns [1] 0.3. Ie, 30% of samples w/ $n=50$ will show as having outliers by this method, even though there actually aren't any. $\endgroup$ – gung - Reinstate Monica May 11 '12 at 1:20
  • 1
    $\begingroup$ @gung: set.seed(1); out = c(); for(i in 1:100) {x = rnorm(500); y = boxplot(x, plot=F); out[i] = length(y$out)}; sum(out)/50000 gives 0.00738 which is closer to what I was describing $\endgroup$ – Henry May 11 '12 at 22:45
1
$\begingroup$

As others have said you have stated the question poorly in terms of confidence. There are statistical tests for outlier's like Grubbs' test and Dixon's ratio test that I have referred to on another post. They assume the population distribution is normal although Dixon's test is robust to the normality assumption in small samples. A boxplot is a nice informal way to spot outliers in your data. Usually the whiskers are set at the 5th and 95th percentile and obsevations plotted beyond the whiskers are usually considered to be possible outliers. However this does not involve formal statistical testing.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Setting the whiskers at these fixed percentiles seems strange to me. Do you have a reference for this? (Tukey, who originated the boxplot, did not use this method: he set the whiskers either at the extremes, if they are sufficiently close to the quartiles, but no further than 1.5 "steps" (equal approximately to 1.5 times the IQR) out from the quartiles.) This is much more robust for outlier detection than using an extreme percentile, which--by definition--would always identify 10% of the data as "outliers," which wouldn't be a very useful procedure. $\endgroup$ – whuber May 10 '12 at 22:28
  • $\begingroup$ I don't know if I should have said usually. I think a lot of different points have been used for the whiskers. I think the 1st percentile and 99th have also been used and the min and max. But if you use min and max you can't find outliers beyond the whiskers. I have no specific reference that come to mind at the moment. I did not mean that anything outside the whiskers would be an outlier when the 5th and 95th percentiles are used. I just meant that visually you can see them because they will be far above or below the whiskers. $\endgroup$ – Michael R. Chernick May 10 '12 at 22:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.