# Finding outliers without assuming normal distribution

I have small datasets of size 40-50 points. Without assuming that the data is normally distributed I wanted to find out the outliers with 90% confidence at least. I thought boxplot could be a good way to do that but I am not sure.

Any help appreciated.

Also with boxplot implementations I could not find a implementation which besides drawing the plot explicitly spits out the outliers.

-
90% confidence of what? – Henry Apr 19 '12 at 23:54
What I also see sometimes is that researchers drop the top and bottom X % of their observations to reduce the influence of extreme cases. But I'm unsure whether I agree with it, it's quite arbitrary isn't it? – C. Pieters May 10 '12 at 19:54
You don't have to assume that your data are normally distributed, but since you know what data you're dealing with, you may be able to use another parametric distribution. For example, waiting times are often Poisson-distributed. Then it makes sense to say whether one Poisson data point likely to be generated by a given distribution of them. – Jack Tanner May 10 '12 at 21:24

That's because such an algorithm can't exist. You require an assumed distribution in order to be able to classify something as lying outside the range of expected values.

Even if you do assume a normal distribution, declaring data points as outliers is a fraught business. In general, you not only need a good estimate of the true distribution, which is often unavailable, but also a good theoretically supported reason for making your decision (i.e. the subject broke the experimental setup somehow). Such a judgement is usually impossible to codify in an algorithm.

-
+1. Also, the use of "with 90% confidence" reveals a misunderstanding of the way the concept of confidence could apply in this case. Without a basis for a degree of confidence, there's no systematic way to quantify the level of confidence one might have. It would come down to an arbitrary thing, as if one were to say "I'm x% confident that this soup is too salty." – rolando2 Apr 20 '12 at 0:40
@rolando2, that is as it may be, but nonetheless, I'm 90% confident that's a good comment. – gung May 10 '12 at 15:13

This does not directly answer your question, but you may learn something from looking at the outliers dataset in the TeachingDemos package for R and working through the examples on the help page. This may give you a better understanding of some of the issues with automatic outlier detection.

-

R will spit out the outliers as in

dat <- c(6,8.5,-12,1,rnorm(40),-1,10,0)
boxplot(dat)$out which will draw the boxplot and give [1] 6.0 8.5 -12.0 10.0 - No - the default definition of "outlier" for a boxplot is anything more than$1.5 \, IQR$below the lower hinge or quartile or above$1.5 \, IQR$above the higher hinge or quartile, where$IQR$is the interquartile range. Since asymmetry will usually affect the relative position of the quartiles and median, you cannot say this assumes a symmetrical distribution. For something like an exponential distribution you will typically only see outliers at the high end, but this is what you would expect anyway. – Henry Apr 20 '12 at 6:37 It's worth noting that finding points$>|1.5IQR|$is something that should be expected to happen fairly often, and doesn't necessarily indicate any problems. – gung May 10 '12 at 15:27 @gung: This is$1.5 IQR$beyond the quartile, so about$2 IQR$from the median for a symmetric distribution. It also depends on what you mean by "fairly often" and the distribution: almost never for a sample from a uniform distribution; about 0.7% of a sample from a normal distribution; about 5% for a sample from an exponential distribution; about 16% for a sample from a Cauchy distribution. – Henry May 10 '12 at 20:24 I remember having seen a brief paper on this a while ago, of course I can't find it now, but here's my thinking: I start w/ 2*(1-pnorm(4*qnorm(.75))), which returns [1] 0.006976603, the value you report above, but then I simulate as follows: Set.seed(1); out = c(); for(i in 1:100) x = rnorm(50) y = boxplot(x, plot=F) out[i] = length(y$out)>=1} sum(out)/100 which returns [1] 0.3. Ie, 30% of samples w/ $n=50$ will show as having outliers by this method, even though there actually aren't any. – gung May 11 '12 at 1:20
@gung: set.seed(1); out = c(); for(i in 1:100) {x = rnorm(500); y = boxplot(x, plot=F); out[i] = length(y\$out)}; sum(out)/50000 gives 0.00738 which is closer to what I was describing – Henry May 11 '12 at 22:45