0
$\begingroup$

What motivates the choice of 1.5 IQRs for finding outliers in a set of data? The the lower quartile minus 1.5 times the IQR would give the lower fence, and the upper quartile plus 1.5 times the IQR gives the upper fence, with any data points falling outside either fence categorized as "outliers". But I don't know why 1.5 is used and not some other value.

$\endgroup$
3
  • 1
    $\begingroup$ Because "significant" has a well-known technical meaning in statistics, but is not applicable here, I have to ask: what do you mean by "significance"? $\endgroup$
    – whuber
    Sep 26, 2016 at 16:47
  • 2
    $\begingroup$ I meant why is the number 1.5 multiplied here? Why not anything else? $\endgroup$
    – Ahnaf
    Sep 27, 2016 at 2:13
  • $\begingroup$ One can surely, change this limit depending upon how much data, he/she is willing to consider as the outlier. These commonly used limits make sure that '0.7%' of data is treated as an outlier; if there's any data point in that region. $\endgroup$ Nov 1, 2019 at 15:12

1 Answer 1

5
$\begingroup$

Like pretty much any method for detecting/defining outliers, the fence at 1.5*IQR is a rule of thumb. It will be a reasonable strategy for detecting outliers in some circumstances, and not in others. You can get an idea for the logic behind it by considering its application to a normal distribution. If the data are normally distributed, the fence will be 2.7 standard deviations from the mean, so cases outside of it will be quite rare (0.4%). So the idea is that unless you have quite a large dataset, it's unlikely that you'll have many cases more than 1.5*IQR below the 1st quartile or above the 3rd quartile; when you see cases there, they may represent some weird deviation in your data (e.g. a coding error, equipment malfunction) and you probably want to exclude them from your analyses. Really, the main goal is to help you identify extreme points in your data, so that you can examine them to a) look for and correct errors and b) make decisions about how to handle cases that may have undue influence on your model estimation, as outliers often do.

If your data are NOT normally distributed, then that all changes --- if you have skewed data, for example, it's very common to detect a lot of "outliers" using standard techniques like the boxplot cutoffs you mention.

For example, check out a boxplot of the skewed data, drawn from an F distribution (Note that if you run this yourself you'll get slightly different results because of the random number generator, unless you use first set.seed() to coordinate with me):

set.seed(24601)
skewed_data <- rf(100, 1, 30)
boxplot(skewed_data)

skewed_boxplot

It looks like there are several outliers there. But if you look at a histogram, you'll see that those outliers are just a continuation of the skewed tail of the data:

hist(skewed_data, breaks=30)

skewed_histogram

Should all of those cases be excluded from your analysis, but not the ones that happen to fall right inside the fence cutoff? Probably not. (In this situation, you may want to transform the data to correct the skew instead, and that will probably take care of the outliers for you.) There is no test that will work regardless of the data and the model to tell you for sure whether a case is a "significant" outlier. Instead, I recommend that you use outlier analyses to better understand your data and guide your decisions about your analysis plan.

$\endgroup$
1
  • 1
    $\begingroup$ Welcome to our site, Rose! $\endgroup$
    – whuber
    Sep 26, 2016 at 18:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.