7
$\begingroup$

Why divide data into 4 parts for IQR, versus into more parts, such as 20 or 10 percent per part?

I know that interquartile range by definition means 25%, but that is not my question.

I think that discarding 50 percentile of data for IQR to remove outliers is too much of a waste of data. But there has to be a reason for it, right?

New contributor
True_Nerd is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
3
  • 8
    $\begingroup$ Calculating a median is a first step to get an idea of the middlemost value in data. Calculating the medians of each half of the data is one second step, giving lower (first) and upper (third) quartiles and so IQR as upper q. minus lower q. That often is extended to other quantiles. Although its focus is on terminology the thread stats.stackexchange.com/questions/235330/… bears witness to terciles (tertiles) (3), quintiles (5), sextiles (6) and many other kinds of quantile binning, including decile (10) and ventiles (vigintiles) (20). $\endgroup$
    – Nick Cox
    May 14 at 8:55
  • 2
    $\begingroup$ The IQR is only the beginning of a description of the data. For more background on this, see my post at stats.stackexchange.com/a/96684/919. $\endgroup$
    – whuber
    May 14 at 12:46
  • 6
    $\begingroup$ This seems to be based on a faulty assumption that outlier detection should be performed by throwing away the 50% of the data that lies outside the IQR. That's simply not the case, an outlier is determined by its relationship to the rest of the data and what you choose to call an outlier - there's no particular reason why the 25% of highest and lowest values should be considered outliers. It would be extremely aggressive to call half of every dataset you see as outliers that should be ignored. $\endgroup$ May 14 at 17:51

4 Answers 4

9
$\begingroup$

I think that discarding 50 percentile of data for IQR to remove outliers is too much of a waste of data.

I don't think the purpose of IQR, or any choice of quantiles, is to remove outliers. If anything, you could remove datapoints much more extreme than above/below 75%/25% of the distribution. But there isn't any rule or a-priori justification for doing that. Perhaps you misunderstood the source stating that?

But there has to be a reason for it, right?

Dividing into 4 parts feels about right for eye-balling a list of numbers or for plotting to get a sense for the data. That, I think, is the main purpose of IQR. For example compare these two sets (4 intervals vs 20), the second is too much to read:

quantile(rivers, c(0, 0.25, 0.50, 0.75, 1))
  0%  25%  50%  75% 100% 
 135  310  425  680 3710
 
> quantile(rivers, seq(0, 1, length.out=21))
  0%   5%  10%  15%  20%  25%  30%  35%  40%  45%  50%  55%  60%  65%  70%  75% 
 135  230  255  276  291  310  330  350  375  392  425  460  505  545  610  680 
 80%  85%  90%  95% 100% 
 735  890 1054 1450 3710

But again, it's a convention that you can override as you feel appropriate.

$\endgroup$
5
$\begingroup$

Regardless of what metric you use for outlier detection, there really isn't any justifiable reason to simply remove the data if there are outliers. Generally, one can just use robust methods, such as robust linear regression, quantile regression, etc. to get around issues related to outliers. As you already mentioned, removing the data will simply cause more problems, not less.

$\endgroup$
4
  • 3
    $\begingroup$ Other choices include transformation of problematic variables; generalized linear models and other models with appropriate link functions; analyses with and without putative outliers to see how much difference omission makes to results. $\endgroup$
    – Nick Cox
    May 14 at 10:09
  • 3
    $\begingroup$ Perhaps needless to state, but outliers can be removed if they are obviously wrong and cannot be corrected. A man 10 metres tall is implausible outside movies. $\endgroup$
    – Nick Cox
    May 14 at 18:22
  • $\begingroup$ Yes that is always an important qualifier regarding outliers. $\endgroup$ May 15 at 0:19
  • 1
    $\begingroup$ Would this not suit more as a comment than an answer? I do not quite see how it answers the question. $\endgroup$ May 15 at 12:29
3
$\begingroup$

I'm interpreting the OP's question as: How did it become so common to report IQRs and quartiles specifically, instead of defaulting to some other percentiles?
The IQR and quartiles are often taught in Intro Statistics courses, and they are often reported by default in statistical software. Why are they, and not something else, commonly used as the standard?
(For instance, we could have settled on a standard of reporting the deciles, and reporting the distance from 10th to 90th or perhaps 20th to 80th percentiles as a measure of spread. But instead, it's much more common to report quartiles, and to report the distance from the 25th to 75th percentiles as a measure of spread.)

So here's my understanding of why this convention arose historically, though I admit I don't have citations to a rigorous historical survey :-)
This is not a history of quartiles in general, just a brief note on why quartiles are the default in so many of today's textbooks and software.

I've heard from older statisticians and statistics educators that quartiles, boxplots, and IQRs were popularized by Tukey's influential Exploratory Data Analysis book in the 1970s, though the ideas themselves are older.

Before modern computers were ubiquitous, people often wanted a quick & simple way to start exploring their (often small) datasets by hand.
(As Nick Cox points out, various mechanical and electrical calculators have existed for a long time, but they were not always as cheap and accessible as they are today. So there was a niche for data summaries which require almost no calculations, just sorting the data and simple arithmetic with 2 numbers at a time.)

Tukey and others recommended quartiles, IQRs, and boxplots because:

  1. Finding the quartiles by hand is really easy for a not-too-large dataset: Just sort the values, find the median, and then find the median of each half. Octiles and other powers of 2 would be easy too, but quartiles get you far enough for initial data exploration: They provide a (very) rough outline of the data's shape very quickly, and they're easy to display as a boxplot.

  2. Once you have the quartiles, the IQR is a decent way to define the "spread" of the data. It can be sensibly used to compare the spreads of different groups in side-by-side boxplots. It's much more robust to individual outliers than the range is, but doesn't require all the tedious calculations needed to find the standard deviation. Besides the computational effort itself, the SD calculations can be error-prone if doing them by pencil-and-paper or with a hand calculator, while finding the quartiles and subtracting them to get the IQR offers less room for calculation errors to arise. (Also, if your data are approximately Normally distributed, there's a simple conversion: the IQR is approximately 1.35 standard deviations.)

So the story I've heard is that Tukey's book popularized the use of quartiles, boxplots, and IQRs. They became part of the standard Intro Statistics textbooks and curricula, which in turn led to demand for reporting them in statistical software as modern computers grew more ubiquitous.

But nowadays, simple hand calculations are no longer a pragmatic skill; everybody has a computer at their desk which can do the calculations much faster & more accurately than you could sort the data by hand. So there is no need to stick with quartile-based defaults if something else works better for your situation.

And certainly do not blindly throw away data outside the 1st and 3rd quartiles as "outliers"!

$\endgroup$
6
  • 1
    $\begingroup$ I am not clear what precisely you're asserting as history, or what you're adding to existing answers, but quartiles as summary statistics go back at least to the 19th century, as do deciles and percentiles and some others. There is a tenuous connection to the idea of probable error, a summary of deviations such that half are less and half are more in absolute value. People often used to cite estimate $\pm$ PE, a practice that faded away some decades ago. The link in my first comment on the question gives details on first known use of terminology. $\endgroup$
    – Nick Cox
    May 15 at 7:45
  • $\begingroup$ "By hand" here rather skates over several decades in which people used several generations of mechanical and electrical calculators. Numerical analysis grew out of all sorts of tricks and algorithms for such machines. $\endgroup$
    – Nick Cox
    May 15 at 8:51
  • $\begingroup$ Of course you are right that quartiles are a very old idea, and that people have used calculation aids for a long time too. I merely meant to address the current widespread reporting of quartiles specifically rather than other percentiles, since that's what the OP appears to be asking about. I'll clarify my answer. $\endgroup$
    – civilstat
    May 15 at 14:17
  • 1
    $\begingroup$ Tukey certainly gave the ideas a boost, but I wouldn't over-emphasise his role. There is an unbroken tradition starting from Galton of using quartiles and other quantiles as a basis for statistical summary and graphics, including Bowley, Yule and others in much used texts and repeated re-inventions of predecessors of box plots avant la lettre in the 1930s, 1940s and 1950s. Unfortunately comments on the history are often quite unhistorical and usually reflect some myth or meme rather than detailed knowledge of the literature. $\endgroup$
    – Nick Cox
    May 15 at 15:56
  • 1
    $\begingroup$ Thanks for the positive comments. stats.stackexchange.com/questions/369393/… has not been revised since 2018, but it's a sketch. Otherwise the thread linked in my first comment on the question details use of various quantile summaries through remarks on first known use of various quantile tems. $\endgroup$
    – Nick Cox
    2 days ago
1
$\begingroup$

Reporting IQR is not throwing out any data. Otherwise, any summary statistics would be throwing out data, e.g. the mean or a median is just one number it doesn't mean that we threw out entire sample.

IQR is simply one way to summarize the spread of a dataset in one number. Instead, you can report deciles or any number of percentiles you like.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.