4
$\begingroup$

My classmate told me he was showing his work in some stuff statistics-based and some time he was showing a boxplot and using it as outlier detection then his professor said 'it's not even correct, the 1.5 from classic outlier formula comes up due to normal data' the classic formula is:

$IC=(Q_1-1.5.IQR, Q_3+1.5.IQR)$ where $IQR=Q_3-Q_1$ and $Q_i$ as $i-th$ quartile.

So all numbers which don't lie in this interval is go as an outlier.

The I get interested where outlier formula come from and why does that assume normality, this image below is as far as I could go in my internet searching:

enter image description here

Finally my question is Has actually outlier classic evaluating come from normal distribution? , and if so where are the scientific works which shows a proof for that?, or Has there been some incorrect stuff which I've been told on conversation with my classmate?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
8
$\begingroup$

Davies and Gather argue in this paper (I think correctly)... https://www.jstor.org/stable/2290763 ...that any definition of outliers has to be made with respect to a reference distribution. Outliers qualify as "outlying" by being further away from the main bulk of the data than what is to be expected, but this depends on what one expects, or in other words, what distributional assumption one makes for this "main bulk". The normal distribution is the default choice not only for historical reasons and because of the CLT, but also because it models "homogeneity" in the sense that on one hand observations in some distance from the centre can occur (which is very often the case in reality), but on the other hand, observations that have a strong gap between themselves and the other observations are extremely unlikely. That is, observations that intuitively qualify as outliers will not normally occur under the normal distribution, which therefore serves to formalise the "expectation" against which outliers are identified. This in particular means that the normal assumption is not made for the data set as a whole, but only for the non-outliers (in fact some people identify outliers based on mean and standard deviation computed on all data, which would assume normality overall and is not a good idea for that reason, however using robust statistics such as IQR or MAD will not (or hardly) be affected by outliers that are not in line with normality).

This makes sense as a default choice, but there are situations in which other distributions could serve as reference for data without outliers, such as the exponential or Poisson distribution for skew and count data. Heavy tailed distributions such as the Cauchy are usually inappropriate though, because they imply already a substantial probability to observe points that intuitively should be treated as outliers. But then, if there is a genuinely heavy-tailed real process, outlier identification based on the normal distribution may often identify perfectly fine observations as outliers.

Improve this answer
$\endgroup$
13
$\begingroup$

I won't speak for "classical outlier detection" (methods), because you did not specify them exactly, but as far as the IQR (rule) is concerned, it has no assumptions of normality. In fact one of the main points of this method is that it is robust, since it makes no use of means, which would be an issue in case of extreme values ("outliers").

So to answer your specific question about IQR (rule), no, there is no such assumption. Whether an IQR can be used to detect "outliers" is a very different question, since you may notice that the IQR (rule) is... quite arbitrary.

Improve this answer
$\endgroup$
11
  • 1
    $\begingroup$ What about this 1.5? $\endgroup$
    – Davi Américo
    Commented Dec 3, 2021 at 7:32
  • 9
    $\begingroup$ This. John W. Tukey proposed lying more than 1.5 IQR away from the nearer quartile as a rule of thumb for deciding which data points should be plotted separately on a box plot. The best reference for this is his Exploratory Data Analysis (Addison-Wesley, Reading, MA, 1977). When asked why 1.5 he replied that 1 is too small and 2 is too large. In earlier work he experimented with various rules. Tukey was not in favour of discarding outliers and indeed he emphasised very strongly the use of transformations (which often subdue apparent outliers) and robust-resistant methods . $\endgroup$
    – Nick Cox
    Commented Dec 3, 2021 at 8:50
  • 7
    $\begingroup$ (ctd) It is especially curious that this rule or convention for the details of box plot construction has somehow been twisted into a hard criterion for what is an outlier (and even further sometimes as a rule for rejection from a dataset). Can anyone supply authoritative references explaining and justifying the latter practice? $\endgroup$
    – Nick Cox
    Commented Dec 3, 2021 at 8:53
  • 2
    $\begingroup$ @Christian In another work (I can't remember which), Tukey partially justified his rules by pointing out that a boxplot outlier is quite rare when the distribution is Normal--a fraction of a percent chance or so. Note two important caveats: (1) this was to placate people used to comparing everything to the Normal distribution and (2) Tukey used multiple rules, defining "outliers" and "far outliers" as data falling beyond unplotted "fences." There's a clear diffidence in both the approach and the terminology that belies a practical, flexible approach to outlier identification. $\endgroup$
    – whuber ♦
    Commented Dec 3, 2021 at 14:48
  • 3
    $\begingroup$ I agree. Indeed, taking the superficially trite but really deep idea from Venables and Ripley that an outlier is surprising, only an explanation of what would not be surprising can take that further. One of many examples is that changing your guess from normal to lognormal can make an outlier seem standard. A more general point is that a symmetric rule like 1.5 IQR does imply a symmetric reference. $\endgroup$
    – Nick Cox
    Commented Dec 3, 2021 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.