4
$\begingroup$

My classmate told me he was showing his work in some stuff statistics-based and some time he was showing a boxplot and using it as outlier detection then his professor said 'it's not even correct, the 1.5 from classic outlier formula comes up due to normal data' the classic formula is:

$IC=(Q_1-1.5.IQR, Q_3+1.5.IQR)$ where $IQR=Q_3-Q_1$ and $Q_i$ as $i-th$ quartile.

So all numbers which don't lie in this interval is go as an outlier.

The I get interested where outlier formula come from and why does that assume normality, this image below is as far as I could go in my internet searching:

enter image description here

Finally my question is Has actually outlier classic evaluating come from normal distribution? , and if so where are the scientific works which shows a proof for that?, or Has there been some incorrect stuff which I've been told on conversation with my classmate?

$\endgroup$
8
$\begingroup$

Davies and Gather argue in this paper (I think correctly)... https://www.jstor.org/stable/2290763 ...that any definition of outliers has to be made with respect to a reference distribution. Outliers qualify as "outlying" by being further away from the main bulk of the data than what is to be expected, but this depends on what one expects, or in other words, what distributional assumption one makes for this "main bulk". The normal distribution is the default choice not only for historical reasons and because of the CLT, but also because it models "homogeneity" in the sense that on one hand observations in some distance from the centre can occur (which is very often the case in reality), but on the other hand, observations that have a strong gap between themselves and the other observations are extremely unlikely. That is, observations that intuitively qualify as outliers will not normally occur under the normal distribution, which therefore serves to formalise the "expectation" against which outliers are identified. This in particular means that the normal assumption is not made for the data set as a whole, but only for the non-outliers (in fact some people identify outliers based on mean and standard deviation computed on all data, which would assume normality overall and is not a good idea for that reason, however using robust statistics such as IQR or MAD will not (or hardly) be affected by outliers that are not in line with normality).

This makes sense as a default choice, but there are situations in which other distributions could serve as reference for data without outliers, such as the exponential or Poisson distribution for skew and count data. Heavy tailed distributions such as the Cauchy are usually inappropriate though, because they imply already a substantial probability to observe points that intuitively should be treated as outliers. But then, if there is a genuinely heavy-tailed real process, outlier identification based on the normal distribution may often identify perfectly fine observations as outliers.

$\endgroup$
12
$\begingroup$

I won't speak for "classical outlier detection" (methods), because you did not specify them exactly, but as far as the IQR (rule) is concerned, it has no assumptions of normality. In fact one of the main points of this method is that it is robust, since it makes no use of means, which would be an issue in case of extreme values ("outliers").

So to answer your specific question about IQR (rule), no, there is no such assumption. Whether an IQR can be used to detect "outliers" is a very different question, since you may notice that the IQR (rule) is... quite arbitrary.

$\endgroup$
11
  • 1
    $\begingroup$ What about this 1.5? $\endgroup$ Dec 3 '21 at 7:32
  • 9
    $\begingroup$ This. John W. Tukey proposed lying more than 1.5 IQR away from the nearer quartile as a rule of thumb for deciding which data points should be plotted separately on a box plot. The best reference for this is his Exploratory Data Analysis (Addison-Wesley, Reading, MA, 1977). When asked why 1.5 he replied that 1 is too small and 2 is too large. In earlier work he experimented with various rules. Tukey was not in favour of discarding outliers and indeed he emphasised very strongly the use of transformations (which often subdue apparent outliers) and robust-resistant methods . $\endgroup$
    – Nick Cox
    Dec 3 '21 at 8:50
  • 7
    $\begingroup$ (ctd) It is especially curious that this rule or convention for the details of box plot construction has somehow been twisted into a hard criterion for what is an outlier (and even further sometimes as a rule for rejection from a dataset). Can anyone supply authoritative references explaining and justifying the latter practice? $\endgroup$
    – Nick Cox
    Dec 3 '21 at 8:53
  • 2
    $\begingroup$ @Christian In another work (I can't remember which), Tukey partially justified his rules by pointing out that a boxplot outlier is quite rare when the distribution is Normal--a fraction of a percent chance or so. Note two important caveats: (1) this was to placate people used to comparing everything to the Normal distribution and (2) Tukey used multiple rules, defining "outliers" and "far outliers" as data falling beyond unplotted "fences." There's a clear diffidence in both the approach and the terminology that belies a practical, flexible approach to outlier identification. $\endgroup$
    – whuber
    Dec 3 '21 at 14:48
  • 2
    $\begingroup$ I agree. Indeed, taking the superficially trite but really deep idea from Venables and Ripley that an outlier is surprising, only an explanation of what would not be surprising can take that further. One of many examples is that changing your guess from normal to lognormal can make an outlier seem standard. A more general point is that a symmetric rule like 1.5 IQR does imply a symmetric reference. $\endgroup$
    – Nick Cox
    Dec 3 '21 at 14:51
0
$\begingroup$

If you thought theoretically like a physicist (also STEM), you might understand why the normal distribution is perfectly fine for outlier determination. Physicists, for example, know that the Central Limit Theorem states that, for a histogram of many averages (means) from many samples:

  • The distribution of sample means approximates a normal distribution as the sample size gets larger, regardless of the population's distribution.
  • Sample sizes equal to or greater than 30 are often considered sufficient for the CLT to hold.
  • A key aspect of CLT is that the average of the sample means and standard deviations will equal the population mean and standard deviation.
  • A sufficiently large sample size can predict the characteristics of a population more accurately.

Also, the Law of Large numbers states that:

  • as a sample size grows, its mean gets closer to the average of the whole population.

In light of the above, physicists commonly assume all data are normally distributed because of the assumption that all sample sizes can be assumed, initially, to be infinitely large $(n=\infty)$.

Now, when looking at the normal distribution, it is known that 95% of the data fall within $\pm$2 standard deviations from the mean. Therefore, if a single observation (or measurement) has a Z-score that exceeds $\pm 2 \sigma$, the observation is assumed to be an outlier. Your IQR values are not needed for such a rule of thumb.

The Z-score for an $i$th observation is equal to

\begin{equation} Z_i=\frac{(X_i - \mu)}{\sigma}, \end{equation}

where $X_i$ is the measurement value, $\mu$ is the average, and $\sigma$ is the standard deviation of the sample that $X_i$ is from. The mean of all $Z_i$ values for all the observations is zero, and variance (and standard deviation) is equal to one, i.e. $\mathcal{N}(0,1)$. The distribution of Zs is called the standard normal distribution.

UPDATE

Regarding the CLT, the simulation plots below show the CLT in action for numerous random draws of samples of size $n$=2, 4, or 8 from the uniform(0,1) or triangle(10,20,50) distribution. For example, at $n=2$, a sample of 2 values are randomly drawn from the upper left distribution. The sample mean $\bar{x}_1$, is then calculated, and this is repeated over and over again to yield $\bar{x}_1, \bar{x}_2,\ldots,\bar{x}_m$. Next, a histogram of all the means $\bar{x}$ is then generated. As you can see, when the sample size of random draws is n=32, the shape of the distribution becomes normally distributed, even when the parent distribution was not normal.

enter image description here

enter image description here

enter image description here

$\endgroup$
3
  • 6
    $\begingroup$ "physicists commonly assume all data are normally distributed" is a very strong statement! It sounds dangerously close to "every sequence of IID random variables has a normal distribution", which is of course false (see stats.stackexchange.com/a/473596/9330) $\endgroup$
    – Adrian
    Dec 3 '21 at 5:32
  • 1
    $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – Sycorax
    Dec 4 '21 at 0:16
  • 2
    $\begingroup$ Boxplots are almost always used to plot data rather than sampling distributions of statistics. Introducing the CLT in this context--where it is irrelevant--seems destined to mislead the many people who do not fully grasp the CLT. $\endgroup$
    – whuber
    Dec 4 '21 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.