2
$\begingroup$

I am wondering if I can use the box plot to explain the Empirical Rule for a normal.distribution. I don't know if the Empirical Rule can be explaines using the box plot or not, but I just looked at some materials on the Internet and found that there is some relationship between the box plot and nor distributions. Hope to hear some explanations.

$\endgroup$
6
  • 1
    $\begingroup$ John Tukey partially justified his formulas for the fences in a boxplot by computing the rates at which outliers and far outliers would be expected for a Normal distribution. $\endgroup$
    – whuber
    Commented Oct 26, 2020 at 21:35
  • $\begingroup$ Why is 1.5 multiplied for the whiskers? $\endgroup$ Commented Oct 26, 2020 at 21:55
  • 1
    $\begingroup$ It's a relatively simple multiple and it results in an expected outlier rate of just under 1% for Normal distributions. If it were changed to 2.0, the rate would drop to 0.07% and if it were set only at 1.0, the rate would soar to over 4%. Since Tukey used boxplots to analyze smallish batches of data (comprising five to a few hundred values), a rate of 1% would tend not to produce outliers or, for the largest batches, only a handful of outliers -- at least for Normally distributed data. $\endgroup$
    – whuber
    Commented Oct 26, 2020 at 22:04
  • $\begingroup$ Just under 1% of the ...? What do you by that? $\endgroup$ Commented Oct 26, 2020 at 22:06
  • $\begingroup$ My computation predicts about 1.4 outliers in a normal sample of size 200. My simulation, just now added, shows an average of about 1.6 outliers in normal sample of size 200. Either way, you might say that's a little 'under 1%' of the observations. $\endgroup$
    – BruceET
    Commented Oct 26, 2020 at 22:40

1 Answer 1

1
$\begingroup$

The Empirical Rule with which I am familiar looks at sample means and standard deviations. (For example, the interval $\bar X \pm S$ often contains about 68% of the sample elements.)

Boxplots look at medians and interquartile ranges. Of course, a boxplot is constructed so that (nearly) 50% of the data is 'contained' within the box.

The ER is often said to apply to "mound shaped" (i.e., roughly normal) samples, as you suggest. Boxplots are often used to explore samples suspected of not coming from normal distributions.

So it does not seem an explanation of the ER in terms of boxplots would be straightforward. If you have some particular idea in mind, please explain it in more detail.


Addendum on per Comments.

Computing normal probabilities. If you look at a standard normal curve and the probabilities of intervals with endpoints $=3,-2,-1,0,1,2,3,$ then you get $P(Z \le -3) = 0.0013,\,$ $P(Z \le -2) = 0.0228,\,$ $\dots\,, P(Z \le 3) = 0.9987.$

See the computations in R below. You can get essentially the same values from a printed table of the standard normal PDF.

round(pnorm(-3:3), 4)
[1] 0.0013 0.0228 0.1587 0.5000 0.8413 0.9772 0.9987

Then, with a little arithmetic, you can get

  • $P(-1 \le Z < 1) = 0.8413 - 0.1587 = 0.6826 \approx 68\%,$
  • $P(-2 \le Z < 2) = 0.9772 - 0.0228 = 0.9544 \approx 95\%,$
  • $P(-3 \le Z < 3) = 0.9987 - 0.0013 = 0.9974 \approx 100\%,$

as in the Empirical Rule.

Here is a plot of a standard normal distribution with blue vertical lines at $\pm 1$ (one standard deviation from the mean $0,$ brown vertical lines at $\pm 2 (\mu\pm 2)$ and red vertical lines at $\pm 3.$

curve(dnorm(x), -4, 4, lwd=2, ylab="PDF", 
      main="Standard Normal Density Function")
 abline(v=0, col="green2"); abline(h=0, col="green2")
  abline(v=c(-1,1), col="blue")
  abline(v=c(-2,2), col="brown")
  abline(v=c(-3,3), col="red")

enter image description here

The colored lines divide the area under the standard normal density curve into eight regions. From left to right the areas are as follows: $$0.0013, 0.0215, 0.1359, 0.3413, 0.3413, 0.1359, 0.0215, 0.0013,$$ If you paste these areas together properly, you will get the probabilties mentioned in the Empirical rule. For example, the area between the two blue lines is $0.3413+0,3413 = 0.6826,$ the area between the two brown lines is $0.1359 + 0.3413 +0.3413 +0.1359 = 0.9544,$ etc.

diff(round(pnorm(-3:3), 4))
[1] 0.0215 0.1359 0.3413 0.3413 0.1359 0.0215

For samples of at least moderate size for a normal distribution approximately the same proportions apply. For example, a sample of size $n = 200$ from $\mathsf{Norm}(\mu = 100, 15),$ you would expect to see about $2(68)=126$ observations in the interval $[85,115].$ Such a sample in R shows $74 + 55 = 129,$ which is not far from the approximate value from the Empirical rule (where the last probability is sometimes given as "all or almost all".)

set.seed(2020)
x = rnorm(200, 100, 15)
summary(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  54.15   89.03  101.01   99.95  110.91  148.02 

cp = seq(40, 160, by=15)
hist(x, br=cp, col="skyblue2", label=T, ylim=c(0,100))

enter image description here

Frequency of outliers in boxplots. Here is a brief discussion of a somewhat similar approach for boxplot outliers. Consider again the distribution $\mathsf{Norm}(100, 15).$ Its lower and upper quartiles are $89.883$ and $110.117,$ respectively. So the IQR of the distribution is $20.235.$

qtl = qnorm(c(.25,.75), 100, 15)
qtl; diff(qtl)
[1]  89.88265 110.11735
[1] 20.23469

Then according to the "1.5IQR rule" for declaring outliers, we can predict that the lower and upper fences (below and above which observations are considered outliers) are anticipated to be $59.53$ and $140.47,$ respectively. Consequently, we might anticipate $99.3\%$ of a sample from this distribution will not be outliers.

fnc = c(89.883-1.5*20.235, 110.117+1.5*20.235)
fnc
[1]  59.5305 140.4695
diff(pnorm(fnc, 100, 15))
[1] 0.9930236

So in the sample of size $n=200$ observations x above from this distribution we might expect about $.007(200) = 1.4$ outliers. Actually, there are exactly three outliers in our sample.

boxplot.stats(x)$out
[1]  54.41853 148.02448  54.14975

A simulation in R can show the average number of outliers seen in 100,000 samples of size $n=200$ from this distribution is about $1.63,$ as follows:

set.seed(1026)
nr.out = replicate(10^5, length(boxplot.stats(rnorm(200, 150, 15))$out))
mean(nr.out)
[1] 1.62818
summary(nr.out)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  0.000   0.000   1.000   1.628   2.000  14.000 
$\endgroup$
6
  • $\begingroup$ Thank you for the answer. While I am looking on the Internet, I have found this blog and it explains the ER using the integration. It says that the probabity is 68% between -1 and +1 standard deviation and also data values lie within the range but I don't really get it. I know that the integration means the probability for the fixed range but the blog also says that there are 68% of the data values. How come it shows the intefration and says that there 68% of the data values? towardsdatascience.com/… $\endgroup$ Commented Oct 26, 2020 at 17:47
  • $\begingroup$ 'Integration' for normal probabilities has to be done by some sort of approximation (or simulation) because the CDF of a standard normal distribution cannot be written closed form for use with ordinary integration. That is why we have printed tables of normal CDF and computer procedures to get normal probabilities (The function pnorm in R, used in my Addendum, is based on a very accurate 'rational approximation' to the standard normal CDF.) Possibly relevant link $\endgroup$
    – BruceET
    Commented Oct 26, 2020 at 21:47
  • $\begingroup$ I don't know why it does not come straight to my head. I feel that I understand it half way but I don't know what makes still confused about it. Anyway, thanks for the appendum and I will thank more about it. $\endgroup$ Commented Oct 27, 2020 at 0:15
  • $\begingroup$ What is the first thing in the half you don't understand? $\endgroup$
    – BruceET
    Commented Oct 27, 2020 at 5:20
  • $\begingroup$ What I am realy confused about is rhat how come the probability means the number of observations in the normal and standard normal distributions. I didn't read your additional writing yet. $\endgroup$ Commented Oct 27, 2020 at 6:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.