As I understand it the IQR specifies the dispersion of the data by taking the difference between Q3 and Q1 (i.e., the range that the middle 50% of the data lies), while std specifies dispersion using all the data, to where 68% of the data belong. Does this imply that the std is always larger than (or equal to?) iqr?
-
3$\begingroup$ Where does this 68% figure come from? Regardless, consider a Binomial$(1/4+\epsilon)$ variable for a small value of $\epsilon.$ $\endgroup$– whuber ♦Dec 3, 2019 at 0:51
-
1$\begingroup$ The rule for sd corresponding to 68% of the data refers to a normal distribution, but its 2 x sd that corresponds to 68%... Is that what the question is trying to ask about? $\endgroup$– Sal MangiaficoDec 3, 2019 at 12:15
2 Answers
Your argument about $68\%$ of the density contained within $1$ standard deviation (of the mean) is true for the Normal distribution, but not in general. There are examples where the standard deviation exceeds the IQR and examples in the other direction as well.
Let $X$ have a t-distribution with $v$ degrees of freedom.
The standard deviation of $X$ is $$\sigma = \sqrt{\text{Var}(X)} = \sqrt{\frac{v}{v-2}}$$
The IQR can be found using the quantile function of the t-distribution. In R, we have
qt(.75, v) - qt(.25, v)
We have that $\sigma = IQR$ for $v\approx 3.61$. When $v < 3.61$, we have $\sigma > IQR$ and we have $\sigma < IQR$ otherwise. Since a normal distribution can be achieved in the limit as $v\rightarrow \infty$, this demonstrates that $\sigma < IQR$ for a normal distribution.
Sal considers the interesting question, can IQR be larger than $2\sigma$?. Indeed, this will occur for the symmetric Beta distribution when the shape parameter is small.
Let $X \sim Beta(\alpha, \alpha)$. Then $IQR = 2\sigma$ when $\alpha = 0.5$, $IQR > 2\sigma$ when $\alpha < 0.5$ and $IQR < 2\sigma$ otherwise.
We have established examples, where $IQR > \sigma$ and examples where $IQR > 2\sigma$. It is not possible, however, for $IQR$ to exceed $4\sigma$ when the distribution of $X$ is symmetric about its mean $\mu$. Applying Chebyshev's Inequality, we have $$P(|X-\mu| \leq 2\sigma) \geq \frac{3}{4}.$$
If the density function of $X$ is symmetric about $\mu$, we have $$P(|X-\mu| \leq \text{IQR}/2) = \frac{3}{4}.$$
This implies that $2\sigma \geq \text{IQR}/2$, or equivalently $$\text{IQR} \leq 4\sigma$$
-
1$\begingroup$ I think I'm missing something -- in your example the std is larger than the IQR. Am I misunderstanding? I was thinking about the opposite case. Moreover, how should I interpret a distribution where the IQR is larger than the std? That's really throwing me off. Clearly, it may not be normally distributed as you suggested. But could it instead mean that the distribution maybe has negative samples, which is why the iqr comes out larger? $\endgroup$ Dec 3, 2019 at 1:17
-
$\begingroup$ FWIW, the SD of the student t distribution is defined and finite for all $\nu \gt 2.$ This includes all $\nu$ in the interval $(2,3)$ that you have excluded. It also is defined and infinite for $1 \lt \nu \le 2.$ $\endgroup$– whuber ♦Dec 3, 2019 at 15:08
-
$\begingroup$ Ah, I think I see the source of your confusion. $v=3.61$ is the value of the degrees of freedom for which $\sigma = IQR$. $\endgroup$– knrumseyDec 3, 2019 at 19:13
-
2$\begingroup$ +1 That (and your edits) much clarify things! $\endgroup$– AlexisDec 3, 2019 at 21:18
Trying some toy distributions, it appears that IQR is often less than standard deviation (sd), but that IQR will be larger than sd for bimodal distributions.
If the question is supposed to be asking Is IQR ever larger than 2 times sd ?, then I have been unsuccessful finding distributions that satisfies this question. But for distributions meeting this criterion, see the answer by @knrumsey-ReinstateMonica .
A = c(1,2,99,100)
sd(A)
### 56.58327
IQR(A)
### 97.5
set.seed(12345)
B = c(rnorm(25,10,1 ), rnorm(25,15,1))
hist(B)
sd(B)
### 2.915855
IQR(B)
### 5.586538
There's also the case of some binomial distributions, such as mentioned by whuber in the comments to the original post.
set.seed(12345)
C = rbinom (100, 30, 0.26)
hist(C)
sd(C)
### 2.569125
IQR(C)
### 3