15
$\begingroup$

Let $\{X_i\}_{i=1}^n$ be a family of i.i.d. random variables taking values in $[0,1]$, having a mean $\mu$ and variance $\sigma^2$. A simple confidence interval for the mean, using $\sigma$ whenever it is known, is given by $$ P( | \bar X - \mu| > \varepsilon) \le \frac{\sigma^2}{n\varepsilon^2} \le\frac{1}{n \varepsilon^2} \qquad (1). $$

Also, because $\frac{\bar X- \mu}{\sigma/\sqrt{n}}$ is asymptotically distributed as a standard normal random variable, the normal distribution is sometimes used to "construct" an approximate confidence interval.


In multiple-choice answer statistics exams, I've had to use this approximation instead of $(1)$ whenever $n \geq 30$. I've always felt very uncomfortable with this (more than you can imagine), as the approximation error is not quantified.


  • Why use the normal approximation rather than $(1)$?

  • I don't want, ever again, to blindly apply the rule $n \geq 30$. Are there good references that can support me in a refusal to do so and provide appropriate alternatives? ($(1)$ is an example of what I consider an appropriate alternative.)

Here, while $\sigma$ and $E[ |X|^3]$ are unknown, they are easily bounded.

Please note that my question is a reference request particularly about confidence intervals and therefore is distinct from the differs from the questions that were suggested as partial duplicates here and here. It is not answered there.

$\endgroup$
  • 2
    $\begingroup$ You may have to improve the approximation found in classical references and exploit the fact that the $X_i$ are in $(0, 1)$ which as you noticed gives information about the moments. The magical tool, I believe, will be the Berry–Esseen theorem! $\endgroup$ – Yves Mar 20 '17 at 10:14
  • 1
    $\begingroup$ with those bounds, variance can't be greater than 0.25, much better than 1, isn't it? $\endgroup$ – carlo Mar 21 '17 at 23:28
3
$\begingroup$

Why use normal approximation?

It's as simple as saying that it's always better to use more information than less. The equation (1) uses Chebyshev's theorem. Note, how it doesn't use any information about your distribution's shape, i.e. it works for any distribution with a given variance. Hence, if you use some information about your distribution's shape you must get a better approximation. If you knew that your distribution is Gaussian, then by using this knowledge you get a better estimate.

Since, you're already applying the central limit theorem, why not use the Gaussian approximation of the bounds? They're going to be better, actually, tighter (or sharper) because these estimates are based on the knowledge of the shape which is an additional piece of information.

The rule of thumb 30 is a myth, which benefits from the confirmation bias. It just keeps being copied from one book to another. Once I found a reference suggesting this rule in a paper in 1950s. It wasn't any kind of solid proof, as I recall. It was some sort of empirical study. Basically, the only reason it's used is because it sort of works. You don't see it violated badly often.

UPDATE Look up the paper by Zachary R. Smith and Craig S. Wells "Central Limit Theorem and Sample Size". They present an empirical study of the convergence to CLT for different kinds of distributions. The magic number 30 doesn't work in many cases, of course.

$\endgroup$
  • $\begingroup$ +1 For a sensible explanation. But isn't there a risk of using information that is not quite right? The CLT doesn't say anything about the distribution of $\bar X $ for a fixed $n$. $\endgroup$ – Olivier Mar 23 '17 at 22:11
  • $\begingroup$ right, CLT doesn't say anything about the distribution of finite sample, but so don't any asympthotic equations. However, undeniably they have useful information, that's why limiting relationships are used everywhere. The problem with Chebyshev's is that it's so wide that it's rarely used outside the classroom. For instance, for one standard deviation the probability it gives is $<1/k^2=1$ - hardly practical information $\endgroup$ – Aksakal Mar 24 '17 at 0:05
  • $\begingroup$ Yet for $X$ taking the values 0 or 1 with equal probability, your application of Chebyshev is sharp. ;) The problem is that Chebyshev, applied to a sample mean, will never stay sharp as $n$ grows. $\endgroup$ – Olivier Mar 24 '17 at 1:27
  • $\begingroup$ I dunno about Smith and Wells's paper, I tried reproducing it in R and couldn't recover their conclusions... $\endgroup$ – Alex Nelson Jun 18 at 19:09
9
+50
$\begingroup$

The issue with using the Chebyshev inequality to obtain an interval for the true value, is that it only gives you a lower bound for the probability, which moreover is sometimes trivial, or, in order not to be trivial, it may give a very wide confidence interval. We have

$$P( | \bar X - \mu| > \varepsilon) = 1 - P(\bar X-\varepsilon \leq \mu \leq \bar X+\varepsilon)$$

$$\implies P(\bar X-\varepsilon \leq \mu \leq \bar X+\varepsilon) \geq 1- \frac{1}{n \varepsilon^2}$$

We see that, depending also on sample size, if we decrease $\varepsilon$ "too much" we will get the trivial answer "the probability is greater than zero".

Apart from that, what we get from this approach is a conclusion of the form ""the probability of $\mu$ falling in $[\bar X \pm \varepsilon]$ is equal or greater than..."

But let's assume that we're good with this, and denote $p_{min}$ the minimum probability with which we are comfortable. So we want

$$ 1- \frac{1}{n \varepsilon^2} = p_{min} \implies \varepsilon = \sqrt {\frac {1}{(1-p_{min})n}}$$

With small sample sizes and high desired minimum probability, this may give an unsatisfactorily wide confidence interval. E.g. for $p_{min} =0.9$ and $n=100$ we will get $\varepsilon \approx .316$, which, for example for the variable treated by the OP that is bounded in $[0,1]$ appears to be too big to be useful.

But the approach is valid, and distribution-free, and so there may be instances where it can be useful.

One may want to check also the Vysochanskij–Petunin inequality mentioned in another answer, which holds for continuous unimodal distributions and refines Chebyshev's inequality.

$\endgroup$
  • $\begingroup$ I don't agree that a problem with Chebychev it that it only gives a lower bound for the probability. In a distribution-free setting, a lower bound is the best we can hope for. The important questions are: is Chebychev sharp? Is the Chebychev C.I.'s length systematically over-estimated for a fixed level $\alpha$? I answered this in my post, from a particular point of view. However, I'm still trying to understand if Chebychev for a sample mean will always fail to be sharp, in a stronger sense. $\endgroup$ – Olivier Mar 22 '17 at 22:42
  • $\begingroup$ The length of the CI is not under estimation, since there does not exists some single unknown length, so I am not sure what you mean by using the word "over-estimation" here. Different methods provide different CI's, which then of course we can attempt to evaluate and assess them. $\endgroup$ – Alecos Papadopoulos Mar 23 '17 at 0:39
  • $\begingroup$ Over-estimation was a bad choice of words, thanks for pointing it out. By "systematically over-estimated lenght" I meant that the method for obtaining a C.I. always yields something larger than necessary. $\endgroup$ – Olivier Mar 23 '17 at 0:48
  • 1
    $\begingroup$ @Olivier Generally speaking, the Chebyshev Inequality is known to be a loose inequality, and so used more as a tool in theoretical derivations and proofs rather than in applied work. $\endgroup$ – Alecos Papadopoulos Mar 23 '17 at 1:03
  • 2
    $\begingroup$ @Olivier "Generally speaking" covers your qualification, I would say. $\endgroup$ – Alecos Papadopoulos Mar 23 '17 at 13:20
7
$\begingroup$

The short answer is that it can go pretty badly, but only if one or both tails of the sampling distribution is really fat.

This R code generate a million sets of 30 gamma-distributed variables and take their mean; it can be used to get a sense of what the sampling distribution of the mean looks like. If the normal approximation works as intended, the results should be approximately normal with mean 1 and variance 1/(30 * shape).

f = function(shape){replicate(1E6, mean(rgamma(30, shape, shape)))}

When shape is 1.0, the gamma distribution becomes an exponential distribution, which is pretty non-normal. Nevertheless, the non-Gaussian parts mostly average out and so Gaussian approximation isn't so bad:

histogram & density plot

There's clearly some bias, and it would be good to avoid that when possible. But honestly, that level of bias probably won't be the biggest problem facing a typical study.

That said, things can get much worse. With f(0.01), the histogram looks like this:

histogram

Log-transforming the 30 sampled data points before averaging helps a lot, though:

histogram

In general, distributions with long tails (on one or both sides of the distribution) will require the most samples before the Gaussian approximation starts to become reliable. There are even pathological cases where there will literally never be enough data for the Gaussian approximation to work, but you'll probably have more serious problems in that case (because the sampling distribution doesn't have a well-defined mean or variance to begin with).

$\endgroup$
  • $\begingroup$ I find the experiment very pertinent and interesting. I won't take this as the answer, however, as it does not address the crux of the problem. $\endgroup$ – Olivier Mar 15 '17 at 3:38
  • 1
    $\begingroup$ what's the crux? $\endgroup$ – David J. Harris Mar 15 '17 at 5:03
  • $\begingroup$ Your answer does not provide rigorous footing for sound statistical practice. It only gives examples. Note, also, that the random variables I consider are bounded, greatly changing what is the worst possible case. $\endgroup$ – Olivier Mar 15 '17 at 21:31
  • $\begingroup$ @Glen_b: this answer isn't so relevant to your revised version of the question. Should I just leave it here, or would you recommend something else? $\endgroup$ – David J. Harris Mar 22 '17 at 0:02
3
$\begingroup$

Problem with the Chebyshev confidence interval

As mentioned by Carlo, we have $\sigma^2 \le \frac{1}{4}$. This follows from $\text{Var}(X) \le \mu(1-\mu)$. Therefore a confidence interval for $\mu$ is given by $$ P(|\bar{X}-\mu| \geq \varepsilon) \le \frac{1}{4n\varepsilon^2}. $$ The problem is that the inequality is, in a certain sense, quite loose when $n$ gets large. An improvement is given by Hoeffding's bound and shown below. However, we can also demonstrate how bad it can get using the Berry-Esseen theorem, pointed out by Yves. Let $X_i$ have a variance $\tfrac{1}{4}$, the worst possible case. The theorem implies that $ P(|\bar X - \mu| \geq \tfrac{\varepsilon}{2\sqrt{n}}) \le 2\, \text{SF}(\varepsilon) + \tfrac{8}{\sqrt{n}}, $ where $\text{SF}$ is the survival function of the standard normal distribution. In particular, with $\varepsilon = 16$, we get $\text{SF}(16) \approx e^{-58}$ (according to Scipy), so that essentially $$ P(|\bar X - \mu| \geq \tfrac{8}{\sqrt{n}}) \le \tfrac{8}{\sqrt{n}} + 0, \qquad (*) $$ whereas the Chebyshev inequality implies $$ P(|\bar X - \mu| \geq \tfrac{8}{\sqrt{n}}) \le \tfrac{1}{256}. $$ Note that I did not try to optimize the bound given in $(*)$, the result here is only of conceptual interest.


Comparing the lengths of the confidence intervals

Consider the $(1-\alpha)$-level confidence interval lengths $\ell_Z(\alpha, n)$ and $\ell_C(\alpha, n)$ obtained using the normal approximation ($\sigma = \tfrac{1}{2}$) and the Chebyshev inequality, repectively. It turns out that $\ell_C(\alpha, n)$ is a constant times bigger than $\ell_Z(\alpha, n)$, independently of $n$. Precisely, for all $n$, $$ \ell_C(\alpha, n) = \kappa(\alpha) \ell_Z(\alpha, n), \quad \kappa(\alpha) = \left(\text{ISF}\left(\tfrac{\alpha}{2}\right) \sqrt{\alpha}\right)^{-1}, $$ where $\text{ISF}$ is the inverse survival function of the standard normal distribution. I plot below the multiplicative constant.

$\hskip 1in$enter image description here

In particular, the $95\%$ level confidence interval obtained using the Chebyshev inequality is about $2.3$ times bigger than the same level confidence interval obtained using the normal approximation.


Using Hoeffding's bound

Hoeffding's bound gives $$ P(|\bar X - \mu| \geq \varepsilon) \leq 2e^{-2n \varepsilon^2}. $$ Thus an $(1-\alpha)$-level confidence interval for $\mu$ is $$ (\bar X - \varepsilon, \bar X + \varepsilon), \quad \varepsilon = \sqrt{\frac{-\ln \tfrac{\alpha}{2}}{2n}}, $$ of length $\ell_H (\alpha, n) = 2\varepsilon$. I plot below the lengths of the different confidence intervals (Chebyshev inequality: $\ell_C$; normal approximation ($\sigma = 1/2$): $\ell_Z$; Hoeffding's inequality: $\ell_H$) for $\alpha = 0.05$.

$\hskip 0.5in$enter image description here

$\endgroup$
  • $\begingroup$ Very interesting! I have though some corrections to suggest you toghether with a big puzzlement: first, you should take out absolute value from the Hoeffding's unequality definition, it's $ P(\bar{X}-\mu \ge \varepsilon) \le e^{-2 n \varepsilon^2} $ or $P(|\bar{X}-\mu| \ge \varepsilon) \le 2 e^{-2 n \varepsilon^2} ; $ the second correction is less important, $\alpha$ is generally taken to be 0.05 or lower, while 0.95 is addressed as $1-\alpha,$ it's a bit confusing to see them switched in your post. $\endgroup$ – carlo Mar 23 '17 at 17:10
  • $\begingroup$ Last and more important: I found your result incredible, so I tried to replicate it in R and I got a completely opposite result: normal approximation gives smaller confidence intervals to me! this is the code I used: curve(sqrt(-log(.025)/2/x), to= 100, col= 'red', xlab= 'n', ylab= 'half interval') #Hoeffding ; curve(qnorm(.975, 0, .5/sqrt(x)), to= 100, add= T, col= 'darkgreen') #normal approximation $\endgroup$ – carlo Mar 23 '17 at 17:13
0
$\begingroup$

let's start with the number 30: it's, as anyone will say, a rule of thumb. but how can we find a number that fits better to our data? It's actually mostly a matter of skewness: even the strangest distribution will fast converge to normal if they are simmetric and continuous, skewed data will be much slower. I remember learning that a binomial distribution can be properly approximated to normal when its variance is greater than 9; for this example it's to be considered that discrete distribution also have the problem that they need great numbers to simulate continuity, but think to this: a simmetric binomial distribution will reach that variance with n = 36, if p = 0.1 instead, n must go up to 100 (variabile trasformation, however, would help a lot)!

If you only want to use variance instead, dropping gaussian approximation, consider Vysochanskij–Petunin inequality over Chebichev's, it needs the assumption of unimodal distribution of the mean, but this is a very safe one with any sample size, I'd say, greater than 2.

$\endgroup$
  • $\begingroup$ Could you add a reference for " Vysochanskij–Petunin inequality "? Never heard of it! $\endgroup$ – kjetil b halvorsen Mar 21 '17 at 19:08
  • $\begingroup$ wikipedia docet $\endgroup$ – carlo Mar 21 '17 at 19:08
  • $\begingroup$ Can you express the rate of convergence in terms of the skewdness? Why is a sample size of, you'd say 2, enough for unimodality? How is the Vysochanskij–Petunin inequality an improvement over Chebychev if you need to double or triple the sample size for it to apply? $\endgroup$ – Olivier Mar 22 '17 at 13:10
  • $\begingroup$ I made a fast google search and I found out that binomial distribution is actually often used to explain different sample size need for skewed data, but I didn't find, and I guess there is no accepted "rate of convergence in terms of the skewdness". $\endgroup$ – carlo Mar 22 '17 at 14:23
  • $\begingroup$ Vysochanskij–Petunin inequality is more efficent than Chebychev's, so it doesn't need a greater sample at all, but it has some use constraints: first, you have to have a continuous distribution, than, it has to be unimodal (no local modes are allowed). It may seem strange to drop normality assumption to adopt another one, but if your data is not discrete, sample mean should eliminate local modes even with very small samples. Fact is that mean has much of a bell distribution and, also if it can be skewed or have fat tails, it quickly comes to only have one mode. $\endgroup$ – carlo Mar 22 '17 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.