31
$\begingroup$

Wikipedia says -

In probability theory, the central limit theorem (CLT) establishes that, in most situations, when independent random variables are added, their properly normalized sum tends toward a normal distribution (informally a "bell curve") even if the original variables themselves are not normally distributed...

When it says "in most situations", in which situations does the central limit theorem not work?

$\endgroup$
32
$\begingroup$

To understand this, you need to first state a version of the Central Limit Theorem. Here's the "typical" statement of the central limit theorem:

Lindeberg–Lévy CLT. Suppose ${X_1, X_2, \dots}$ is a sequence of i.i.d. random variables with $E[X_i] = \mu$ and $Var[X_i] = \sigma^2 < \infty$. Let $S_{n}:={\frac {X_{1}+\cdots +X_{n}}{n}}$. Then as $n$ approaches infinity, the random variables $\sqrt{n}(S_n − \mu)$ converge in distribution to a normal $N(0,\sigma^2)$ i.e.

$${\displaystyle {\sqrt {n}}\left(\left({\frac {1}{n}}\sum _{i=1}^{n}X_{i}\right)-\mu \right)\ {\xrightarrow {d}}\ N\left(0,\sigma ^{2}\right).}$$

So, how does this differ from the informal description, and what are the gaps? There are several differences between your informal description and this description, some of which have been discussed in other answers, but not completely. So, we can turn this into three specific questions:

  • What happens if the variables are not identically distributed?
  • What if the variables have infinite variance, or infinite mean?
  • How important is independence?

Taking these one at a time,

Not identically distributed, The best general results are the Lindeberg and Lyaponov versions of the central limit theorem. Basically, as long as the standard deviations don't grow too wildly, you can get a decent central limit theorem out of it.

Lyapunov CLT.[5] Suppose ${X_1, X_2, \dots}$ is a sequence of independent random variables, each with finite expected value $\mu_i$ and variance $\sigma^2$ Define: $s_{n}^{2}=\sum _{i=1}^{n}\sigma _{i}^{2}$

If for some $\delta > 0$, Lyapunov’s condition ${\displaystyle \lim _{n\to \infty }{\frac {1}{s_{n}^{2+\delta }}}\sum_{i=1}^{n}\operatorname {E} \left[|X_{i}-\mu _{i}|^{2+\delta }\right]=0}$ is satisfied, then a sum of $X_i − \mu_i / s_n$ converges in distribution to a standard normal random variable, as n goes to infinity:

${{\frac {1}{s_{n}}}\sum _{i=1}^{n}\left(X_{i}-\mu_{i}\right)\ {\xrightarrow {d}}\ N(0,1).}$

Infinite Variance Theorems similar to the central limit theorem exist for variables with infinite variance, but the conditions are significantly more narrow than for the usual central limit theorem. Essentially the tail of the probability distribution must be asymptotic to $|x|^{-\alpha-1}$ for $0 < \alpha < 2$. In this case, appropriate scaled summands converge to a Levy-Alpha stable distribution.

Importance of Independence There are many different central limit theorems for non-independent sequences of $X_i$. They are all highly contextual. As Batman points out, there's one for Martingales. This question is an ongoing area of research, with many, many different variations depending upon the specific context of interest. This Question on Math Exchange is another post related to this question.

$\endgroup$
  • 2
    $\begingroup$ I have removed a stray ">" from a formula that I think has crept in because of the quoting system - feel free to reverse my edit if it was intentional! $\endgroup$ – Silverfish May 30 '18 at 18:43
  • $\begingroup$ A triangular array CLT is probably a more representative CLT than the one stated. As for not independent, martingale CLT's are reasonably commonly used case. $\endgroup$ – Batman May 30 '18 at 23:13
  • $\begingroup$ @Batman, what's an example of a triangular array CLT? Feel free to edit my response, to add it. I'm not familiar with that one. $\endgroup$ – John May 31 '18 at 20:20
  • $\begingroup$ Something like sec. 4.2.3 in personal.psu.edu/drh20/asymp/lectures/p93to100.pdf $\endgroup$ – Batman Jun 1 '18 at 1:24
  • 1
    $\begingroup$ "as long as the standard deviations don't grow too wildly" Or shrink (eg: $\sigma_i^2 = \sigma_{i-1}^2/2$) $\endgroup$ – leonbloy Jun 2 '18 at 19:28
21
$\begingroup$

Although I'm pretty sure that it has been answered before, here's another one:

There are several versions of the central limit theorem, the most general being that given arbitrary probability density functions, the sum of the variables will be distributed normally with a mean value equal to the sum of mean values, as well as the variance being the sum of the individual variances.

A very important and relevant constraint is that the mean and the variance of the given pdfs have to exist and must be finite.

So, just take any pdf without mean value or variance -- and the central limit theorem will not hold anymore. So take a Lorentzian distribution for example.

$\endgroup$
  • $\begingroup$ +1 Or take a distribution with an infinite variance, like the distribution of a random walk. $\endgroup$ – Alexis May 30 '18 at 14:35
  • 2
    $\begingroup$ @Alexis - assuming you are looking at a random walk at a finite point in time, I would have thought it would have a finite variance, being the sum of $n$ i.i.d steps each with finite variance $\endgroup$ – Henry May 30 '18 at 19:33
  • 1
    $\begingroup$ @Henry: Nope, am not assuming at a point in time, but the variance of the distribution of all possible random walks of infinite lengths. $\endgroup$ – Alexis May 30 '18 at 21:28
  • 1
    $\begingroup$ @Alexis If each step $X_i$ of the random walk is $+1$ or $-1$ i.i.d. with equal probability and the positions are $Y_n =\sum_1^n X_i$ then the Central Limit Theorem implies correctly that as $n \to \infty$ you have the distribution of $\sqrt{n}\left(\frac1n Y_n\right) = \frac{Y_n}{\sqrt{n}}$ converging in distribution to $\mathcal N(0,1)$ $\endgroup$ – Henry May 30 '18 at 23:35
  • 1
    $\begingroup$ @Alexis Doesn't matter for the CLT, because each individual distribution still has a finite variance. $\endgroup$ – Cubic May 31 '18 at 11:08
15
$\begingroup$

No, CLT always holds when its assumptions hold. Qualifications such as "in most situations" are informal references to the conditions under which CLT should be applied.

For instance, a linear combination of independent variables from Cauchy distribution will not add up to Normal distributed variable. One of the reasons is that the variance is undefined for Cauchy distribution, while CLT puts certain conditions on the variance, e.g. that it has to be finite. An interesting implication is that since Monte Carlo simulations is motivated by CLT, you have to be careful with Monte Carlo simulations when dealing with fat tailed distributions, such as Cauchy.

Note, that there is a generalized version of CLT. It works for infinite or undefined variances, such as Cauchy distribution. Unlike many well behaving distributions, the properly normalized sum of Cauchy numbers remains Cauchy. It doesn't converge to Gaussian.

By the way, not only Gaussian but many other distributions have bell shaped PDFs, e.g. Student t. That's why the description you quoted is quite liberal and imprecise, perhaps on purpose.

$\endgroup$
7
$\begingroup$

Here is an illustration of cherub's answer, a histogram of 1e5 draws from scaled (by $\sqrt{n}$) sample means of t-distributions with two degrees of freedom, such that the variance does not exist.

If the CLT did apply, the histogram for $n$ as large as $n=1000$ should resemble the density of a standard normal distribution (which, e.g., has density $1/\sqrt{2\pi}\approx0.4$ at its peak), which it evidently does not.

enter image description here

library(MASS)
n <- 1000
samples.from.t <- replicate(1e5, sqrt(n)*mean(rt(n, df = 2)))
truehist(samples.from.t, xlim = c(-10,10), col="salmon")
$\endgroup$
  • 3
    $\begingroup$ You have to be slightly careful here as if you did this with a $t$-distribution with say $3$ degrees of freedom then the Central Limit theorem would apply but your graph would not have a peak density around $0.4$ but instead around $\frac1{\sqrt{6\pi}}\approx 0.23$ because the original variance would not be $1$ $\endgroup$ – Henry May 30 '18 at 19:43
  • $\begingroup$ That is a good point, one might standardize the mean by sd(x) to get something which, if the CLT works, converges by Slutzky's theorem, to a N(0,1) variate. I wanted to keep the example simple, but you are of course right. $\endgroup$ – Christoph Hanck May 31 '18 at 5:08
6
$\begingroup$

A simple case where the CLT cannot hold for very practical reasons, is when the sequence of random variables approaches its probability limit strictly from the one side. This is encountered for example in estimators that estimate something that lies on a boundary.

The standard example here perhaps is the estimation of $\theta$ in a sample of i.i.d. Uniforms $U(0,\theta)$. The maximum likelihood estimator will be the maximum order statistic, and it will approach $\theta$ necessarily only from below: naively thinking, since its probability limit will be $\theta$, the estimator cannot have a distribution "around" $\theta$ - and the CLT is gone.

The estimator properly scaled does have a limiting distribution - but not of the "CLT variety".

$\endgroup$
3
$\begingroup$

You can find a quick solution here.

Exceptions to the central-limit theorem arise

  1. When there are multiple maxima of the same height, and
  2. Where the second derivative vanishes at the maximum.

There are certain other exceptions which are outlined in the answer of @cherub.


The same question has already been asked on math.stackexchange. You can check the answers there.

$\endgroup$
  • 4
    $\begingroup$ By "maxima", do you mean modes? Being bimodal has nothing to do with failing to satisfy CLT. $\endgroup$ – Acccumulation May 30 '18 at 15:08
  • $\begingroup$ @Acccumulation: The wording here is confusing because it actually refers to the PGF of a discrete r.v. $M(z)=\sum_{n=-\infty}^\infty P(X=n)z^n$ $\endgroup$ – Alex R. May 30 '18 at 18:17
  • $\begingroup$ @AlexR. The answer doesn't make sense at all without reading through the link, and is far from clear even with the link. I'm leaning towards downvoting as being even worse than a link-only answer. $\endgroup$ – Acccumulation May 30 '18 at 18:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.