Are there any examples of where the central limit theorem does not hold?

Question

Wikipedia says -

In probability theory, the central limit theorem (CLT) establishes that, in most situations, when independent random variables are added, their properly normalized sum tends toward a normal distribution (informally a "bell curve") even if the original variables themselves are not normally distributed...

When it says "in most situations", in which situations does the central limit theorem not work?

Answer 1

To understand this, you need to first state a version of the Central Limit Theorem. Here's the "typical" statement of the central limit theorem:

Lindeberg–Lévy CLT. Suppose ${X_1, X_2, \dots}$ is a sequence of i.i.d. random variables with $E[X_i] = \mu$ and $Var[X_i] = \sigma^2 < \infty$. Let $S_{n}:={\frac {X_{1}+\cdots +X_{n}}{n}}$. Then as $n$ approaches infinity, the random variables $\sqrt{n}(S_n − \mu)$ converge in distribution to a normal $N(0,\sigma^2)$ i.e.

$${\displaystyle {\sqrt {n}}\left(\left({\frac {1}{n}}\sum _{i=1}^{n}X_{i}\right)-\mu \right)\ {\xrightarrow {d}}\ N\left(0,\sigma ^{2}\right).}$$

So, how does this differ from the informal description, and what are the gaps? There are several differences between your informal description and this description, some of which have been discussed in other answers, but not completely. So, we can turn this into three specific questions:

• What happens if the variables are not identically distributed?
• What if the variables have infinite variance, or infinite mean?
• How important is independence?

Taking these one at a time,

Not identically distributed, The best general results are the Lindeberg and Lyaponov versions of the central limit theorem. Basically, as long as the standard deviations don't grow too wildly, you can get a decent central limit theorem out of it.

Lyapunov CLT.[5] Suppose ${X_1, X_2, \dots}$ is a sequence of independent random variables, each with finite expected value $\mu_i$ and variance $\sigma^2$ Define: $s_{n}^{2}=\sum _{i=1}^{n}\sigma _{i}^{2}$

If for some $\delta > 0$, Lyapunov’s condition ${\displaystyle \lim _{n\to \infty }{\frac {1}{s_{n}^{2+\delta }}}\sum_{i=1}^{n}\operatorname {E} \left[|X_{i}-\mu _{i}|^{2+\delta }\right]=0}$ is satisfied, then a sum of $X_i − \mu_i / s_n$ converges in distribution to a standard normal random variable, as n goes to infinity:

${{\frac {1}{s_{n}}}\sum _{i=1}^{n}\left(X_{i}-\mu_{i}\right)\ {\xrightarrow {d}}\ N(0,1).}$

Infinite Variance Theorems similar to the central limit theorem exist for variables with infinite variance, but the conditions are significantly more narrow than for the usual central limit theorem. Essentially the tail of the probability distribution must be asymptotic to $|x|^{-\alpha-1}$ for $0 < \alpha < 2$. In this case, appropriate scaled summands converge to a Levy-Alpha stable distribution.

Importance of Independence There are many different central limit theorems for non-independent sequences of $X_i$. They are all highly contextual. As Batman points out, there's one for Martingales. This question is an ongoing area of research, with many, many different variations depending upon the specific context of interest. This Question on Math Exchange is another post related to this question.

Answer 2

Although I'm pretty sure that it has been answered before, here's another one:

There are several versions of the central limit theorem, the most general being that given arbitrary probability density functions, the sum of the variables will be distributed normally with a mean value equal to the sum of mean values, as well as the variance being the sum of the individual variances.

A very important and relevant constraint is that the mean and the variance of the given pdfs have to exist and must be finite.

So, just take any pdf without mean value or variance -- and the central limit theorem will not hold anymore. So take a Lorentzian distribution for example.

Answer 3

No, CLT always holds when its assumptions hold. Qualifications such as "in most situations" are informal references to the conditions under which CLT should be applied.

For instance, a linear combination of independent variables from Cauchy distribution will not add up to Normal distributed variable. One of the reasons is that the variance is undefined for Cauchy distribution, while CLT puts certain conditions on the variance, e.g. that it has to be finite. An interesting implication is that since Monte Carlo simulations is motivated by CLT, you have to be careful with Monte Carlo simulations when dealing with fat tailed distributions, such as Cauchy.

Note, that there is a generalized version of CLT. It works for infinite or undefined variances, such as Cauchy distribution. Unlike many well behaving distributions, the properly normalized sum of Cauchy numbers remains Cauchy. It doesn't converge to Gaussian.

By the way, not only Gaussian but many other distributions have bell shaped PDFs, e.g. Student t. That's why the description you quoted is quite liberal and imprecise, perhaps on purpose.

Answer 4

Here is an illustration of cherub's answer, a histogram of 1e6 draws from scaled (by $\sqrt{n}$) and standardized (by the sample standard deviation) sample means of t-distributions with two degrees of freedom, such that the variance does not exist.

If the CLT did apply, the histogram for $n$ as large as $n=2000$ should resemble the density of a standard normal distribution (which, e.g., has density $1/\sqrt{2\pi}\approx0.4$ at its peak), which it evidently does not.

library(MASS)
n <- 2000
std.t <- function(n){
  x <- rt(n, df = 2)
  sqrt(n)*mean(x)/sd(x)
}
samples.from.t <- replicate(1e6, std.t(n))
xax <- seq(-5,5, by=0.01)
truehist(samples.from.t, xlim = c(-5,5), ylim = c(0,0.4), 
         col="salmon")
lines(xax, dnorm(xax), col="blue", lwd=2)

Answer 5

A simple case where the CLT cannot hold for very practical reasons, is when the sequence of random variables approaches its probability limit strictly from the one side. This is encountered for example in estimators that estimate something that lies on a boundary.

The standard example here perhaps is the estimation of $\theta$ in a sample of i.i.d. Uniforms $U(0,\theta)$. The maximum likelihood estimator will be the maximum order statistic, and it will approach $\theta$ necessarily only from below: naively thinking, since its probability limit will be $\theta$, the estimator cannot have a distribution "around" $\theta$ - and the CLT is gone.

The estimator properly scaled does have a limiting distribution - but not of the "CLT variety".

Answer 6

You can find a quick solution here.

Exceptions to the central-limit theorem arise

1. When there are multiple maxima of the same height, and
2. Where the second derivative vanishes at the maximum.

---

There are certain other exceptions which are outlined in the answer of @cherub.

---

The same question has already been asked on math.stackexchange. You can check the answers there.

Answer 7

The (usual) central limit theorem applies only if the random variables involved are mutually independent with the same distribution and finite mean and variance. If the variables are merely pairwise independent (meaning any two of them are independent of each other, but more than two are not necessarily independent), the theorem need not hold true, and Avanzi et al. (2020) show some examples that the theorem does not work for pairwise independent random variables in general.

REFERENCES:

• Avanzi, Benjamin, Guillaume Boglioni Beaulieu, Pierre Lafaye de Micheaux, Frédéric Ouimet, and Bernard Wong. "A counterexample to the central limit theorem for pairwise independent random variables having a common arbitrary margin" arXiv preprint arXiv:2003.01350 (2020).