# Central limit theorem seems counterintuitive given Law of large number

From what I understand, the Central limit theorem says the sample mean is distributed normally when sample number tends to infinity.

However, the Law of large number says sample mean converges in probability to the population mean. I would imagine this means $$P(\bar X_n = \mu) \to 1$$ and $$P(\bar X_n = \text{anything else}) \to 0$$ as sample number increases. And this doesn't look like a normal distribution at all.

• The CLT is about the distribution of a standardized mean (note that the variance of that is constant at any sample size). The LLN is about what happens to an unstandardized mean (whose variance decreases with sample size), more specifically, about how a centered mean stays close to 0. Commented May 26, 2020 at 13:49

There's no contradiction. By central limit theorem we know that as $$n$$ increases, it converges in distribution to Gaussian. We also know that as sample size grows, the variance of this distribution gets smaller and smaller, by a factor of $$n$$:

$$\bar{X}_n - \mu\ \xrightarrow{d}\ \mathcal{N}\left(0,\tfrac{\sigma^2}{n}\right)$$

The strong law of large numbers says that in the end as $$n \to \infty$$ we would end up estimating $$\mu$$ precisely

$$\Pr\!\left( \lim_{n\to\infty}\overline{X}_n = \mu \right) = 1$$

Notice that as $$n \to \infty$$ then $$\tfrac{\sigma^2}{n} \to 0$$, so you can think of it as a normal distribution shrinking all the way down, until it reaches Dirac delta function with all the probability mass at $$\mu$$ and zero probability otherwise (see the image taken from the Wikipedia article on Dirac delta). They can be seen as one being natural consequence of the other, rather then contradiction. Law of large numbers would be the case of "bigger infinity" then the central limit theorem.

• @innisfree "you can think of it", forgive me lack of formality, it is an intuition that can be applied to join the two together. I'm not claiming that they are the same, but that they do not counter each other.
– Tim
Commented May 26, 2020 at 12:27
• Thanks, wasn’t a criticism, just a comment. I agree that it’s sometimes intuitive, but also can be counterintuitive! Commented May 26, 2020 at 12:30
• Mathematically, it is better to write $\sqrt{n}(\bar{X}_n - \mu) \overset{d}{\to} N(0, \sigma^2)$ instead of $\bar{X}_n - \mu \overset{d}{\to} N(0, \sigma^2/n)$. Commented May 26, 2020 at 13:12
• @Zhanxiong agree, but to clarify the notation and make the point more clear I used the altered form, since it emphasizes the shrinkage of variance.
– Tim
Commented May 26, 2020 at 13:17
• @HarisGušić as said in the comments, the answer is more about intuition than formal argument. I cannot recall any reference of this kind at the moment.
– Tim
Commented Jan 11, 2021 at 15:14

Be aware: the Central limit theorem does not say that the sample mean is distributed normally when sample number tends to infinity. The CLT says that if $$\sigma > 0$$ then: $$\frac{X_1+\dots+X_n-n\mu}{\sigma\sqrt{n}}\overset{d}{\to}N(0,1)\tag{1}$$ You can divide numerator and denominator by $$n$$ and write: $$\frac{\sqrt{n}(\overline{X}_n-\mu)}{\sigma}\overset{d}{\to}N(0,1)\tag{2}$$ You also can multiply by $$\sigma$$ and write: $$\sqrt{n}(\overline{X}_n-\mu)\overset{d}{\to}N(0,\sigma^2)\tag{3}$$ but you can't go farther: $$\overline{X}_n-\mu\overset{d}{\to}N(0,\sigma^2/n)\quad\text{or}\quad \overline{X}_n\overset{d}{\to}N(\mu,\sigma^2/n)\tag{4}$$ because "$$\overset{d}{\to}$$" means that as $$n$$ goes to infinity the left CDF goes to the right CDF, buf as $$n$$ goes to infinity $$\sigma^2/n$$ goes to $$0$$, so you get a degenerate distribution, not a normal distribution.

However, as long as $$n$$ is a finite number, you can write that if $$n$$ is large then $$\overline{X}_n\mathrel{\dot\sim} N(\mu,\sigma^2/n),\quad 1\ll n<\infty$$ where "$$\mathrel{\dot\sim}$$" (dot over sim) means "approximately distributed as", because the CDF of $$\overline{X}_n$$ is obtained by scaling and shifting the CDF of $$\sqrt{n}(\overline{X}_n-\mu)/\sigma$$, thus the two CDFs have similar shapes (see https://www.probabilitycourse.com/chapter7/7_1_2_central_limit_theorem.php .)

Notice that (4) is not "false" (it is true if you "forget" that "$$N$$" stands for "normal distribution", which requires a strictly positive variance), i.e. you can think that, according to CLT, as $$n$$ goes to infinity the CDF of $$\overline{X}_n$$ goes to the CDF of a variable with $$\mu$$ mean and $$0$$ variance. And that is just what LLN says: $$\overline{X}_n\overset{p}{\to}\mu\quad\Rightarrow\quad\overline{X}_n\overset{d}{\to}\mu$$ where $$\mu$$ is just a number, and has zero variance.

Tim already argued that intuitively speaking, the pdf of $$X_n$$ actually converges to a Dirac delta, but I want to make a slightly more formal argument from a different angle.

The Law of Large Numbers says that the sequence of random variables $$X_n$$ converges to a constant random variable $$X_{lim}$$ that is equal to the population mean $$\mu$$. Its cdf $$F$$ is a step function, since $$F = \text{P}(X_{lim} \le x)$$ is 0 for $$x < \mu$$ and is 1 for $$x \ge \mu$$.

Now, a step function is not an absolutely continuous function, so it doesn't have a well defined density function, but it does have a derivative in the sense of distributions and that is precisely the Dirac delta.