# Debunking wrong CLT statement

The central limit theorem (CLT) gives some nice properties about converging to a normal distribution. Prior to studying statistics formally, I was under the extremely wrong impression that the CLT said that data approached normality.

I now find myself arguing with collaborators about this. I say that $$68\%$$ of the data need not be within one standard deviation of the mean when we have non-normal distributions. They agree but then say that, by the CLT, since we have many observations (probably 50,000), our data are very close to normal, so we can use the empirical rule and say that $$68\%$$ of the data are within one standard deviation of the mean. This is, of course, false. The population does not care how many observations are drawn from it; the population is the population, whether we sample from it or not!

What would be a good way to explain why the central limit theorem is not about the empirical distribution converging?

• Well, the sampling distribution (distribution of $X_1, X_2, \cdots, X_n$, or of $\bar{X}$, the sample mean for that matter) doesn't converge to a normal distribution either. So, you need to be more precise as to what you wish to cavil about. – Dilip Sarwate Jun 22 '20 at 16:25
• focus on what is exactly converging to normal according to CLT. this way you easily point to what is not converging – Aksakal Jun 22 '20 at 16:53
• Ask your collaborators to check their conclusions when the data are binary, such as the indicator of the flip of a fair coin. – whuber Jun 22 '20 at 17:59
• I don't think there's a good way to explain this to people who are unable to follow logical reasoning. So, maybe just learn to live with your collaborators – Aksakal Jun 22 '20 at 21:37
• So, what I am trying to say is that, maybe your colleagues are not correct in their words (or maybe you are not correctly paraphrasing them), but we shouldn't weight those words to heavily (we can not blame the thoughts behind them when they aren't expressed with a lot of rigor) and instead we should try to understand the underlying thoughts. Therefore, why wouldn't you clarify the underlying issue and share the problem where this CLT is supposed to apply? That is much more clear than a one-sided view of the discussion between you and your colleagues. en.wikipedia.org/wiki/XY_problem – Sextus Empiricus Jun 23 '20 at 8:51

As whuber notes, you can always point your collaborators to a binary discrete distribution. But they might consider that "cheating" and retreat to the weaker claim that the proposed statement only applied to continuous distributions.

So use the uniform distribution on the unit interval $$[0,1]$$. It has a mean of $$\mu=0.5$$, a variance of $$\frac{1}{12}$$, thus a standard deviation of $$\sigma=\frac{1}{\sqrt{12}}\approx 0.289$$. But of course the interval $$[\mu-\sigma,\mu+\sigma]\approx[0.211,0.789]$$ of length $$2\sigma\approx 0.577$$ only contains $$57.7\%$$ of your data (more specifically: as the sample size increases, the proportion approaches $$0.577$$), not $$68\%$$, no matter how many data points you sample.

• In your uniform example, using the sample statistics, the proportion of observations in $[m-s,m+s]$ gets closer to $0.577$ as the sample size increases – Henry Jun 23 '20 at 10:26
• @Henry: thank you. I have made my statement a little more precise. – Stephan Kolassa Jun 23 '20 at 11:48
• @Alexis: thanks! I'm not quite sure where you think a "because" would make sense... do you want to just edit it in? – Stephan Kolassa Jun 26 '20 at 6:35
• @Alexis, sorry, I misunderstood what you meant. Now cleared up. – Richard Hardy Jun 26 '20 at 15:05
• @Alexis: thanks for elaborating. I actually don't think there is anything particularly interesting here. It's just often the case that A makes a false claim, B provides a counterexample (like whuber's one here), and then A retreats to a weaker (and still false) claim. "Don't be silly, of course I'm not talking about discrete distributions here!" That would actually be an interesting conversation to have, if the original claim and its weaker version didn't exhibit such a deep misunderstanding. – Stephan Kolassa Jun 26 '20 at 15:13

This is quite a ubiquitous misunderstanding of the central limit theorem, which I have also encountered in my statistical teaching. Over the years I have encountered this problem so often that I have developed a Socratic method to deal with it. I identify a student that has accepted this idea and then engage the student to tease out what this would logically imply. It is fairly simple to get to the reductio ad absurdum of the false version of the theorem, which is that every sequence of IID random variables has a normal distribution. A typical conversation would go something like this.

Teacher: I noticed in this assignment question that you said that because $$n$$ is large, the data are approximately normally distributed. Can you take me through your reasoning for that bit?

Student: Is that wrong?

Teacher: I don't know. Let's have a look at it.

Student: Well, I used that theorem you talked about in class; that main one you mentioned a bunch of times. I forget the name.

Teacher: The central limit theorem?

Student: Yeah, the central limit theorem.

Teacher: Great, and when does that theorem apply?

Student: I think if the variables are IID.

Teacher: And have finite variance.

Student: Yeah, and finite variance.

Teacher: Okay, so the random variables have some fixed distribution with finite variance, is that right?

Student: Yeah.

Teacher: And the distribution isn't changing or anything?

Student: No, they're IID with a fixed distribution.

Teacher: Okay great, so let me see if I can state the theorem. The central limit theorem says that if you have an IID sequence of random variables with finite variance, and you take a sample of $$n$$ of them, then as that sample size $$n$$ gets large the distribution of the random variables converges to a normal distribution. Is that right?

Student: Yeah, I think so.

Teacher: Okay great, so let's think about what that would mean. Suppose I have a sequence like that. If I take say, a thousand sample values, what is the distribution of those random variables?

Student: It's approximately a normal distribution.

Teacher: How close?

Student: Pretty close I think.

Teacher: Okay, what if I take a billion sample values. How close now?

Student: Really close I'd say.

Teacher: And if we have a sequence of these things, then in theory we can take $$n$$ as high as we want can't we? So we can make the distribution as close to a normal distribution as we want.

Student: Yeah.

Teacher: So let's say we take $$n$$ big enough that we're happy to say that the random variables basically have a normal distribution. And that's a fixed distribution right?

Student: Yeah.

Teacher: And they're IID right? These random variables are IID?

Student: Yeah, they're IID.

Teacher: Okay, so they all have the same distribution.

Student: Yeah.

Teacher: Okay, so that means the first value in the sequence, it also has a normal distribution. Is that right?

Student: Yeah. I mean, it's an approximation, but yeah, if $$n$$ is really large then it effectively has a normal distribution.

Teacher: Okay great. And so does the second value in the sequence, and so on, right?

Student: Yeah.

Teacher: Okay, so really, as soon as we started sampling, we were already getting values that are essentially normal distributed. We didn't really need to wait until $$n$$ gets large before that started happening.

Student: Hmmm. I'm not sure. That sounds wrong. The theorem says you need a large $$n$$, so I guess I think you can't apply it if you only sampled a small number of values.

Teacher: Okay, so let's say we are sampling a billion values. Then we have large $$n$$. And we've established that this means that the first few random variables in the sequence are normally distributed, to a very close approximation. If that's true, can't we just stop sampling early? Say we were going to sample a billion values, but then we stop sampling after the first value. Was that random variable still normally distributed?

Student: I think maybe it isn't.

Teacher: Okay, so at some point its distribution changes?

Student: I'm not sure. I'm a bit confused about it now.

Teacher: Hmmm, well it seems we have something strange going on here. Why don't you have another read of the material on the central limit theorem and see if you can figure out how to resolve that contradiction. Let's talk more about it then.

That is one possible approach, which seeks to reduce the false theorem down to the reductio which says that every IID sequence (with finite variance) must be composed of normal random variables. Either the student will get to this conclusion, and realise something is wrong, or they will defend against this conclusion by saying that the distribution changes as $$n$$ gets large. Either way, this usually provokes some further thinking that can lead them to re-read the theorem. Here is another approach:

Teacher: Let's look at this another way. Suppose we have an IID sequence of random variables from some other distribution; one that is not a normal distribution. Is that possible? For example, could we have a sequence of random variables representing outcome of coin flip, from the Bernoulli distribution?

Student: Yeah, we can have that.

Teacher: Okay, great. And these are all IID values, so again, they all have the same distribution. So every random variable in that sequence is going to have a distribution that is not a normal distribution, right?

Student: Yeah.

Teacher: In fact, in this case, every value in the sequence will be the outcome of a coin flip, which we set as zero or one. Is that right?

Student: Yeah, as long as we label them that way.

Teacher: Okay, great. So if all the values in the sequence are zeroes or ones, no matter how many of them we sample, we are always going to get a histogram showing values at zero and one, right?

Student: Yeah.

Teacher: Okay. And do you think if we sample more and more values, we will get closer and closer to the true distribution? Like, if it is a fair coin, does the histogram eventually converge to where the relative frequency bars are the same height?

Student: I guess so. I think it does.

Teacher: I think you're right. In fact, we call that result the "law of large numbers". Anyway, it seems like we have a bit of a problem here doesn't it. If we sample a large number of the values then the central limit theorem says we converge to a normal distribution, but it sounds like the "law of large numbers" says we actually converge to the true distribution, which isn't a normal distribution. In fact, it's a distribution that is just probabilities on the zero value and the one value, which looks nothing like the normal distribution. So which is it?

Student: I think when $$n$$ is large it looks like a normal distribution.

Teacher: So describe it to me. Let's say we have flipped the coin a billion times. Describe the distribution of the outcomes and explain why that looks like a normal distribution.

Student: I'm not really sure how to do that.

Teacher: Okay. Well, do you agree that if we have a billion coin flips, all those outcomes are zeroes and ones?

Student: Yeah.

Teacher: Okay, so describe what its histogram looks like.

Student: It's just two bars on those values.

Teacher: Okay, so not "bell curve" shaped?

Student: Yeah, I guess not.

Teacher: Hmmm, so perhaps the central limit theorem doesn't say what we thought. Why don't you read the material on the central limit theorem again and see if you can figure out what it says. Let's talk more about it then.

• I like dialogs. But I think the first one doesn't recognize the student's misperception. It sounds like they have a grasp of the idea that the empirical distribution of a small sample is likely to depart appreciably from the underlying distribution. The first dialog seems to dance around that--mainly through the teacher's unmodified repetition of one attempted explanation--without addressing it. – whuber Jun 23 '20 at 12:31
• This teacher is very patient. I guess you need to be when teaching undergrad – Aksakal Jun 23 '20 at 19:37
• @whuber: The general idea of these dialogues is not to address the problem, but rather, to merely bring attention to certain contradictions, so that the student will be motivated to go and re-read the theorem to try to resolve the apparent paradox. In the first instance, you are drawing attention to the fact that the IID assumption forces all the distributions of the random variables to be the same, so if they "converge in distribution to the normal" then they must all have been normal. There is usually some repetition in these consersations to assist the student. – Ben Jun 23 '20 at 22:13

The central limit theorem states that the mean of the data will become normally distributed as the sample size increases, it says nothing about the data itself. Another way to put it is the distribution of the parameter (the mean) is normal, but that is entirely separate from the distribution of the underlying data.

Most of the value from the CLT comes from the fact that you can compare samples that are not normally distributed to one another (based solely on the fact that, due to the CLT, you know how their means should behave).

I think where this gets confusing is that just because you can compare two sample means to each other based on some test that assumes normality (eg. t-test) doesn't mean that you should. (ie comparing the means of two exponential distributions might not tell you what you think it does, or two bi-modal distributions, or a bi-modal with a uni-modal distribution, ect).

The question most people should ask is, "is the mean (or a difference in means) a useful metric given the distribution of my data". Only if the answer to this question is yes, should one proceed to compare means (thus relying on the CLT).

By not asking this question, many people fall into the following (roughly stated) logical fallacy:

The CLT applies, so I can compare means. And I can compare means because they are normally distributed. This comparison must be meaningful, because the CLT says I can do it (and the CLT is very powerful). The comparison/test I am using most intuitively (/only) makes sense when the data is normally distributed, and after all, the mean is normally distributed, so my data must be normally distributed too!

To directly answer the question, you can:

1. Show them the definition, point out that the CLT only makes a claim about the distribution of the mean approaching normality, emphasize the distribution of a parameter can be very different from the distribution of the data from which it is derived.

2. Show them this video which provides a nice visual representation of how the CLT works using several different distributions for the underlying data. (its a bit quirky, but communicated very clearly)

I glossed over some technical details in my explanation in order to make it more understandable to someone who is less familiar with statistics. Several commenters have pointed this out and so I thought I would include their feedback here:

• A more accurate statement of the CLT would be:

"The central limit theorem states that the mean of the data will become normally distributed (more specifically the difference between the mean of the data/sample and the true mean, multiplied by the square root of the sample size $$\sqrt{n}$$ is normal distributed)"

I have also seen this explained as "the properly normalized sum tends toward a normal distribution"

It is also worth pointing out that the data must be composed of independent and identically distributed random variables with finite variance in order for the CLT to apply.

• A more accurate and/or less Bayesian way to say "the distribution of the parameter (mean)" would be "the distribution of the parameter estimate by the regular sample mean"
• Doesn't CTL also apply to the distribution of any parameter estimate ? Like mean but also median or SD or any other summarizing parameters like regression coefficient and the associated measure of error ? – Rodolphe Jun 23 '20 at 7:05
• I believe it is only the mean. Many parametric hypothesis tests rely on the mean being normally distributed (or some metric derived from the mean). This goes back to my point about can vs should. Just because the CLT lets you perform a parametric test, doesn't mean that the test will give you "meaningful" results from your data, that relies more on how appropriate of a metric the mean is for the kind of comparison you are trying to make. – Cole Jun 23 '20 at 7:33
• "states that the mean of the data..." this is a bit more like the law or large numbers, the mean of the data will approach a degenerate distribution. It would be more elegant to add a side-note stating that more specifically a scaled and shifted sample mean approaches a normal distribution "The central limit theorem states that the mean of the data will become normally distributed (more specifically the difference between the mean of the data/sample and the true mean, multiplied by the square root of the sample size $\sqrt{n}$ is normal distributed)".... – Sextus Empiricus Jun 23 '20 at 9:04
• I’m with @SextusEmpiricus. Consider $U(0,1)$. $\bar{X}$ can’t have any density outside of $[0,1]$, and the law of large numbers gives an even stronger convergence in probability (CLT is convergence in distribution) of $\bar{X}$ to the true expected value. Also, “distribution of the parameter” is a controversial, inherently Bayesian comment. I think you meant the distribution of the parameter estimate by the regular sample mean. – Dave Jun 23 '20 at 9:42
• I think this is the best answer so far. This is the only one that actually clarifies what the CLT says (and from the question, it is quite possible the asker is confused about this as well, not just the colleagues). – usul Jun 24 '20 at 12:44

CLT is about convergence of a sum of random variables. If we have an iid sample $$X_1,...,X_n$$, where $$EX_i=\mu$$ and $$Var(X_i)<\infty$$ then

$$\frac{1}{\sqrt{n}}\left(X_1+...+X_n-n\mu\right) \to N(0, Var(X_i))$$

This statement is solely about closeness of a distribution of suitably normalized sum $$(X_1+...+X_n)$$ to the normal distribution. It does not say that nothing about convergence of distribution of $$X_i$$. Since $$X_i$$ do not depend on $$n$$ why should they converge anywhere?

Empirical distribution of a sample $$X_i$$ will actually converge (as sample size increases) to the actual distribution of $$X_i$$ according to Donsker theorem, so unless the actual distribution is not close to normal, the empirical distribution will not be close to it either.

• Donsker seems to be about stochastic processes. Did you mean Glivenko–Cantelli? – Dave Jun 23 '20 at 21:14
• Glivenko-Cantelli is law of large numbers for stochastic processes, Donsker is the central limit theorem. First is about convergence in probability, second is about convergence in distribution. – mpiktas Jun 27 '20 at 19:19

This is how I like to visualize the CLT. I'm not 100% sure the argument is correct though, please check.

Start with a population of values whose distribution is nowhere near normal. E.g., a uniform distribution:

X <- runif(n= 50000)
hist(X)


Now, take $$n$$ samples from this population, calculate the mean of each sample, shift the sample mean by the mean of the population and scale it by $$\sqrt{n}$$, plot a histogram of these $$n$$ means. That histogram is (close to) normal:

mu <- 1/2 # Mean of population X
x <- rep(NA, 1000)
size <- 10
for(i in 1:length(x)) {
x[i] <- sqrt(size) * (mean(sample(X, size= size)) - mu)
}


• The CLT is not about sample means, which converge in probability (therefore in distribution like the CLT’s convergence) to the population mean. CLT says $\sqrt{n}(\bar{X}_n-\mu) \overset{d}{\rightarrow}N(0,\sigma^2)$. – Dave Jun 23 '20 at 10:58
• @Dave, thanks - I knew I was confused myself. However, I read from here: The CLT states that if you have a population with mean μ and standard deviation σ and take sufficiently large random samples from the population with replacement, then the distribution of the sample means will be approximately normally distributed Isn't it what I've done above? – dariober Jun 23 '20 at 11:15
• Ask yourself how the sample mean of a Bernoulli distribution can have a normal distribution when there sample mean literally can’t ever be $-1$. That website gives the wrong statement of the central limit theorem. – Dave Jun 23 '20 at 11:17
• @Dave (I'm trying to understand - not arguing). If I replace runif(n= 50000) with rbinom(n= 50000, size= 1, prob= 0.5) (50k coin flips), then the histogram of 1000 sample means still turns out roughly normal – dariober Jun 23 '20 at 11:30
• Binomial in particular can be wrangled to converge to normal, so consider $U(0,1)$ to make life easy. You literally can’t have a sample mean of $-1$. The sample means cluster around $1/2$ and, by the law of large numbers (not CLT), $\bar{X}$ converges to $1/2$. My suggestion is to delete this post so newcomers don’t read it and get confused about the central limit theorem, and then post about your confusion as a distinct question. As you can see, you’re not the only person who posted an answer who has your same confusion. – Dave Jun 23 '20 at 11:38

The point of confusion here is what is actually converging to a normal distribution. I think the easiest way to overcome this is to explain examples of the extremes of a sampling distribution, one with one measurement per sample (just as if taking measurements straight from the population as you describe) and one where each sample is the entire population. From there it is easier to understand what happens in the middle ground.