Debunking wrong CLT statement

Question

The central limit theorem (CLT) gives some nice properties about converging to a normal distribution. Prior to studying statistics formally, I was under the extremely wrong impression that the CLT said that data approached normality.

I now find myself arguing with collaborators about this. I say that $68\%$ of the data need not be within one standard deviation of the mean when we have non-normal distributions. They agree but then say that, by the CLT, since we have many observations (probably 50,000), our data are very close to normal, so we can use the empirical rule and say that $68\%$ of the data are within one standard deviation of the mean. This is, of course, false. The population does not care how many observations are drawn from it; the population is the population, whether we sample from it or not!

What would be a good way to explain why the central limit theorem is not about the empirical distribution converging?

Answer 1

This is quite a ubiquitous misunderstanding of the central limit theorem, which I have also encountered in my statistical teaching. Over the years I have encountered this problem so often that I have developed a Socratic method to deal with it. I identify a student that has accepted this idea and then engage the student to tease out what this would logically imply. It is fairly simple to get to the reductio ad absurdum of the false version of the theorem, which is that every sequence of IID random variables has a normal distribution. A typical conversation would go something like this.

Teacher: I noticed in this assignment question that you said that because $n$ is large, the data are approximately normally distributed. Can you take me through your reasoning for that bit?

Student: Is that wrong?

Teacher: I don't know. Let's have a look at it.

Student: Well, I used that theorem you talked about in class; that main one you mentioned a bunch of times. I forget the name.

Teacher: The central limit theorem?

Student: Yeah, the central limit theorem.

Teacher: Great, and when does that theorem apply?

Student: I think if the variables are IID.

Teacher: And have finite variance.

Student: Yeah, and finite variance.

Teacher: Okay, so the random variables have some fixed distribution with finite variance, is that right?

Student: Yeah.

Teacher: And the distribution isn't changing or anything?

Student: No, they're IID with a fixed distribution.

Teacher: Okay great, so let me see if I can state the theorem. The central limit theorem says that if you have an IID sequence of random variables with finite variance, and you take a sample of $n$ of them, then as that sample size $n$ gets large the distribution of the random variables converges to a normal distribution. Is that right?

Student: Yeah, I think so.

Teacher: Okay great, so let's think about what that would mean. Suppose I have a sequence like that. If I take say, a thousand sample values, what is the distribution of those random variables?

Student: It's approximately a normal distribution.

Teacher: How close?

Student: Pretty close I think.

Teacher: Okay, what if I take a billion sample values. How close now?

Student: Really close I'd say.

Teacher: And if we have a sequence of these things, then in theory we can take $n$ as high as we want can't we? So we can make the distribution as close to a normal distribution as we want.

Student: Yeah.

Teacher: So let's say we take $n$ big enough that we're happy to say that the random variables basically have a normal distribution. And that's a fixed distribution right?

Student: Yeah.

Teacher: And they're IID right? These random variables are IID?

Student: Yeah, they're IID.

Teacher: Okay, so they all have the same distribution.

Student: Yeah.

Teacher: Okay, so that means the first value in the sequence, it also has a normal distribution. Is that right?

Student: Yeah. I mean, it's an approximation, but yeah, if $n$ is really large then it effectively has a normal distribution.

Teacher: Okay great. And so does the second value in the sequence, and so on, right?

Student: Yeah.

Teacher: Okay, so really, as soon as we started sampling, we were already getting values that are essentially normal distributed. We didn't really need to wait until $n$ gets large before that started happening.

Student: Hmmm. I'm not sure. That sounds wrong. The theorem says you need a large $n$, so I guess I think you can't apply it if you only sampled a small number of values.

Teacher: Okay, so let's say we are sampling a billion values. Then we have large $n$. And we've established that this means that the first few random variables in the sequence are normally distributed, to a very close approximation. If that's true, can't we just stop sampling early? Say we were going to sample a billion values, but then we stop sampling after the first value. Was that random variable still normally distributed?

Student: I think maybe it isn't.

Teacher: Okay, so at some point its distribution changes?

Student: I'm not sure. I'm a bit confused about it now.

Teacher: Hmmm, well it seems we have something strange going on here. Why don't you have another read of the material on the central limit theorem and see if you can figure out how to resolve that contradiction. Let's talk more about it then.

That is one possible approach, which seeks to reduce the false theorem down to the reductio which says that every IID sequence (with finite variance) must be composed of normal random variables. (You should be prepared to encounter some variations in the student's views, which might include confusion between the probability distribution of a random variable and the sampling distribution of a sample. Additional confusions or issues may take the conversation off on a tangent, but the above captures a common flow of argument.) Assuming that the conversation goes as it should, either the student will get to the false (and absurd) conclusion and realise something is wrong, or they will defend against this conclusion by saying that the distribution changes as $n$ gets large (or they may handwave a bit, and you might have to lawyer them to a conclusion). Either way, this usually provokes some further thinking that can lead them to re-read the theorem. Here is another approach:

Teacher: Let's look at this another way. Suppose we have an IID sequence of random variables from some other distribution; one that is not a normal distribution. Is that possible? For example, could we have a sequence of random variables representing outcome of coin flip, from the Bernoulli distribution?

Student: Yeah, we can have that.

Teacher: Okay, great. And these are all IID values, so again, they all have the same distribution. So every random variable in that sequence is going to have a distribution that is not a normal distribution, right?

Student: Yeah.

Teacher: In fact, in this case, every value in the sequence will be the outcome of a coin flip, which we set as zero or one. Is that right?

Student: Yeah, as long as we label them that way.

Teacher: Okay, great. So if all the values in the sequence are zeroes or ones, no matter how many of them we sample, we are always going to get a histogram showing values at zero and one, right?

Student: Yeah.

Teacher: Okay. And do you think if we sample more and more values, we will get closer and closer to the true distribution? Like, if it is a fair coin, does the histogram eventually converge to where the relative frequency bars are the same height?

Student: I guess so. I think it does.

Teacher: I think you're right. In fact, we call that result the "law of large numbers". Anyway, it seems like we have a bit of a problem here doesn't it. If we sample a large number of the values then the central limit theorem says we converge to a normal distribution, but it sounds like the "law of large numbers" says we actually converge to the true distribution, which isn't a normal distribution. In fact, it's a distribution that is just probabilities on the zero value and the one value, which looks nothing like the normal distribution. So which is it?

Student: I think when $n$ is large it looks like a normal distribution.

Teacher: So describe it to me. Let's say we have flipped the coin a billion times. Describe the distribution of the outcomes and explain why that looks like a normal distribution.

Student: I'm not really sure how to do that.

Teacher: Okay. Well, do you agree that if we have a billion coin flips, all those outcomes are zeroes and ones?

Student: Yeah.

Teacher: Okay, so describe what its histogram looks like.

Student: It's just two bars on those values.

Teacher: Okay, so not "bell curve" shaped?

Student: Yeah, I guess not.

Teacher: Hmmm, so perhaps the central limit theorem doesn't say what we thought. Why don't you read the material on the central limit theorem again and see if you can figure out what it says. Let's talk more about it then.

Answer 2

As whuber notes, you can always point your collaborators to a binary discrete distribution. But they might consider that "cheating" and retreat to the weaker claim that the proposed statement only applied to continuous distributions.

So use the uniform distribution on the unit interval $[0,1]$. It has a mean of $\mu=0.5$, a variance of $\frac{1}{12}$, thus a standard deviation of $\sigma=\frac{1}{\sqrt{12}}\approx 0.289$. But of course the interval $[\mu-\sigma,\mu+\sigma]\approx[0.211,0.789]$ of length $2\sigma\approx 0.577$ only contains $57.7\%$ of your data (more specifically: as the sample size increases, the proportion approaches $0.577$), not $68\%$, no matter how many data points you sample.

Answer 3

The central limit theorem states that the mean of the data will become normally distributed as the sample size increases, it says nothing about the data itself. Another way to put it is the distribution of the parameter (the mean) is normal, but that is entirely separate from the distribution of the underlying data.

Most of the value from the CLT comes from the fact that you can compare samples that are not normally distributed to one another (based solely on the fact that, due to the CLT, you know how their means should behave).

I think where this gets confusing is that just because you can compare two sample means to each other based on some test that assumes normality (eg. t-test) doesn't mean that you should. (ie comparing the means of two exponential distributions might not tell you what you think it does, or two bi-modal distributions, or a bi-modal with a uni-modal distribution, ect).

The question most people should ask is, "is the mean (or a difference in means) a useful metric given the distribution of my data". Only if the answer to this question is yes, should one proceed to compare means (thus relying on the CLT).

By not asking this question, many people fall into the following (roughly stated) logical fallacy:

The CLT applies, so I can compare means. And I can compare means because they are normally distributed. This comparison must be meaningful, because the CLT says I can do it (and the CLT is very powerful). The comparison/test I am using most intuitively (/only) makes sense when the data is normally distributed, and after all, the mean is normally distributed, so my data must be normally distributed too!

To directly answer the question, you can:

1. Show them the definition, point out that the CLT only makes a claim about the distribution of the mean approaching normality, emphasize the distribution of a parameter can be very different from the distribution of the data from which it is derived.

2. Show them this video which provides a nice visual representation of how the CLT works using several different distributions for the underlying data. (its a bit quirky, but communicated very clearly)

Addendum:

I glossed over some technical details in my explanation in order to make it more understandable to someone who is less familiar with statistics. Several commenters have pointed this out and so I thought I would include their feedback here:

• A more accurate statement of the CLT would be:

"The central limit theorem states that the mean of the data will become normally distributed (more specifically the difference between the mean of the data/sample and the true mean, multiplied by the square root of the sample size $\sqrt{n}$ is normal distributed)"

I have also seen this explained as "the properly normalized sum tends toward a normal distribution"

It is also worth pointing out that the data must be composed of independent and identically distributed random variables with finite variance in order for the CLT to apply.

• A more accurate and/or less Bayesian way to say "the distribution of the parameter (mean)" would be "the distribution of the parameter estimate by the regular sample mean"

Answer 4

CLT is about convergence of a sum of random variables. If we have an iid sample $X_1,...,X_n$, where $EX_i=\mu$ and $Var(X_i)<\infty$ then

$$ \frac{1}{\sqrt{n}}\left(X_1+...+X_n-n\mu\right) \to N(0, Var(X_i)) $$

This statement is solely about closeness of a distribution of suitably normalized sum $(X_1+...+X_n)$ to the normal distribution. It does not say that nothing about convergence of distribution of $X_i$. Since $X_i$ do not depend on $n$ why should they converge anywhere?

Empirical distribution of a sample $X_i$ will actually converge (as sample size increases) to the actual distribution of $X_i$ according to Donsker theorem, so unless the actual distribution is not close to normal, the empirical distribution will not be close to it either.

Answer 5

This is how I like to visualize the CLT. I'm not 100% sure the argument is correct though, please check.

Start with a population of values whose distribution is nowhere near normal. E.g., a uniform distribution:

X <- runif(n= 50000)
hist(X)

Now, take $n$ samples from this population, calculate the mean of each sample, shift the sample mean by the mean of the population and scale it by $\sqrt{n}$, plot a histogram of these $n$ means. That histogram is (close to) normal:

mu <- 1/2 # Mean of population X
x <- rep(NA, 1000) 
size <- 10
for(i in 1:length(x)) {
    x[i] <- sqrt(size) * (mean(sample(X, size= size)) - mu)
}

Answer 6

The point of confusion here is what is actually converging to a normal distribution. I think the easiest way to overcome this is to explain examples of the extremes of a sampling distribution, one with one measurement per sample (just as if taking measurements straight from the population as you describe) and one where each sample is the entire population. From there it is easier to understand what happens in the middle ground.