# What intuitive explanation is there for the central limit theorem?

In several different contexts we invoke the central limit theorem to justify whatever statistical method we want to adopt (e.g., approximate the binomial distribution by a normal distribution). I understand the technical details as to why the theorem is true but it just now occurred to me that I do not really understand the intuition behind the central limit theorem.

So, what is the intuition behind the central limit theorem?

Layman explanations would be ideal. If some technical detail is needed please assume that I understand the concepts of a pdf, cdf, random variable etc but have no knowledge of convergence concepts, characteristic functions or anything to do with measure theory.

-
Good question, although my immediate reaction, backed up by my limited experience of teaching this, is that the CLT isn't initially at all intuitive to most people. If anything, it's counter-intuitive! –  onestop Oct 19 '10 at 2:39
@onestop AMEN! staring at the binomial distribution with p = 1/2 as n increases does show the CLT is lurking - but the intuition for it has always escaped me. –  ronaf Oct 19 '10 at 3:18
Similar question with some nice ideas: stats.stackexchange.com/questions/643/… –  mbq Oct 19 '10 at 6:42

I apologize in advance for the length of this post: it is with some trepidation that I let it out in public at all, because it takes some time and attention to read through and undoubtedly has typographic errors and expository lapses. But here it this for those who are interested in the fascinating topic, offered in the hope that it will encourage you to identify one or more of the many parts of the CLT for further elaboration in responses of your own.

Most attempts at "explaining" the CLT are illustrations or just restatements that assert it is true. A really penetrating, correct explanation would have to explain an awful lot of things.

Before looking at this further, let's be clear about what the CLT says. As you all know, there are versions that vary in their generality. The common context is a sequence of random variables. For intuitive explanations that hold up rigorously I find it helpful to think of a probability space as a box with distinguishable objects. It doesn't matter what those objects are but I will call them "tickets." We make one "observation" of a box by thoroughly mixing up the tickets and drawing one out; that ticket constitutes the observation. After recording it for later analysis we return the ticket to the box so that its contents remain unchanged. A "random variable" is just a number written on each ticket.

In 1735, Abraham de Moivre considered the case of a single box where the numbers on the tickets were only zeros and ones ("Bernoulli trials"), with some of each number present. He imagined making $n$ physically independent observations, yielding a sequence of values $x_1, x_2, \ldots, x_n$, all of which are zero or one. The sum of those values, $y_n = x_1 + x_2 + \ldots + x_n$ is random because the terms in the sum are. Therefore, if we could repeat this procedure many times, various sums (ranging from $0$ through $n$) would appear with various frequencies. Now one would expect--and it's true--that for very large values of $n$, all the relative frequencies would be quite small. If we were to be so bold (or foolish) as to attempt to "take a limit" or "let $n$ go to $\infty$", we would conclude that all frequencies reduce to $0$. But if we simply draw a histogram of the frequencies, without paying any attention to how its axes are labeled, we see that the histograms for large $n$ all begin to look the same: in some sense, they approach a limit.

The insight here is to draw the histogram first and label its axes later. With large $n$ the histogram covers a large range of values centered around $n/2$ (on the horizontal axis) and a vanishingly small range of values (on the vertical axis). Fitting this curve into the plotting region has required both a shifting and rescaling of the histogram. The mathematical description of this is that for each $n$ we can choose some central value $m_n$ (not necessarily unique!) to position the histogram and some scale value $s_n$ (not necessarily unique!) to make it fit within the axes. This is done mathematically by changing $y_n$ to $z_n = (y_n - m_n) / s_n$.

Remember that a histogram represents frequencies by areas between it and the horizontal axis. The eventual stability of these histograms for large values of $n$ should therefore be stated in terms of area. So, pick any interval of values you like, say from $a$ to $b \gt a$ and, as $n$ increases, track the area of the part of the histogram of $z_n$ that horizontally spans the interval $(a, b]$. The CLT asserts several things:

1. No matter what $a$ and $b$ are, if we choose the sequences $m_n$ and $s_n$ appropriately (in a way that does not depend on $a$ or $b$ at all), this area indeed approaches a limit as $n$ gets large.

2. The sequences $m_n$ and $s_n$ can be chosen in a way that depends only on $n$, the average of values in the box, and some measure of spread of those values--but on nothing else--so that regardless of what is in the box, the limit is always the same.

3. Specifically, that limiting area is the area under the curve $y = \exp(-z^2/2) / \sqrt{2 \pi}$ between $a$ and $b$: this is the formula of that universal limiting histogram.

The first generalization of the CLT adds,

4. When the box can contain numbers in addition to zeros and ones, exactly the same conclusions hold (provided that the proportions of extremely large or small numbers in the box are not "too great," a criterion that has a precise quantitative statement).

The next generalization, and perhaps the most amazing one, replaces this single box of tickets with an ordered indefinitely long array of boxes with tickets. Each box can have different numbers on its tickets in different proportions. The observation $x_1$ is made by drawing a ticket from the first box, $x_2$ comes from the second box, and so on.

5. Exactly the same conclusions hold provided the contents of the boxes are "not too different" (there are several precise, but different, quantitative characterizations of what "not too different" has to mean; they allow an astonishing amount of latitude).

These five assertions, at a minimum, need explaining. There's more. Several intriguing aspects of the setup are implicit in all the statements. For example,

• What is special about the sum? Why don't we have central limit theorems for other mathematical combinations of numbers such as their product or their maximum? (It turns out we do, but they are not quite so general nor do they always have such a clean, simple conclusion unless they can be reduced to the CLT.) The sequences of $m_n$ and $s_n$ are not unique but they're almost unique in the sense that eventually they have to approximate the expectation of the sum of $n$ tickets and the standard deviation of the sum, respectively (which, in the first two statements of the CLT, equals $\sqrt{n}$ times the standard deviation of the box).

The standard deviation is one measure of the spread of values, but it is by no means the only one nor is it the most "natural," either historically or for many applications. (Many people would choose something like a median absolute deviation from the median, for instance.)

• Why does the SD appear in such an essential way?

• Consider the formula for the limiting histogram: who would have expected it to take such a form? It says the logarithm of the probability density is a quadratic function. Why? Is there some intuitive or clear, compelling explanation for this?

I confess I am unable to reach the ultimate goal of supplying answers that are simple enough to meet Srikant's challenging criteria for intuitiveness and simplicity, but I have sketched this background in the hope that others might be inspired to fill in some of the many gaps. I think a good demonstration will ultimately have to rely on an elementary analysis of how values between $\alpha_n = a s_n + m_n$ and $\beta_n = b s_n + m_n$ can arise in forming the sum $x_1 + x_2 + \ldots + x_n$. Going back to the single-box version of the CLT, the case of a symmetric distribution is simpler to handle: its median equals its mean, so there's a 50% chance that $x_i$ will be less than the box's mean and a 50% chance that $x_i$ will be greater than its mean. Moreover, when $n$ is sufficiently large, the positive deviations from the mean ought to compensate for the negative deviations in the mean. (This requires some careful justification, not just hand waving.) Thus we ought primarily to be concerned about counting the numbers of positive and negative deviations and only have a secondary concern about their sizes. (Of all the things I have written here, this might be the most useful at providing some intuition about why the CLT works. Indeed, the technical assumptions needed to make the generalizations of the CLT true essentially are various ways of ruling out the possibility that rare huge deviations will upset the balance enough to prevent the limiting histogram from arising.)

This shows, to some degree anyway, why the first generalization of the CLT does not really uncover anything that was not in de Moivre's original Bernoulli trial version.

At this point it looks like there nothing for it but to do a little math: we need to count the number of distinct ways in which the number of positive deviations from the mean can differ from the number of negative deviations by any predetermined value $k$, where evidently $k$ is one of $-n, -n+2, \ldots, n-2, n$. But because vanishingly small errors will disappear in the limit, we don't have to count precisely; we only need to approximate the counts. To this end it suffices to know that

$$\text{The number of ways to obtain } k \text{ positive and } n-k \text{ negative values out of } n$$

$$\text{equals } \frac{n-k}{k}$$

$$\text{times the number of ways to get } k-1 \text{ positive and } n-k+1 \text { negative values.}$$

(That's a perfectly elementary result so I won't bother to write down the justification.) Now we approximate wholesale. The maximum frequency occurs when $k$ is as close to $n/2$ as possible (also elementary). Let's write $m = n/2$. Then, relative to the maximum frequency, the frequency of $m+j$ positive deviations ($j \ge 0$) is estimated by the product

$$\frac{m-1}{m+1} \frac{m-2}{m+2} \cdots \frac{m-j}{m+j}$$

$$=\frac{1 - 1/m}{1 + 1/m} \frac{1-2/m}{1+2/m} \cdots \frac{1-j/m}{1+j/m}.$$

135 years before de Moivre was writing, John Napier invented logarithms to simplify multiplication, so let's take advantage of this. Using the approximation

$$\log(\frac{1-x}{1+x}) \sim -2x,$$

we find that the log of the relative frequency is approximately

$$-2/m - 4/m - \cdots - (2j)/m = -\frac{j(j+1)}{m} \sim -\frac{j^2}{m}.$$

Because the cumulative error is proportional to $j^4/m^3$, this ought to work well provided $j^4$ is small relative to $m^3$. That covers a greater range of values of $j$ than is needed. (It suffices for the approximation to work for $j$ only on the order of $\sqrt{m} \ll m^{3/4}$.)

Obviously much more analysis of this sort should be presented to justify the other assertions in the CLT, but I'm running out of time, space, and energy and I've probably lost 90% of the people who started reading this anyway. This simple approximation, though, suggests how de Moivre might originally have suspected that there is a universal limiting distribution, that its logarithm is a quadratic function, and that the proper scale factor $s_n$ must be proportional to $\sqrt{n}$ (because $j^2/m = 2 j^2 / n = 2 (j/\sqrt{n})^2$). It is difficult to imagine how this important quantitative relationship could be explained without invoking some kind of mathematical information and reasoning; anything less would leave the precise shape of the limiting curve a complete mystery.

-
+1 It will take me some time to digest your answer. I admit that asking for an intuition for the CLT within the constraints I imposed may be nearly impossible. –  user28 Oct 23 '10 at 1:42
+1 that was a great read, thank you. –  ars Oct 30 '10 at 3:03

Intuition is a tricky thing. It's even trickier with theory in our hands tied behind our back.

The CLT is all about sums of tiny, independent disturbances. "Sums" in the sense of the sample mean, "tiny" in the sense of finite variance (of the population), and "disturbances" in the sense of plus/minus around a central (population) value.

For me, the device that appeals most directly to intuition is the quincunx, or 'Galton box', see Wikipedia (for 'bean machine'?) The idea is to roll a tiny little ball down the face of a board adorned by a lattice of equally spaced pins. On its way down the ball diverts right and left (...randomly, independently) and collects at the bottom. Over time, we see a nice bell shaped mound form right before our eyes.

The CLT says the same thing. It is a mathematical description of this phenomenon (more precisely, the quincunx is physical evidence for the normal approximation to the binomial distribution). Loosely speaking, the CLT says that as long as our population is not overly misbehaved (that is, if the tails of the PDF are sufficiently thin), then the sample mean (properly scaled) behaves just like that little ball bouncing down the face of the quincunx: sometimes it falls off to the left, sometimes it falls off to the right, but most of the time it lands right around the middle, in a nice bell shape.

The majesty of the CLT (to me) is that the shape of the underlying population is irrelevant. Shape only plays a role insofar as it delegates the length of time we need to wait (in the sense of sample size).

-

The nicest animation I know: http://www.ms.uky.edu/~mai/java/stat/GaltonMachine.html

The simplest words I have read: http://elonen.iki.fi/articles/centrallimit/index.en.html

If you sum the results of these ten throws, what you get is likely to be closer to 30-40 than the maximum, 60 (all sixes) or on the other hand, the minumum, 10 (all ones).

The reason for this is that you can get the middle values in many more different ways than the extremes. Example: when throwing two dice: 1+6 = 2+5 = 3+4 = 7, but only 1+1 = 2 and only 6+6 = 12.

That is: even though you get any of the six numbers equally likely when throwing one die, the extremes are less probable than middle values in sums of several dice.

-

I gave up on trying to come up with an intuitive version and came up with some simulations. I have one that does a quincunx and some others that do things like show how even a skewed raw reaction time distribution will become normal if you collect enough RT's per subject. I think they help but they're new in my class this year and I haven't graded the first test yet.

One thing that I thought was good was being able to show the law of large numbers as well. I could show how variable things are with small sample sizes and then show how they stabilize with large ones. I do a bunch of other large number demos as well. I can show the interaction in the quincunx between the numbers of random processes and the numbers of samples.

(turns out not being able to use a chalk or white board in my class may have been a blessing)

-

I think it's important to realize that a histogram of coin flips already approximates the normal distribution, that's just a fact of reality (in fact this is probably why the normal distribution is called the normal distribution): a simple plot of a 1/2 bar in the middle, 1/4 bars on either side of that middle, 1/8 bars next to the 1/4 bars and so on (a histogram of the probabilities of getting heads or tails x number of times in a row) is already pretty close to the normal distribution, but here's the thing: when you have a loaded coin with 2/3 probability of getting tails, the histogram is also normally distributed. When you add a lot of histograms of random distributions together you either maintain the normal distribution shape because all of the individual histograms already have that shape or you get that shape because fluctuations in the individual histograms tend to cancel each other out if you add a large number of histograms. A histogram of a random distribution of one variable is already approximately distributed in a way that people have started calling the normal distribution because it's so common and that's a microcosm of the central limit theorem.

This is not the whole story but I think it's as intuitive as it gets.

-
Your description of a "normal distribution" sounds instead like a discrete version of the double exponential, which is not even remotely like a Gaussian normal distribution (except insofar both are unimodal and symmetric). The histogram of coin flips does not have bars that decrease by a factor of $2$ with each step! That suggests there may be some difficulties lurking in this explanation that have been papered over by an appeal to "intuition." –  whuber Aug 23 at 18:18
This answer is mostly nonsense. No number of flips of a fair coin will result in a distribution of number of heads that has probabilities $\frac 18, \frac 14, \frac 12, \frac 14, \frac 18$; indeed that is not even a probability mass function! Nor does the number of heads in a row have anything to do with the question. –  Dilip Sarwate Aug 23 at 22:32