In several different contexts we invoke the central limit theorem to justify whatever statistical method we want to adopt (e.g., approximate the binomial distribution by a normal distribution). I understand the technical details as to why the theorem is true but it just now occurred to me that I do not really understand the intuition behind the central limit theorem.

So, what is the intuition behind the central limit theorem?

Layman explanations would be ideal. If some technical detail is needed please assume that I understand the concepts of a pdf, cdf, random variable etc but have no knowledge of convergence concepts, characteristic functions or anything to do with measure theory.

  • 7
    Good question, although my immediate reaction, backed up by my limited experience of teaching this, is that the CLT isn't initially at all intuitive to most people. If anything, it's counter-intuitive! – onestop Oct 19 '10 at 2:39
  • 2
    @onestop AMEN! staring at the binomial distribution with p = 1/2 as n increases does show the CLT is lurking - but the intuition for it has always escaped me. – ronaf Oct 19 '10 at 3:18
  • 2
    Similar question with some nice ideas: stats.stackexchange.com/questions/643/… – mbq Oct 19 '10 at 6:42
  • 1
    Not an explanation but this simulation can be helpful understanding it. – David Lane May 12 '17 at 18:12

I apologize in advance for the length of this post: it is with some trepidation that I let it out in public at all, because it takes some time and attention to read through and undoubtedly has typographic errors and expository lapses. But here it is for those who are interested in the fascinating topic, offered in the hope that it will encourage you to identify one or more of the many parts of the CLT for further elaboration in responses of your own.


Most attempts at "explaining" the CLT are illustrations or just restatements that assert it is true. A really penetrating, correct explanation would have to explain an awful lot of things.

Before looking at this further, let's be clear about what the CLT says. As you all know, there are versions that vary in their generality. The common context is a sequence of random variables, which are certain kinds of functions on a common probability space. For intuitive explanations that hold up rigorously I find it helpful to think of a probability space as a box with distinguishable objects. It doesn't matter what those objects are but I will call them "tickets." We make one "observation" of a box by thoroughly mixing up the tickets and drawing one out; that ticket constitutes the observation. After recording it for later analysis we return the ticket to the box so that its contents remain unchanged. A "random variable" basically is a number written on each ticket.

In 1733, Abraham de Moivre considered the case of a single box where the numbers on the tickets are only zeros and ones ("Bernoulli trials"), with some of each number present. He imagined making $n$ physically independent observations, yielding a sequence of values $x_1, x_2, \ldots, x_n$, all of which are zero or one. The sum of those values, $y_n = x_1 + x_2 + \ldots + x_n$, is random because the terms in the sum are. Therefore, if we could repeat this procedure many times, various sums (whole numbers ranging from $0$ through $n$) would appear with various frequencies--proportions of the total. (See the histograms below.)

Now one would expect--and it's true--that for very large values of $n$, all the frequencies would be quite small. If we were to be so bold (or foolish) as to attempt to "take a limit" or "let $n$ go to $\infty$", we would conclude correctly that all frequencies reduce to $0$. But if we simply draw a histogram of the frequencies, without paying any attention to how its axes are labeled, we see that the histograms for large $n$ all begin to look the same: in some sense, these histograms approach a limit even though the frequencies themselves all go to zero.

Histograms

These histograms depict the results of repeating the procedure of obtaining $y_n$ many times. $n$ is the "number of trials" in the titles.

The insight here is to draw the histogram first and label its axes later. With large $n$ the histogram covers a large range of values centered around $n/2$ (on the horizontal axis) and a vanishingly small interval of values (on the vertical axis), because the individual frequencies grow quite small. Fitting this curve into the plotting region has therefore required both a shifting and rescaling of the histogram. The mathematical description of this is that for each $n$ we can choose some central value $m_n$ (not necessarily unique!) to position the histogram and some scale value $s_n$ (not necessarily unique!) to make it fit within the axes. This can be done mathematically by changing $y_n$ to $z_n = (y_n - m_n) / s_n$.

Remember that a histogram represents frequencies by areas between it and the horizontal axis. The eventual stability of these histograms for large values of $n$ should therefore be stated in terms of area. So, pick any interval of values you like, say from $a$ to $b \gt a$ and, as $n$ increases, track the area of the part of the histogram of $z_n$ that horizontally spans the interval $(a, b]$. The CLT asserts several things:

  1. No matter what $a$ and $b$ are, if we choose the sequences $m_n$ and $s_n$ appropriately (in a way that does not depend on $a$ or $b$ at all), this area indeed approaches a limit as $n$ gets large.

  2. The sequences $m_n$ and $s_n$ can be chosen in a way that depends only on $n$, the average of values in the box, and some measure of spread of those values--but on nothing else--so that regardless of what is in the box, the limit is always the same. (This universality property is amazing.)

  3. Specifically, that limiting area is the area under the curve $y = \exp(-z^2/2) / \sqrt{2 \pi}$ between $a$ and $b$: this is the formula of that universal limiting histogram.

    The first generalization of the CLT adds,

  4. When the box can contain numbers in addition to zeros and ones, exactly the same conclusions hold (provided that the proportions of extremely large or small numbers in the box are not "too great," a criterion that has a precise and simple quantitative statement).

    The next generalization, and perhaps the most amazing one, replaces this single box of tickets with an ordered indefinitely long array of boxes with tickets. Each box can have different numbers on its tickets in different proportions. The observation $x_1$ is made by drawing a ticket from the first box, $x_2$ comes from the second box, and so on.

  5. Exactly the same conclusions hold provided the contents of the boxes are "not too different" (there are several precise, but different, quantitative characterizations of what "not too different" has to mean; they allow an astonishing amount of latitude).

These five assertions, at a minimum, need explaining. There's more. Several intriguing aspects of the setup are implicit in all the statements. For example,

  • What is special about the sum? Why don't we have central limit theorems for other mathematical combinations of numbers such as their product or their maximum? (It turns out we do, but they are not quite so general nor do they always have such a clean, simple conclusion unless they can be reduced to the CLT.) The sequences of $m_n$ and $s_n$ are not unique but they're almost unique in the sense that eventually they have to approximate the expectation of the sum of $n$ tickets and the standard deviation of the sum, respectively (which, in the first two statements of the CLT, equals $\sqrt{n}$ times the standard deviation of the box).

    The standard deviation is one measure of the spread of values, but it is by no means the only one nor is it the most "natural," either historically or for many applications. (Many people would choose something like a median absolute deviation from the median, for instance.)

  • Why does the SD appear in such an essential way?

  • Consider the formula for the limiting histogram: who would have expected it to take such a form? It says the logarithm of the probability density is a quadratic function. Why? Is there some intuitive or clear, compelling explanation for this?


I confess I am unable to reach the ultimate goal of supplying answers that are simple enough to meet Srikant's challenging criteria for intuitiveness and simplicity, but I have sketched this background in the hope that others might be inspired to fill in some of the many gaps. I think a good demonstration will ultimately have to rely on an elementary analysis of how values between $\alpha_n = a s_n + m_n$ and $\beta_n = b s_n + m_n$ can arise in forming the sum $x_1 + x_2 + \ldots + x_n$. Going back to the single-box version of the CLT, the case of a symmetric distribution is simpler to handle: its median equals its mean, so there's a 50% chance that $x_i$ will be less than the box's mean and a 50% chance that $x_i$ will be greater than its mean. Moreover, when $n$ is sufficiently large, the positive deviations from the mean ought to compensate for the negative deviations in the mean. (This requires some careful justification, not just hand waving.) Thus we ought primarily to be concerned about counting the numbers of positive and negative deviations and only have a secondary concern about their sizes. (Of all the things I have written here, this might be the most useful at providing some intuition about why the CLT works. Indeed, the technical assumptions needed to make the generalizations of the CLT true essentially are various ways of ruling out the possibility that rare huge deviations will upset the balance enough to prevent the limiting histogram from arising.)

This shows, to some degree anyway, why the first generalization of the CLT does not really uncover anything that was not in de Moivre's original Bernoulli trial version.

At this point it looks like there is nothing for it but to do a little math: we need to count the number of distinct ways in which the number of positive deviations from the mean can differ from the number of negative deviations by any predetermined value $k$, where evidently $k$ is one of $-n, -n+2, \ldots, n-2, n$. But because vanishingly small errors will disappear in the limit, we don't have to count precisely; we only need to approximate the counts. To this end it suffices to know that

$$\text{The number of ways to obtain } k \text{ positive and } n-k \text{ negative values out of } n$$

$$\text{equals } \frac{n-k+1}{k}$$

$$\text{times the number of ways to get } k-1 \text{ positive and } n-k+1 \text { negative values.}$$

(That's a perfectly elementary result so I won't bother to write down the justification.) Now we approximate wholesale. The maximum frequency occurs when $k$ is as close to $n/2$ as possible (also elementary). Let's write $m = n/2$. Then, relative to the maximum frequency, the frequency of $m+j+1$ positive deviations ($j \ge 0$) is estimated by the product

$$\frac{m+1}{m+1} \frac{m}{m+2} \cdots \frac{m-j+1}{m+j+1}$$

$$=\frac{1 - 1/(m+1)}{1 + 1/(m+1)} \frac{1-2/(m+1)}{1+2/(m+1)} \cdots \frac{1-j/(m+1)}{1+j/(m+1)}.$$

135 years before de Moivre was writing, John Napier invented logarithms to simplify multiplication, so let's take advantage of this. Using the approximation

$$\log\left(\frac{1-x}{1+x}\right) \sim -2x,$$

we find that the log of the relative frequency is approximately

$$-2/(m+1) - 4/(m+1) - \cdots - 2j/(m+1) = -\frac{j(j+1)}{m+1} \sim -\frac{j^2}{m}.$$

Because the cumulative error is proportional to $j^4/m^3$, this ought to work well provided $j^4$ is small relative to $m^3$. That covers a greater range of values of $j$ than is needed. (It suffices for the approximation to work for $j$ only on the order of $\sqrt{m}$ which asymptotically is much smaller than $m^{3/4}$.)


Obviously much more analysis of this sort should be presented to justify the other assertions in the CLT, but I'm running out of time, space, and energy and I've probably lost 90% of the people who started reading this anyway. This simple approximation, though, suggests how de Moivre might originally have suspected that there is a universal limiting distribution, that its logarithm is a quadratic function, and that the proper scale factor $s_n$ must be proportional to $\sqrt{n}$ (because $j^2/m = 2 j^2 / n = 2 (j/\sqrt{n})^2$). It is difficult to imagine how this important quantitative relationship could be explained without invoking some kind of mathematical information and reasoning; anything less would leave the precise shape of the limiting curve a complete mystery.

  • 5
    +1 It will take me some time to digest your answer. I admit that asking for an intuition for the CLT within the constraints I imposed may be nearly impossible. – user28 Oct 23 '10 at 1:42
  • 2
    Thank you for taking the time to write this, it's the most helpful exposition of the CLT I've seen that is also very accessible mathematically. – jeremy radcliff Jul 11 '16 at 3:25
  • 1
    Yes, quite dense.... so many questions. How does the first histogram have 2 bars (there was only 1 trial!) ; can I just ignore that? And the convention is usually to avoid horizontal gaps between bars of a histogram, right? (because , as you say, area is important, and the area will eventually be calculated over a continuous (i.e. no gaps) domain) ? So I'll ignore the gaps, too...? Even I had gaps when I first tried to understand it :) – The Red Pea May 12 '17 at 19:15
  • @TheRed Thank you for your questions. I have edited the first part of this post to make these points a little clearer. – whuber May 12 '17 at 19:41
  • 3
    Ah, yes, I confused "number of trials= $n$ = "observations"" with "number of times (this entire procedure) is repeated". So if a ticket can only have the a value of the two values, 0 or 1, and you only observe one ticket, the sum of those tickets' values can only be one of two things: 0, or 1. Hence your first histogram has two bars. Moreover, these bars are roughly equal in height because we expect 0 and 1 to occur in equal proportions. – The Red Pea May 12 '17 at 20:27

The nicest animation I know: http://www.ms.uky.edu/~mai/java/stat/GaltonMachine.html

8 horizontal layers of equally spaced pins, each layer staggered, results in a "pachinko/pinball" style obstacle for balls dropped through these pins. Each ball falls at the bottom, and as the balls stack, their height approaches an outline of the Gaussian curve. This illustrates that the sum of many independent random events (the layers), will result in a Gaussian distribution of results (the stacked ball height)

The simplest words I have read: http://elonen.iki.fi/articles/centrallimit/index.en.html

If you sum the results of these ten throws, what you get is likely to be closer to 30-40 than the maximum, 60 (all sixes) or on the other hand, the minumum, 10 (all ones).

The reason for this is that you can get the middle values in many more different ways than the extremes. Example: when throwing two dice: 1+6 = 2+5 = 3+4 = 7, but only 1+1 = 2 and only 6+6 = 12.

That is: even though you get any of the six numbers equally likely when throwing one die, the extremes are less probable than middle values in sums of several dice.

Intuition is a tricky thing. It's even trickier with theory in our hands tied behind our back.

The CLT is all about sums of tiny, independent disturbances. "Sums" in the sense of the sample mean, "tiny" in the sense of finite variance (of the population), and "disturbances" in the sense of plus/minus around a central (population) value.

For me, the device that appeals most directly to intuition is the quincunx, or 'Galton box', see Wikipedia (for 'bean machine'?) The idea is to roll a tiny little ball down the face of a board adorned by a lattice of equally spaced pins. On its way down the ball diverts right and left (...randomly, independently) and collects at the bottom. Over time, we see a nice bell shaped mound form right before our eyes.

The CLT says the same thing. It is a mathematical description of this phenomenon (more precisely, the quincunx is physical evidence for the normal approximation to the binomial distribution). Loosely speaking, the CLT says that as long as our population is not overly misbehaved (that is, if the tails of the PDF are sufficiently thin), then the sample mean (properly scaled) behaves just like that little ball bouncing down the face of the quincunx: sometimes it falls off to the left, sometimes it falls off to the right, but most of the time it lands right around the middle, in a nice bell shape.

The majesty of the CLT (to me) is that the shape of the underlying population is irrelevant. Shape only plays a role insofar as it delegates the length of time we need to wait (in the sense of sample size).

An observation concerning the CLT may be the following. When you have a sum $$ S = X_1 + X_2 + \ldots + X_n $$ of a lot of random components, if one is "smaller than usual" then this is mostly compensated for by some of the other components being "larger than usual". In other words, negative deviations and positive deviations from the component means cancel each other out in the summation. Personally, I have no clear-cut intuition why exactly the remaining deviations form a distribution that looks more and more normal the more terms you have.

There are many versions of the CLT, some stronger than others, some with relaxed conditions such as a moderate dependence between the terms and/or non-identical distributions for the terms. In the simplest-to-prove versions of the CLT, the proof is usually based on the moment-generating function (or Laplace-Stieltjes transform or some other appropriate transform of the density) of the sum $S$. Writing this as a Taylor expansion and keeping only the most dominant term gives you the moment-generating function of the normal distribution. So for me personally, the normality is something that follows from a bunch of equations and I can not provide any further intuition than that.

It should be noted however that the sum's distribution, never really is normally distributed, nor does the CLT claims that it would be. If $n$ is finite, there is still some distance to the normal distribution and if $n=\infty$ both the mean and the variance are infinite as well. In the latter case you could take the mean of the infinite sum, but then you get a deterministic number without any variance at all, which could hardly be labelled as "normally distributed".

This may pose problems with practical applications of the CLT. Usually, if you are interested in the distribution of $S/n$ close to its center, CLT works fine. However, convergence to the normal is not uniform everywhere and the further you get away from the center, the more terms you need to have a reasonable approximation.

With all the "sanctity" of the Central Limit Theorem in statistics, its limitations are often overlooked all too easily. Below I give two slides from my course making the point that CLT utterly fails in the tails, in any practical use case. Unfortunately, a lot of people specifically use CLT to estimate tail probabilities, knowingly or otherwise.

enter image description here enter image description here

  • 3
    This is great material and wise advice. I cannot upvote it, unfortunately, because the assertions in "This normality is a mathematical artifact and I think it is not useful to search for any deeper truth or intuition behind it" are deeply troubling. They seem to suggest that (1) we shouldn't rely on mathematics to help us theoretically and (2) there is no point to understanding the math in the first place. I hope that other posts in this thread already go a long way towards disproving the second assertion. The first is so self-inconsistent it hardly bears further analysis. – whuber Mar 22 '15 at 17:56
  • 2
    @whuber. You are right, I am out of my league perhaps. I'll edit. – StijnDeVuyst Mar 22 '15 at 20:32
  • 2
    Thank you for reconsidering the problematic part, and a big +1 for the rest. – whuber Mar 22 '15 at 21:17

I gave up on trying to come up with an intuitive version and came up with some simulations. I have one that does a quincunx and some others that do things like show how even a skewed raw reaction time distribution will become normal if you collect enough RT's per subject. I think they help but they're new in my class this year and I haven't graded the first test yet.

One thing that I thought was good was being able to show the law of large numbers as well. I could show how variable things are with small sample sizes and then show how they stabilize with large ones. I do a bunch of other large number demos as well. I can show the interaction in the quincunx between the numbers of random processes and the numbers of samples.

(turns out not being able to use a chalk or white board in my class may have been a blessing)

I think it's important to realize that a histogram of coin flips already approximates the normal distribution, that's just a fact of reality (in fact this is probably why the normal distribution is called the normal distribution): a simple plot of a 1/2 bar in the middle, 1/4 bars on either side of that middle, 1/8 bars next to the 1/4 bars and so on (a histogram of the probabilities of getting heads or tails x number of times in a row) is already pretty close to the normal distribution, but here's the thing: when you have a loaded coin with 2/3 probability of getting tails, the histogram is also normally distributed. When you add a lot of histograms of random distributions together you either maintain the normal distribution shape because all of the individual histograms already have that shape or you get that shape because fluctuations in the individual histograms tend to cancel each other out if you add a large number of histograms. A histogram of a random distribution of one variable is already approximately distributed in a way that people have started calling the normal distribution because it's so common and that's a microcosm of the central limit theorem.

This is not the whole story but I think it's as intuitive as it gets.

  • 2
    Your description of a "normal distribution" sounds instead like a discrete version of the double exponential, which is not even remotely like a Gaussian normal distribution (except insofar both are unimodal and symmetric). The histogram of coin flips does not have bars that decrease by a factor of $2$ with each step! That suggests there may be some difficulties lurking in this explanation that have been papered over by an appeal to "intuition." – whuber Aug 23 '13 at 18:18
  • 5
    This answer is mostly nonsense. No number of flips of a fair coin will result in a distribution of number of heads that has probabilities $\frac 18, \frac 14, \frac 12, \frac 14, \frac 18$; indeed that is not even a probability mass function! Nor does the number of heads in a row have anything to do with the question. – Dilip Sarwate Aug 23 '13 at 22:32

protected by Glen_b Jan 14 '17 at 1:00

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?