3
$\begingroup$

Both CLT and LLN are stated in terms of a fixed probability space that admits an infinite sequence of IID RVs.

It is a common-place in many probability and statistics texts/notes that such a sequence does not exist on a finite probability space, but that the CLT and LLN apply anyway as "approximations" provided that there are "enough" samples (rules of thumb like "at least 30 samples" are often bandied about).

Is there a mathematically rigorous argument that shows this approximation is justified and, if so, how good it is?

In most sources (pick your favorite), if these issues are even brought up, the various assurances are stated without any proof or explanation beyond that it all can be observed empirically, perhaps by looking at some graph or computer simulation.

Sometimes there are references to the Berry-Esseen theorem but even there it appears the assumption of an infinite sequence remains.

Occasionally, some set of notes acknowledge that there might be another probabilistic formulation that might fit better (ex: Remark 4.6 in https://people.cs.uchicago.edu/~laci/reu02/prob.pdf; Shiryaev's Probability GTM ca p.48), but that is not generally pursued in the text and there may not be references to follow.

I've seen certain references online to "finite" versions of the CLT as far back as the 50s (including by Erdos et al) but I haven't been able to understand them.

I'm surprised that so many texts state a version of a workhorse theorem that strictly speaking doesn't apply to its most commonly suggested applications. I'm not above using things that work even though we don't really know why, but I'd just like to understand them, if that's possible, even if it requires much fancier math.

$\endgroup$
11
  • 1
    $\begingroup$ Re "Is there a mathematically rigorous argument that shows this approximation is justified and, if so, how good it is?" That's what (literally) any proof of the CLT offers. It's unclear what you mean by "doesn't apply to." Application of any mathematical theorem to reality is always a simplification and an approximation. Suggesting that "we really don't know why" completely misses the point, indicating it might be worthwhile taking a closer look at your sources -- or finding more authoritative ones. $\endgroup$
    – whuber
    Commented Jul 20, 2023 at 18:06
  • $\begingroup$ I'm studying in isolation, so I could be confused. I believe there is no infinite sequence of IID RVs on a finite probability space, ergo since the CLT statement requires such a sequence then it doesn't apply directly. Others noted that the CLT is intended to be applied not directly to the finite population $\mathcal P$ (ex the RVs are not the functions that assign a height to a sampled human from $\mathcal P$) but to an infinite probability space $\Omega$ derived from the finite population $\mathcal P$ (ex $\Omega=$ infinite sequences of height measurements associated to each human). $\endgroup$
    – ac1501
    Commented Jul 20, 2023 at 18:28
  • $\begingroup$ @ac1501 You get the infinite sequence by sampling with replacement from distributions over a finite set of outcomes. It is not clear what you mean about a probability space on the infinite sequence. $\endgroup$
    – Dave
    Commented Jul 20, 2023 at 18:30
  • $\begingroup$ @Dave yes I was referring to your point below. I use "probability space" to mean a triplet of a set $\Omega$ of "outcomes", a $\sigma$-algebra of "events" on $\Omega$, and a probability measure defined on the $\sigma$-algebra. If the finite population $\mathcal P$ itself is used as the probability space then presumably one takes the power set as $\sigma$-algebra and probabilities on the individual elements as generators of the measure. If, as you suggest, one considers the infinite sequence space derived from $\mathcal P$, then presumably one takes the infinite product of the finite space. $\endgroup$
    – ac1501
    Commented Jul 20, 2023 at 18:37
  • $\begingroup$ @whuber: Lots of proofs of the CLT don't give any explicit error bounds, so you can see why OP might object. $\endgroup$ Commented Jul 21, 2023 at 3:30

2 Answers 2

4
$\begingroup$

You're right, the CLT and such "soft" asymptotic theorems don't apply to a finite probability space, per se. They are inherently statements about an infinite sequence of independent random variables, and as you point out, in a finite probability space, no such sequence exists. So, when someone is "applying the CLT" on such a space, normally one of two things is happening:

  1. They are using the CLT just as a heuristic, and are not truly making any mathematical claim;

  2. They are actually applying some theorem other than the classical abstract CLT. The phrase "central limit theorem" is often used generically to refer to any theorem of a similar form.


Let's do an example. Suppose I roll $n=50$ fair 6-sided dice, letting $S$ be their sum, and I want to say something about $P(S \le 360)$. Clearly I can define the independent random variables $X_1, \dots, X_{50}$ on a finite probability space with $6^{50}$ outcomes.

Now I might write (ignore my careless rounding):

Let $\mu = E[X_i] = 3.5$ and $\sigma^2 = \operatorname{Var}(X_i) = 2.917$. Set $Z = (S-n\mu) / \sigma\sqrt{n}$, so that $P(S \le 375) = P(Z \le 0.82)$. By the central limit theorem, we have $Z \to N(0,1)$ in distribution, so $P(Z \le 0.82) \approx \Phi(0.82) = 0.794$. Thus $P( S \le 375)$ is approximately 0.794.

This is very familiar looking, but in fact I have not done any mathematics. Indeed, merely saying that $P( S \le 375)$ is "approximately" 0.794, is a statement with no rigorous mathematical content. I have a heuristic that suggests a value that might be close, for some unknown sense of the word "close", but I have not asserted anything certain about $P(S \le 375)$.

So this is an example of #1 above. True, the hypotheses of the CLT don't apply to this setting, but I didn't actually claim that the conclusion applies either: mathematically, I didn't really claim anything at all. So as far as math is concerned, no harm, no foul.


To get an actual result, we need a "hard" theorem providing explicit error bounds. Berry-Esseen is exactly such a theorem. It tells us that $$|P(S \le 360) - \Phi(0.82)| \le \frac{C \rho}{\sigma^3 \sqrt{n}}$$ where $\rho = E|X_i - \mu|^3 = 6.375$ and $C$ is a constant for which it is known that $C \le 0.5$. Therefore, we learn that $$|P(S \le 360) - \Phi(0.82)| \le \frac{0.5 \cdot 6.375}{(2.917)^{3/2} \sqrt{50}} = 0.09$$ which is to say that $$0.703 \le P(S \le 360) \le 0.884.$$ Ahh. Now that is a meaningful mathematical statement. This is an example of #2.


One thing that might be worrying you is that if you look carefully at Wikipedia's statement, it starts out "if $X_1, X_2, \dots$ are iid random variables...". Indeed, we don't have an infinite iid sequence, so strictly speaking we cannot apply this theorem. But this is actually just the Wikipedia editor being sloppy. They should have written "if $X_1, \dots, X_n$ are iid random variables...", because in fact that is the only hypothesis that the proof uses.

Even if somehow the proof does assume an entire infinite iid sequence, which I doubt, there would be a workaround. Suppose we have a proof of "if $Y_1,Y_2, \dots$ are iid, then the Berry-Esseen bound holds for $(Y_1 + \dots + Y_n)/n$". Suppose $X_1, \dots, X_n$ are iid random variables on some probability space $(\Omega, P)$ (possibly finite). Let $(\Omega', P')$ be another probability space on which we have an infinite sequence of iid random variables $X_1', X_2', \dots$, having the same distribution as the $X_i$. (You can construct such an $(\Omega', P')$ using Lebesgue measure, or with the Kolmogorov extension theorem if you prefer.) Consider the product probability space $(\Omega \times \Omega', P \otimes P')$; the random variables $X_i, X_i'$ extend to this space in a natural way. Apply the theorem on this probability space, taking $Y_1 = X_1, \dots, Y_n = X_n, Y_{n+1} = X_1', \dots$. Obtain a conclusion about $(Y_1 + \dots + Y_n)/n$. This is in fact a conclusion about $(X_1 + \dots + X_n)/n$, which makes sense as a random variable on $\Omega$.

This trick of "extending the probability space" is occasionally useful in other settings. Any time you are trying to prove something about some random variables $X_i$ on a probability space, if it would be helpful within the proof to have another infinite family $X_i'$ of random variables with any particular joint distribution, you can assume without loss of generality that they too exist on the given space - just so long as your conclusion doesn't mention the $X_i'$. It will remain a valid statement about the $X_i$.

$\endgroup$
1
$\begingroup$

SAMPLING WITH REPLACEMENT

Think of sampling from such a distribution as follows.

  1. Each of the finite number of events has a probability (measure) attached to it.

  2. Throw the events into a hat. This is how it is sampling with replacement.

  3. Draw an event from the hat, with the probability of drawing an event given by its probability (measure). The act of performing this draw is your $X_1$.

  4. Throw that event back into the hat.

  5. Repeat steps 3 and 4 as many times as is desired to produce $X_2$, $X_3$, etc.

There is no limit to how many times you can repeat this procedure, so the consideration of an infinite sequence for convergence theorems is reasonable.

If you skip step 4, then $X_2$ does not have the same distribution as $X_1$. As an extreme example, consider flipping a coin: you flip a certain side for the first sequence element, and this locks in the second sequence element as being the other side with probability $1$, which was not the case for the first flip (or else the coin would have landed with this side up on the first flip).

$\endgroup$
4
  • $\begingroup$ Thanks for helping me understand this! So, the probability space in this case is not the original finite population $\mathcal P$ itself but rather a new $\Omega$ defined as sequences drawn from $\mathcal P$, right? If so, does this mean there is no "finite CLT" that can be applied to a finite $\Omega$ ? $\endgroup$
    – ac1501
    Commented Jul 20, 2023 at 17:49
  • 2
    $\begingroup$ @ac1501 The CLT operates on the infinite product space of the original space (which might be finite). So far you are right (but arguably this is not so problematic). Berry-Esseen gives approximations that can be evaluated for finite $n$, and I believe they also hold if you ignore the (infinite) reminder of the sequence. $\endgroup$ Commented Jul 20, 2023 at 18:15
  • $\begingroup$ Thanks @ChristianHennig! So, there is no "finite CLT"? i.e., no direct application of CLT where the set $\Omega$ is the finite population and the RVs are literally functions from $\Omega\to\mathbb R$? Rather, one only applies CLT to the "derived" infinite spaces of sequences from $\mathcal P$ and it's only "abuse of language" that leads us to say that the probability space to which we apply the CLT is actually finite? $\endgroup$
    – ac1501
    Commented Jul 20, 2023 at 18:47
  • 2
    $\begingroup$ @ac1501 This is really about the use of language. Somebody may mean by "the probability space to which we apply CLT" the finite space for a single observation. But the space on which infinite sequences live can of course not be finite. The CLT is an asymptotic theorem; there is no way to do asymptotics without modelling infinite sequences. So I agree technically with you but I don't really get why this bothers you so much or why you call it "abuse". (And the situation for Berry-Esseen type theory is different as this makes valid statements for finite sequences.) $\endgroup$ Commented Jul 20, 2023 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.