11
$\begingroup$

I've read somewhere that the reason we square the differences instead of taking absolute values when calculating variance is that variance defined in the usual way, with squares in the nominator, plays a unique role in Central Limit Theorem.

Well, then what exactly is the role of variance in CLT? I was unable to find more about this, or understand it properly.

We could also ask what makes us think that variance is a measure of how far a set of numbers is spread out. I could define other quantities, similar to variance, and convince you they measure the spread of numbers. For this to happen, you would have to state what exactly is meant by spread of numbers, what behaviour you expect from measure of spread etc. There's no formal definition of spread, thus we may treat variance as the definition. However, for some reason variance is considered 'the best' measure of spread.

$\endgroup$
3
  • $\begingroup$ I specifically attempted to answer this question in my response at stats.stackexchange.com/a/3904/919. $\endgroup$
    – whuber
    Commented Sep 1, 2015 at 14:01
  • 1
    $\begingroup$ Now I remember I've seen your answer before, but the problem is I can't really find the word 'variance' in your answer. Which part exactly explains the problem? Maybe I should read it again. $\endgroup$ Commented Sep 1, 2015 at 15:14
  • 3
    $\begingroup$ Look for "SD," which is equivalent to variance, and to the term "scale factor." The (rather deep) point here is that the variance itself is not a unique choice: for any given distribution, you may choose (almost) any measure of spread you like! Assuming that measure converges to the spread of the underlying distribution, what really matters is that when you standardize the sum (or mean) of $n$ iid samples from that distribution, you must rescale its spread by a factor that asymptotically is $\sqrt{n}$. In so doing you will achieve a limiting Normal distribution. $\endgroup$
    – whuber
    Commented Sep 1, 2015 at 15:22

4 Answers 4

11
$\begingroup$

The classical statement of the Central Limit Theorem (CLT) considers a sequence of independent, identically distributed random variables $X_1, X_2, \ldots, X_n, \ldots$ with common distribution $F$. This sequence models the situation we confront when designing a sampling program or experiment: if we can obtain $n$ independent observations of the same underlying phenomenon, then the finite collection $X_1, X_2, \ldots, X_n$ models the anticipated data. Allowing the sequence to be infinite is a convenient way to contemplate arbitrarily large sample sizes.

Various laws of large numbers assert that the mean

$$m(X_1, X_2, \ldots, X_n) = \frac{1}{n}(X_1 + X_2 + \cdots + X_n)$$

will closely approach the expectation of $F$, $\mu(F)$, with high probability, provided $F$ actually has an expectation. (Not all distributions do.) This implies the deviation $m(X_1, X_2, \ldots, X_n) - \mu(F)$ (which, as a function of these $n$ random variables, is also a random variable) will tend to get smaller as $n$ increases. The CLT adds to this in a much more specific way: it states (under some conditions, which I will discuss below) that if we rescale this deviation by $\sqrt{n}$, it will have a distribution function $F_n$ that approaches some zero-mean Normal distribution function as $n$ grows large. (My answer at https://stats.stackexchange.com/a/3904 attempts to explain why this is and why the factor of $\sqrt{n}$ is the right one to use.)

This is not a standard statement of the CLT. Let's connect it with the usual one. That limiting zero-mean Normal distribution will be completely determined by a second parameter, which is usually chosen to be a measure of its spread (naturally!), such as its variance or standard deviation. Let $\sigma^2$ be its variance. Surely it must have some relationship to a similar property of $F$. To discover what this might be, let $F$ have a variance $\tau^2$--which might be infinite, by the way. Regardless, because the $X_i$ are independent, we easily compute the variance of the means:

$$\eqalign{ \text{Var}(m(X_1, X_2, \ldots, X_n)) &= \text{Var}(\frac{1}{n}(X_1 + X_2 + \cdots + X_n)) \\ &= \left(\frac{1}{n}\right)^2(\text{Var}(X_1) + \text{Var}(X_2) + \cdots + \text{Var}(X_n)) \\ &= \left(\frac{1}{n}\right)^2(\tau^2 + \tau^2 + \cdots + \tau^2) \\ &= \frac{\tau^2}{n}. }$$

Consequently, the variance of the standardized residuals equals $\tau^2/n \times (\sqrt{n})^2 = \tau^2$: it is constant. The variance of the limiting Normal distribution, then, must be $\tau^2$ itself. (This immediately shows that the theorem can hold only when $\tau^2$ is finite: that is the additional assumption I glossed over earlier.)

(If we had chosen any other measure of spread of $F$ we could still succeed in connecting it to $\sigma^2$, but we would not have found that the corresponding measure of spread of the standardized mean deviation is constant for all $n$, which is a beautiful--albeit inessential--simplification.)

If we had wished, we could have standardized the mean deviations all along by dividing them by $\tau$ as well as multiplying them by $\sqrt{n}$. That would have ensured the limiting distribution is standard Normal, with unit variance. Whether you elect to standardize by $\tau$ in this way or not is really a matter of taste: it's the same theorem and the same conclusion in the end. What mattered was the multiplication by $\sqrt{n}$.

Note that you could multiply the deviations by some factor other than $\sqrt{n}$. You could use $\sqrt{n} + \exp(-n)$, or $n^{1/2 + 1/n}$, or anything else that asymptotically behaves just like $\sqrt{n}$. Any other asymptotic form would, in the limit, reduce $\sigma^2$ to $0$ or blow it up to $\infty$. This observation refines our appreciation of the CLT by showing the extent to which it is flexible concerning how the standardization is performed. We might want to state the CLT, then, in the following way.

Provided the deviation between the mean of a sequence of IID variables (with common distribution $F$) and the underlying expectation is scaled asymptotically by $\sqrt{n}$, this scaled deviation will have a zero-mean Normal limiting distribution whose variance is that of $F$.

Although variances are involved in the statement, they appear only because they are needed to characterize the limiting Normal distribution and relate its spread to that of $F$. This is only an incidental aspect. It has nothing to do with variance being "best" in any sense. The crux of the matter is the asymptotic rescaling by $\sqrt{n}$.

$\endgroup$
0
6
$\begingroup$

Variance is NOT essential to Central Limit Theorems. It is essential to the the garden variety beginner's i.i.d., Central Limit Theorem, the one most folks know and love, use and abuse.

There is not "the" Central Limit Theorem, there are many Central Limit Theorems:

The garden variety beginner's i.i.d. Central Limit Theorem. Even here, judicious choice of norming constant (so an advanced variant of the beginner's CLT) can allow Central Limit Theorems to be proved for certain random variables having infinite variance (see Feller Vol. II http://www.amazon.com/Introduction-Probability-Theory-Applications-Edition/dp/0471257095 p. 260).

The triangular array Lindeberg-Feller Central Limit Theorem. http://sites.stat.psu.edu/~dhunter/asymp/lectures/p93to100.pdf
https://en.wikipedia.org/wiki/Central_limit_theorem .

The wild world of anything goes everything in sight dependent Central Limit Theorems for which variance need not even exist. I once proved a Central Limit Theorem for which not only variance didn't exist, but neither did the mean, and in fact not even a 1 - epsilon moment for epsilon arbitrarily small positive. That was a hairy proof, because it "barely" converged, and did so very slowly. Asymptotically it converged to a Normal, in reality, a sample size of millions of terms would be needed for the Normal to be a good approximation.

$\endgroup$
2
  • $\begingroup$ Is the CLT you proved accessible somewhere on the web? It sounds very interesting, and I would like to read it. $\endgroup$ Commented Sep 19, 2015 at 17:22
  • 2
    $\begingroup$ It was a homework assignment in a theoretical probability course almost 35 years ago, lost to the times of sand. Well, it might be in one of my boxes somewhere, but I'm not likely to dig it up any time soon. I was barely smart enough to prove it (with many hours of hard slogging), not nearly smart enough to have formulated it. There are infinitely many different Central Limit Theorems, norming is the key. $\endgroup$ Commented Sep 19, 2015 at 17:46
1
$\begingroup$

What is the best measure of spread depends on the situation. Variance is a measure of spread which is a parameter of the normal distribution. So if you models your data with a nornal distribition, the (arithmetic) mean and the empirical variance is the best estimators (they are "sufficient") of the parameters of that normal distribution. That also gives the link to the central limit theorem, since that is about a normal limit, that is, the limit is a normal distribution. So if yoy have enough observations that the central limit theorem is relevant, again you can use the normal distribution, and the empirical variance is the natural description of variability, because it is tied to the normal distribution.

Without this link to the normal distribution, there is no sense in which the varoiance is best or even a natual descriptor of variability.

$\endgroup$
1
  • $\begingroup$ It is unclear why the theory of "best" estimators (in any sense of "best") should have any connection with the central limit theorem. If one were to use a non-quadratic loss function, for instance, then mean and variance might not be "best" estimators of a normal distribution's parameters--instead, the median and IQR might be best. $\endgroup$
    – whuber
    Commented Sep 1, 2015 at 14:03
1
$\begingroup$

Addressing the second question only:

I guess that variance has been the dispersion measure of choice for most of the statistician mainly for historical reasons and then because of inertia for most of non statistician practitioners.

Although I cannot cite by heart a specific reference with some rigorous definition of spread, I can offer heuristic for its mathematical characterization: central moments (i.e., $ E[(X-\mu)^k]$) are very useful for weighing deviations from the distribution's center and their probabilities/frequencies, but only if $k$ is integer and even.

Why? Because that way deviations below the center (negative) will sum up with deviations above the center (positive), instead of partially canceling them, like average does, for instance. As you can think, absolute central moments (i.e., $E(|X-\mu|^k)$) can also do that job and, more, for any $k>0$ (ok, both moments are equal if $k$ is even).

So a large amount of small deviations (both positive and negative) with few large deviations are characteristics of little dispersion, which will yield a relatively small even central moment. Lots of large deviations will yield a relatively large even central moment.

Remember when I said about the historical reasons above? Before computational power became cheap and available, one needed to rely only on mathematical, analytical skills to deal with the development of statistical theories.

Problems involving central moments were easier to tackle than ones involving absolute central moments. For instance, optimization problems involving central moments (e.g., least squares) require only calculus, while optimization involving absolute central moments with $k$ odd (for $k=1$ you get a simplex problem), which cannot be solved with calculus alone.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.