50
$\begingroup$

A very simple version of central limited theorem as below $$ \sqrt{n}\bigg(\bigg(\frac{1}{n}\sum_{i=1}^n X_i\bigg) - \mu\bigg)\ \xrightarrow{d}\ \mathcal{N}(0,\;\sigma^2) $$ which is Lindeberg–Lévy CLT. I do not understand why there is a $\sqrt{n}$ on the left handside. And Lyapunov CLT says $$ \frac{1}{s_n} \sum_{i=1}^{n} (X_i - \mu_i) \ \xrightarrow{d}\ \mathcal{N}(0,\;1) $$ but why not $\sqrt{s_n}$? Would anyone tell me what are these factors, such $\sqrt{n}$ and $\frac{1}{s_n}$? how do we get them in the theorem?

$\endgroup$
2
  • 4
    $\begingroup$ This is explained at stats.stackexchange.com/questions/3734. That answer is long, because it asks for "intuition." It concludes, "This simple approximation, though, suggests how de Moivre might originally have suspected that there is a universal limiting distribution, that its logarithm is a quadratic function, and that the proper scale factor $s_n$ must be proportional to $\sqrt{n}$... ." $\endgroup$
    – whuber
    Sep 11, 2012 at 18:06
  • 2
    $\begingroup$ Intuitively, if all $\sigma_i=\sigma$ then $s_n = \sqrt{\sum\sigma_i^2}=\sqrt{n}\sigma$ and the 2nd line follows from the 1st line: $$ \sqrt{n}\bigg(\bigg(\frac{1}{n}\sum_{i=1}^n X_i\bigg)-\mu\bigg)=\frac{1}{\sqrt{n}}\sum_{i=1}^n \bigg(X_i-\mu\bigg)\xrightarrow{d}\ \mathcal{N}(0,\;\sigma^2)$$divide by $\sigma = \frac{s_n}{\sqrt{n}}$$$\frac{\frac{1}{\sqrt{n}}\sum_{i=1}^n \bigg(X_i-\mu\bigg)}{\frac{s_n}{\sqrt{n}}}=\frac{1}{s_n}\sum_{i=1}^{n}(X_i-\mu_i)\xrightarrow{d}\ \mathcal{N}(0,\;1) $$ (of course the Lyapunov condition, combination off all $\sigma_i$, is another question) $\endgroup$ May 21, 2018 at 16:13

4 Answers 4

46
$\begingroup$

Nice question (+1)!!

You will remember that for independent random variables $X$ and $Y$, $Var(X+Y) = Var(X) + Var(Y)$ and $Var(a\cdot X) = a^2 \cdot Var(X)$. So the variance of $\sum_{i=1}^n X_i$ is $\sum_{i=1}^n \sigma^2 = n\sigma^2$, and the variance of $\bar{X} = \frac{1}{n}\sum_{i=1}^n X_i$ is $n\sigma^2 / n^2 = \sigma^2/n$.

This is for the variance. To standardize a random variable, you divide it by its standard deviation. As you know, the expected value of $\bar{X}$ is $\mu$, so the variable

$$ \frac{\bar{X} - E\left( \bar{X} \right)}{\sqrt{ Var(\bar{X}) }} = \sqrt{n} \frac{\bar{X} - \mu}{\sigma}$$ has expected value 0 and variance 1. So if it tends to a Gaussian, it has to be the standard Gaussian $\mathcal{N}(0,\;1)$. Your formulation in the first equation is equivalent. By multiplying the left hand side by $\sigma$ you set the variance to $\sigma^2$.

Regarding your second point, I believe that the equation shown above illustrates that you have to divide by $\sigma$ and not $\sqrt{\sigma}$ to standardize the equation, explaining why you use $s_n$ (the estimator of $\sigma)$ and not $\sqrt{s_n}$.

Addition: @whuber suggests to discuss the why of the scaling by $\sqrt{n}$. He does it there, but because the answer is very long I will try to capture the essense of his argument (which is a reconstruction of de Moivre's thoughts).

If you add a large number $n$ of +1's and -1's, you can approximate the probability that the sum will be $j$ by elementary counting. The log of this probability is proportional to $-j^2/n$. So if we want the probability above to converge to a constant as $n$ goes large, we have to use a normalizing factor in $O(\sqrt{n})$.

Using modern (post de Moivre) mathematical tools, you can see the approximation mentioned above by noticing that the sought probability is

$$P(j) = \frac{{n \choose n/2+j}}{2^n} = \frac{n!}{2^n(n/2+j)!(n/2-j)!}$$

which we approximate by Stirling's formula

$$ P(j) \approx \frac{n^n e^{n/2+j} e^{n/2-j}}{2^n e^n (n/2+j)^{n/2+j} (n/2-j)^{n/2-j} } = \left(\frac{1}{1+2j/n}\right)^{n+j} \left(\frac{1}{1-2j/n}\right)^{n-j}. $$

$$ \log(P(j)) = -(n+j) \log(1+2j/n) - (n-j) \log(1-2j/n) \\ \sim -2j(n+j)/n + 2j(n-j)/n \propto -j^2/n.$$

$\endgroup$
7
  • $\begingroup$ Please see my comments to previous answers by Michael C. and guy. $\endgroup$
    – whuber
    Sep 11, 2012 at 18:11
  • $\begingroup$ Seems like the first equation (LL CLT) s/b $\sqrt{n}\bigg(\bigg(\frac{1}{n}\sum_{i=1}^n X_i\bigg) - \mu\bigg)\ \xrightarrow{d}\ \mathcal{N}(0,\;1)$? That confused me as well that $\sigma^2$ appeared as the variance. $\endgroup$
    – B_Miner
    Sep 11, 2012 at 18:37
  • $\begingroup$ If you parametrize the Gaussian with mean and variance (not standard deviation) then I believe OP's formula is correct. $\endgroup$
    – gui11aume
    Sep 11, 2012 at 18:41
  • 1
    $\begingroup$ Ahh..Given that $\frac{\bar{X} - E\left( \bar{X} \right)}{\sqrt{ Var(\bar{X}) }} = \sqrt{n} \frac{\bar{X} - \mu}{\sigma} \xrightarrow{d}\ \mathcal{N}(0,\;1)$ if we multiply $\frac{\bar{X} - E\left( \bar{X} \right)}{\sqrt{ Var(\bar{X}) }}$ by $\sigma$ we get what was shown by the OP ($\sigma$ cancel): namely $\sqrt{n}\bigg(\bigg(\frac{1}{n}\sum_{i=1}^n X_i\bigg) - \mu\bigg)$. But we know that VAR(aX) = a^2Var(X) where in this case a= $\sigma^2$ and Var(X) is 1 so the distribution is $\mathcal{N}(0,\;\sigma^2)$. $\endgroup$
    – B_Miner
    Sep 11, 2012 at 19:00
  • $\begingroup$ Gui,If not too late I wanted to make sure I had this correct. If we assume $\frac{\bar{X} - E\left( \bar{X} \right)}{\sqrt{ Var(\bar{X}) }} = \sqrt{n} ({\bar{X} - \mu}) \xrightarrow{d}\ \mathcal{N}(0,\;1)$ and we multiply by a constant ($\sigma$), the expected value of this quantity (i.e. $\sqrt{n} ({\bar{X} - \mu})$), which was zero is still zero as E[aX]=a*E[X] => $\sigma$*0=0. Is this correct? $\endgroup$
    – B_Miner
    Dec 27, 2012 at 20:52
8
$\begingroup$

There is a nice theory of what kind of distributions can be limiting distributions of sums of random variables. The nice resource is the following book by Petrov, which I personally enjoyed immensely.

It turns out, that if you are investigating limits of this type $$\frac{1}{a_n}\sum_{i=1}^n(X_i-b_i), \quad (1)$$ where $X_i$ are independent random variables, the distributions of limits are only certain distributions.

There is a lot of mathematics going around then, which boils to several theorems which completely characterizes what happens in the limit. One of such theorems is due to Feller:

Theorem Let $\{X_n;n=1,2,...\}$ be a sequence of independent random variables, $V_n(x)$ be the distribution function of $X_n$, and $a_n$ be a sequence of positive constant. In order that

$$\max_{1\le k\le n}P(|X_k|\ge\varepsilon a_n)\to 0, \text{ for every fixed } \varepsilon>0$$

and

$$\sup_x\left|P\left(a_n^{-1}\sum_{k=1}^nX_k<x\right)-\Phi(x)\right|\to 0$$

it is necessary and sufficient that

$$\sum_{k=1}^n\int_{|x|\ge \varepsilon a_n}dV_k(x)\to 0 \text{ for every fixed }\varepsilon>0,$$

$$a_n^{-2}\sum_{k=1}^n\left(\int_{|x|<a_n}x^2dV_k(x)-\left(\int_{|x|<a_n}xdV_k(x)\right)^2\right)\to 1$$

and

$$a_n^{-1}\sum_{k=1}^n\int_{|x|<a_n}xdV_k(x)\to 0.$$

This theorem then gives you an idea of what $a_n$ should look like.

The general theory in the book is constructed in such way that norming constant is restricted in any way, but final theorems which give necessary and sufficient conditions, do not leave any room for norming constant other than $\sqrt{n}$.

$\endgroup$
4
$\begingroup$

$s_n$ represents the sample standard deviation for the sample mean. $s_n$$^2$ is the sample variance for the sample mean and it equals $S_n^2/n$, where $S_n^2$ is the sample estimate of the population variance. Since $s_n =S_n/\sqrt{n}$ that explains how $\sqrt n$ appears in the first formula. Note there would be a $\sigma$ in the denominator if the limit were $N(0,1)$ but the limit is given as $N(0, \sigma^2)$. Since $S_n$ is a consistent estimate of $\sigma$ it is used in the second equation to take $\sigma$ out of the limit.

$\endgroup$
6
  • $\begingroup$ What about the other (more basic and important) part of the question: why $s_n$ and not some other measure of dispersion? $\endgroup$
    – whuber
    Sep 11, 2012 at 18:10
  • $\begingroup$ @whuber That may be up for discussion but it was not part of the question. The OP just wanted to known why s$_n$ and √n appear in the formula for the CLT. Of course S$_n$ is there because it is consistent for σ and in that form of the CLT σ is removed. $\endgroup$ Sep 19, 2012 at 15:40
  • 1
    $\begingroup$ To me it's not at all clear that $s_n$ is present because it is "consistent for $\sigma$". Why wouldn't that also imply, say, that $s_n$ should be used to normalize extreme-value statistics (which would not work)? Am I missing something simple and self-evident? And, to echo the OP, why not use $\sqrt{s_n}$--after all, that is consistent for $\sqrt{\sigma}$! $\endgroup$
    – whuber
    Sep 19, 2012 at 17:32
  • $\begingroup$ The theorem as stated has convergence to N(0,1), so to accomplish that you either have to know σ and use it or use a consistent estimate of it which works by Slutsky's theorem I think. Was I that unclear? $\endgroup$ Sep 19, 2012 at 17:53
  • $\begingroup$ I don't think you were unclear; I just think that an important point may be missing. After all, for many distributions we can obtain a limiting normal distribution by using the IQR instead of $s_n$--but then the result is not as neat (the SD of the limiting distribution depends on the distribution we begin with). I'm just suggesting that this deserves to be called out and explained. It will not be quite as obvious to someone who does not have the intuition developed by 40 years of standardizing all the distributions they encounter! $\endgroup$
    – whuber
    Sep 19, 2012 at 17:59
2
$\begingroup$

Intuitively, if $Z_n \to \mathcal N(0, \sigma^2)$ for some $\sigma^2$ we should expect that $\mbox{Var}(Z_n)$ is roughly equal to $\sigma^2$; it seems like a pretty reasonable expectation, though I don't think it is necessary in general. The reason for the $\sqrt n$ in the first expression is that the variance of $\bar X_n - \mu$ goes to $0$ like $\frac 1 n$ and so the $\sqrt n$ is inflating the variance so that the expression just has variance equal to $\sigma^2$. In the second expression, the term $s_n$ is defined to be $\sqrt{\sum_{i = 1} ^ n \mbox{Var}(X_i)}$ while the variance of the numerator grows like $\sum_{i = 1} ^ n \mbox{Var}(X_i)$, so we again have that the variance of the whole expression is a constant ($1$ in this case).

Essentially, we know something "interesting" is happening with the distribution of $\bar X_n := \frac 1 n \sum_i X_i$, but if we don't properly center and scale it we won't be able to see it. I've heard this described sometimes as needing to adjust the microscope. If we don't blow up (e.g.) $\bar X - \mu$ by $\sqrt n$ then we just have $\bar X_n - \mu \to 0$ in distribution by the weak law; an interesting result in it's own right but not as informative as the CLT. If we inflate by any factor $a_n$ which is dominated by $\sqrt n$, we still get $a_n(\bar X_n - \mu) \to 0$ while any factor $a_n$ which dominates $\sqrt n$ gives $a_n(\bar X_n - \mu) \to \infty$. It turns out $\sqrt n$ is just the right magnification to be able to see what is going on in this case (note: all convergence here is in distribution; there is another level of magnification which is interesting for almost sure convergence, which gives rise to the law of iterated logarithm).

$\endgroup$
2
  • 4
    $\begingroup$ A more fundamental question, which ought to be addressed first, is why the SD is used to measure dispersion. Why not the absolute central $k^\text{th}$ moment for some other value of $k$? Or why not the IQR or any of its relatives? Once that is answered, then simple properties of covariance immediately give the $\sqrt{n}$ dependence (as @Gui11aume has recently explained.) $\endgroup$
    – whuber
    Sep 11, 2012 at 18:09
  • 1
    $\begingroup$ @whuber I agree, which is why I presented this as heuristic. I'm not certain it is amenable to a simple explanation, though I'd love to hear one. For me I'm not sure that I have a simpler, explainable reason past "because the square term is the relevant term in the Taylor expansion of the characteristic function once you subtract off the mean." $\endgroup$
    – guy
    Sep 11, 2012 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.