5
$\begingroup$

I have a quick question about the central limit theorem. Lets say I measure some value that comes from an arbitrary distribution N times and I repeat this M times. I understand that if I calculcate the mean from the N values I will have a set of M values that follows a normal distribution. But what if I measure the sample standard deviation from N, will my resulting distribution also be normal? Following the derivation of the CLT I do not see this to be the case, but intuitively I think this is true, at least for some distributions. Any light on the issue would be greatly appreciated.

First I will quote the CLT from wiki:

the central limit theorem (CLT) states that, given certain conditions, the arithmetic mean of a sufficiently large number of iterates of independent random variables, each with a well-defined expected value and well-defined variance, be approximately normally distributed.

My question then is a variant on the quote from the wiki page: will the the central limit theorem (CLT) state that, given certain conditions, the STANDARD DEVIATION of a sufficiently large number of iterates of independent random variables, each with a well-defined expected value and well-defined variance, be approximately normally distributed? Here is a series of plots I made from random numbers that follow a beta distribution. I have generated 1000 sets of 5000 points each. The first plot is a histogram of the first set. The second is a histogram of the 1000 calculated means, and the 3rd is a histogram of the 1000 calculated std.

Simulation results

Results 2

Results 3

$\endgroup$
  • $\begingroup$ I am sorry, perhaps what I wrote was confusing. The CLT says that if I measure the mean of a set of values a sufficiently large number of times these means will follow a normal distribution. My question is instead if I took the same data set but calculated the sample standard deviation would these values come from a normal distribution? I.e., can the CLT be applied to moments beyond the first moment? Please let me know if this clears up any confusion. Thanks for your help. $\endgroup$ – user49060 Jun 26 '14 at 16:27
  • $\begingroup$ Would mathworld.wolfram.com/StandardDeviationDistribution.html be relevant? $\endgroup$ – Gene Arboit Jun 26 '14 at 18:22
  • 3
    $\begingroup$ @Gene That can be used to demonstrate a special case: because the sampling distribution of the SD from a Normal population is $\chi^2$, which asymptotically is Normal, a CLT-like result holds in this case. However, when sampling from a Bernoulli$(1/2)$ variate, the sampling distribution of the SD does not approach normality. $\endgroup$ – whuber Jun 26 '14 at 18:29
  • 1
    $\begingroup$ "The CLT says that if I measure the mean of a set of values a sufficiently large number of times these means will follow a normal distribution." --- well, actually, that's not quite what any of the versions of the CLT say. The quote from Wikipedia* is closer to accurate (but the claim is strictly speaking false - the CLT doesn't quite say that either). $\quad\quad\quad\quad$ *(not 'wiki' - that's a bit like calling the Library of Congress 'building' when referring to it) $\endgroup$ – Glen_b Jun 27 '14 at 0:39
  • $\begingroup$ Under appropriate conditions, you can apply the CLT to the variance (e.g. see p3-4 here). $\endgroup$ – Glen_b Jul 1 '14 at 1:28
8
$\begingroup$

Yes, the sample standard deviation is asymptotically normal. Let the sample standard deviation be $\hat{\sigma} = \sqrt{\frac{1}{n} \sum_{i=1}^n (x_i - \bar{x})^2}$, and let $\sigma$ be the population standard deviation. Let's use the central limit theorem to show that $$ \sqrt{n}(\hat{\sigma} - \sigma) \xrightarrow{d} N(0, V). $$ First write things as $$ \sqrt{n}(\hat{\sigma} - \sigma) = \sqrt{n}(\sqrt{ \frac{1}{n} \sum_{i=1}^n (x_i - \bar{x})^2} - \sqrt{\sigma^2} )$$ The central limit theorem tells is about how sample moments minus population moments behave. If we didn't have square roots above, we'd just have something like sample moments minus population ones, and we could use the central limit theorem. To get rid of the square roots, let's take a Taylor expansion of the first square root around $\sigma^2$. $$ \sqrt{ \frac{1}{n} \sum_{i=1}^n (x_i - \bar{x})^2} = \sqrt{\sigma^2} + \frac{1}{2\sigma} \left( \frac{1}{n} \sum_{i=1}^n (x_i - \bar{x})^2 - \sigma^2\right) + O\left(\left( \frac{1}{n} \sum_{i=1}^n (x_i - \bar{x})^2 - \sigma^2\right)^2\right) $$ Plugging this into the above, we have $$\sqrt{n}(\hat{\sigma} - \sigma) = \frac{\sqrt{n}}{2 \sigma} \left( \frac{1}{n} \sum_{i=1}^n (x_i - \bar{x})^2 - \sigma^2\right) + O\left(\frac{1}{n^{3/2}} \left(\sum_{i=1}^n (x_i - \bar{x})^2 - \sigma^2\right)^2\right)$$ Rearranging the first term on the right gives $$\frac{\sqrt{n}}{2 \sigma} \left( \frac{1}{n} \sum_{i=1}^n (x_i - \bar{x})^2 - \sigma^2\right) = \frac{1}{2\sigma \sqrt{n}} \sum_{i=1}^n ((x_i - E[X])^2 - \sigma^2) - \frac{\sqrt{n}}{2\sigma}(\bar{x} - E[x])^2 $$ The central limit theorem tells us that under some conditions $\frac{1}{\sqrt{n}} \sum_{i=1}^n (y_i - E[y]) \xrightarrow{d} N(0, \sigma_y^2)$. With $y_i = (x_i - E[x])^2$ and $E[y] = \sigma^2$, then the first term on the right converges in distribution to a normal.

The second term we can write as $\sqrt{n}(\bar{x} - E[x]) (\bar{x} - E[x])$, and since $\sqrt{n}(\bar{x} - E[x]) \xrightarrow{d} N$ and $(\bar{x} - E[x]) \xrightarrow{p} 0$, by Slutsky's lemma, the product converges in probability to 0.

Similarly, we could show that, $\frac{1}{n^{3/2}} \left(\sum_{i=1}^n (x_i - \bar{x})^2 - \sigma^2\right)^2 \xrightarrow{p} 0$, so the remainder from the Taylor expansion vanishes.

This Taylor expansion trick comes up often, so it has a name. It's called the delta method.

$\endgroup$
  • 2
    $\begingroup$ I mistrust this argument. I can't put my finger on the error, but I invite you to consider the case where the underlying distribution is Bernoulli$(1/2)$. In this case the sampling distribution of the SD will never be close to Normal: it is always J-shaped and highly negatively skewed. $\endgroup$ – whuber Jun 26 '14 at 21:06
  • 5
    $\begingroup$ That's a really interesting observation @whuber. The argument breaks down for the Bernoulli(1/2) because then $(x_i - E[x])^2 - \sigma^2= 0$ is constant. Strangely enough, it works for Bernoulli(p) with $p \neq 0.5$. The Bernoulli(1/2) counter-example is a great illustration of why it's important to check those annoying regularity conditions instead of just saying "the central limit theorem tells us that under some conditions ..." like I did. $\endgroup$ – paul Jun 26 '14 at 23:18
  • 1
    $\begingroup$ I have the suspicion that Slutsky's theorem may not be straightforward to apply when the product of r.v.'s that we consider involves the exact same r.v. Namely, when we have $Y_n$ (converging in probability to zero), and $X_n = \sqrt n Y_n$ (converging in distribution to a random variable), and we consider the product $X_nY_n$. I am looking up various proofs of Slutsky's and I will return. (cc @whuber) $\endgroup$ – Alecos Papadopoulos Jun 27 '14 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.