58
$\begingroup$

According to the Wikipedia article on unbiased estimation of standard deviation the sample SD

$$s = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (x_i - \overline{x})^2}$$

is a biased estimator of the SD of the population. It states that $E(\sqrt{s^2}) \neq \sqrt{E(s^2)}$.

NB. Random variables are independent and each $x_{i} \sim N(\mu,\sigma^{2})$

My question is two-fold:

  • What is the proof of the biasedness?
  • How does one compute the expectation of the sample standard deviation

My knowledge of maths/stats is only intermediate.

$\endgroup$
  • 4
    $\begingroup$ You will find both questions are answered in the Wikipedia article on the Chi distribution. $\endgroup$ – whuber Jun 8 '11 at 12:58
58
$\begingroup$

@NRH's answer to this question gives a nice, simple proof of the biasedness of the sample standard deviation. Here I will explicitly calculate the expectation of the sample standard deviation (the original poster's second question) from a normally distributed sample, at which point the bias is clear.

The unbiased sample variance of a set of points $x_1, ..., x_n$ is

$$ s^{2} = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \overline{x})^2 $$

If the $x_i$'s are normally distributed, it is a fact that

$$ \frac{(n-1)s^2}{\sigma^2} \sim \chi^{2}_{n-1} $$

where $\sigma^2$ is the true variance. The $\chi^2_{k}$ distribution has probability density

$$ p(x) = \frac{(1/2)^{k/2}}{\Gamma(k/2)} x^{k/2 - 1}e^{-x/2} $$

using this we can derive the expected value of $s$;

$$ \begin{align} E(s) &= \sqrt{\frac{\sigma^2}{n-1}} E \left( \sqrt{\frac{s^2(n-1)}{\sigma^2}} \right) \\ &= \sqrt{\frac{\sigma^2}{n-1}} \int_{0}^{\infty} \sqrt{x} \frac{(1/2)^{(n-1)/2}}{\Gamma((n-1)/2)} x^{((n-1)/2) - 1}e^{-x/2} \ dx \end{align} $$

which follows from the definition of expected value and fact that $ \sqrt{\frac{s^2(n-1)}{\sigma^2}}$ is the square root of a $\chi^2$ distributed variable. The trick now is to rearrange terms so that the integrand becomes another $\chi^2$ density:

$$ \begin{align} E(s) &= \sqrt{\frac{\sigma^2}{n-1}} \int_{0}^{\infty} \frac{(1/2)^{(n-1)/2}}{\Gamma(\frac{n-1}{2})} x^{(n/2) - 1}e^{-x/2} \ dx \\ &= \sqrt{\frac{\sigma^2}{n-1}} \cdot \frac{ \Gamma(n/2) }{ \Gamma( \frac{n-1}{2} ) } \int_{0}^{\infty} \frac{(1/2)^{(n-1)/2}}{\Gamma(n/2)} x^{(n/2) - 1}e^{-x/2} \ dx \\ &= \sqrt{\frac{\sigma^2}{n-1}} \cdot \frac{ \Gamma(n/2) }{ \Gamma( \frac{n-1}{2} ) } \cdot \frac{ (1/2)^{(n-1)/2} }{ (1/2)^{n/2} } \underbrace{ \int_{0}^{\infty} \frac{(1/2)^{n/2}}{\Gamma(n/2)} x^{(n/2) - 1}e^{-x/2} \ dx}_{\chi^2_n \ {\rm density} } \end{align} $$

now we know the integrand the last line is equal to 1, since it is a $\chi^2_{n}$ density. Simplifying constants a bit gives

$$ E(s) = \sigma \cdot \sqrt{ \frac{2}{n-1} } \cdot \frac{ \Gamma(n/2) }{ \Gamma( \frac{n-1}{2} ) } $$

Therefore the bias of $s$ is

$$ \sigma - E(s) = \sigma \bigg(1 - \sqrt{ \frac{2}{n-1} } \cdot \frac{ \Gamma(n/2) }{ \Gamma( \frac{n-1}{2} ) } \bigg) \sim \frac{\sigma}{4 n} \>$$ as $n \to \infty$.

It's not hard to see that this bias is not 0 for any finite $n$, thus proving the sample standard deviation is biased. Below the bias is plot as a function of $n$ for $\sigma=1$ in red along with $1/4n$ in blue:

enter image description here

$\endgroup$
  • 1
    $\begingroup$ (+1) Nice answer. I hope you don't mind, I tweaked a couple of very minor things and added an asymptotic result regarding the bias. I suppose you could superimpose the curve $(4n)^{-1}$ onto your plot, but it's probably unnecessary. Cheers. :) $\endgroup$ – cardinal May 8 '12 at 1:49
  • $\begingroup$ You really went to a lot of pains to do this Macro. When I first saw the post about a minute ago I was thinking of showing the bias using Jensen's rule but someone already did it. $\endgroup$ – Michael R. Chernick May 8 '12 at 2:23
  • 2
    $\begingroup$ of course this is a round-a-bout way to show that the standard deviation is biased - I was mainly answering the original poster's second question: "How does one compute the expectation of the standard deviation?". $\endgroup$ – Macro May 8 '12 at 2:24
  • 2
    $\begingroup$ Another point perhaps worth mentioning is that this calculation allows one to read off immediately what the UMVU estimator of the standard deviation is in the Gaussian case: One simply multiplies $s$ by the reciprocal of the scale factor that appears in the proof. This generalizes to UMVU estimators of $\sigma^k$ fairly readily. $\endgroup$ – cardinal May 8 '12 at 11:42
  • 2
    $\begingroup$ Sorry, Macro. The same basic integral approach you've used will work, you'll just end up with a different scaling factor of $s^k$, with the gamma arguments you get being functions of $k$. That's what I meant, but it came out a bit too terse. :) $\endgroup$ – cardinal May 8 '12 at 15:13
43
$\begingroup$

You don't need normality. All you need is that $$s^2 = \frac{1}{n-1} \sum_{i=1}^n(x_i - \bar{x})^2$$ is an unbiased estimator of the variance $\sigma^2$. Then use that the square root function is strictly concave such that (by a strong form of Jensen's inequality)
$$E(\sqrt{s^2}) < \sqrt{E(s^2)} = \sigma$$ unless the distribution of $s^2$ is degenerate at $\sigma^2$.

$\endgroup$
20
$\begingroup$

Complementing NRH's answer, if someone is teaching this to a group of students who haven't studied Jensen's inequality yet, one way to go is to define the sample standard deviation $$ S_n = \sqrt{\sum_{i=1}^n\frac{(X_i-\bar{X}_n)^2}{n-1}} , $$ suppose that $S_n$ is non degenerate (therefore, $\mathrm{Var}[S_n]\ne0$), and notice the equivalences $$ 0 < \mathrm{Var}[S_n] = \mathrm{E}[S_n^2] - \mathrm{E}^2[S_n] \;\;\Leftrightarrow\;\; \mathrm{E}^2[S_n] < \mathrm{E}[S_n^2] \;\;\Leftrightarrow\;\; \mathrm{E}[S_n] < \sqrt{\mathrm{E}[S_n^2]} =\sigma. $$

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.