According to the Wikipedia article on unbiased estimation of standard deviation the sample SD

$$s = \sqrt{\frac{1}{n-1} \sum_{i=1}^n (x_i - \overline{x})^2}$$

is a biased estimator of the SD of the population. It states that $E(\sqrt{s^2}) \neq \sqrt{E(s^2)}$.

NB. Random variables are independent and each $x_{i} \sim N(\mu,\sigma^{2})$

My question is two-fold:

  • What is the proof of the biasedness?
  • How does one compute the expectation of the sample standard deviation

My knowledge of maths/stats is only intermediate.

  • 4
    $\begingroup$ You will find both questions are answered in the Wikipedia article on the Chi distribution. $\endgroup$ – whuber Jun 8 '11 at 12:58

@NRH's answer to this question gives a nice, simple proof of the biasedness of the sample standard deviation. Here I will explicitly calculate the expectation of the sample standard deviation (the original poster's second question) from a normally distributed sample, at which point the bias is clear.

The unbiased sample variance of a set of points $x_1, ..., x_n$ is

$$ s^{2} = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \overline{x})^2 $$

If the $x_i$'s are normally distributed, it is a fact that

$$ \frac{(n-1)s^2}{\sigma^2} \sim \chi^{2}_{n-1} $$

where $\sigma^2$ is the true variance. The $\chi^2_{k}$ distribution has probability density

$$ p(x) = \frac{(1/2)^{k/2}}{\Gamma(k/2)} x^{k/2 - 1}e^{-x/2} $$

using this we can derive the expected value of $s$;

$$ \begin{align} E(s) &= \sqrt{\frac{\sigma^2}{n-1}} E \left( \sqrt{\frac{s^2(n-1)}{\sigma^2}} \right) \\ &= \sqrt{\frac{\sigma^2}{n-1}} \int_{0}^{\infty} \sqrt{x} \frac{(1/2)^{(n-1)/2}}{\Gamma((n-1)/2)} x^{((n-1)/2) - 1}e^{-x/2} \ dx \end{align} $$

which follows from the definition of expected value and fact that $ \sqrt{\frac{s^2(n-1)}{\sigma^2}}$ is the square root of a $\chi^2$ distributed variable. The trick now is to rearrange terms so that the integrand becomes another $\chi^2$ density:

$$ \begin{align} E(s) &= \sqrt{\frac{\sigma^2}{n-1}} \int_{0}^{\infty} \frac{(1/2)^{(n-1)/2}}{\Gamma(\frac{n-1}{2})} x^{(n/2) - 1}e^{-x/2} \ dx \\ &= \sqrt{\frac{\sigma^2}{n-1}} \cdot \frac{ \Gamma(n/2) }{ \Gamma( \frac{n-1}{2} ) } \int_{0}^{\infty} \frac{(1/2)^{(n-1)/2}}{\Gamma(n/2)} x^{(n/2) - 1}e^{-x/2} \ dx \\ &= \sqrt{\frac{\sigma^2}{n-1}} \cdot \frac{ \Gamma(n/2) }{ \Gamma( \frac{n-1}{2} ) } \cdot \frac{ (1/2)^{(n-1)/2} }{ (1/2)^{n/2} } \underbrace{ \int_{0}^{\infty} \frac{(1/2)^{n/2}}{\Gamma(n/2)} x^{(n/2) - 1}e^{-x/2} \ dx}_{\chi^2_n \ {\rm density} } \end{align} $$

now we know the integrand the last line is equal to 1, since it is a $\chi^2_{n}$ density. Simplifying constants a bit gives

$$ E(s) = \sigma \cdot \sqrt{ \frac{2}{n-1} } \cdot \frac{ \Gamma(n/2) }{ \Gamma( \frac{n-1}{2} ) } $$

Therefore the bias of $s$ is

$$ \sigma - E(s) = \sigma \bigg(1 - \sqrt{ \frac{2}{n-1} } \cdot \frac{ \Gamma(n/2) }{ \Gamma( \frac{n-1}{2} ) } \bigg) \sim \frac{\sigma}{4 n} \>$$ as $n \to \infty$.

It's not hard to see that this bias is not 0 for any finite $n$, thus proving the sample standard deviation is biased. Below the bias is plot as a function of $n$ for $\sigma=1$ in red along with $1/4n$ in blue:

enter image description here

  • $\begingroup$ (+1) Nice answer. I hope you don't mind, I tweaked a couple of very minor things and added an asymptotic result regarding the bias. I suppose you could superimpose the curve $(4n)^{-1}$ onto your plot, but it's probably unnecessary. Cheers. :) $\endgroup$ – cardinal May 8 '12 at 1:49
  • $\begingroup$ You really went to a lot of pains to do this Macro. When I first saw the post about a minute ago I was thinking of showing the bias using Jensen's rule but someone already did it. $\endgroup$ – Michael Chernick May 8 '12 at 2:23
  • 2
    $\begingroup$ of course this is a round-a-bout way to show that the standard deviation is biased - I was mainly answering the original poster's second question: "How does one compute the expectation of the standard deviation?". $\endgroup$ – Macro May 8 '12 at 2:24
  • 2
    $\begingroup$ Another point perhaps worth mentioning is that this calculation allows one to read off immediately what the UMVU estimator of the standard deviation is in the Gaussian case: One simply multiplies $s$ by the reciprocal of the scale factor that appears in the proof. This generalizes to UMVU estimators of $\sigma^k$ fairly readily. $\endgroup$ – cardinal May 8 '12 at 11:42
  • 2
    $\begingroup$ Sorry, Macro. The same basic integral approach you've used will work, you'll just end up with a different scaling factor of $s^k$, with the gamma arguments you get being functions of $k$. That's what I meant, but it came out a bit too terse. :) $\endgroup$ – cardinal May 8 '12 at 15:13

You don't need normality. All you need is that $$s^2 = \frac{1}{n-1} \sum_{i=1}^n(x_i - \bar{x})^2$$ is an unbiased estimator of the variance $\sigma^2$. Then use that the square root function is strictly concave such that (by a strong form of Jensen's inequality)
$$E(\sqrt{s^2}) < \sqrt{E(s^2)} = \sigma$$ unless the distribution of $s^2$ is degenerate at $\sigma^2$.


Complementing NRH's answer, if someone is teaching this to a group of students who haven't studied Jensen's inequality yet, one way to go is to define the sample standard deviation $$ S_n = \sqrt{\sum_{i=1}^n\frac{(X_i-\bar{X}_n)^2}{n-1}} , $$ suppose that $S_n$ is non degenerate (therefore, $\mathrm{Var}[S_n]\ne0$), and notice the equivalences $$ 0 < \mathrm{Var}[S_n] = \mathrm{E}[S_n^2] - \mathrm{E}^2[S_n] \;\;\Leftrightarrow\;\; \mathrm{E}^2[S_n] < \mathrm{E}[S_n^2] \;\;\Leftrightarrow\;\; \mathrm{E}[S_n] < \sqrt{\mathrm{E}[S_n^2]} =\sigma. $$


protected by whuber Apr 27 '16 at 11:50

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.