3
$\begingroup$

I am currently studying Introduction to Probability, second edition, by Blitzstein and Hwang. In studying the Jensen inequality, the following example is presented:

Example 10.1.6 (Bias of sample standard deviation). Let $X_1, \dots, X_n$ be i.i.d. random variables with variance $\sigma^2$. Recall from Theorem 6.3.4 that the sample variance $S^2_n$ is unbiased for estimating $\sigma^2$. That is, $E(S^2_n) = \sigma^2$. However, we are often more interested in estimating the standard deviation $\sigma$. A natural estimator for $\sigma$ is the sample standard deviation, $S_n$. Jensen’s inequality shows us that $S_n$ is biased for estimating $\sigma$. Moreover, it tells us which way the inequality goes:

$$E(S_n) = E(\sqrt{S^2_n}) \le \sqrt{E(S^2_n)} = \sigma,$$

so the sample standard deviation tends to underestimate the true standard deviation. How biased it is depends on the distribution (so there is no universal way to fix the bias, in contrast to the fact that defining sample variance with $n − 1$ in the denominator makes it unbiased for all distributions). Fortunately, the bias is typically minor if the sample size is reasonably large. $\square$

I'm wondering if there is a (perhaps, universal) way to measure (the amount of) this bias? But, thinking about this further, I guess that, if we can measure exactly the amount of this bias, then we would be able to exactly account for it (and therefore fix it), right?

I would appreciate it if people would please take the time to elaborate on this.

$\endgroup$
9
  • $\begingroup$ en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation has some more detail. $\endgroup$
    – Nick Cox
    Feb 23, 2020 at 9:22
  • $\begingroup$ @NickCox Eh, which part of this article informs us on quantifying the bias? $\endgroup$ Feb 23, 2020 at 9:24
  • $\begingroup$ Take the correction factor, etc. If $cs$ is more nearly unbiased than $s$, then $s - cs = s(1 - c)$ is an approximation to the bias, but $c$ is not constant, so formulas are messy as well as approximate. $\endgroup$
    – Nick Cox
    Feb 23, 2020 at 11:13
  • $\begingroup$ The expectation value expressions are explicitly given here: stats.stackexchange.com/a/410311/247352 . The “unbiasing constants”, as I called them, are a double-edged sword, in my opinion. $\endgroup$
    – Ed V
    Feb 23, 2020 at 14:09
  • $\begingroup$ @EdV so the “unbiasing constants”, as you refer to them, are the amount of bias? Please clarify, because it is not clear specifically which part of this relates to my post. $\endgroup$ Feb 23, 2020 at 14:21

1 Answer 1

6
$\begingroup$

The text you quote from Blitzstein and Hwang already tells you that there is no universal way to estimate the bias:

How biased it is depends on the distribution (so there is no universal way to fix the bias

In the canonical case that the $X_i$ are normally distributed, there is an analytic formula for the bias in terms of the gamma function. For most other distributions, the bias has to be computed numerically.

In practice, statisticians almost never correct $S_n$ for bias, and not because we can't but because there is no advantage in doing so. Amongst other considerations, fixing the bias for $S_n$ would generally increase the mean-square-error, as the reduction in bias is offset by an increase in variance.

Unbiasedness is an unambiguously desirable property only for estimators with symmetric distributions. For estimators with skewed distributions, unbiasedness is more a mathematical curiousity than an essential property. For estimators with right-skew distributions (like $S_n$) the mean-square-error can often be reduced by introducing a slight negative bias.

Proof that fixing the bias increases the MSE

I don't know any references that discuss mean-square-error (MSE) for the standard deviation, but the fact that the mean-corrected estimator has a higher MSE is easy to prove from first principles.

Suppose that $$E(S_n)=(1-\delta)\sigma$$ with $0<\delta<1$. We can conclude that \begin{eqnarray} {\rm var}(S_n)&=&E(S_n^2)-E(S_n)^2\\ &=&\sigma^2-(1-\delta)^2\sigma^2\\ &=&(2\delta-\delta^2)\sigma^2 \end{eqnarray}

The MSE of $S_n$ is \begin{eqnarray} {\rm MSE}(S_n)&=&E\left([S_n-\sigma]^2\right)\\ &=&{\rm var}(S_n)+{\rm bias}(S_n)^2\\ &=&(2\delta-\delta^2)\sigma^2 + (\delta\sigma)^2\\ &=&2\delta\sigma^2 \end{eqnarray}

The mean-corrected estimator is $$\tilde S_n=S_n/(1-\delta).$$ The MSE of $\tilde S_n$ is \begin{eqnarray} {\rm MSE}(\tilde S_n)&=&{\rm var}(\tilde S_n)+{\rm bias}(\tilde S_n)^2\\ &=&{\rm var}(S_n)/(1-\delta)^2+0^2\\ &=&(2\delta-\delta^2)\sigma^2/(1-\delta)^2\\ &=&(2\delta-\delta^2)\sigma^2\{1+2\delta +O(\delta^3)\}\\ &=&\{2\delta+3\delta^2+O(\delta^3)\}\sigma^2\\ &=&{\rm MSE}(S_n) + \{3\delta^2+O(\delta^3)\}\sigma^2 \end{eqnarray} which is greater than MSE$(S_n)$.

$\endgroup$
7
  • 1
    $\begingroup$ Interesting. Thanks for the answer. Can you redirect me to some resource that elaborates upon your claim that "fixing the bias generally increases the mean-square-error, as the reduction in bias is offset by an increase in variance"? $\endgroup$ Feb 23, 2020 at 10:13
  • 1
    $\begingroup$ @ThePointer I've added a proof to my answer. $\endgroup$ Feb 24, 2020 at 1:02
  • $\begingroup$ Thanks for that. How did you get that $\text{bias}(S_n)^2 = (\delta \sigma)^2$? $\endgroup$ Feb 24, 2020 at 1:32
  • 1
    $\begingroup$ @ThePointer Bias is defined as $E(S_n)-\sigma$. I will let you figure the rest out for yourself. $\endgroup$ Feb 24, 2020 at 2:40
  • 1
    $\begingroup$ @ThePointer Well, yes, but I thought you would do it easier way with ${\rm bias}(S_n)=(1-\delta)\sigma-\sigma=\sigma-\delta\sigma-\sigma=-\delta\sigma$, i.e., simplify the bias before squaring. $\endgroup$ Feb 25, 2020 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.