Why is the unbiased sample variance estimator so ubiquitous in science?

Question

The bias-corrected sample variance $$ \hat{\sigma}^2 = \dfrac{1}{n-1} \sum_{i=1}^n (X_i - \bar{X})^2 $$ is commonly used in natural sciences to estimate the variance $ \sigma^2 $ of a Gaussian random variable $ X $ from an i.i.d. sample $ X_1, \dots, X_n $.

One of the earliest things I learned in statistical inference classes is that bias is only one of many relevant properties when comparing estimators. My understanding is that mean square error (MSE) is generally a better measure of estimator performance than bias alone.

Why then is the minimum MSE estimator $$ \hat{\sigma}_{MMSE}^2 = \dfrac{1}{n+1} \sum_{i=1}^n (X_i - \bar{X})^2 $$ not the standard choice for this problem?

Answer 1

Dividing by (n+1) minimizes the MSE only for normally distributed data. In general, a variance estimator of the form $$s_k^2 = \frac{1}{k}\sum_{i=1}^n (x_i-\overline{x})^2$$ has minimal MSE for $$k=(n-1)\left[ \frac{1}{n}\left(\frac{\mu_4}{\mu_2^2} - \frac{n-3}{n-1}\right) + 1\right]$$ where $\mu_2$ and $\mu_4$ are the second and fourth central moment of the distribution (see here for a proof), which means that $\mu_4/\mu_2^2$ is the kurtosis. For the normal distribution, it is $\mu_4 = 3\mu_2^2$ and the above formula reduces to $k=n+1$.

As you cannot know whether your data is normally distributed, it is thus not clear whether n+1 actually is the best choice. The bias correction by dividing through n-1, however, is universal and does not require normality. This might be a reason why the bias corrected version is preferred.

From Jensen's inequality, it follows that $E(X^4)\geq \big(E(X^2)\big)^2$, and therefore $\mu_4/\mu_2^2\geq 1$. The optimal choice for k is thus always greater than n-1: $$k \geq \frac{n-1}{n} \left(1 - \frac{n-3}{n-1}\right) + (n-1) = \frac{2}{n} + (n-1) > n-1$$

There might even be distributions (those restricted to a small range without outliers) for which k=n-1 actually is close to the optimal choice with respect to the MSE. The example of the normal distribution shows, however, that in most practical situations the optimal k is greater than this value and choosing the empirical variance (dividing by n) typically yields an estimate that is on average closer to the true value than the bias corrected empirical variance (dividing by n-1). Unlike the bias correction, this is not guaranteed in all cases, though.

Answer 2

Personally I prefer $\hat\sigma^2$ over $\hat\sigma_{MMSE}^2$ for a reason different from unbiasedness. If the estimation problem were in fact symmetric, i.e., too low values would be as bad as too high values of the same size, I'd think that the MSE would be a good measure, and optimum MSE would be in fact good. (Note that also the expected value, and by extension the concept of unbiasedness, implicitly treats the estimation problem as symmetric, as deviations on both sides count the same also for the computation of the expected value.)

But actually variances are bounded from below, and differences between small variances should be taken as more important as the same differences between large variances; also, by implication, there should be more loss for the same deviation in negative direction than in positive direction (if the true $\sigma^2$ is 1, 0.5 should be seen as worse than 1.5; in fact, in robust statistics, $\hat\sigma^2\to 0$ is seen as "breakdown" along with $\hat\sigma^2\to \infty$).

So in fact one could think that we'd need an estimator to optimise an asymmetric loss function rather than MSE, and correspondingly there would even need to be an alternative definition to unbiasedness. Now this would be hard to do; as it isn't at all obvious how this loss function should look like, and different choices would have different implications. One could probably do some research work on this and publish a nice paper, but in practical analysis situations, that's not really what we want to do.

So in standard routine data analysis I bite the bullet and use $\hat\sigma^2$ as this is implemented everywhere and I can explain its features in a fairly generally understandable way, even though secretly I think that, because of the unbiasedness feature ignoring asymmetry, $\hat\sigma^2$ likely tends to be too small.

I wouldn't worry much about any "complication" as a result of using $\hat\sigma_{MMSE}^2$; just using a different factor would be easy to do, and the optimum MSE argument is appealing on the surface, but in fact $\hat\sigma_{MMSE}^2$ is even smaller than $\hat\sigma^2$, which I honestly think is rather too small already, if anything.

So no, $\hat\sigma_{MMSE}^2$ is not a good alternative!

PS: Of course one could have the same discussion involving the Maximum Likelihood variance estimator with factor $\frac{1}{n}$, which is sometimes used, and has a better MSE than $\hat\sigma^2$.

PPS: One could actually interpret the difference between the optimum unbiased, the ML, and the minimum MSE estimator as an expression of the asymmetry of the problem.

Answer 3

Calculating sample variance as squared deviation divided by n + 1 (instead of n - 1) will lead to underestimating variance, which will lead to confidence intervals with lower than expected probability coverage. That's probably at least part of the reason that an unbiased estimator of variance is preferred over minimizing MSE.