# Why do we prefer unbiased estimators instead of minimizing MSE?

I was thinking about why, usually, $$\hat{\sigma}^2=\hat{p}(1-\hat{p})$$ is used to estimate the variance in a Bernoulli population instead of $$s^2=\hat{p}(1-\hat{p})\frac{n}{n-1}$$.

$$s^2$$ is unbiased, but in some cases would lead to strange situations.

Suppose you find $$\hat{p}=0.5$$. $$s^2$$ would be grater than $$0.25$$, which is obviously wrong (since $$0<=\sigma^2<=0.25$$). In this situation, it would be fair to cap $$s^2$$ at $$0.25$$ to avoid results that don't make sense at all. This would also reduce the $$MSE$$ but would make the estimator biased again. Instead, it makes more sense to just use $$\hat{\sigma}^2$$.

This made me think about why unbiased estimators are usually preferred, even if they have a higher MSE.

I have 3 questions:

1. Is this the reason why biased $$\hat{\sigma}^2$$ is usually used instead of udjusted $$s^2$$ for Bernoulli populations? Are there any other reasons?
2. Why would you want to choose an esimator which is further away from the truth on average, just to make it unbiased?
3. Why is udjusted $$s^2$$ preferred over $$\hat{\sigma}^2$$ for other distributions, even if its $$MSE$$ is higher? (The estimator that minimizes MSE divides the sum of the squared residuals by $$n+1$$, but I've never seen it used, why?)

Moreover, $$s^2$$ is unbiased, but $$\sqrt{s^2}$$ is biased for estimating the standard deviation, so it isn't that obvious why we should be choosing it (why isn't a different adjusted estimator used just for the standard deviation?).

I'm editing the question based on the comments I've received: It seems like the second question is too general because in the majority of cases an unbiased estimator doesn't exists, also there isn't usually a single estimator that minimizes MSE. So, please, consider the second point I've asked only for situations where there is an unbiased estimator which isn't the best one in terms of MSE.

• Answering your last question about why we don't use an unbiased estimator of standard deviation, there is no such universal estimator. $s^2$ is unbiased for variance for any distribution that has a variance. There is no such unbiased estimator for standard deviation.
– Dave
Jul 8 at 19:30
• Why should mse be preferred? Why not mae? These questions are impossible to answer. Also, the claim that usually preferred unbiased needs support itself. I don’t prefer them Jul 8 at 19:30
• ML people regularly use biased estimators (regularisation) for prediction. IMO biased estimators are a problem for significance testing.. Jul 8 at 19:37
• As @Dave pointed out, there exist very few transforms of the parameter that allow for the existence of an unbiased estimator. Thus, in many situations, there is no option but using biased estimators (such as Bayes estimators, which are almost always biased). Similarly, there are few situations where there exists a single best MSE estimator. Jul 8 at 19:47
• In schools mse is a common measure because it’s convenient mathematically for white board derivations of equations. Jul 8 at 20:44

We do often seek the unbiased estimator with the smallest variability. But sometimes convenience (or habit) leads to use of estimators that are not "optimal" according to some criterion.

In the case of estimating the mean $$\mu$$ of a normal population where $$\sigma$$ is known, sample average $$A= \bar X$$ is used instead of the sample median $$H.$$ Both are unbiased: $$E(A) = E(H) = \mu.$$ But $$Var(A) < Var(H),$$ so $$A$$ is more widely used.

Suppose we have a sample of size $$n=9$$ from $$\mathsf{Norm}(\mu=10,\sigma=1).$$ An easy formula gives $$Var(A) = 1/n = 1/9.$$ There is no such easy formula for $$Var(H),$$ but one can show that $$Var(H) > Var(A).$$ One way to get a rough estimate is by simulation:

set.seed(2021)
h = replicate(10^5, median(rnorm(9,10,1)))
mean(h);  var(h)
[1] 10.00027    # aprx E(H) = 10
[1] 0.1668142   # aprx Var(H) > 1/9

set.seed(2021)
a = replicate(10^5, mean(rnorm(9,10,1)))
mean(a);  var(a)
[1] 10.0009     # aprx E(A) = 10
[1] 0.1118979   # aprx Var(A) = 1/9


By contrast, the sample variance $$S^2 = V_{n-1} = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2$$ is unbiased with $$E(S^2) = \sigma^2.$$ However, even though the estimate $$V_n = \frac{1}{n}\sum_{i=1}^n (X_i - \bar X)^2$$ is biased, some still argue that it is better because its mean squared error $$MSE(V_n) = E[(V_n-\sigma^2)^2]$$ is smaller than for $$V_{n-1}.$$

set.seed(2021)
v.8 = replicate(10^5, var(rnorm(9,10,1)))
mean(v.8);  var(v.8);  mean((v.8-1)^2)
[1] 1.001419    # aprx E(V.8) = 1
[1] 0.2528645   # aprx Var(V.8)
[1] 0.252864    # aprx MSE(V.8) = Var(V.8)

set.seed(2021)
v.9 = replicate(10^5, (8/9)*var(rnorm(9,10,1)))
mean(v.9);  var(v.9);  mean((v.9-1)^2)
[1] 0.8901506  # aprx E(V.9) < 1: too small
[1] 0.1997942
[1] 0.2118591  # aprx MSE(V.9) < MSE(V.8)


Perhaps more remarkable is that the estimator $$V_{n+1} = \frac{1}{n+1}\sum_{i=1}^n (X_i-\bar X)^2.$$ has even smaller MSE.

set.seed(2021)
v.10 = replicate(10^5, (8/10)*var(rnorm(9,10,1)))
mean(v.10);  var(v.10);  mean((v.10-1)^2)
[1] 0.8011355  # more serious downward bias
[1] 0.1618333
[1] 0.2013788  # aprx MSE(V.10) < MSE(V.9) < MSE(V.8)


In practice, $$V_{n+1}$$ is not much used. Some statisticians feel MSE puts too much emphasis on far deviations. And many would object to such large underestimates of $$\sigma^2.$$

The basic lesson is that there are frequently trade-offs to be made in choosing an 'ideal' estimator, and it is not always clear whether unbiasedness or small MSE is more important.

I think one historical reason $$s^2$$ with an $$n-1$$ denominator is used is the analogy to regression settings where the denominator is $$n-p$$, and especially to the Neyman-Scott problem. The difference between $$n$$ and $$n-1$$ doesn't matter much but the difference between $$n$$ and $$n-p$$ may.

The Neyman-Scott problem is the extreme case of $$n$$ pairs of Normal observations: $$Y_{ij}=\mu_i+\epsilon_{ij}$$, where $$\epsilon_{ij}\sim N(0,\sigma^2)$$. In this model, the MLE of $$\mu_i$$ is the pairwise mean $$\bar Y_i$$ and of $$\sigma^2$$ is $$\hat\sigma^2_\textrm{MLE}=\frac{\sum_i\sum_j (Y_{ij}-\bar Y_{i})^2}{2n}$$ The MLE has expectation $$\sigma^2/2$$, which is Not Good.

A 'degrees of freedom' correction gives $$\hat\sigma^2_\textrm{df}=\frac{\sum_i\sum_j (Y_{ij}-\bar Y_{i})^2}{2n-n}$$ which is unbiased, but in this case is clearly better than the MLE.

So, the actual advantage of the df-corrected estimator in various structured experimental designs lends some weight to also using the df-corrected estimator where it doesn't matter.

• I would argue that the $n-1$ denominator estimator is exactly the $n-p$ estimator in a regression with just an intercept…in which case, why do we care that the $n-p$ estimator is unbiased for the variance of the error term?
– Dave
Jul 8 at 23:25
• Because when $p$ is of the same order as $n$, the denominator of $n$ is really bad -- it's not even a consistent estimator under some reasonable asymptotics. So it's not that the $n-p$ estimator is exactly unbiased, it's that it isn't horribly far from the truth and the $n$ estimator is. Jul 9 at 0:51