Why do we prefer unbiased estimators instead of minimizing MSE?

Question

I was thinking about why, usually, $\hat{\sigma}^2=\hat{p}(1-\hat{p})$ is used to estimate the variance in a Bernoulli population instead of $s^2=\hat{p}(1-\hat{p})\frac{n}{n-1}$.

$s^2$ is unbiased, but in some cases would lead to strange situations.

Suppose you find $\hat{p}=0.5$. $s^2$ would be grater than $0.25$, which is obviously wrong (since $0<=\sigma^2<=0.25$). In this situation, it would be fair to cap $s^2$ at $0.25$ to avoid results that don't make sense at all. This would also reduce the $MSE$ but would make the estimator biased again. Instead, it makes more sense to just use $\hat{\sigma}^2$.

This made me think about why unbiased estimators are usually preferred, even if they have a higher MSE.

I have 3 questions:

1. Is this the reason why biased $\hat{\sigma}^2$ is usually used instead of udjusted $s^2$ for Bernoulli populations? Are there any other reasons?
2. Why would you want to choose an esimator which is further away from the truth on average, just to make it unbiased?
3. Why is udjusted $s^2$ preferred over $\hat{\sigma}^2$ for other distributions, even if its $MSE$ is higher? (The estimator that minimizes MSE divides the sum of the squared residuals by $n+1$, but I've never seen it used, why?)

Moreover, $s^2$ is unbiased, but $\sqrt{s^2}$ is biased for estimating the standard deviation, so it isn't that obvious why we should be choosing it (why isn't a different adjusted estimator used just for the standard deviation?).

I'm editing the question based on the comments I've received: It seems like the second question is too general because in the majority of cases an unbiased estimator doesn't exists, also there isn't usually a single estimator that minimizes MSE. So, please, consider the second point I've asked only for situations where there is an unbiased estimator which isn't the best one in terms of MSE.

Answer 1

We do often seek the unbiased estimator with the smallest variability. But sometimes convenience (or habit) leads to use of estimators that are not "optimal" according to some criterion.

In the case of estimating the mean $\mu$ of a normal population where $\sigma$ is known, sample average $A= \bar X$ is used instead of the sample median $H.$ Both are unbiased: $E(A) = E(H) = \mu.$ But $Var(A) < Var(H),$ so $A$ is more widely used.

Suppose we have a sample of size $n=9$ from $\mathsf{Norm}(\mu=10,\sigma=1).$ An easy formula gives $Var(A) = 1/n = 1/9.$ There is no such easy formula for $Var(H),$ but one can show that $Var(H) > Var(A).$ One way to get a rough estimate is by simulation:

set.seed(2021)
h = replicate(10^5, median(rnorm(9,10,1)))
mean(h);  var(h)
[1] 10.00027    # aprx E(H) = 10
[1] 0.1668142   # aprx Var(H) > 1/9

set.seed(2021)
a = replicate(10^5, mean(rnorm(9,10,1)))
mean(a);  var(a)
[1] 10.0009     # aprx E(A) = 10
[1] 0.1118979   # aprx Var(A) = 1/9

By contrast, the sample variance $S^2 = V_{n-1} = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2$ is unbiased with $E(S^2) = \sigma^2.$ However, even though the estimate $V_n = \frac{1}{n}\sum_{i=1}^n (X_i - \bar X)^2$ is biased, some still argue that it is better because its mean squared error $MSE(V_n) = E[(V_n-\sigma^2)^2]$ is smaller than for $V_{n-1}.$

set.seed(2021)
v.8 = replicate(10^5, var(rnorm(9,10,1)))
mean(v.8);  var(v.8);  mean((v.8-1)^2)
[1] 1.001419    # aprx E(V.8) = 1
[1] 0.2528645   # aprx Var(V.8)
[1] 0.252864    # aprx MSE(V.8) = Var(V.8)

set.seed(2021)
v.9 = replicate(10^5, (8/9)*var(rnorm(9,10,1)))
mean(v.9);  var(v.9);  mean((v.9-1)^2)
[1] 0.8901506  # aprx E(V.9) < 1: too small
[1] 0.1997942  
[1] 0.2118591  # aprx MSE(V.9) < MSE(V.8)

Perhaps more remarkable is that the estimator $V_{n+1} = \frac{1}{n+1}\sum_{i=1}^n (X_i-\bar X)^2.$ has even smaller MSE.

set.seed(2021)
v.10 = replicate(10^5, (8/10)*var(rnorm(9,10,1)))
mean(v.10);  var(v.10);  mean((v.10-1)^2)
[1] 0.8011355  # more serious downward bias
[1] 0.1618333
[1] 0.2013788  # aprx MSE(V.10) < MSE(V.9) < MSE(V.8)

In practice, $V_{n+1}$ is not much used. Some statisticians feel MSE puts too much emphasis on far deviations. And many would object to such large underestimates of $\sigma^2.$

The basic lesson is that there are frequently trade-offs to be made in choosing an 'ideal' estimator, and it is not always clear whether unbiasedness or small MSE is more important.

Answer 2

I think one historical reason $s^2$ with an $n-1$ denominator is used is the analogy to regression settings where the denominator is $n-p$, and especially to the Neyman-Scott problem. The difference between $n$ and $n-1$ doesn't matter much but the difference between $n$ and $n-p$ may.

The Neyman-Scott problem is the extreme case of $n$ pairs of Normal observations: $Y_{ij}=\mu_i+\epsilon_{ij}$, where $\epsilon_{ij}\sim N(0,\sigma^2)$. In this model, the MLE of $\mu_i$ is the pairwise mean $\bar Y_i$ and of $\sigma^2$ is $$\hat\sigma^2_\textrm{MLE}=\frac{\sum_i\sum_j (Y_{ij}-\bar Y_{i})^2}{2n}$$ The MLE has expectation $\sigma^2/2$, which is Not Good.

A 'degrees of freedom' correction gives $$\hat\sigma^2_\textrm{df}=\frac{\sum_i\sum_j (Y_{ij}-\bar Y_{i})^2}{2n-n}$$ which is unbiased, but in this case is clearly better than the MLE.

So, the actual advantage of the df-corrected estimator in various structured experimental designs lends some weight to also using the df-corrected estimator where it doesn't matter.