Why is "unbiased" estimator more important than min-error estimator?

Question

According to Edwin Jaynes (Chapter 17 of his book Probability Theory: the Logic of Science), the mean squared error of an estimator consists of bias term and variance term, that is:

$$L =E[(\beta - \alpha)^2]=E[(E[\beta] - \alpha)^2]+(E[\beta^2] - E[\beta]^2)$$

where $\beta$ is an estimator for quantity $\alpha$.

For "unbiased" estimator, we aim to minimize first term $E[(E[\beta] - \alpha)^2]$, however, this does not usually lead to minimization of the total error $L$. Jaynes suggests it would be better to use estimator $\beta$ to minimize $L$ such that we make best use of limited available data.

Given this reasoning, I am a bit confused about why we want to get "unbiased" estimator rather than minimal-error estimator in the first place, can someone share your opinions?

Answer 1

We don't, in general, want to get unbiased estimators. It's extremely common to use biased estimators, especially when we don't have very much data: random-effects models, Bayesian estimators, penalised regression, small-area estimation, smoothing, density estimation...

However, when you have a smooth parametric model and a lot of data, we do know that the best estimator in the sense of mean squared error is actually an asymptotically unbiased estimator. The Convolution Theorem(s) and the Local Asymptotic Minimax Theorem tell us that for squared error and for any other "symmetric bowl-shaped loss", no estimator can do better[1] in large samples than an efficient estimator, where an efficient estimator is an estimator with the same asymptotic distribution as the MLE.

This large-data, nice-model setting is also the setting where reasonable Bayesian estimators have the same asymptotically unbiased distribution as the MLE (this is called the Bernstein-von Mises theorem).

[1] There are technical details here, but they don't affect the point you're asking about.