Are there parameters where a biased estimator is considered "better" than the unbiased estimator? [duplicate]

Question

A perfect estimator would be accurate (unbiased) and precise (good estimation even with small samples).

I never really thought of the question of precision but only the one of accuracy (as I did in Estimator of $\frac{\sigma^2}{\mu (1 - \mu)}$ when sampling without replacement for example).

Are there cases where the unbiased estimator is less precise (and therefore eventually "less good") than a biased estimator? If yes, I would love a simple example proving mathematically that the less accurate estimator is so much more precise that it could be considered better.

Answer 1

One example is estimates from ordinary least squares regression when there is collinearity. They are unbiased but have huge variance. Ridge regression on the same problem yields estimates that are biased but have much lower variance. E.g.

install.packages("ridge")
library(ridge)
set.seed(831)

data(GenCont)
ridgemod <- linearRidge(Phenotypes ~ ., data = as.data.frame(GenCont))
summary(ridgemod)
linmod <- lm(Phenotypes ~ ., data = as.data.frame(GenCont))
summary(linmod)

The t values are much larger for ridge regression than linear regression. The bias is fairly small.

Answer 2

Yes there are plenty of cases; you're beating around the bush that is the topic of Bias-Variance tradeoff (in particular, the graphic to the right is a good visualization).

As for a mathematical example, I am pulling the following example from the excellent Statistical Inference by Casella and Berger to show that a biased estimator has lower Mean Squared Error and thus is considered better.

Let $X_1, ..., X_n$ be i.i.d. n$(\mu, \sigma^2)$ (i.e. Gaussian with mean $\mu$ and variance $\sigma^2$ in their notation). We will compare two estimators of $\sigma^2$: the first, unbiased, estimator is $$\hat{\sigma}_{unbiased}^2 := \frac{1}{n-1}\sum_{i=1}^{n} (X_i - \bar{X})^2$$ usually called $S^2$, the canonical sample variance, and the second is $$\hat{\sigma}_{biased}^2 := \frac{1}{n}\sum_{i=1}^{n} (X_i - \bar{X})^2 = \frac{n-1}{n}\hat{\sigma}_{unbiased}^2$$ which is the Maximum Likelihood estimate of $\sigma^2$. First, the MSE of the unbiased estimator:

$$\begin{align} \text{MSE}(\hat{\sigma}^2_{unbiased}) &= \text{Var} \ \hat{\sigma}^2_{unbiased} + \text{Bias}(\hat{\sigma}^2_{unbiased})^2 \\ &= \frac{2\sigma^4}{n-1}\end{align}$$ The MSE of the biased, maximum likelihood estimate of $\sigma^2$ is: $$\begin{align}\text{MSE}(\hat{\sigma}_{biased}^2) &= \text{Var}\ \hat{\sigma}_{biased}^2 + \text{Bias}(\hat{\sigma}_{biased}^2)^2\\ &=\text{Var}\left(\frac{n-1}{n}\hat{\sigma}^2_{unbiased}\right) + \left(\text{E}\hat{\sigma}_{biased}^2 - \sigma^2\right)^2 \\ &=\left(\frac{n-1}{n}\right)^2\text{Var} \ \hat{\sigma}^2_{unbiased} \, + \left(\text{E}\left(\frac{n-1}{n}\hat{\sigma}^2_{unbiased}\right) - \sigma^2\right)^2\\ &= \frac{2(n-1)\sigma^4}{n^2} + \left(\frac{n-1}{n}\sigma^2 - \sigma^2\right)^2\\ &= \left(\frac{2n-1}{n^2}\right)\sigma^4\end{align}$$ Hence, $$\text{MSE}(\hat{\sigma}_{biased}^2) = \frac{2n-1}{n^2}\sigma^4 < \frac{2}{n-1}\sigma^4 = \text{MSE}(\hat{\sigma}_{unbiased}^2)$$

Answer 3

There are numerous examples where the MLE has smaller mean square error (MSE) than the best available unbiased estimator (though often there are estimators with smaller MSE still). The "standard" example in normal sampling arises in estimating the variance for example.

There are many cases where no unbiased estimator exists, such as the inverse of the rate parameter in a Poisson.

It's also possible to find situations where an unbiased estimator can sometimes give "impossible" estimates (give estimated values that the parameter simply cannot take, such as sometimes giving negative estimates for necessarily positive quantities for example).

One widely known example (see for example [1], but it often appears in student exercises) is for $X\sim \text{Pois}(\lambda)$ where the only unbiased estimator of $e^{-2\lambda}$ is $(-1)^X$. On observing an odd value for $X$, the estimate is negative.

[1] Romano, J. P. and Siegel, A. F. (1986),
Counterexamples in Probability and Statistics.
Boca Raton: Chapman and Hall/CRC.