15
$\begingroup$

This question already has an answer here:

A perfect estimator would be accurate (unbiased) and precise (good estimation even with small samples).

I never really thought of the question of precision but only the one of accuracy (as I did in Estimator of $\frac{\sigma^2}{\mu (1 - \mu)}$ when sampling without replacement for example).

Are there cases where the unbiased estimator is less precise (and therefore eventually "less good") than a biased estimator? If yes, I would love a simple example proving mathematically that the less accurate estimator is so much more precise that it could be considered better.

$\endgroup$

marked as duplicate by kjetil b halvorsen, user158565, Michael Chernick, jpmuc, mkt Jul 22 at 10:57

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Could that be a duplicate? Anyone? $\endgroup$ – Richard Hardy Sep 15 '17 at 7:27
  • $\begingroup$ It depends on what is important. If you are estimating anything gaussian distributed then unbiasedness is important, but for any other distribution in our world some bias could make the estimate fit the true distribution better. $\endgroup$ – mathreadler Sep 15 '17 at 9:58
  • $\begingroup$ @RichardHardy I discuss this at some length here: stats.stackexchange.com/questions/207760/… and almost surely that's not the first place on this site where this discussion occurs $\endgroup$ – jld Sep 15 '17 at 14:57
  • $\begingroup$ @RichardHardy The complete question text of "When is a biased estimator preferable to unbiased one?" is "It's obvious many times why one prefers an unbiased estimator. But, are there any circumstances under which we might actually prefer a biased estimator over an unbiased one?" I wonder whether there is really a difference between the questions; the answers here all seem relevant there, which suggests not just closing as a duplicate but potentially a full merge would be a good idea. $\endgroup$ – Silverfish Sep 15 '17 at 19:05
34
$\begingroup$

One example is estimates from ordinary least squares regression when there is collinearity. They are unbiased but have huge variance. Ridge regression on the same problem yields estimates that are biased but have much lower variance. E.g.

install.packages("ridge")
library(ridge)
set.seed(831)

data(GenCont)
ridgemod <- linearRidge(Phenotypes ~ ., data = as.data.frame(GenCont))
summary(ridgemod)
linmod <- lm(Phenotypes ~ ., data = as.data.frame(GenCont))
summary(linmod)

The t values are much larger for ridge regression than linear regression. The bias is fairly small.

$\endgroup$
  • 10
    $\begingroup$ Not sure why this answer was down voted, penalized regression is a canonical example of great practical importance. $\endgroup$ – Matthew Drury Sep 15 '17 at 1:04
20
$\begingroup$

Yes there are plenty of cases; you're beating around the bush that is the topic of Bias-Variance tradeoff (in particular, the graphic to the right is a good visualization).

As for a mathematical example, I am pulling the following example from the excellent Statistical Inference by Casella and Berger to show that a biased estimator has lower Mean Squared Error and thus is considered better.

Let $X_1, ..., X_n$ be i.i.d. n$(\mu, \sigma^2)$ (i.e. Gaussian with mean $\mu$ and variance $\sigma^2$ in their notation). We will compare two estimators of $\sigma^2$: the first, unbiased, estimator is $$\hat{\sigma}_{unbiased}^2 := \frac{1}{n-1}\sum_{i=1}^{n} (X_i - \bar{X})^2$$ usually called $S^2$, the canonical sample variance, and the second is $$\hat{\sigma}_{biased}^2 := \frac{1}{n}\sum_{i=1}^{n} (X_i - \bar{X})^2 = \frac{n-1}{n}\hat{\sigma}_{unbiased}^2$$ which is the Maximum Likelihood estimate of $\sigma^2$. First, the MSE of the unbiased estimator:

$$\begin{align} \text{MSE}(\hat{\sigma}^2_{unbiased}) &= \text{Var} \ \hat{\sigma}^2_{unbiased} + \text{Bias}(\hat{\sigma}^2_{unbiased})^2 \\ &= \frac{2\sigma^4}{n-1}\end{align}$$ The MSE of the biased, maximum likelihood estimate of $\sigma^2$ is: $$\begin{align}\text{MSE}(\hat{\sigma}_{biased}^2) &= \text{Var}\ \hat{\sigma}_{biased}^2 + \text{Bias}(\hat{\sigma}_{biased}^2)^2\\ &=\text{Var}\left(\frac{n-1}{n}\hat{\sigma}^2_{unbiased}\right) + \left(\text{E}\hat{\sigma}_{biased}^2 - \sigma^2\right)^2 \\ &=\left(\frac{n-1}{n}\right)^2\text{Var} \ \hat{\sigma}^2_{unbiased} \, + \left(\text{E}\left(\frac{n-1}{n}\hat{\sigma}^2_{unbiased}\right) - \sigma^2\right)^2\\ &= \frac{2(n-1)\sigma^4}{n^2} + \left(\frac{n-1}{n}\sigma^2 - \sigma^2\right)^2\\ &= \left(\frac{2n-1}{n^2}\right)\sigma^4\end{align}$$ Hence, $$\text{MSE}(\hat{\sigma}_{biased}^2) = \frac{2n-1}{n^2}\sigma^4 < \frac{2}{n-1}\sigma^4 = \text{MSE}(\hat{\sigma}_{unbiased}^2)$$

$\endgroup$
9
$\begingroup$

There are numerous examples where the MLE has smaller mean square error (MSE) than the best available unbiased estimator (though often there are estimators with smaller MSE still). The "standard" example in normal sampling arises in estimating the variance for example.

There are many cases where no unbiased estimator exists, such as the inverse of the rate parameter in a Poisson.

It's also possible to find situations where an unbiased estimator can sometimes give "impossible" estimates (give estimated values that the parameter simply cannot take, such as sometimes giving negative estimates for necessarily positive quantities for example).

One widely known example (see for example [1], but it often appears in student exercises) is for $X\sim \text{Pois}(\lambda)$ where the only unbiased estimator of $e^{-2\lambda}$ is $(-1)^X$. On observing an odd value for $X$, the estimate is negative.

[1] Romano, J. P. and Siegel, A. F. (1986),
Counterexamples in Probability and Statistics.
Boca Raton: Chapman and Hall/CRC.

$\endgroup$
  • $\begingroup$ Can you give specific examples of the latter situation? $\endgroup$ – Felix Goldberg Sep 17 '17 at 11:02
  • $\begingroup$ done . . . . . $\:$ $\endgroup$ – Glen_b Sep 17 '17 at 18:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.