50
$\begingroup$

It's obvious many times why one prefers an unbiased estimator. But, are there any circumstances under which we might actually prefer a biased estimator over an unbiased one?

$\endgroup$
6
  • 5
    $\begingroup$ Related: Why does shrinkage work? $\endgroup$ Apr 17 '16 at 8:48
  • 17
    $\begingroup$ Actually it is not obvious to me why one prefers an unbiased estimator. Bias is like the boogeyman in statistic books, creating unnecessary fear among students of statistics. In reality information theoretic approach to learning always leads to a biased estimation in small samples, and is consistent in the limit. $\endgroup$ Apr 17 '16 at 15:17
  • 9
    $\begingroup$ I have had clients (especially in legal cases) who would strongly prefer biased estimators, provided the bias were systematically in their favor! $\endgroup$
    – whuber
    Apr 20 '16 at 22:16
  • 4
    $\begingroup$ Section 17.2 ("Unbiased estimators") of Jaynes's Probability Theory: The Logic of Science is a very insightful discussion, with examples, of whether the bias of an estimator really is or is not important, and why a biased one may be preferable (in line with Chaconne's great answer below). $\endgroup$
    – pglpm
    Sep 16 '17 at 22:25
  • 2
    $\begingroup$ If I can summarize Chaconne-Jaynes's answer: an "unbiased" estimator can err to the right or to the left of the true value by equal amounts; a "biased" one can err more to the right than to the left or vice versa. But the error of the unbiased one, though symmetric, can be far greater than that of the biased one. See Chaconne's first figure. In many situations it's much more important that an estimator have a small error, rather than that this error be symmetric. $\endgroup$
    – pglpm
    Sep 16 '17 at 22:45
65
$\begingroup$

Yes. Often it is the case that we are interested in minimizing the mean squared error, which can be decomposed into variance + bias squared. This is an extremely fundamental idea in machine learning, and statistics in general. Frequently we see that a small increase in bias can come with a large enough reduction in variance that the overall MSE decreases.

A standard example is ridge regression. We have $\hat \beta_R = (X^T X + \lambda I)^{-1}X^T Y$ which is biased; but if $X$ is ill conditioned then $Var(\hat \beta) \propto (X^T X)^{-1}$ may be monstrous whereas $Var(\hat \beta_R)$ can be much more modest.

Another example is the kNN classifier. Think about $k = 1$: we assign a new point to its nearest neighbor. If we have a ton of data and only a few variables we can probably recover the true decision boundary and our classifier is unbiased; but for any realistic case, it is likely that $k = 1$ will be far too flexible (i.e. have too much variance) and so the small bias is not worth it (i.e. the MSE is larger than more biased but less variable classifiers).

Finally, here's a picture. Suppose that these are the sampling distributions of two estimators and we are trying to estimate 0. The flatter one is unbiased, but also much more variable. Overall I think I'd prefer to use the biased one, because even though on average we won't be correct, for any single instance of that estimator we'll be closer.

bias-variance

$$ \ $$ Update

I mention the numerical issues that happen when $X$ is ill conditioned and how ridge regression helps. Here's an example.

I'm making a matrix $X$ which is $4 \times 3$ and the third column is nearly all 0, meaning that it is almost not full rank, which means that $X^T X$ is really close to being singular.

    x <- cbind(0:3, 2:5, runif(4, -.001, .001)) 
           ## almost reduced rank
    
    > x
         [,1] [,2]        [,3]
    [1,]    0    2 0.000624715
    [2,]    1    3 0.000248889
    [3,]    2    4 0.000226021
    [4,]    3    5 0.000795289

    (xtx <- t(x) %*% x) 
     ## the inverse of this is proportional to Var(beta.hat)

               [,1]        [,2]        [,3]
    [1,] 14.0000000 26.00000000 3.08680e-03
    [2,] 26.0000000 54.00000000 6.87663e-03
    [3,]  0.0030868  0.00687663 1.13579e-06
    
    eigen(xtx)$values 
      ## all eigenvalues > 0 so it is PD, but not by much
    
    [1] 6.68024e+01 1.19756e+00 2.26161e-07


    solve(xtx) ## huge values
    
               [,1]        [,2]        [,3]
    [1,]   0.776238   -0.458945     669.057
    [2,]  -0.458945    0.352219    -885.211
    [3,] 669.057303 -885.210847 4421628.936

    solve(xtx + .5 * diag(3)) ## very reasonable values

                 [,1]         [,2]         [,3]
    [1,]  0.477024087 -0.227571147  0.000184889
    [2,] -0.227571147  0.126914719 -0.000340557
    [3,]  0.000184889 -0.000340557  1.999998999

Update 2

As promised, here's a more thorough example.

First, remember the point of all of this: we want a good estimator. There are many ways to define 'good'. Suppose that we've got $X_1, ..., X_n \sim \ iid \ \mathcal N(\mu, \sigma^2)$ and we want to estimate $\mu$.

Let's say that we decide that a 'good' estimator is one that is unbiased. This isn't optimal because, while it is true that the estimator $T_1(X_1, ..., X_n) = X_1$ is unbiased for $\mu$, we have $n$ data points so it seems silly to ignore almost all of them. To make that idea more formal, we think that we ought to be able to get an estimator that varies less from $\mu$ for a given sample than $T_1$. This means that we want an estimator with a smaller variance.

So maybe now we say that we still want only unbiased estimators, but among all unbiased estimators we'll choose the one with the smallest variance. This leads us to the concept of the uniformly minimum variance unbiased estimator (UMVUE), an object of much study in classical statistics. IF we only want unbiased estimators, then choosing the one with the smallest variance is a good idea. In our example, consider $T_1$ vs. $T_2(X_1, ..., X_n) = \frac{X_1 + X_2}{2}$ and $T_n(X_1, ..., X_n) = \frac{X_1 + ... + X_n}{n}$. Again, all three are unbiased but they have different variances: $Var(T_1) = \sigma^2$, $Var(T_2) = \frac{\sigma^2}{2}$, and $Var(T_n) = \frac{\sigma^2}{n}$. For $n > 2$ $T_n$ has the smallest variance of these, and it's unbiased, so this is our chosen estimator.

But often unbiasedness is a strange thing to be so fixated on (see @Cagdas Ozgenc's comment, for example). I think this is partly because we generally don't care so much about having a good estimate in the average case, but rather we want a good estimate in our particular case. We can quantify this concept with the mean squared error (MSE) which is like the average squared distance between our estimator and the thing we're estimating. If $T$ is an estimator of $\theta$, then $MSE(T) = E((T - \theta)^2)$. As I've mentioned earlier, it turns out that $MSE(T) = Var(T) + Bias(T)^2$, where bias is defined to be $Bias(T) = E(T) - \theta$. Thus we may decide that rather than UMVUEs we want an estimator that minimizes MSE.

Suppose that $T$ is unbiased. Then $MSE(T) = Var(T) = Bias(T)^2 = Var(T)$, so if we are only considering unbiased estimators then minimizing MSE is the same as choosing the UMVUE. But, as I showed above, there are cases where we can get an even smaller MSE by considering non-zero biases.

In summary, we want to minimize $Var(T) + Bias(T)^2$. We could require $Bias(T) = 0$ and then pick the best $T$ among those that do that, or we could allow both to vary. Allowing both to vary will likely give us a better MSE, since it includes the unbiased cases. This idea is the variance-bias trade-off that I mentioned earlier in the answer.

Now here are some pictures of this trade-off. We're trying to estimate $\theta$ and we've got five models, $T_1$ through $T_5$. $T_1$ is unbiased and the bias gets more and more severe until $T_5$. $T_1$ has the largest variance and the variance gets smaller and smaller until $T_5$. We can visualize the MSE as the square of the distance of the distribution's center from $\theta$ plus the square of the distance to the first inflection point (that's a way to see the SD for normal densities, which these are). We can see that for $T_1$ (the black curve) the variance is so large that being unbiased doesn't help: there's still a massive MSE. Conversely, for $T_5$ the variance is way smaller but now the bias is big enough that the estimator is suffering. But somewhere in the middle there is a happy medium, and that's $T_3$. It has reduced the variability by a lot (compared with $T_1$) but has only incurred a small amount of bias, and thus it has the smallest MSE.

VBtradeoff

You asked for examples of estimators that have this shape: one example is ridge regression, where you can think of each estimator as $T_\lambda(X, Y) = (X^T X + \lambda I)^{-1} X^T Y$. You could (perhaps using cross-validation) make a plot of MSE as a function of $\lambda$ and then choose the best $T_\lambda$.

$\endgroup$
3
  • 2
    $\begingroup$ Probably the most effort anyone ever put into answering one of my questions. Thanks a lot. $\endgroup$ Apr 21 '16 at 0:08
  • $\begingroup$ @Chaconne Can I know when and why we should prefer unbiased estimator( e.g., for covariance matrix estimation) instead of biased regarding analysis and practical applications? $\endgroup$
    – olivia
    Apr 16 '17 at 8:39
  • 2
    $\begingroup$ @olivia i can't think of a single non-trivial case where bias is the only criterion i care about (although there may be such cases that I just don't know about!), although there are times when bias is known to be a dominating factor (consider REML, for instance, where the bias is severe enough that it's worth doing something about). I think no matter what you're doing you just want your one particular estimator to be close to the truth, and that's what MSE does. $\endgroup$
    – jld
    Apr 18 '17 at 22:13
5
$\begingroup$

This paper [1] gives a simple example demostrating that a biased estimator can even achieve a lower variance than the Cramér–Rao bound (CRB).

Consider $i.i.d. X_1,...,X_n\sim N(0,\sigma^2)$, and let $k=\sigma^2$.

The maximum likelihood estimator for $k$ is $\hat{k}_{ML}=\frac{1}{n}\sum{X_i^2}$. It is unbiased with a variance of $MSE_{ML}=E[(\hat{k}_{ML}-k)^2]=\frac{2\sigma^4}{n}=CRB$.

Estimator $\hat{k}=\frac{1}{n+2}\sum{X_i^2}$ is biased but its variance is $MSE=E[(\hat{k}-k)^2]=\frac{2\sigma^4}{n+2}<MSE_{ML}=CRB$.


[1] Stoica, P. , and R. L. Moses . "On biased estimators and the unbiased Cramér-Rao lower bound." Signal Processing 21.4(1990):349-350.

$\endgroup$
1
  • $\begingroup$ This entire discussion seems to be devoid of context. Bias is important in the frequentist domain where you have the possibility of repeated samples. In general, the data generating mechanism may be capable of generating only one sample (i.e., bias cannot be computed) , and minimizing Bayes risk is the goal. Bias plays no direct role then. $\endgroup$ Sep 30 at 11:36
4
$\begingroup$

The other examples in this thread are fantastic, but I wanted to provide an extremely simple example that illustrates that a biased estimator can sometimes have drastically smaller variance.


Let $X_1, X_2, \ldots X_n \stackrel{\text{iid}}{\sim} \text{Unif}(0, \theta)$.

First we consider the Method of Moments estimator $$\hat\theta_1 = 2\bar X.$$

This estimator is intuitive and it is unbiased, but it is an estimator with relatively large variance.

\begin{align*} \text{bias}(\hat\theta_1) &= 0 \\ \text{Var}(\hat\theta_1) &= \frac{\theta^2}{3n} \\ \text{MSE}(\hat\theta_1) &= \frac{\theta^2}{3n} = \mathcal O(n^{-1}) \end{align*}

The maximum likelihood estimator, on the other hand, is given by $$\hat\theta_2 = X_{(n)} = \text{max}_{i}\{X_i\}$$ This estimator is clearly biased since all $X_i < \theta$. But it turns out that the bias is relatively small, and the variance is much smaller that of $\hat\theta_1$.

\begin{align*} \text{bias}(\hat\theta_2) &= \frac{-\theta}{n+1} \\ \text{Var}(\hat\theta_2) &= \frac{n\theta^2}{(n+1)^2(n+2)} \\ \text{MSE}(\hat\theta_2) &= \frac{2\theta^2}{(n+1)(n+2)} = \mathcal O(n^{-2}) \end{align*}

The MSE of the second estimator tends to zero much faster than the first estimator. This example shows that bias should not be the only thing we consider when choosing an estimator.


Further discussion:

While the MLE ($\hat\theta_2$) (for this problem) is generally considered a better estimator than MOM ($\hat\theta_1$), neither would be a reasonable choice in practice. This is because the MLE can be adjusted so that it is unbiased. Consider $$\hat\theta_3 = \frac{n+1}{n}X_{(n)}.$$ Here, we have reduced the bias to zero, but in doing so we have inflated the variance. \begin{align*} \text{bias}(\hat\theta_3) &= 0 \\ \text{Var}(\hat\theta_3) &= \frac{\theta^2}{n(n+2)} \\ \text{MSE}(\hat\theta_3) &= \frac{\theta^2}{n(n+2)} = \mathcal O(n^{-2}) \end{align*} Still, this estimator is preferable (from the perspective of MSE) to either of the previous estimators.

So now we notice: (i) $\hat\theta_2$ is an estimator with high bias and low variance and (ii) $\hat\theta_3 = c\hat\theta_2$ is an estimator with low bias and high variance. This begs the question, is there an estimator "in between" these two that achieves smaller MSE?"

The answer is yes. Consider $$\hat\theta_4 = \frac{(n+1)(n+2)}{n(n+2) + 1}X_{(n)}.$$ This estimator reintroduces some bias to reduce the variance. It is provably the estimator of the form $cX_{(n)}$ which minimizes MSE.

The takeaway here, again, is that bias and variance are two separate quantities which we would like to minimize. Often reducing one metric leads to an increase in the other. An estimator should be chosen with this tradeoff in mind. MSE is a popular (but certainly not the only) metric which takes this tradeoff into account.

$\endgroup$
1
  • 1
    $\begingroup$ +1 great example! $\endgroup$
    – jld
    Sep 30 at 15:15
3
$\begingroup$

Two reasons come to mind, aside from the MSE explanation above (the commonly accepted answer to the question):

  • Managing risk
  • Efficient testing

Risk, roughly, is the sense of how much something can explode when certain conditions aren't met. Take superefficient estimators: $T(X) = \bar{X}_n$ if $\bar{X}_n$ lies beyond an $\epsilon$-ball of 0, 0 otherwise. You can show that this statistic is more efficient than the UMVUE, since it has the same asymptotic variance as the UMVUE with $\theta \ne 0$ and infinite efficiency otherwise. This is a stupid statistic, and Hodges threw it out there as a strawman. Turns out that if you take $\theta_n$ on the boundary of the ball, it becomes an inconsistent test, it never knows what's going on and the risk explodes.

In the minimax world, we try to minimize risk. It can give us biased estimators, but we don't care, they still work because there are fewer ways to break the system. Suppose, for instance, I were interested in inference on a $\Gamma(\alpha, \beta_n)$ distribution, and once in a while the distribution threw curve balls. A trimmed mean estimate $$T_\theta(X) = \sum X_i \mathcal{I} (\|X_i\| < \theta) / \sum \mathcal{I} (\|X_i\| < \theta)$$ systematically throws out the high leverage points.

Efficient testing means you don't estimate the thing you're interested in, but an approximation thereof, because this provides a more powerful test. The best example I can think of here is logistic regression. People always confuse logistic regression with relative risk regression. For instance an odds ratio of 1.6 for cancer comparing smokers to non-smokers does NOT mean that "smokers had a 1.6 greater risk of cancer". BZZT wrong. That's a risk ratio. They technically had a 1.6 fold odds of the outcome (reminder: odds = probability / (1-probability)). However, for rare events, the odds ratio approximates the risk ratio. There is relative risk regression, but it has a lot of issues with converging and is not as powerful as logistic regression. So we report the OR as a biased estimate of the RR (for rare events), and calculate more efficient CIs and p-values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.