When is a biased estimator preferable to unbiased one? It's obvious many times why one prefers an unbiased estimator. But, are there any circumstances under which we might actually prefer a biased estimator over an unbiased one?
 A: This paper [1] gives a simple example demostrating that a biased estimator can even achieve a lower variance than the Cramér–Rao bound (CRB).
Consider $i.i.d. X_1,...,X_n\sim N(0,\sigma^2)$, and let $k=\sigma^2$.
The maximum likelihood estimator for $k$ is $\hat{k}_{ML}=\frac{1}{n}\sum{X_i^2}$. It is unbiased with a variance of $MSE_{ML}=E[(\hat{k}_{ML}-k)^2]=\frac{2\sigma^4}{n}=CRB$.
Estimator $\hat{k}=\frac{1}{n+2}\sum{X_i^2}$ is biased but its variance is $MSE=E[(\hat{k}-k)^2]=\frac{2\sigma^4}{n+2}<MSE_{ML}=CRB$.

[1] Stoica, P. , and  R. L. Moses . "On biased estimators and the unbiased Cramér-Rao lower bound." Signal Processing 21.4(1990):349-350.
A: The other examples in this thread are fantastic, but I wanted to provide an extremely simple example that illustrates that a biased estimator can sometimes have drastically smaller variance.

Let $X_1, X_2, \ldots X_n \stackrel{\text{iid}}{\sim} \text{Unif}(0, \theta)$.
First we consider the Method of Moments estimator
$$\hat\theta_1 = 2\bar X.$$
This estimator is intuitive and it is unbiased, but it is an estimator with relatively large variance.
\begin{align*}
\text{bias}(\hat\theta_1) &= 0 \\
\text{Var}(\hat\theta_1) &= \frac{\theta^2}{3n} \\
\text{MSE}(\hat\theta_1) &= \frac{\theta^2}{3n} = \mathcal O(n^{-1})
\end{align*}
The maximum likelihood estimator, on the other hand, is given by
$$\hat\theta_2 = X_{(n)} = \text{max}_{i}\{X_i\}$$
This estimator is clearly biased since all $X_i < \theta$. But it turns out that the bias is relatively small, and the variance is much smaller that of $\hat\theta_1$.
\begin{align*}
\text{bias}(\hat\theta_2) &= \frac{-\theta}{n+1} \\
\text{Var}(\hat\theta_2) &= \frac{n\theta^2}{(n+1)^2(n+2)} \\
\text{MSE}(\hat\theta_2) &= \frac{2\theta^2}{(n+1)(n+2)} = \mathcal O(n^{-2})
\end{align*}
The MSE of the second estimator tends to zero much faster than the first estimator. This example shows that bias should not be the only thing we consider when choosing an estimator.

Further discussion:
While the MLE ($\hat\theta_2$) (for this problem) is generally considered a better estimator than MOM ($\hat\theta_1$), neither would be a reasonable choice in practice. This is because the MLE can be adjusted so that it is unbiased. Consider
$$\hat\theta_3 = \frac{n+1}{n}X_{(n)}.$$
Here, we have reduced the bias to zero, but in doing so we have inflated the variance.
\begin{align*}
\text{bias}(\hat\theta_3) &= 0 \\
\text{Var}(\hat\theta_3) &= \frac{\theta^2}{n(n+2)} \\
\text{MSE}(\hat\theta_3) &= \frac{\theta^2}{n(n+2)} = \mathcal O(n^{-2})
\end{align*}
Still, this estimator is preferable (from the perspective of MSE) to either of the previous estimators.
So now we notice: (i) $\hat\theta_2$ is an estimator with high bias and low variance and (ii) $\hat\theta_3 = c\hat\theta_2$ is an estimator with low bias and high variance. This begs the question, is there an estimator "in between" these two that achieves smaller MSE?"
The answer is yes. Consider
$$\hat\theta_4 = \frac{(n+1)(n+2)}{n(n+2) + 1}X_{(n)}.$$
This estimator reintroduces some bias to reduce the variance. It is provably the estimator of the form $cX_{(n)}$ which minimizes MSE.
The takeaway here, again, is that bias and variance are two separate quantities which we would like to minimize. Often reducing one metric leads to an increase in the other. An estimator should be chosen with this tradeoff in mind. MSE is a popular (but certainly not the only) metric which takes this tradeoff into account.
A: Two reasons come to mind, aside from the MSE explanation above (the commonly accepted answer to the question):


*

*Managing risk

*Efficient testing


Risk, roughly, is the sense of how much something can explode when certain conditions aren't met. Take superefficient estimators: $T(X) = \bar{X}_n$ if $\bar{X}_n$ lies beyond an $\epsilon$-ball  of 0, 0 otherwise. You can show that this statistic is more efficient than the UMVUE, since it has the same asymptotic variance as the UMVUE with $\theta \ne 0$ and infinite efficiency otherwise. This is a stupid statistic, and Hodges threw it out there as a strawman. Turns out that if you take $\theta_n$ on the boundary of the ball, it becomes an inconsistent test, it never knows what's going on and the risk explodes. 
In the minimax world, we try to minimize risk. It can give us biased estimators, but we don't care, they still work because there are fewer ways to break the system. Suppose, for instance, I were interested in inference on a $\Gamma(\alpha, \beta_n)$ distribution, and once in a while the distribution threw curve balls. A trimmed mean estimate $$T_\theta(X) = \sum X_i \mathcal{I} (\|X_i\| < \theta) / \sum \mathcal{I} (\|X_i\| < \theta)$$ systematically throws out the high leverage points.
Efficient testing means you don't estimate the thing you're interested in, but an approximation thereof, because this provides a more powerful test. The best example I can think of here is logistic regression. People always confuse logistic regression with relative risk regression. For instance an odds ratio of 1.6 for cancer comparing smokers to non-smokers does NOT mean that "smokers had a 1.6 greater risk of cancer". BZZT wrong. That's a risk ratio. They technically had a 1.6 fold odds of the outcome (reminder: odds = probability / (1-probability)). However, for rare events, the odds ratio approximates the risk ratio. There is relative risk regression, but it has a lot of issues with converging and is not as powerful as logistic regression. So we report the OR as a biased estimate of the RR (for rare events), and calculate more efficient CIs and p-values.
A: The maximum-likelihood estimator $\frac 1 n \sum_{i=1}^n (X_i - \overline X)^2$ of the population variance for a normally distributed population has a lower mean squared error than does the commonplace unbiased estimator, in which the denominator is $n-1.$ But that's a somewhat weak example.
I wrote a paper addressing this question via a counterexample of my own devising: https://arxiv.org/pdf/math/0206006.pdf
