Smarter example of biased but consistent estimator?

Question

I am trying to get an intuitive understanding of why there exists biased but consistent estimator.

Suppose $X_i \sim \mathcal{N}(\mu, 1)$. One example I came across is this:

$$W_n(X_1,\cdots,X_n) = \frac{1}{n} \sum_{i=1}^n X_i + \frac{1}{n}$$

I see that $W_n$ is biased due to the $\frac{1}{n}$ term, but is consistent. However, I feel that this example is somewhat "silly" because few people would use $W_n$ to estimate $\mu$.

Do we have any other example of biased but consistent estimator $W_n$, where people are more likely to use it?

Answer 1

Here's a straightforward one.

Consider a uniform population with unknown upper bound

$$ X \sim U(0, \theta) $$

A simple estimator of $\theta$ is the sample maximum

$$ \hat \theta = \max(x_1, x_2, \ldots, x_n) $$

This is a biased estimator. With a little math you can show that

$$ E[\hat \theta] = \frac{n}{n+1} \theta $$

Which is a little smaller than $\theta$ itself.

This also shows that the estimator is consistent, since $\frac{n}{n+1} \rightarrow 1$ as $n \rightarrow \infty$.

An natural unbiased estimator of the maximum is twice the sample mean. You can show that this unbiased estimator has much higher variance than the slightly biased on above.

Answer 2

A very commonly used consistent but biased estimator used is that of the estimated standard deviation.

If we are looking at a simple situation in our data is distributed as $x_i \sim N(\mu, \sigma^2)$, then sometimes the MLE estimate of $\sigma$ is used, ie

$\hat \sigma^2 = \frac{1}{n} \sum_{i = 1}^n (x_i - \bar x)^2$

This is, of course, a biased but consistent estimator of the variance. Some people may try to account for this bias by using

$\hat s^2 = \frac{1}{n-1} \sum_{i = 1}^n (x_i - \bar x)^2 $

which is now unbiased for $\sigma^2$...but people don't usually look at variances, they usually look at standard deviations! Jensen's inequality will tell us that if $\hat s^2$ is an unbiased estimator of $\sigma^2$ with positive variance, then $E[s] > \sqrt{E[s^2]}$...so even though we had a unbiased estimator for $\sigma^2$, by taking the square root of this estimator, we now have a biased estimator for $\sigma$!

More generally (and stated without proof), it is very common that MLE estimates of variance components will be downwardly biased but consistent. Hopefully, this bias is ignorable; in the example above, we can see that the fix is almost inconsequential for decent sized $n$. However, if the number of parameters estimated becomes very large, it is quite possible that this bias can be especially problematic; this manifests itself as overfitting.