2
$\begingroup$

I am trying to get an intuitive understanding of why there exists biased but consistent estimator.

Suppose $X_i \sim \mathcal{N}(\mu, 1)$. One example I came across is this:

$$W_n(X_1,\cdots,X_n) = \frac{1}{n} \sum_{i=1}^n X_i + \frac{1}{n}$$

I see that $W_n$ is biased due to the $\frac{1}{n}$ term, but is consistent. However, I feel that this example is somewhat "silly" because few people would use $W_n$ to estimate $\mu$.

Do we have any other example of biased but consistent estimator $W_n$, where people are more likely to use it?

$\endgroup$
  • $\begingroup$ Many commonly-used estimators are biased but consistent, for example, the MLE of the two parameters of the Negative Binomial distribution, or, for that matter, the method-of-moments estimator of same. $\endgroup$ – jbowman Sep 15 '17 at 23:51
7
$\begingroup$

Here's a straightforward one.

Consider a uniform population with unknown upper bound

$$ X \sim U(0, \theta) $$

A simple estimator of $\theta$ is the sample maximum

$$ \hat \theta = \max(x_1, x_2, \ldots, x_n) $$

This is a biased estimator. With a little math you can show that

$$ E[\hat \theta] = \frac{n}{n+1} \theta $$

Which is a little smaller than $\theta$ itself.

This also shows that the estimator is consistent, since $\frac{n}{n+1} \rightarrow 1$ as $n \rightarrow \infty$.

An natural unbiased estimator of the maximum is twice the sample mean. You can show that this unbiased estimator has much higher variance than the slightly biased on above.

$\endgroup$
6
$\begingroup$

A very commonly used consistent but biased estimator used is that of the estimated standard deviation.

If we are looking at a simple situation in our data is distributed as $x_i \sim N(\mu, \sigma^2)$, then sometimes the MLE estimate of $\sigma$ is used, ie

$\hat \sigma^2 = \frac{1}{n} \sum_{i = 1}^n (x_i - \bar x)^2$

This is, of course, a biased but consistent estimator of the variance. Some people may try to account for this bias by using

$\hat s^2 = \frac{1}{n-1} \sum_{i = 1}^n (x_i - \bar x)^2 $

which is now unbiased for $\sigma^2$...but people don't usually look at variances, they usually look at standard deviations! Jensen's inequality will tell us that if $\hat s^2$ is an unbiased estimator of $\sigma^2$ with positive variance, then $E[s] > \sqrt{E[s^2]}$...so even though we had a unbiased estimator for $\sigma^2$, by taking the square root of this estimator, we now have a biased estimator for $\sigma$!

More generally (and stated without proof), it is very common that MLE estimates of variance components will be downwardly biased but consistent. Hopefully, this bias is ignorable; in the example above, we can see that the fix is almost inconsequential for decent sized $n$. However, if the number of parameters estimated becomes very large, it is quite possible that this bias can be especially problematic; this manifests itself as overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.