7
$\begingroup$

We have a few questions and answers about when one would prefer a biased estimate over a unbiased one, but I have not found anything on the reverse question:

In what situations is it important to consider only unbiased estimators?

A lot of emphasis is put on the concept of unbiasedness is introductory statistical courses, but I've never read a compelling defense of this. Since we generally only collect data once, when is it useful to be correct on average (besides the possible psychological comfort it provides)? In what situations would one need to be correct on average?

I'm open to philosophical arguments, but would prefer concrete examples from research or industry.

$\endgroup$
  • 2
    $\begingroup$ It's difficult to imagine any definite answer could exist, for the simple reason that although an unbiased estimator is well-quantified--its bias is zero--a biased estimator is not: its bias could be anything. Where do you draw the line? Would an estimator with a bias of $10^{-100}\%$ be unacceptable compared to an unbiased estimator? Of course not. $\endgroup$ – whuber Jul 13 '17 at 18:56
  • $\begingroup$ Are you taking issue with my use of the word "need"? How about "in what situations would one have a strong compelling desire to be correct on average"? $\endgroup$ – Matthew Drury Jul 13 '17 at 19:11
  • 2
    $\begingroup$ Same comment: any circumstance will come with a nonzero amount of acceptable error. If anyone claims they do have a compelling need to be correct on average, you could prove them wrong by showing that they will do just as well by being $\epsilon\%$ away from average for a sufficiently small nonzero $\epsilon$. This suggests it would be better to ask questions like "What properties do unbiased estimators enjoy? How much could those properties be improved by considering biased estimators and what would be the cost of carrying out such improvements?" $\endgroup$ – whuber Jul 13 '17 at 20:01
  • 3
    $\begingroup$ There's definitely compelling evidence when not to use unbiased estimators: en.wikipedia.org/wiki/… $\endgroup$ – Alex R. Jul 13 '17 at 20:13
  • $\begingroup$ +1 but i still feel this question is too high level... $\endgroup$ – Haitao Du Jul 14 '17 at 2:30
5
$\begingroup$

I think it's safe to say there's no situation when one needs an unbiased estimator; for example, if $\mu = 1$ and we have $E[\hat \mu] = \mu + \epsilon$, there has got be an $\epsilon$ small enough that you cannot possibly care.

With that said, I think it's important to see unbiased estimators as more of the limit of something that is good. All else remaining the same, less bias is better. And there are plenty of consistent estimators in which the bias is so high in moderate samples that the estimator is greatly impacted. For example, in most maximum likelihood estimators, the estimate of variance components is often downward biased. In the cases of prediction intervals, for example, this can be a really big problem in the face of over fitting.

In short, I would extremely hard pressed to find a situation in which truly unbiased estimates are needed. However, it's quite easy to come up with problems in which the bias of an estimator is the crucial problem. Having an estimator be unbiased is probably never an absolute requirement, but having an estimator unbiased does mean that there's one potentially serious issue taken care of.

EDIT:

After thinking about it a little more, it occurred to me that out-of-sample error is the perfect answer for your request. The "classic" method for estimating out-of-sample error is the maximum likelihood estimator, which in the case of normal data, reduces to the in-sample error. While this estimator is consistent, with models with large degrees of freedom, the bias is so bad that it will recommend degenerate models (i.e. estimate 0 out-of-sample error with models that are heavily over fit). Cross-validation is a clever way of getting an unbiased estimate of the out-of-sample error. If you use cross-validation to do model selection, you again downwardly bias your out-of-sample error estimate...which is why you hold a validation dataset to get an unbiased estimate of the final selected model.

Of course, my comment about truly unbiased still remains: if I had an estimator had expected value of the out-of-sample error + $\epsilon$, I would happily use it instead for small enough $\epsilon$. But the method of cross-validation is motivated by trying to get an unbiased estimator of the out-of-sample error. And without cross-validation, the field of machine learning would look completely different than it does now.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.