2
$\begingroup$

I am familiar with the layperson explanation of an unbiased estimator as follows: if we repeat an experiment under identical conditions many times, the average value of the estimate will be close to the true value (see the following question and answers).

I find that, when I explain this to people, the common answer I get is something along the lines of: "But I only ever have one sample and I compute the estimator based on this one sample, so what implication does unbiasedness have for my one sample?"

Intuitively, my answer would be something like the following: if we have two estimators with equal variance but the first is unbiased and the second is biased, then for a given random sample, the unbiased estimator is more likely to be closer to the true value than the biased estimator.

My question is whether (1) this is actually correct and (2) if it is correct, is there an explanation of unbiasedness that does not also involve the variance of the estimator as well?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Related: stats.stackexchange.com/questions/236328/…, $\endgroup$
    – kjetil b halvorsen ♦
    Commented Aug 19, 2021 at 17:45
  • 1
    $\begingroup$ Have you seen the target visualization? $\endgroup$
    – dimitriy
    Commented Aug 19, 2021 at 17:58
  • $\begingroup$ Yes, I'm familiar with the visualization, but the target visualization uses many points (samples) and looks at the cluster. I'm interested in the concept as applied to any particular point/sample and wondering if the interpretation in my question is correct. $\endgroup$
    – Mr Saltine
    Commented Aug 19, 2021 at 18:02
  • 3
    $\begingroup$ Unless the bias is constant (in the parameter) this is not necessarily true. And anyway the same counterargument applies: a probability statement does not say anything about my current sample and estimator. $\endgroup$
    – Xi'an
    Commented Aug 19, 2021 at 19:46

1 Answer 1

Reset to default
3
$\begingroup$

if we repeat an experiment under identical conditions many times, the average value of the estimate will be close to the true value

was said, and the following figure from Understanding "variance" intuitively shown

enter image description here

and the laymen countered

But I only ever have one sample and I compute the estimator based on this one sample, so what implication does unbiasedness have for my one sample?

I would say something along the lines of : well, this estimate could be an overestimate, or it could be an underestimate, but if this sample is all we know, we are in no position to guess about either. So we can as well bet on the unbiased estimate, as it could as well be to low as to high.

Then try to imagine how our laymen could be countering:

So you are betting? But I want to know the truth!

Me: statistics is about betting. If you want the truth, certainty, go get a larger sample!

the sample ... but how do you know the sample is representative?

Me: You need to know how your sample was obtained! If it was obtained by simple random sampling (or some elaboration ...), it has a high probability of being representative. If it wasn't obtained in some such way that we can analyze, unbiased estimation might not be the correct tool. Then we would have to look closer at how you obtained the sample first ...

OK, I used simple random sampling. But still, what if an underestimation would be a small error, but an overestimation might be huge?

Me: That would mean that your distribution is not symmetric. Indeed, unbiased estimators works best in symmetric cases. For asymmetric cases, maybe look into median unbiased estimators.

Continuing the dialogue, I would point out that betting on the unbiased estimator is reasonable provided that we have really used all available information. So, the estimator must be optimal in some sense (maybe sufficient), so that is it really using all the relevant information in the sample. Still there might be non-sample prior information, which in some cases could show us that the realized value of the unbiased estimator is unreasonable. And at this point I could show an example from What is the difference between finite and infinite variance

But then, finally, back to your first question:

Intuitively, my answer would be something like the following: if we have two estimators with equal variance but the first is unbiased and the second is biased, then for a given random sample, the unbiased estimator is more likely to be closer to the true value than the biased estimator.

Is that true? Mostly, yes, assuming that we have an optimal unbiased estimator, such as an unbiased minimum-variance estimator. But, in many situations, there are (slightly) biased estimators that are better, for instance in having lower mean-square error. These are often known as shrinkage estimators, or found via regularization. See

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ The practical conclusion then is that whenever a lower MSE estimator is available, we should prefer it over an unbiased one as unbiasedness holds no practical merit in most studies? (Maybe a case can be roughly made for meta-analysis) Then the question is why such an intuitive conclusion is so absent from discourse. There are still on this site unanswered questions (e.g. stats.stackexchange.com/questions/239333/…) on the merit of unbiasedness and the same repeated asymptotic justifications are invoked as if avoiding the evident. $\endgroup$
    – Kuku
    Commented Oct 2, 2021 at 11:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.