10
$\begingroup$

Suppose $\hat{\theta}$ is an unbiased estimator for $\theta$. Then of course, $\mathbb{E}[\hat{\theta} \mid \theta] = \theta$.

How does one explain this to a layperson? In the past, what I have said is if you average a bunch of values of $\hat{\theta}$, as the sample size gets larger, you get a better approximation of $\theta$.

To me, this is problematic. I think what I'm actually describing here is this phenomenon of being asymptotically unbiased, rather than solely being unbiased, i.e., $$\lim_{n \to \infty}\mathbb{E}[\hat{\theta} \mid \theta] = \theta\text{,}$$ where $\hat{\theta}$ is likely dependent on $n$.

So, how does one explain what an unbiased estimator is to a layperson?

$\endgroup$
  • 2
    $\begingroup$ Its a way of making an estimate that's just about right: it's usually not exactly right but on the whole it doesn't produce overestimates more often than underestimates. I realize that this makes it sound more like $\theta$ is the median of $\hat \theta$ than the mean, but I think it captures the essential point. $\endgroup$ – jwimberley Sep 22 '16 at 12:13
  • 3
    $\begingroup$ I like the "three statisticians hunting" joke (a version here) for this ... $\endgroup$ – Ben Bolker Sep 22 '16 at 12:31
  • 2
    $\begingroup$ Your explanation is the Law of Large Numbers, it has nothing to do with unbiasedness. $\endgroup$ – Xi'an Sep 22 '16 at 13:42
  • $\begingroup$ @Xi'an: If the estimator was biased, the limit wouldn't be $\theta$. $\endgroup$ – user2357112 Sep 22 '16 at 18:23
  • $\begingroup$ @user2357112: in my understanding (and others', as shown by the answers so far), as the sample size gets larger means considering $\hat\theta_n$ as $n$ grows to infinity, i.e. an estimator based on $n$ observations. I now see the sentence can be interpreted differently. $\endgroup$ – Xi'an Sep 22 '16 at 18:48
14
$\begingroup$

Technically what you're describing when you say that your estimator gets closer to the true value as the sample size grows is (as others have mentioned) consistency, or convergence of statistical estimators. This convergence can either be convergence in probability, which says that $\lim_{n \to \infty} P(|\hat{\theta}_n - \theta| > \epsilon) = 0$ for every $\epsilon > 0$, or almost sure convergence which says that $P(\lim_{n \to \infty} |\hat{\theta}_n - \theta| > \epsilon) = 0$. Notice how the limit is actually inside the probability in the second case. It turns out this latter form of convergence is stronger than the other, but both of them mean essentially the same thing, which is that the estimate tends to get closer and closer to the thing we're estimating as we gather more samples.

A subtle point here is that even when $\hat{\theta}_n \to \theta$ either in probability or almost surely, it is not true in general that $\lim_{n \to \infty} \text{E}(\hat{\theta}_n) = \theta$, so consistency does not imply asymptotic unbiasedness as you're suggesting. You have to be careful when moving between sequences of random variables (which are functions) to sequences of expectations (which are integrals).

All the technical stuff aside, unbiased only means that $\text{E}(\hat{\theta}_n) = \theta$. So when you explain it to someone just say that if the experiment were repeated under identical conditions many times that the average value of the estimate would be close to the true value.

$\endgroup$
  • 5
    $\begingroup$ Your vision of the layperson is quite admirable. He knows what's "convergence in probability", "a.s. convergence", limits... It's the man from the future. $\endgroup$ – Aksakal Sep 22 '16 at 19:24
  • 2
    $\begingroup$ I don't think a layperson knows any of these things, I was trying to correct some misunderstanding in the original post. My suggestion as to how to explain things to a layperson is in the last paragraph. $\endgroup$ – dsaxton Sep 22 '16 at 19:29
  • $\begingroup$ that last paragraph though entangles the bias concept with consistency of an estimator, which probably was one of the confusions of OP to start with. $\endgroup$ – Aksakal Sep 22 '16 at 19:31
  • 3
    $\begingroup$ How so? Repeating an experiment under identical conditions would mean that the sample size is fixed so we obviously aren't talking about consistency. $\endgroup$ – dsaxton Sep 22 '16 at 19:42
  • 1
    $\begingroup$ Ok, you're right about that, but then it means you're bringing in frequentist view of a probability $\endgroup$ – Aksakal Sep 22 '16 at 19:49
9
$\begingroup$

I am not sure if you confuse consistency and unbiasedness.

Consistency: The larger the sample size the smaller the variance of the estimator.

  • Depends on sample size

Unbiasedness: The expected value of the estimator equals the true value of the parameters

  • Does not depend on sample size

So your sentence

if you average a bunch of values of $\hat\theta$, as the sample size gets larger, you get a better approximation of $\theta$.

Is not correct. Even if the sample size gets infinite an unbiased estimator will stay an unbiased estimator, e.g. If you estimate the mean as "mean +1" you can add one billion observations to your sample and your estimator will still not give you the true value.

Here you can find a more profound discussion about the difference between consistency and unbiasedness.

What is the difference between a consistent estimator and an unbiased estimator?

$\endgroup$
  • 2
    $\begingroup$ I actually don't know anything about consistency, but thank you nevertheless. $\endgroup$ – Clarinetist Sep 22 '16 at 12:19
  • 1
    $\begingroup$ @Clarinetist Consistency is perhaps the most important property of an estimator, that with enough data, you will get arbitrarily close to the right answer. $\endgroup$ – Matthew Gunn Sep 22 '16 at 14:27
7
$\begingroup$

@Ferdi already provided clear answer to your question, but let's make it a little bit more formal.

Let $X_1,\dots,X_n$ be your sample of independent and identically distributed random variables from distribution $F$. You are interested in estimating unknown but fixed quantity $\theta$, using estimator $g$ being a function of $X_1,\dots,X_n$. Since $g$ is a function of random variables, estimate

$$ \hat\theta_n = g(X_1,\dots,X_n)$$

is also a random variable. We define bias as

$$ \mathrm{bias}(\hat\theta_n) = \mathbb{E}_\theta(\hat\theta_n) - \theta $$

estimator is unbiased when $\mathbb{E}_\theta(\hat\theta_n) = \theta$.

Saying it in plain English: we are dealing with random variables, so unless it's degenerate, if we took different samples, we could expect to observe different data and so different estimates. Nonetheless, we could expect that across different samples "on average" estimated $\hat\theta_n$ would be "right" if the estimator is unbiased. So it would not be always right, but "on average" it would be right. It simply cannot always be "right" because of randomness associated with the data.

As others already noted, the fact that your estimate gets "closer" to estimated quantity as your sample grows, i.e. that in converges in probability

$$ \hat\theta_n \overset{P}{\to} \theta $$

has to do with estimators consistency, not unbiasedness. Unbiasedness alone does not tell us anything about sample size and its relation to obtained estimates. Moreover, unbiased estimators are not always available and not always preferable over biased ones. For example, after considering bias-variance tradeoff you may be willing to consider using estimator with greater bias, but smaller variance -- so "on average" it would be farther from the true value, but more often (smaller variance) the estimates would be closer to the true value, then in case of unbiased estimator.

$\endgroup$
  • $\begingroup$ (+1): very good point of bringing in the fact that there are rarely unbiased estimators available. And mentioning the bias/variance opposition. $\endgroup$ – Xi'an Sep 22 '16 at 18:49
2
$\begingroup$

First you must distinguish misunderstanding bias from statistical bias, especially for a lay person.

The choice of say using median, mean or mode as your estimator for a population average, often contains a political, religious or science theory belief bias. The computation as to which estimator is the best form of average is of a different type to the arithmetic that affects statistical bias.

Once you have got past the method selection bias, then you can address the potential biases in the estimation method. First you have to pick a method that can have a bias, and a mechanism that leads easily toward that bias.

It can be easier to use a divide a conquer viewpoint where it becomes obvious as the sample size gets smaller, the estimate becomes clearly biased. For example the n-1 factor (vs 'n' factor) in sample spread estimators becomes obvious as n drops from 3 to 2 to 1 !

It all depends on how 'lay' the person is.

$\endgroup$
  • $\begingroup$ I'm afraid that you may be talking about different kind of biases that the one in the question. Could you try being more specific about what bias is? You write about "potential biases in the estimation method" and this does not seems to correspond to the definition of bias (given in the question and answers above). In the end, this makes your answer confusing... $\endgroup$ – Tim Sep 23 '16 at 14:14
  • $\begingroup$ @Tim, the first step was to just ensure that human biases had been covered. The second step was (and partially follows the issues of step 1) to make sure that the lay person's teaching wasn't already that method X (the unbiased one) was to be chosen. e.g. The standard deviation is 1/n*sum((x-mean)^2), but that (carefully) doesn't distinguish between population and sample. Most 'lay people' are taught the unthinking 1/(N-1) version for a sample. If you have just one method, you (the lay person) has no choice to make, so estimator bias can't be an issue... It's the Kruger-Dunning step. $\endgroup$ – Philip Oakley Sep 24 '16 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.