Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 11 down vote favorite
6

I'm really surprised that nobody appears to have asked this already...

When discussing estimators, two terms frequently used are "consistent" and "unbiased". My question is simple: what's the difference?

The precise technical definitions of these terms are fairly complicated, and it's difficult to get an intuitive feel for what they mean. I can imagine a good estimator, and a bad estimator. But I'm having trouble seeing how any estimator could satisfy one condition and not the other.

share|improve this question
3  
Have you looked at the very first figure in the Wikipedia article on consistent estimators, which specifically explains this distinction? – whuber♦ Jun 24 '12 at 16:45
I've read the articles for both consistency and bias, but I still don't really understand the distinction. (The figure you refer to claims that the estimator is consistent but biased, but doesn't explain why.) – MathematicalOrchid Jun 24 '12 at 16:47
1  
Which part of the explanation do you need help with? The caption points out that each of the estimators in the sequence is biased and it also explains why the sequence is consistent. Do you need an explanation of how the bias in these estimators is apparent from the figure? – whuber♦ Jun 24 '12 at 16:50
1  
+1 The comment thread following one of these answers is very illuminating, both for what it reveals about the subject matter and as an interesting example of how an online community can work to expose and rectify misconceptions. – whuber♦ Jan 12 at 17:51

3 Answers

up vote 13 down vote accepted

To define the two terms without using too much technical language:

  • An estimator is consistent if, as the sample size increases, the estimator "converges" to the true value of the parameter being estimated. To be slightly more precise - consistency means that, as the sample size increases, the sampling distribution of the estimator becomes increasingly concentrated at the true parameter value.

  • An estimator is unbiased if, on average, it hits the true parameter value. That is, the mean of the sampling distribution of the estimator is equal to the true parameter value.

  • The two are not equivalent: Unbiasedness is a statement about the expected value of the sampling distribution of the estimator. Consistency is a statement about "where the sampling distribution of the estimator is going" as the sample size increases.

It certainly is possible for one condition to be satisfied but not the other - I will give two examples. For both examples consider a sample $X_1, ..., X_n$ from a $N(\mu, \sigma^2)$ population.

  • Unbiased but not consistent: Suppose you're estimating $\mu$. Then $X_1$ is an unbiased estimator of $\mu$ since $E(X_1) = \mu$. But, $X_1$ is not consistent since its distribution does not become more concentrated around $\mu$ as the sample size increases - it's always $N(\mu, \sigma^2)$!

  • Consistent but not unbiased: Suppose you're estimating $\sigma^2$. The maximum likelihood estimator is $$ \hat{\sigma}^2 = \frac{1}{n} \sum_{i=1}^{n} (X_i - \overline{X})^2 $$ where $\overline{X}$ is the sample mean. It is a fact that $$ E(\hat{\sigma}^2) = \frac{n-1}{n} \sigma^2 $$ herefore, $\hat{\sigma}^2$ which can be derived using the information here. Therefore $\hat{\sigma}^2$ is biased for any finite sample size. We can also easily derive that $${\rm var}(\hat{\sigma}^2) = \frac{ 2\sigma^4(n-1)}{n^2}$$ From these facts we can informally see that the distribution of $\hat{\sigma}^2$ is becoming more and more concentrated at $\sigma^2$ as the sample size increases since the mean is converging to $\sigma^2$ and the variance is converging to $0$. (Note: This does constitute a proof of consistency, using the same argument as the one used in the answer here)

share|improve this answer
5  
(+1) Not all MLEs are consistent though: the general result is that there exists a consistent subsequence in the sequence of MLEs. For proper consistency a few additional requirements, e.g. identifiability, are needed. Examples of MLEs that aren't consistent are found in certain errors-in-variables models (where the "maximum" turns out to be a saddle-point). – MånsT Jun 25 '12 at 6:42
1  
Well, the EIV MLEs that I mentioned are perhaps not good examples, since the likelihood function is unbounded and no maximum exists. They're good examples of how the ML approach can fail though :) I'm sorry that I can't give a relevant link right now - I'm on vacation. – MånsT Jun 25 '12 at 6:59
Thank you @MånsT. The necessary conditions were outlined in the link but that wasn't clear from the wording. – Macro Jun 25 '12 at 11:12
1  
Just a side note: The parameter space is certainly not compact in this case, in contrast to the conditions at that link, nor is the log likelihood concave wrt $\sigma^2$ itself. The stated consistency result still holds, of course. – cardinal Jun 25 '12 at 12:43
1  
You're right, @cardinal, I'll delete that reference. It's clear enough that $E(\hat{\sigma}^2) \rightarrow \sigma^2$ and ${\rm var}(\hat{\sigma}^2) \rightarrow 0$ but I don't want to stray from the point by turning this into an exercise of proving the consistency of $\hat{\sigma}^2$. – Macro Jun 25 '12 at 12:54
up vote 5 down vote

Consistency of an estimator means that as the sample size gets large the estimate gets closer and closer to the true value of the parameter. Unbiasedness is a finite sample property that is not affected by increasing sample size. An estimate is unbiased if its expected value equals the true parameter value. This will be true for all sample sizes and is exact whereas consistency is asymptotic and only is approximately equal and not exact. To say that an estimator is unbiased means that if you took many samples of size n and computed the estimate each time the average of all these estimates would be close to the true parameter value and will get closer as the number times you do this increases. The sample mean is both consistent and unbiased. The sample estimate of standard deviation is biased but consistent. As described below there are apparently pathological cases where the variance does not have to go to 0 for the estimator to be strongly consistent and the bias doesn't even have to go to 0 either.

share|improve this answer
6  
@MichaelChernick +1 for your answer but, regarding your comment, the variance of a consistent estimator does not necessarily goes to $0$. For example if $(X_1,...,X_n)$ is a sample from $\mbox{Normal}(\mu,1)$, $\mu\neq 0$, then $1/{\bar X}$ is a (strong) consistent estimator of $1/\mu$, but $\mbox{var}(1/{\bar X})=\infty$, for all $n$. – user10525 Jun 24 '12 at 20:38
3  
6  
Unfortunately, the first two sentences in your first comment and the entire second comment are false. But, I fear it is not fruitful to further try to convince you of these facts. – cardinal Jun 25 '12 at 13:48
5  
Here is an admittedly absurd, but simple example. The idea is to illustrate exactly what can go wrong and why. It does have practical applications. Example: Consider the typical iid model with finite second moment. Let $\hat\theta_n = \bar X_n + Z_n$ where $Z_n$ is independent of $\bar X_n$ and $Z_n = \pm a n$ each with probability $1/n^2$ and is zero otherwise, with $a > 0$ arbitrary. Then $\hat\theta_n$ is unbiased, has variance bounded below by $a^2$, and $\hat\theta_n \to \mu$ almost surely (it's strongly consistent). I leave as an exercise the case regarding the bias. – cardinal Jun 25 '12 at 14:48
3  
Michael, The estimator is strongly consistent for $\mu$; the second term converges to zero almost surely! Recall the whole point was you asked for an example of a consistent estimator where the bias did not vanish! I've shown it not only does not vanish, but can be made arbitrarily bad. A very small tweak produces an example such that the bias diverges to $\infty$. You could also make it oscillate at your whim. – cardinal Jun 25 '12 at 17:24
show 41 more comments
up vote 0 down vote

I may be 7 months too late here but I just came across this thread. I thought it was a great tutorial as I get this question all the time, and the exchange shows the pitfalls in intuitions lacking the background in real analysis. But I wanted to see added some explicit discussion of convergence in distribution and the use of point estimators in centering Wald confidence intervals (e.g., estimate -/+ 1.96 SEs). Here's what I say and I look forward to correction if needed: Standard asymptotic CI theory, which is the mainstay of health and social-science apps, hinges on convergence in distribution, not convergence in probability (as focal in point estimation) which guarantees nothing about Wald CI coverage: Cardinal's counterexample estimators could not be used to center valid asymptotic Wald CIs, which need an asymptotically unbiased center. But then the best CIs are not based on point estimators; hence in my opinion, CI theory is better understood as test inversion or summarization of a P-value function as that leads to better finite-sample approximate CIs than Wald intervals (e.g., score and profile-likelihood intervals). And finite samples are all we get in practice.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.