I am trying to to get an intuitive understanding and feel for the difference and practical difference between the term consistent and asymptotically unbiased. I know their mathematical/statistical definitions, but I'm looking for something intuitive. To me, looking at their individual definitions, they almost seem to be the same thing. I realize the difference must be subtle but I just don't see it. I'm try to visualize the differences, but just can't. Can anyone help?
Intuitive understanding of the difference between consistent and asymptotically unbiased [duplicate]
$\begingroup$
$\endgroup$
3
asked May 20, 2017 at 6:19
-
1$\begingroup$ Just remember that these are frequentist and not general ideas. $\endgroup$– Frank HarrellCommented May 20, 2017 at 16:00
-
1$\begingroup$ See also this thread, stats.stackexchange.com/a/239919/28746 $\endgroup$– Alecos PapadopoulosCommented May 20, 2017 at 16:08
-
$\begingroup$ Thank you @AlecosPapadopoulos. I'm not sure how I missed that thread! $\endgroup$– StatsStudentCommented May 20, 2017 at 16:39
Add a comment
|
6 Answers
$\begingroup$
$\endgroup$
5
Asymptotic unbiasedness $\impliedby$ consistency + bounded variance
Consider an estimator $\hat{\theta}_n$ for a parameter $\theta$. Asymptotic unbiasedness means that the bias of the estimator goes to zero as $n \rightarrow \infty$, which means that the expected value of the estimator converges to the true value of the parameter. Consistency is a stronger condition than this; it requires the estimator (not just its expected value) to converge to the true value of the parameter (with convergence interpreted in various ways). Since there is generally some non-zero variance in the estimator, it will not generally be equal to (or converge to) its expected value. Assuming the variance of the estimator is bounded, consistency ensures asymptotic unbiasedness (proof), but asymptotic unbiasedness is not enough to get consistency. To put it another way, under some mild conditions, asymptotic unbiasedness is a necessary but not sufficient condition for consistency.
Asymptotic unbiasedness + vanishing variance $\implies$ consistency
If you have an asymptotically unbiased estimator, and its variance converges to zero, this is sufficient to give weak consistency. (This follows from Markov's inequality, which ensures that convergence in mean-square implies convergence in probability). Intuitively, this reflects the fact that a vanishing variance means that the sequence of random variables is converging closer and closer to the expected value, and if the expected value converges to the true parameter (as it does under asymptotic unbiasedness) then the random variable is converging to the true parameter.
-
1$\begingroup$ Your answer seems to be at odds with this one. Is the other answer wrong? Or maybe yours is? Or are the answers not in conflict? $\endgroup$ Commented Jan 30, 2021 at 18:43
-
1$\begingroup$ @RichardHardy: Thanks for pointing that out. I have made a minor edit to clarify the (mild) additional condition required for the stated implications. The pathological case in the other answer occurs because of an unbounded variance. I have now edited to add bounded variance as an assumption for the present implications. (My goal here is to give a general "intuitive" answer that avoids pathological cases.) $\endgroup$– BenCommented Jan 31, 2021 at 1:18
-
$\begingroup$ very nice post! it would be nice if you could sketch the proof for the first implication. $\endgroup$– lmaosomeCommented Feb 18, 2021 at 11:57
-
1$\begingroup$ @lmaosome -- see math.stackexchange.com/questions/239146/… for a proof. Basically just apply C.S. $\endgroup$– JamesCommented Jul 22, 2021 at 4:51
-
$\begingroup$ for asymptotic unbiasedness it suffices to have consistency + uniformly bounded $1+\epsilon$ moments for some $\epsilon>0$ (see math.stackexchange.com/questions/3909755/…) $\endgroup$ Commented Dec 4, 2023 at 1:46
$\begingroup$
$\endgroup$
1
They are related ideas, but an asymptotically unbiased estimator doesn't have to be consistent.
For example, imagine an i.i.d. sample of size $n$ ($X_1, X_2, ..., X_n$) from some distribution with mean $\mu$ and variance $\sigma^2$. As an estimator of $\mu$ consider $T = X_1 + 1/n$.
(Edit: Note the $X_1$ there, not $\bar{X}$)
The bias is $1/n$ so $T$ is asymptotically unbiased, but it is not consistent.
-
4$\begingroup$ I have come across this several times and each time I think it is wrong at first because I miss that you use X_1, rather than the sample mean, in constructing T (the Wikipedia example for "biased but consistent" uses the sample mean + 1/n, so this is similar enough to be confusing). I'm putting this note here in case others have the same thing happen to them. $\endgroup$ Commented Nov 30, 2019 at 6:27
$\begingroup$
$\endgroup$
5
I would like to clarify that consistency in general does not imply asymptotic unbiasedness. Consider an estimator for $0$ taking value $0$ with probability $(n-1)/n$ and value $n$ with probability $1/n$. It is a biased estimator since the expected value is always equal to $1$ and the bias does not disappear even if $n\to\infty$. However, it is a consistent estimator since it converges to $0$ in probability as $n\to\infty$.
Asymptotic unbiasedness does not imply consistency either as it is mentioned in other answers. For example, the periodogram is an asymptotically unbiased estimator of the spectral density, but it is not consistent.
Roughly speaking, consistency means that for large values of $n$ we are going to be close to the true value of the parameter with a high probability, i.e. estimates are going to be close to the true value of the parameter. Asymptotic unbiasedness means that for large values of $n$ on average we are going to be close to the true value of the parameter, i.e. the average of estimates is going to be close to the true value of the parameter, but not necessarily the estimates themselves.
-
1$\begingroup$ I think you mean that the probability is $(n-1)/n$, otherwise the probability of 0 is greater than 1 for all $n$. $\endgroup$– SantiagoCommented Dec 14, 2020 at 2:54
-
$\begingroup$ @Santiago You're absolutely right! Thanks! $\endgroup$– Cm7F7BbCommented Dec 14, 2020 at 3:25
-
$\begingroup$ @RichardHardy It is not asymptotically unbiased, the bias does not disappear even when $n\to\infty$. It is an example of an estimator that is consistent but not asymptotically unbiased. This comes from the fact that convergence in probability does not imply convergence in mean without additional assumptions. The first part of Ben's answer is wrong. $\endgroup$– Cm7F7BbCommented Jan 30, 2021 at 18:11
-
$\begingroup$ @Cm7F7Bb, shame on me! I read your answer too hastily and what I wrote was wrong. I will see if Ben has any comment on that so that we can reconcile our views. $\endgroup$ Commented Jan 30, 2021 at 18:53
-
$\begingroup$ You are correct that we need an additional (mild) condition for consistency to imply asymptotic variance. I have added this additional condition into my answer. $\endgroup$– BenCommented Jan 31, 2021 at 1:19
$\begingroup$
$\endgroup$
1
There are "unbiased but not consistent" estimators as well as "biased but consistent" estimators:
https://en.wikipedia.org/wiki/Consistent_estimator#Unbiased_but_not_consistent
So, they are not the same thing.
Also, there is a long discussion about this topic here:
What is the difference between a consistent estimator and an unbiased estimator?
-
2$\begingroup$ I believe this answer misses the mark as as the question is about the difference between asymptotic unbiasedness and consistency and not between biasedness and consistency $\endgroup$ Commented Mar 31, 2020 at 2:06
$\begingroup$
$\endgroup$
If the estimator is bounded then consistency implies asymptotic unbiasness by the dominated convergence theorem.
$\begingroup$
$\endgroup$
1
Asymptotic unbiased: As $n \rightarrow \infty$, bias converges to $0$.
Consistent: As $n \rightarrow \infty$, variance of the estimator converges to $0$.
-
4$\begingroup$ I have issue with this characterization of consistency. By this definition, a constant estimator, i.e. $\hat\theta = 1$, would be consistent for every parameter. $\endgroup$– knrumseyCommented Nov 30, 2019 at 6:48