19
$\begingroup$

The issue has come up before, but I want to ask a specific question that will attempt to elicit an answer that will clarify (and classify) it:

In "Poor Man's Asymptotics", one keeps a clear distinction between

  • (a) a sequence of random variables that converges in probability to a constant

as contrasted to

  • (b) a sequence of random variables that converges in probability to a random variable (and hence in distribution to it).

But in "Wise Man's Asymptotics", we can also have the case of

  • (c) a sequence of random variables that converges in probability to a constant while maintaining a non-zero variance at the limit.

My question is (stealing from my own exploratory answer below):

How can we understand an estimator that is asymptotically consistent but also has a non-zero, finite variance? What does this variance reflects? How its behavior differs from a "usual" consistent estimator?

Threads related to the phenomenon described in (c) (look also in the comments):

$\endgroup$
2
  • 1
    $\begingroup$ The way you capitalize "Poor Man's Asymptotics" makes me think I must be missing knowledge of a reference (or possibly have seen it but forgotten it, which amounts to much the same thing); either an actual book or paper, or possibly even just a cultural reference. I know of "Poor Man's Data Augmentation" (Tanner and Wei), but I don't think this is connected to what you're getting at. What am I missing? $\endgroup$
    – Glen_b
    Oct 18, 2014 at 23:52
  • $\begingroup$ @Glen_B You don't miss anything - I just made the term up to contrast the level of knowledge of (=intellectual access to) Asymptotic Theory that people like me have, against, say, that of people like cardinal. Capitalization was just a marketing tactic. $\endgroup$ Oct 18, 2014 at 23:59

3 Answers 3

10
+200
$\begingroup$

I won't give a very satisfactory answer to your question because it seems to me to be a little bit too open, but let me try to shed some light on why this question is a hard one.

I think you are struggling with the fact that the conventional topologies we use on probability distributions and random variables are bad. I've written a bigger piece about this on my blog but let me try to summarize: you can converge in the weak (and the total-variation) sense while violating commonsensical assumptions about what convergence means.

For example, you can converge in weak topology towards a constant while having variance = 1 (which is exactly what your $Z_n$ sequence is doing). There is then a limit distribution (in the weak topology) that is this monstruous random variable which is most of the time equal to 0 but infinitesimally rarely equal to infinity.

I personally take this to mean that the weak topology (and the total-variation topology too) is a poor notion of convergence that should be discarded. Most of the convergences we actually use are stronger than that. However, I don't really know what should we use instead of the weak topology sooo ...

If you really want to find an essential difference between $\hat \theta= \bar X+Z_n$ and $\tilde \theta=\bar X$, here is my take: both estimators are equivalent for the [0,1]-loss (when the size of your mistake doesn't matter). However, $\tilde \theta $ is much better if the size of your mistakes matter, because $\hat \theta$ sometimes fails catastrophically.

$\endgroup$
0
9
$\begingroup$

27-10-2014: Unfortunately (for me that is), no-one has as yet contributed an answer here -perhaps because it looks like a weird, "pathological" theoretical issue and nothing more?

Well to quote a comment for user Cardinal (which I will subsequently explore)

"Here is an admittedly absurd, but simple example. The idea is to illustrate exactly what can go wrong and why. It does have practical applications (my emphasis). Example: Consider the typical i.i.d. model with finite second moment. Let $\hat θ_n=\bar X_n+Z_n$ where $Z_n$ is independent of $\bar X_n$ and $Z_n=\pm an$ each with probability $1/n^2$ and is zero otherwise, with $a>0$ arbitrary. Then $\hat θ_n$ is unbiased, has variance bounded below by $a^2$, and $\hat θ_n→\mu$ almost surely (it's strongly consistent). I leave as an exercise the case regarding the bias".

The maverick random variable here is $Z_n$, so let's see what we can say about it.
The variable has support $\{-an,0,an\}$ with corresponding probabilities $\{1/n^2,1-2/n^2,1/n^2\}$. It is symmetric around zero, so we have

$$E(Z_n) = 0,\;\; \text{Var}(Z_n) = \frac {(-an)^2}{n^2} + 0 + \frac {(an)^2}{n^2} = 2a^2$$

These moments do not depend on $n$ so I guess we are allowed to trivially write

$$\lim_{n\rightarrow \infty} E(Z_n) = 0,\;\;\lim_{n\rightarrow \infty}\text{Var}(Z_n) = 2a^2$$

In Poor Man's Asymptotics, we know of a condition for the limits of moments to equal the moments of the limiting distribution. If the $r$-th moment of the finite case distribution converges to a constant (as is our case), then, if moreover,

$$\exists \delta >0 :\lim \sup E(|Z_n|^{r+\delta}) < \infty $$

the limit of the $r$-th moment will be the $r$-th moment of the limiting distribution. In our case

$$E(|Z_n|^{r+\delta}) = \frac {|-an|^{r+\delta}}{n^2} + 0 + \frac {|an|^{r+\delta}}{n^2} = 2a^{r+\delta}\cdot n^{r+\delta-2}$$

For $r\geq2$ this diverges for any $\delta >0$, so this sufficient condition does not hold for the variance (it does hold for the mean).
Take the other way: What is the asymptotic distribution of $Z_n$? Does the CDF of $Z_n$ converge to a non-degenerate CDF at the limit?

It doesn't look like it does: the limiting support will be $\{-\infty, 0, \infty\}$ (if we are permitted to write this), and the corresponding probabilities $\{0,1,0\}$. Looks like a constant to me.
But if we don't have a limiting distribution in the first place, how can we talk about its moments?

Then, going back to the estimator $\hat \theta_n$, since $\bar X_n$ also converges to a constant, it appears that

$\hat \theta_n$ does not have a (non-trivial) limiting distribution, but it does have a variance at the limit. Or, maybe this variance is infinite? But an infinite variance with a constant distribution?

How can we understand this? What does it tell us about the estimator? What is the essential difference, at the limit, between $\hat \theta_n = \bar X_n + Z_n$ and $\tilde \theta_n = \bar X_n$?

$\endgroup$
5
  • $\begingroup$ Stupid reference request: do you have a (good) source for: "if the r-th moment converges to a constant, then all the moments with index lower than r converge to the moments of the limiting distribution ?". I know it's true, but I never found a good source $\endgroup$ Oct 15, 2015 at 12:18
  • $\begingroup$ Secondly, the theorem you try to use can't be applied in this case: for r=2 (which is the case you want to use: you want to prove that the variance converges), a for any strictly positive $\delta$, the $E(|Z_n|^{r+\delta}$ diverge ! $\endgroup$ Oct 15, 2015 at 12:20
  • $\begingroup$ Perhaps it would be good to somehow ping @cardinal (in chat?) so that he joins this discussion. $\endgroup$
    – amoeba
    Oct 15, 2015 at 14:46
  • $\begingroup$ @amoeba Cardinal is an estimator that converges a.s. to the true answer here, but I remember trying to engage him in the past without success. $\endgroup$ Oct 15, 2015 at 14:59
  • 1
    $\begingroup$ @GuillaumeDehaene A reference is A.W. Van der Vaart (1998) "Asymptotic Statistics", ch. 2.5 "Convergence of Moments". It is given as an example 2.21 of Theorem 2.20. And you are right: I was under the impression that it was enough to have boundedness for finite $n$ - but it is the limsup that must be finite. I am correcting my post. $\endgroup$ Oct 15, 2015 at 18:19
4
$\begingroup$

An estimator is consistent in probability but not in MSE if there is an arbitrarily small probability of the estimator "exploding". While an interesting mathematical curiosity, for any practical purpose this should not bother you. For any practical purpose, estimators have finite supports and thus cannot explode (the real world is not infinitesimally small, nor large).

If you still wish to call upon a continuous approximation of the "real world", and your approximation is such that is converges in probability and not in MSE, then take it as it is: Your estimator can be right with arbitrarily large probability, but there will always be an arbitrarily small chance of it exploding. Luckily, when it does, you will notice, so that otherwise, you can trust it. :-)

$\endgroup$
3
  • $\begingroup$ It is my impression that $\hat \theta = \bar X + Z_n$ does converge in mean square, since $$\lim E(\hat \theta^2) = 2a^2$$ $\endgroup$ Oct 15, 2015 at 18:54
  • $\begingroup$ The question specifically deals with the interpretation of an estimator that converges in probability and not in MSE (due to a non vanishing variance). $\endgroup$
    – JohnRos
    Oct 16, 2015 at 5:56
  • $\begingroup$ You're right, I just confused a plus sign with a minus one. $\endgroup$ Oct 17, 2015 at 2:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.