8
$\begingroup$

We know that if an estimator is an unbiased estimator of theta and if its variance tends to 0 as n tends to infinity then it is a consistent estimator for theta. But this is a sufficient and not a necessary condition. I am looking for an example of an estimator which is consistent but whose variance does not tend to 0 as n tends to infinity. Any suggestions?

$\endgroup$
  • 3
    $\begingroup$ See, for example, this comment and the related discussion. $\endgroup$ – cardinal Jan 31 '14 at 1:43
6
$\begingroup$

Glad to see that my (incorrect) answer generated two more, and turned a dead question into a lively Q&A thread. So it's time to try to offer something worthwhile, I guess).

Consider a serially correlated, covariance-stationary stochastic process $\{y_t\},\;\; t=1,...,n$, with mean $\mu$ and autocovariances $\{\gamma_j\},\;\; \gamma_j\equiv \operatorname{Cov}(y_t,y_{t-j})$. Assume that $\lim_{j\rightarrow \infty}\gamma_j= 0$ (this bounds the "strength" of autocorrelation as two realizations of the process are further and further away in time). Then we have that

$$\bar y_n = \frac 1n\sum_{t=1}^ny_t\rightarrow_{m.s} \mu,\;\; \text{as}\; n\rightarrow \infty$$

i.e. the sample mean converges in mean square to the true mean of the process, and therefore it also converges in probability: so it is a consistent estimator of $\mu$.

The variance of $\bar y_n$ can be found to be

$$\operatorname{Var}(\bar y_n) = \frac 1n \gamma_0+\frac 2n \sum_{j=1}^{n-1}\left(1-\frac {j}{n}\right)\gamma_j$$

which is easily shown to go to zero as $n$ goes to infinity.

Now, making use of Cardinal's comment let's randomize further our estimator of the mean, by considering the estimator

$$\tilde \mu_n = \bar y_n + z_n$$

where $\{z_t\}$ is an stochastic process of independent random variables which are also independent from the $y_i$'s, taking the value $at$ (parameter $a>0$ to be specified by us) with probability $1/t^2$, the value $-at$ with probability $1/t^2$, and zero otherwise. So $\{z_t\}$ has expected value and variance

$$E(z_t) = at\frac 1{t^2} -at\frac 1{t^2} + 0\cdot \left (1-\frac 2{t^2}\right)= 0,\;\;\operatorname{Var}(z_t) = 2a^2$$

The expected value and the variance of the estimator is therefore

$$E(\tilde \mu) = \mu,\;\;\operatorname{Var}(\tilde \mu) = \operatorname{Var}(\bar y_n) + 2a^2$$

Consider the probability distribution of $|z_n|$, $P\left(|z_n| \le \epsilon\right),\;\epsilon>0$: $|z_n|$ takes the value $0$ with probability $(1-2/n^2)$ and the value $an$ with probability $2/n^2$. So

$$P\left(|z_n| <\epsilon\right) \ge 1-2/n^2 = \lim_{n\rightarrow \infty}P\left(|z_n| < \epsilon\right) \ge 1 = 1$$

which means that $z_n$ converges in probability to $0$ (while its variance remains finite). Therefore

$$\operatorname{plim}\tilde \mu_n = \operatorname{plim}\bar y_n+\operatorname{plim} z_n = \mu$$

so this randomized estimator of the mean value of the $y$-stochastic process remains consistent. But its variance does not go to zero as $n$ goes to infinity, neither does it go to infinity.

Closing, why all the apparently useless elaboration with an autocorrelated stochastic process? Because Cardinal qualified his example by calling it "absurd", like "just to show that mathematically, we can have a consistent estimator with non-zero and finite variance".
I wanted to give a hint that it isn't necessarily a curiosity, at least in spirit: There are times in real life that new processes begin, man-made processes, that had to do with how we organize our lives and activities. While we usually have designed them, and can say a lot about them, still, they may be so complex that they are reasonably treated as stochastic (the illusion of complete control over such processes, or of complete a priori knowledge on their evolution, processes that may represent new ways to trade or produce, or arrange the rights-and-obligations structure between humans, is just that, an illusion). Being also new, we do not have enough accumulated realizations of them in order to do reliable statistical inference on how they will evolve. Then, ad hoc and perhaps "suboptimal" corrections are nevertheless an actual phenomenon, when for example we have a process where we strongly believe that its present depends on the past (hence the auto-correlated stochastic process), but we really don't know how as yet (hence the ad hoc randomization, while we wait for data to accumulate in order to estimate the covariances). And maybe a statistician would find a better way to deal with such kind of severe uncertainty -but many entities have to function in an uncertain environment without the benefit of such scientific services.

What follows is the initial (wrong) answer (see especially Cardinal's comment)

Estimators that converge in probability to a random variable do exist: the case of "spurious regression" comes to mind, where if we attempt to regress two independent random walks (i.e. non-stationary stochastic processes) on each other by using ordinary least squares estimation, the OLS estimator will converge to a random variable.

But a consistent estimator with non-zero variance does not exist, because consistency is defined as the convergence in probability of an estimator to a constant, which, by conception, has zero variance.

$\endgroup$
  • 1
    $\begingroup$ @cardinal Thanks for the intervention, and I will be happy to correct it. Can I have a hint on how could I start looking for a consistent estimator whose variance converges to a finite number? (The infinite/undefined variance case is a known case and should have been mentioned -but the finite non-zero case is the really interesting one). Or did I describe the property of consistency wrongly? $\endgroup$ – Alecos Papadopoulos Jan 31 '14 at 7:13
  • 1
    $\begingroup$ The example I gave in the comment linked to in my note to the OP has finite limiting variance. Consistency deals with convergence in probability, which you've correctly noted. But for the variance to go to zero, we have to control the tails (too). This is related to the relationship between $L_p$ convergence and convergence in probability. $\endgroup$ – cardinal Jan 31 '14 at 11:22
  • $\begingroup$ I put an example of convergence in probability with always positive, finite variance in my answer here as well. $\endgroup$ – ekvall Jan 31 '14 at 13:39
  • $\begingroup$ @cardinal If you no longer believe the current answer to be incorrect, perhaps you could delete your comment, or post a fresh comment to confirm that the current answer no longer is incorrect? From a reader's point of view, having an upvoted answer saying that an answer is incorrect is confusing (and forces one to start checking the edit chronologies). $\endgroup$ – Silverfish Sep 25 '15 at 17:17
  • $\begingroup$ @Silverfish Cardinal's comment indeed refers to my initial answer (the part under the grey bar near the end of the post). Exactly because this initial answer generated comments that are still present, I have left it un-deleted, below the new answer. I added something on the grey bar to help a bit with the confusion. $\endgroup$ – Alecos Papadopoulos Sep 25 '15 at 18:44
2
$\begingroup$

Take any sample from the distribution with finite expectation and infinite variance (Pareto with $\alpha\in(1,2]$ for example). Then the sample mean will converge to the expectation due to the law or large numbers (which requires only the existence of mean) and the variance will be infinite.

$\endgroup$
  • $\begingroup$ Is the variance infinite when, say, $\alpha = 1.5$? Or is it undefined in such a case? $\endgroup$ – Alecos Papadopoulos Jan 31 '14 at 7:16
  • $\begingroup$ Well it is infinite, if we look at the area beneath the curve for intepretation of the integral. $\endgroup$ – mpiktas Jan 31 '14 at 7:20
1
$\begingroup$

Let me give an example of a sequence of random variable converging to zero in probability but with infinite variance. In essence, an estimator is just a random variable so with a little abstraction, you can see that convergence in probability to a constant does not imply variance approaching zero.

Consider the random variable $\xi_n(x):=\chi_{[0,1/n]}(x)x^{-1/2}$ on $[0,1]$ where the probability measure considered is the Lebesgue measure. Clearly, $P(\xi_n(x)>0)=1/n\to0$ but $$\int\xi_n^2dP=\int_{0}^{1/n}x^{-1}dx=\log(x)\mid _{0}^{1/n}=\infty,$$ for all $n$ so its variance does not go to zero.

Now, just make up an estimator where as your sample grows you estimate the true value $\mu=0$ by a draw of $\xi_n$. Note that this estimator is not unbiased for 0, but to make it unbiased you can just set $\eta_n:=\pm\xi_n$ with equal probability 1/2 and use that as your estimator. The same argument for convergence and variance clearly holds.

Edit: If you want an example in which the variance is finite, take $$\xi_n(x):=\chi_{[0,1/n]}(x)\sqrt{n},$$ and again consider $\eta_n:=\pm\xi_n$ w.p. 1/2.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.