Asymptotic Normality and Consistency

Question

I have difficulties understanding the concept of asymptotic normality and consistency.

Take an estimator of a parameter which is consistent and asymptotically normally distributed. Because it is consistent, it converges to the true parameter. I don't understand how it then is still asymptotically distributed normally when it converges to a constant?

Thank you very much in advance!!

Answer 1

Convergence to a constant does not mean that you have an estimator that is exactly equal to this constant at any given time. It just means that given a big enough sample size, you can expect that you estimator will be close to the true value of the parameter.

Asymptotic normality most often includes some sort of a scaling (often by $\sqrt{n}$, where $n$ is the sample size). It means that we are not looking at the estimator per se, but at a scaled version of it which does not converge to a constant.

You can look at it from this way (this explanation is not very precise): without such scaling, when you sample size is big enough, the distribution of the estimator can be approximated by a normal distribution with variance which is decreasing with the sample size. The bigger the sample size, the smaller the variance $\Rightarrow$ in the limit, the variance vanishes completely, hence, convergence to a constant is achieved.

Answer 2

I think you may either be confused with the modes of convergence associated with consistency and asymptotic normality or with the definition of consistency. It is important to note that consistency is defined as convergence to the true value of the parameter of interest. Regardless, let us begin with the definitions,

We say that an estimator, $\theta_n$, is consistent if it converges in probability to the true value, $\theta_0$, of the parameter which can be denoted by,

$$\theta_n \overset{p}{\to} \theta_0 \iff \lim_{n\to\infty}\Pr[||\theta_n -\theta||>\epsilon]=0$$

We say that an estimator, $\theta_n$, is asymptotically normal if it converges in distribution (or in law, or weakly) to a normally distributed random variable. That is,

$$\sqrt{n}(\theta_n - \theta_0)\overset{d}{\to} N(0,\sigma^2)$$

There are several definitions of convergence in distribution depending on the level of abstraction you want to consider, for now, let us use the most commonly used definition:

We say that a sequence of random variables $Y_1, Y_2,\dots$ converges in distribution if $F_{Y_n}(y)\to F_Y(y)$ for all continuity points $y$ of the cdf $F_{Y}$.

To answer your question you need to understand the differences between these modes of convergence. The first thing to note is that convergence in the distribution is weaker than convergence in probability. In fact, convergence in probability will imply convergence in distribution. That is,

$$|Y_n - X_n|\overset{p}{\to}0, X_n \overset{d}{\to}X \implies Y_n \overset{d}{\to} X$$

This takes some work to show for purposes of keeping this answer succinct here is the proof on Wikipedia.

Intuitively, this makes sense since if your sequence of random variables converges to another sequence of random variables you would expect that they both share the same limiting distribution. This is not generally true in reverse.

However, and quite importantly, if a random variable converges in distribution to a constant then it also converges in probability to that constant. That is,

$$X_n \overset{d}{\to}c\implies X_n \overset{p}{\to}c$$

Here is a proof of this fact.

Important bits: Notice that we have only been talking about convergence in probability and convergence in the distribution not consistency or asymptotic normality! This is very important because it actually turns out that,

$$\text{Asymptotic Normality} \implies \text{Consistency}$$

But not vice-versa! Well, this is counter-intuitive given what we just saw but it actually makes sense given the definitions of the two concepts. The main idea is that consistency does not say that our estimator converges to the true estimator just to the true value of a parameter, as you say, a constant. Thus, consistency only takes a stand on where the distribution concentrates not the whole distribution. However, this is usually not enough to do inference because we want to know not just where our estimator concentrates but also some sense of the precision with which it concentrates (i.e. we want to get some approximation of the estimator so we can determine standard errors). This can be done with convergence in distribution.

Asymptotic normality says that our scaled and differenced estimator converges in distribution to a random variable. This means that we can do the following computation,

$$X_n - X = O_p(\frac{1}{\sqrt{n}})=o_p(1) \text{ as } n\to\infty$$

The first equality follows because by the definition of convergence in distribution,

$$\Pr[||X_n-X||> M]\to\Pr[||Z||>M]$$

where $Z=N(0,\sigma^2)$ then by Markov's inequality,

$$\Pr[||Z||>M]\leq \frac{\mathbb{E}[||Z||]}{M}$$

So we can just choose $M$ sufficiently large such that $\Pr[||X_n-X||\geq M] < \epsilon$.

Thus, we can even think of consistency as a necessary condition for asymptotic normality. Of course, the reverse is not true. For a counterexample take a consistent and asymptotically normal estimator $\theta_n$ and define $\tilde\theta_n = \theta_n + n^{-1/3}$. Clearly $\tilde\theta_n \to \theta_0$ and is thus consistent but $\sqrt{n}(\tilde\theta_n -\theta_0)$ will not converge.

The idea here is that consistency is a weaker notion and refers only to the value of our estimator as $n\to\infty$. Thus, it does not affect our ability to talk about the asymptotic normality of our estimator which is really talking about the limiting behavior of our estimator beyond the value it concentrates around.