Parameter estimation and model selection consistencies

Question

In a highly cited paper by Zhao (2006) it is stated that (Section 2)

An estimate which is consistent in terms of parameter estimation does not necessarily consistently selects the correct model (or even attempt to do so) where the reverse is also true. The former requires $$\hat\beta^n - \beta^n \xrightarrow{p}0, \quad \text{as} \quad n\to \infty $$ while the latter requires $$\Pr\left( \{i\colon \hat\beta_i^n \neq 0 \} = \{i\colon \beta_i^n \neq 0 \} \right) \to 1 \quad \text{as} \quad n\to \infty. $$

I do not understand how it could be possible that an estimator is consistent in parameter estimation but selects a wrong model? If asymptotically estimates converge to the true values then they also should converge to zeros whenever the true value is zero, hence selecting the right model.

---

EDIT: One more question related to this one is "If an estimator $\hat\beta^n$ has oracle properties does this imply that $\hat\beta^n - \beta^n \to 0$? That is, do oracle properties imply consistency in parameter estimation?". For clarity I list oracle properties below.

Define $\mathcal{A} = \{ j\colon \beta_j \neq 0 \}$ and further assume that $\vert \mathcal{A}\vert < p$ where $p$ is a number of coefficients in vector $\beta$. We say that fitting procedure $\delta$ that produces estimate $\hat\beta(\delta)$ is an oracle procedure if $\hat\beta(\delta)$ satisfies following oracle properties:

• Identifies the right subset model, $\{ j\colon \hat\beta_j \neq 0 \} = \mathcal{A}$
• Has optimal estimation rate, $\sqrt{n}\left( \hat\beta(\delta)_{\mathcal{A}} - \beta_{\mathcal{A}} \right) \xrightarrow{d} N(0, \Sigma)$, where $\Sigma$ is the covariance matrix of the true subset model.

Answer 1

With respect to your first question - first an example, then the logic.

Consider a model of the form $y_i = a + bx_i + e_i$, but with parameter estimation performed by a linear regression that includes an extra variable: $y_i = a + bx_i + cz_i + \epsilon_i$. As is well-known, the estimated parameters $\hat{a}, \hat{b}, $and $\hat{c}$ are consistent; with respect to $\hat{c}$ in particular, $\hat{c}_n \xrightarrow{p} 0$ as $n \rightarrow \infty$.

Now, what is $\Pr(\hat{c}_n =0)$? It's $0$ for all $n$ (assuming continuous data). Therefore, as is well-known, linear regression with too many regressors is consistent in parameter estimation but always selects a wrong model, because the probability that it selects any model other than the one passed to it is zero.

At this point the logic should be clear. Consistency means that the probability that the estimate is any given distance from zero $\to 0$ as the sample size $\to \infty$, but it says nothing about whether the estimate is ever exactly equal to zero. Model selection is done on the basis of whether the parameter estimate is exactly equal to zero, so consistency has nothing to say about whether the model selection itself is consistent.

With respect to your second question - to see that the converse need not be true either, consider an estimator for $c$ of the form:

$\text{Set }\hat{c}_n = 0 \text{ with probability } {n-1\over n}, \space n \text{ with probability }{1 \over n}$

Clearly the expected value of $\hat{c}_n = 1 \space \forall \space n$, so $\hat{c}$ is not consistent. On the other hand, $\Pr(\hat{c}_n = 0) \to 1$ as $n \to \infty$, so the model selection procedure is consistent.

Once again, the logic rests on the fact that model selection is done on the basis of the parameter estimate being equal to zero, but consistency in estimation is based on the parameter estimate just being more likely to be close to zero as the sample size gets larger, writing loosely.

Edit: It occurred to me on rereading the question that you may be confusing the limit of the sequence of function values $\lim f(x_n)$ with the value of the function at the limit: $f(\lim x_n)$. Often these are the same, but not always. Consider the (relevant but simple) example $f(x_n) = \text{I}(x_n=0)$, the indicator function which takes on the value 1 if its argument is "true" and 0 otherwise, combined with the sequence $x_n = 1/n$. Since for all $n$, $f(x_n) = 0$, $\lim f(x_n) = 0$ as well (the limit of the sequence $0, 0, 0, ...$ is $0$.) However, since the limit of $x_n = 0$, and $\text{I}(0=0) = 1$, $f(\lim x_n) = 1$. That's basically what's happening in the case of the linear regression example above, just with probabilities added on top.