5
$\begingroup$

In a highly cited paper by Zhao (2006) it is stated that (Section 2)

An estimate which is consistent in terms of parameter estimation does not necessarily consistently selects the correct model (or even attempt to do so) where the reverse is also true. The former requires $$\hat\beta^n - \beta^n \xrightarrow{p}0, \quad \text{as} \quad n\to \infty $$ while the latter requires $$\Pr\left( \{i\colon \hat\beta_i^n \neq 0 \} = \{i\colon \beta_i^n \neq 0 \} \right) \to 1 \quad \text{as} \quad n\to \infty. $$

I do not understand how it could be possible that an estimator is consistent in parameter estimation but selects a wrong model? If asymptotically estimates converge to the true values then they also should converge to zeros whenever the true value is zero, hence selecting the right model.


EDIT: One more question related to this one is "If an estimator $\hat\beta^n$ has oracle properties does this imply that $\hat\beta^n - \beta^n \to 0$? That is, do oracle properties imply consistency in parameter estimation?". For clarity I list oracle properties below.

Define $\mathcal{A} = \{ j\colon \beta_j \neq 0 \}$ and further assume that $\vert \mathcal{A}\vert < p$ where $p$ is a number of coefficients in vector $\beta$. We say that fitting procedure $\delta$ that produces estimate $\hat\beta(\delta)$ is an oracle procedure if $\hat\beta(\delta)$ satisfies following oracle properties:

  • Identifies the right subset model, $\{ j\colon \hat\beta_j \neq 0 \} = \mathcal{A}$
  • Has optimal estimation rate, $\sqrt{n}\left( \hat\beta(\delta)_{\mathcal{A}} - \beta_{\mathcal{A}} \right) \xrightarrow{d} N(0, \Sigma)$, where $\Sigma$ is the covariance matrix of the true subset model.
$\endgroup$
5
  • 1
    $\begingroup$ Trying to consider the practical implications of the issue, $n$ just doesn't go to $\infty$. And I've yet to see a procedure that has a truly high probability of selecting the "right" features. Feature selection is overused IMHO. $\endgroup$ Jun 7, 2018 at 11:24
  • 1
    $\begingroup$ To answer your second question, yes, by definition of the oracle property. If consistency in parameter estimation isn't defined to be part of the oracle property, then no. Note the "where the reverse is also true" part of the block quote above, also the second answer to stats.stackexchange.com/questions/229142/…. $\endgroup$
    – jbowman
    Jun 7, 2018 at 16:17
  • $\begingroup$ @jbowman I see your point, but in the definition above the second oracle property states that an estimator is consistent in parameter estimation given that the true model is known (see subscript $\mathcal{A}$). And I believe this is not the same as general parameter estimation consistency (without conditioning on true model knowledge). $\endgroup$
    – tosik
    Jun 7, 2018 at 18:48
  • $\begingroup$ Is that what that means? I thought it just meant the subset of the parameters that were nonzero had the optimal estimation rate given the procedure. It doesn't mean much if you have to know the model to get the optimal estimation rate, since you don't. $\endgroup$
    – jbowman
    Jun 7, 2018 at 18:52
  • $\begingroup$ Your condition on the second question "given that the true model is known" is not correct (as I suspected.) The optimal estimation rate should hold for the parameters in $\mathcal{A}$ for the actual procedure, not just for one that makes use of knowledge of $\mathcal{A}$ (e.g., the MLE given the true model.) $\endgroup$
    – jbowman
    Jun 8, 2018 at 22:52

1 Answer 1

2
$\begingroup$

With respect to your first question - first an example, then the logic.

Consider a model of the form $y_i = a + bx_i + e_i$, but with parameter estimation performed by a linear regression that includes an extra variable: $y_i = a + bx_i + cz_i + \epsilon_i$. As is well-known, the estimated parameters $\hat{a}, \hat{b}, $and $\hat{c}$ are consistent; with respect to $\hat{c}$ in particular, $\hat{c}_n \xrightarrow{p} 0$ as $n \rightarrow \infty$.

Now, what is $\Pr(\hat{c}_n =0)$? It's $0$ for all $n$ (assuming continuous data). Therefore, as is well-known, linear regression with too many regressors is consistent in parameter estimation but always selects a wrong model, because the probability that it selects any model other than the one passed to it is zero.

At this point the logic should be clear. Consistency means that the probability that the estimate is any given distance from zero $\to 0$ as the sample size $\to \infty$, but it says nothing about whether the estimate is ever exactly equal to zero. Model selection is done on the basis of whether the parameter estimate is exactly equal to zero, so consistency has nothing to say about whether the model selection itself is consistent.

With respect to your second question - to see that the converse need not be true either, consider an estimator for $c$ of the form:

$\text{Set }\hat{c}_n = 0 \text{ with probability } {n-1\over n}, \space n \text{ with probability }{1 \over n}$

Clearly the expected value of $\hat{c}_n = 1 \space \forall \space n$, so $\hat{c}$ is not consistent. On the other hand, $\Pr(\hat{c}_n = 0) \to 1$ as $n \to \infty$, so the model selection procedure is consistent.

Once again, the logic rests on the fact that model selection is done on the basis of the parameter estimate being equal to zero, but consistency in estimation is based on the parameter estimate just being more likely to be close to zero as the sample size gets larger, writing loosely.

Edit: It occurred to me on rereading the question that you may be confusing the limit of the sequence of function values $\lim f(x_n)$ with the value of the function at the limit: $f(\lim x_n)$. Often these are the same, but not always. Consider the (relevant but simple) example $f(x_n) = \text{I}(x_n=0)$, the indicator function which takes on the value 1 if its argument is "true" and 0 otherwise, combined with the sequence $x_n = 1/n$. Since for all $n$, $f(x_n) = 0$, $\lim f(x_n) = 0$ as well (the limit of the sequence $0, 0, 0, ...$ is $0$.) However, since the limit of $x_n = 0$, and $\text{I}(0=0) = 1$, $f(\lim x_n) = 1$. That's basically what's happening in the case of the linear regression example above, just with probabilities added on top.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.