I would appreciate help in understanding the following theorem from Knight and Fu (2002) paper:

Consider linear regression model of the form $$Y_i = \beta_0 + x_i'\beta + \varepsilon_i,$$ where $\varepsilon_i \sim i.i.d.(0, \sigma^2)$. We estimate the model using the lasso type estimator $\hat\beta$ $$\hat\beta = \text{arg min}_\phi \sum_{i=1}^{n}(Y_i - x_i'\phi)^2 + \lambda_n\sum_{j=1}^p\vert\phi_j\vert .$$

Further assume that $C_n = n^{-1}\sum_{i=1}^nx_ix_i' \to C$, where $C$ is nonsingular. Let $\lambda_n/n \to \lambda_0 \geq 0$. Then $\hat\beta \xrightarrow{p} \text{arg min}(Z)$ where $$Z(\phi) = (\phi - \beta)'C(\phi - \beta) + \lambda_0\sum_{j=1}^p\vert \phi_j\vert. $$

Thus if $\lambda_n = o(n)$, $argmin(Z) = \beta$ and so $\hat\beta$ is consistent.

Could you explain how does consistency follow from the fact that $\hat\beta \xrightarrow{p} \text{arg min}(Z)$. And also do we talk about consistency in parameter estimation or other types of consistency (e.g. oracle properties)?

  • 1
    $\begingroup$ It’s simply that if it’s $o(n)$, $\lambda_0=0$ and so asymptotically you have just the quadratic form to minimize. Intuitively, it means that you have a penalized estimator for which the penalization is loosened with $n$. $\endgroup$ – hejseb May 26 '18 at 18:39
  • $\begingroup$ @hejseb thank you, this is helpful. If you could also explain how $\sum_{i=1}^n(Y_i - x_i'\phi)^2$ converges to $(\phi - \beta)'C(\phi - \beta)$ and post everything as an answer, I would accept it. $\endgroup$ – tosik May 27 '18 at 9:19
  • $\begingroup$ If you write the quadratic as $(Y-X\phi)'(Y-X\phi)$ you get (using $Y=X\beta+\epsilon$) that it's $(\beta-\phi)'X'X(\beta-\phi)$ plus terms which disappear if $\epsilon$ and $X$ are independent. Divide by $n$ and then using $X'X/n\to C$ you get it. I'll write up an answer in a few days. $\endgroup$ – hejseb May 29 '18 at 6:24

You'll be disappointed to find that the consistency that matters the most with lasso is the consistency about which predictors are chosen. If you simulate two moderately large datasets and perform lasso independently and compare the results, the low degree of overlap will reveal the difficulty of the task in selecting features. This is even more true when co-linearities are present. lasso spends too much of its energy on feature selection intead of estimation, and the L1 norm results in too much shrinkage of truly important predictors (hence the popularity of the horseshoe prior in Bayesian high-dimensional modeling). I wouldn't be too interested in the type of consistency you described above until these more fundamental issues are addressed. I discuss these issues in general, and show how the bootstrap can help uncover them, here.

  • 1
    $\begingroup$ Thank you, I am aware of things that you discuss, but I am actually interested in the consistency described in my question. I need it for my thesis where these results are used. Hence, I would appreciate the help with understanding this type of consistency. $\endgroup$ – tosik May 26 '18 at 12:11
  • 1
    $\begingroup$ I get that, but you are running the risk of skipping over the most important part of the analysis. Users of lasso results constantly claim to learn from the list of features selected, and this can be a grave mistake. $\endgroup$ – Frank Harrell May 26 '18 at 12:27
  • 3
    $\begingroup$ I mean this with no disrespect, but the question is a pretty well-explained question about a specific result in a relatively famous research paper. I don’t see the relevance of this answer. $\endgroup$ – hejseb May 26 '18 at 18:38
  • $\begingroup$ You're completely correct that I didn't answer the original question. I was answering a question that should have preceded that question. Glad at least I got some votes :-) $\endgroup$ – Frank Harrell May 26 '18 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.