From the seminal paper on elastic net regularization from Zou and Hastie 2005, I read

For this kind of
p>>n and grouped variables situation, the lasso
is not the ideal method, because it can only select at most
n variables out of p candidates (Efron et al., 2004).

However, in Efron et al., 2004 I can not fin the proof/demonstration? Any hint?


Consider a linear model $Y = X\beta + \varepsilon$ with $p$ variables and $n$ observations, $p>n$. Assuming the variables are not linear dependent, i.e. the matrix $X$ has rank $n$, $Y$ can be perfectly predictet ($Y = \hat{Y}$) using only $n$ variables. So LASSO will ideally choose the $n$ variables such that $\lambda ||\beta||_1$ is minimal. This solution should be unique (because of linear independence on the individual variable level) and threrefore, all perfect fits of the linear model that include more than $n$ variables will have a higher $\lambda ||\beta||_1$ and are therefore not optimal.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.