Skip to main content
Cross Validated

Return to Question

formatting+MathjaxMagic
Source Link
edit approved Aug 28, 2017 at 15:16
Mooncrater
edit approved Aug 28, 2017 at 15:16
Mooncrater
  • 797
  • 2
  • 9
  • 20

One of the motivations for the elastic net was the following limitation of LASSO:

In the $p > n$ case, the lasso selects at most n variables before it saturates, because of the nature of the convex optimization problem. This seems to be a limiting feature for a variable selection method. Moreover, the lasso is not well defined unless the bound on the L1-norm of the coefficients is smaller than a certain value.

"In the p > n case, the lasso selects at most n variables before it saturates, because of the nature of the convex optimization problem. This seems to be a limiting feature for a variable selection method. Moreover, the lasso is not well defined unless the bound on the L1-norm of the coefficients is smaller than a certain value." (http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2005.00503.x/full)

I understand that LASSO is a quadratic programming problem but also can be solved via LARS or element-wise gradient descent. But I do not understand where in these algorithms I encounter a problem if p > n$p > n$ where p$p$ is the number of predictors and n$n$ is the sample size. And why is this problem solved using elastic net where I augment the problem to p+n$p+n$ variables which clearly exceeds p$p$.

One of the motivations for the elastic net was the following limitation of LASSO:

"In the p > n case, the lasso selects at most n variables before it saturates, because of the nature of the convex optimization problem. This seems to be a limiting feature for a variable selection method. Moreover, the lasso is not well defined unless the bound on the L1-norm of the coefficients is smaller than a certain value." (http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2005.00503.x/full)

I understand that LASSO is a quadratic programming problem but also can be solved via LARS or element-wise gradient descent. But I do not understand where in these algorithms I encounter a problem if p > n where p is the number of predictors and n is the sample size. And why is this problem solved using elastic net where I augment the problem to p+n variables which clearly exceeds p.

One of the motivations for the elastic net was the following limitation of LASSO:

In the $p > n$ case, the lasso selects at most n variables before it saturates, because of the nature of the convex optimization problem. This seems to be a limiting feature for a variable selection method. Moreover, the lasso is not well defined unless the bound on the L1-norm of the coefficients is smaller than a certain value.

(http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2005.00503.x/full)

I understand that LASSO is a quadratic programming problem but also can be solved via LARS or element-wise gradient descent. But I do not understand where in these algorithms I encounter a problem if $p > n$ where $p$ is the number of predictors and $n$ is the sample size. And why is this problem solved using elastic net where I augment the problem to $p+n$ variables which clearly exceeds $p$.

Tweeted twitter.com/#!/StackStats/status/252286932950863873
Source Link
user1137731
user1137731
  • 505
  • 2
  • 4
  • 13

If p > n, the lasso selects at most n variables

One of the motivations for the elastic net was the following limitation of LASSO:

"In the p > n case, the lasso selects at most n variables before it saturates, because of the nature of the convex optimization problem. This seems to be a limiting feature for a variable selection method. Moreover, the lasso is not well defined unless the bound on the L1-norm of the coefficients is smaller than a certain value." (http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2005.00503.x/full)

I understand that LASSO is a quadratic programming problem but also can be solved via LARS or element-wise gradient descent. But I do not understand where in these algorithms I encounter a problem if p > n where p is the number of predictors and n is the sample size. And why is this problem solved using elastic net where I augment the problem to p+n variables which clearly exceeds p.