I know Lasso eventually set some parameters to zero, acting like variable selection. I also read from paper talking about automated variable selection method like Stepwise AIC can be troublesome. So what are the advantages of using Lasso for variable selection over using the automated procedures like Stepwise AIC?
$\begingroup$
$\endgroup$
2
-
2$\begingroup$ ESL and Introduction to Statistical Learning both summarize regularization - and are freely available. $\endgroup$– charlesCommented Aug 22, 2014 at 23:11
-
2$\begingroup$ A lot has been written about why stepwise regression is a poor approach. If you believe that (and you should), then lasso would be "better" simply because it is less bad. $\endgroup$– shadowtalkerCommented Aug 23, 2014 at 19:04
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
Selection of variables on the basis of the magnitude of their observed regression coefficients, or on the basis of their statistical "significance" (and AIC is a 1-1 function of the P-value) will result in selection of variables because their effects were measured with error in the direction of being too far from zero. The lasso as well as other penalized methods penalize you for the context of how hard you have to work to find good predictors. Penalization lowers the chance of overstating a regression coefficient.
answered Aug 23, 2014 at 20:12
-
1$\begingroup$ This makes sense if you are interested in the individual coefficients. What if you only care about prediction? $\endgroup$– DaveCommented Nov 10, 2022 at 16:41
$\begingroup$
$\endgroup$
1
I am not 100% sure that this is right.
Stepwise AIC is designed to obtained a sparse estimator that works well on the training set (if you compute the MSE part of the AIC error on the training set)
The lasso where the penalty is chosen with cross validation is design to obtain the sparse linear model that minimize the prediction error
-
$\begingroup$ There is a cost in terms of predictive discrimination due to an attempt to be parsimonious as with lasso. Stepwise AIC is worse than that as it forgot to penalize. But variable selection is most problematic when attempting to explain something. $\endgroup$ Commented Nov 10, 2022 at 21:37