What is the difference between Stepwise regression and Lasso regression in terms of variable selection? Is the difference just the way in which the variables are selected or is there any significant difference. I know that Lasso is a regularization method. and stepwise is a way to select variables. But in the end Lasso can also be used for variable selection. In both the cases, the parameters are penalized. Please help me in understanding these concepts. Thanks in advance.

Also, how is AIC/BIC different from L1/L2?

  • 1
    $\begingroup$ In what way do you think that parameters are penalized in standard stepwise regression? $\endgroup$ – EdM Nov 8 '17 at 17:38
  • $\begingroup$ Given a set of candidate models for the data, the preferred model is the one with the minimum AIC value. AIC rewards goodness of fit (as assessed by the likelihood function), but it also includes a penalty that is an increasing function of the number of estimated parameters. The penalty discourages overfitting, because increasing the number of parameters in the model almost always improves the goodness of the fit. Sorce $\endgroup$ – cutepanda Nov 8 '17 at 17:40
  • 1
    $\begingroup$ Minimum AIC/BIC reduces the number of predictors selected, but it does not penalize the magnitudes of their coefficients, leading to overfitting. LASSO (L1) and ridge regression (L2) also penalize the sum of coefficient magnitudes (L1) or the sum of their squares (L2). Other pages on this site have addressed your issues, for example: stepwise regression; LASSO; AIC vs BIC. Please read those and revise this question if you have unresolved issues. $\endgroup$ – EdM Nov 8 '17 at 17:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.