I obtained three reduced models from a original full model using

  • forward selection
  • backward elimination
  • L1 penalization technique (LASSO)

For the models obtained using forward selection/backward elimination, I obtained the cross validated estimate of prediction error using CVlm in package DAAG available in R. For the model selected via LASSO, I used cv.glm.

The prediction error for LASSO was less than than the ones obtained for the others. So the model obtained via LASSO seems to be better in terms of its predictive capacity and variability. Is this a general phenomenon that always occurs or is it problem specific? What is the theoretical reasoning for this if this is a general phenomenon?

  • 4
    $\begingroup$ Make sure that you do not use an improper accuracy scoring rule such as proportion classified correct, as this rewards inappropriate prediction/models. And compare with L2 norm. I'll bet it will be better then the 3 approaches you tried. $\endgroup$ Mar 7, 2014 at 22:30

2 Answers 2


The LASSO and forward/backward model selection both have strengths and limitations. No far sweeping recommendation can be made. Simulation can always be explored to address this.

Both can be understood in the sense of dimensionality: referring to $p$ the number of model parameters and $n$ the number of observations. If you were able to fit models using backwards model selection, you probably didn't have $p \gg n$. In that case, the "best fitting" model is the one using all parameters... when validated internally! This is simply a matter of overfitting.

Overfitting is remedied using split sample cross validation (CV) for model evaluation. Since you didn't describe this, I assume you didn't do it. Unlike stepwise model selection, LASSO uses a tuning parameter to penalize the number of parameters in the model. You can fix the tuning parameter, or use a complicated iterative process to choose this value. By default, LASSO does the latter. This is done with CV so as to minimize the MSE of prediction. I am not aware of any implementation of stepwise model selection that uses such sophisticated techniques, even the BIC as a criterion would suffer from internal validation bias. By my account, that automatically gives LASSO leverage over "out-of-the-box" stepwise model selection.

Lastly, stepwise model selection can have different criteria for including/excluding different regressors. If you use the p-values for the specific model parameters' Wald test or the resultant model R^2, you will not do well, mostly because of internal validation bias (again, could be remedied with CV). I find it surprising that this is still the way such models tend to be implemented. AIC or BIC are much better criteria for model selection.

There are a number of problems with each method. Stepwise model selection's problems are much better understood, and far worse than those of LASSO. The main problem I see with your question is that you are using feature selection tools to evaluate prediction. They are distinct tasks. LASSO is better for feature selection or sparse model selection. Ridge regression may give better prediction since it uses all variables.

LASSO's great strength is that it can estimate models in which $p \gg n$, as can be the case forward (but not backward) stepwise regression. In both cases, these models can be effective for prediction only when there is a handful of very powerful predictors. If an outcome is better predicted by many weak predictors, then ridge regression or bagging/boosting will outperform both forward stepwise regression and LASSO by a long shot. LASSO is much faster than forward stepwise regression.

There is obviously a great deal of overlap between feature selection and prediction, but I never tell you about how well a wrench serves as a hammer. In general, for prediction with a sparse number of model coefficients and $p \gg n$, I would prefer LASSO over forward stepwise model selection.

  • $\begingroup$ What do you mean by validation bias and do you have recommended readings to become more familiar with this issue? $\endgroup$ Apr 21, 2021 at 16:51
  • 1
    $\begingroup$ @grisaitis internal validation bias is the bias that arises from using the same data to validate the model as was used to generate the model. While the various information criteria penalize the number of parameters in the model to try to get at this, I think the community consensus is that it's no free lunch. I.e. AIC and BIC don't work in every scenario all the time. This is discussed in Elements of Statistical Learning in the introduction to cross validation. $\endgroup$
    – AdamO
    Apr 21, 2021 at 17:00

You want to choose a subset of predictors according to some criteria. Might be in-sample AIC or adjusted R^2, or cross-validation, doesn't matter.

You could test every single predictor subset combination and pick the best subset. However

  • Very time-consuming due to combinatorial explosion of parameters.
  • Works if you have more parameters than observations in the sense that you test all predictor combinations that give a solution

You could use forward stepwise selection

  • Less time-consuming, but may not get absolute best combination, esp. when predictors are correlated (may pick one predictor and be unable to get further improvement when adding 2 other predictors would have shown improvement)
  • Works even when you have more parameters than observations

You could use backward elimination

  • Doesn't work if you have more parameters than observations, no single good starting point (in theory you could start from all valid starting points, work backwards, pick best one, but that's not what is normally meant by backwards elimination)
  • Like forward stepwise, less time-consuming than all subsets, but may not get absolute best combination, esp. when predictors are correlated

You could use LASSO

  • Works even when you have more parameters than observations
  • CPU-efficient when you have many parameters and combinatorial explosion of subsets
  • Adds regularization

As to your question of why LASSO performs better on your data in CV

  • One possibility is the path-dependency described above - LASSO may find a better subset. Perhaps it got lucky, perhaps LASSO generally/sometimes gets better subsets, I'm not sure. Perhaps there is literature on the subject.
  • Another (more likely) possibility is the LASSO regularization prevents overfitting, so LASSO performs better in CV/out of sample.

Bottom line, LASSO gives you regularization and efficient subset selection, especially when you have a lot of predictors.

BTW you can do LASSO and select your model using CV (most common) but also using AIC or some other criterion. Run your model with L1 regularization and no constraint, then gradually tighten the constraint until AIC reaches a minimum, or CV error, or the criterion of your choice. See http://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_model_selection.html


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.