Superiority of LASSO over forward selection/backward elimination in terms of the cross validation prediction error of the model

Question

I obtained three reduced models from a original full model using

• forward selection
• backward elimination
• L1 penalization technique (LASSO)

For the models obtained using forward selection/backward elimination, I obtained the cross validated estimate of prediction error using CVlm in package DAAG available in R. For the model selected via LASSO, I used cv.glm.

The prediction error for LASSO was less than than the ones obtained for the others. So the model obtained via LASSO seems to be better in terms of its predictive capacity and variability. Is this a general phenomenon that always occurs or is it problem specific? What is the theoretical reasoning for this if this is a general phenomenon?

Answer 1

The LASSO and forward/backward model selection both have strengths and limitations. No far sweeping recommendation can be made. Simulation can always be explored to address this.

Both can be understood in the sense of dimensionality: referring to $p$ the number of model parameters and $n$ the number of observations. If you were able to fit models using backwards model selection, you probably didn't have $p \gg n$. In that case, the "best fitting" model is the one using all parameters... when validated internally! This is simply a matter of overfitting.

Overfitting is remedied using split sample cross validation (CV) for model evaluation. Since you didn't describe this, I assume you didn't do it. Unlike stepwise model selection, LASSO uses a tuning parameter to penalize the number of parameters in the model. You can fix the tuning parameter, or use a complicated iterative process to choose this value. By default, LASSO does the latter. This is done with CV so as to minimize the MSE of prediction. I am not aware of any implementation of stepwise model selection that uses such sophisticated techniques, even the BIC as a criterion would suffer from internal validation bias. By my account, that automatically gives LASSO leverage over "out-of-the-box" stepwise model selection.

Lastly, stepwise model selection can have different criteria for including/excluding different regressors. If you use the p-values for the specific model parameters' Wald test or the resultant model R^2, you will not do well, mostly because of internal validation bias (again, could be remedied with CV). I find it surprising that this is still the way such models tend to be implemented. AIC or BIC are much better criteria for model selection.

There are a number of problems with each method. Stepwise model selection's problems are much better understood, and far worse than those of LASSO. The main problem I see with your question is that you are using feature selection tools to evaluate prediction. They are distinct tasks. LASSO is better for feature selection or sparse model selection. Ridge regression may give better prediction since it uses all variables.

LASSO's great strength is that it can estimate models in which $p \gg n$, as can be the case forward (but not backward) stepwise regression. In both cases, these models can be effective for prediction only when there is a handful of very powerful predictors. If an outcome is better predicted by many weak predictors, then ridge regression or bagging/boosting will outperform both forward stepwise regression and LASSO by a long shot. LASSO is much faster than forward stepwise regression.

There is obviously a great deal of overlap between feature selection and prediction, but I never tell you about how well a wrench serves as a hammer. In general, for prediction with a sparse number of model coefficients and $p \gg n$, I would prefer LASSO over forward stepwise model selection.

Answer 2

You want to choose a subset of predictors according to some criteria. Might be in-sample AIC or adjusted R^2, or cross-validation, doesn't matter.

You could test every single predictor subset combination and pick the best subset. However

• Very time-consuming due to combinatorial explosion of parameters.
• Works if you have more parameters than observations in the sense that you test all predictor combinations that give a solution

You could use forward stepwise selection

• Less time-consuming, but may not get absolute best combination, esp. when predictors are correlated (may pick one predictor and be unable to get further improvement when adding 2 other predictors would have shown improvement)
• Works even when you have more parameters than observations

You could use backward elimination

• Doesn't work if you have more parameters than observations, no single good starting point (in theory you could start from all valid starting points, work backwards, pick best one, but that's not what is normally meant by backwards elimination)
• Like forward stepwise, less time-consuming than all subsets, but may not get absolute best combination, esp. when predictors are correlated

You could use LASSO

• Works even when you have more parameters than observations
• CPU-efficient when you have many parameters and combinatorial explosion of subsets
• Adds regularization

As to your question of why LASSO performs better on your data in CV

• One possibility is the path-dependency described above - LASSO may find a better subset. Perhaps it got lucky, perhaps LASSO generally/sometimes gets better subsets, I'm not sure. Perhaps there is literature on the subject.
• Another (more likely) possibility is the LASSO regularization prevents overfitting, so LASSO performs better in CV/out of sample.

Bottom line, LASSO gives you regularization and efficient subset selection, especially when you have a lot of predictors.

BTW you can do LASSO and select your model using CV (most common) but also using AIC or some other criterion. Run your model with L1 regularization and no constraint, then gradually tighten the constraint until AIC reaches a minimum, or CV error, or the criterion of your choice. See http://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_model_selection.html