# Variable selection without strong theory: Can we do better than LASSO for prediction?

When a variable of interest has many plausible explanatory variables, and one lakes strong theoretical or subject-matter grounds for selecting among them, it is tempting to build a “kitchen sink” model with a large number of variables, and then use some form of penalization to eliminate the surplus variables. This would seem to be the classic case for LASSO penalties with the tuning parameter chosen by one of the forms of cross-validation selection on data outside the training set on which the equation is estimated. I believe LASSO was originally put forward for more or less this purpose.

However, I understand that the experience of the portion of the statistical, machine learning and data science communities that focuses primarily on pragmatic experience of making forecasts is that LASSO eliminates too many variables, and not always the right ones. As a result, ridge regression generally leads to better forecasts.

But sometimes it seems obvious, for one reason or another, that you have too many variables. Variables may be suggested by mutually incompatible theories, for instance, or be strongly co-linear. Further, even if one is primarily concerned with the quality of forecasts, one may still put some weight on other concerns, such as interpretability, or future data-collection costs, or true causality if one expects to do forecasting over substantially changing conditions.

My question is this: starting from a kitchen sink regression model, I'd like to throw away as many variables as I can, subject to the constraint that I am (probably) not removing any that are important to predictive accuracy. Of the many variable selection procedures proposed, is there one, or are there some, that are best adopted to this purpose? I would include among the valid responses pairs of algorithms, where the first does variable selection and the second does shrinkage on the restricted set.

• From more recent advances you may want to take a look at fast best subset selection. Also, consider Adaptive LASSO, Relaxed LASSO, Sqrt-LASSO or Lq (Bridge, q<1) regularization as possible LASSO alternatives.
– runr
Jun 4 '20 at 10:11
• Thanks, @Nutile! do you have a sense of whether any of these methods are better or worse at tossing variables that are not needed for predictive accuracy and keeping those that are? Jun 4 '20 at 14:54
• All of them should improve the lasso somewhat. Adaptive LASSO has very good theoretical properties, under approximate sparseness and well chosen weights should have very good variable selection, and works well with correlated data, large $p$ and even garch errors. Best subset selection, in theory, should be the best of them all, though I'm not yet sure about the "fast" implementation, as it imposes certain heuristic assumptions to make the estimation time feasible.
– runr
Jun 4 '20 at 22:20
• Relaxed Lasso essentially does what you edited in the last paragraph, ideally you could combine Adaptive/Sqrt/L0 LASSO with Relaxed LASSO for optimal shrinkage/variable selection. But in any case, best way to see what works with your data best is by running a "horce race"
– runr
Jun 4 '20 at 22:24

In many cases the problem is not about removing variables, but about regularizing the model, although some recent results show that maybe some common beliefs about regularization that we held are not exactly true. LASSO is just a one, of many, approaches to regularization and variable selection. LASSO is using $$L_1$$ penalty, but you could use $$L_2$$ penalty, or both as in elastic net regularization, there are model-agnostic algorithms like Boruta, FSelector, or genetic algorithms, there is a ton of ways for regularizing models in deep learning like dropout, early stopping, weight decay, batch normalization, etc. Neither of those is "best", if it was the case we would stop searching for new ones and just stick to the best one.