When a variable of interest has many plausible explanatory variables, and one lakes strong theoretical or subject-matter grounds for selecting among them, it is tempting to build a “kitchen sink” model with a large number of variables, and then use some form of penalization to eliminate the surplus variables. This would seem to be the classic case for LASSO penalties with the tuning parameter chosen by one of the forms of cross-validation selection on data outside the training set on which the equation is estimated. I believe LASSO was originally put forward for more or less this purpose.
However, I understand that the experience of the portion of the statistical, machine learning and data science communities that focuses primarily on pragmatic experience of making forecasts is that LASSO eliminates too many variables, and not always the right ones. As a result, ridge regression generally leads to better forecasts.
But sometimes it seems obvious, for one reason or another, that you have too many variables. Variables may be suggested by mutually incompatible theories, for instance, or be strongly co-linear. Further, even if one is primarily concerned with the quality of forecasts, one may still put some weight on other concerns, such as interpretability, or future data-collection costs, or true causality if one expects to do forecasting over substantially changing conditions.
My question is this: starting from a kitchen sink regression model, I'd like to throw away as many variables as I can, subject to the constraint that I am (probably) not removing any that are important to predictive accuracy. Of the many variable selection procedures proposed, is there one, or are there some, that are best adopted to this purpose? I would include among the valid responses pairs of algorithms, where the first does variable selection and the second does shrinkage on the restricted set.
