1
$\begingroup$

When a variable of interest has many plausible explanatory variables, and one lakes strong theoretical or subject-matter grounds for selecting among them, it is tempting to build a “kitchen sink” model with a large number of variables, and then use some form of penalization to eliminate the surplus variables. This would seem to be the classic case for LASSO penalties with the tuning parameter chosen by one of the forms of cross-validation selection on data outside the training set on which the equation is estimated. I believe LASSO was originally put forward for more or less this purpose.

However, I understand that the experience of the portion of the statistical, machine learning and data science communities that focuses primarily on pragmatic experience of making forecasts is that LASSO eliminates too many variables, and not always the right ones. As a result, ridge regression generally leads to better forecasts.

But sometimes it seems obvious, for one reason or another, that you have too many variables. Variables may be suggested by mutually incompatible theories, for instance, or be strongly co-linear. Further, even if one is primarily concerned with the quality of forecasts, one may still put some weight on other concerns, such as interpretability, or future data-collection costs, or true causality if one expects to do forecasting over substantially changing conditions.

My question is this: starting from a kitchen sink regression model, I'd like to throw away as many variables as I can, subject to the constraint that I am (probably) not removing any that are important to predictive accuracy. Of the many variable selection procedures proposed, is there one, or are there some, that are best adopted to this purpose? I would include among the valid responses pairs of algorithms, where the first does variable selection and the second does shrinkage on the restricted set.

| cite | improve this question | |
$\endgroup$
  • 1
    $\begingroup$ From more recent advances you may want to take a look at fast best subset selection. Also, consider Adaptive LASSO, Relaxed LASSO, Sqrt-LASSO or Lq (Bridge, q<1) regularization as possible LASSO alternatives. $\endgroup$ – Nutle Jun 4 at 10:11
  • $\begingroup$ Thanks, @Nutile! do you have a sense of whether any of these methods are better or worse at tossing variables that are not needed for predictive accuracy and keeping those that are? $\endgroup$ – andrewH Jun 4 at 14:54
  • $\begingroup$ All of them should improve the lasso somewhat. Adaptive LASSO has very good theoretical properties, under approximate sparseness and well chosen weights should have very good variable selection, and works well with correlated data, large $p$ and even garch errors. Best subset selection, in theory, should be the best of them all, though I'm not yet sure about the "fast" implementation, as it imposes certain heuristic assumptions to make the estimation time feasible. $\endgroup$ – Nutle Jun 4 at 22:20
  • $\begingroup$ Relaxed Lasso essentially does what you edited in the last paragraph, ideally you could combine Adaptive/Sqrt/L0 LASSO with Relaxed LASSO for optimal shrinkage/variable selection. But in any case, best way to see what works with your data best is by running a "horce race" $\endgroup$ – Nutle Jun 4 at 22:24
3
$\begingroup$

Removing the variables one-by-one, based on some criteria like AIC, is called stepwise selection and is one of the worst algorithms for variable selection.

In many cases the problem is not about removing variables, but about regularizing the model, although some recent results show that maybe some common beliefs about regularization that we held are not exactly true. LASSO is just a one, of many, approaches to regularization and variable selection. LASSO is using $L_1$ penalty, but you could use $L_2$ penalty, or both as in elastic net regularization, there are model-agnostic algorithms like Boruta, FSelector, or genetic algorithms, there is a ton of ways for regularizing models in deep learning like dropout, early stopping, weight decay, batch normalization, etc. Neither of those is "best", if it was the case we would stop searching for new ones and just stick to the best one.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your answer, @Tim! I have (somewhat unfairly) redrafted the last paragraph of my question in an effort to narrow it enough to reap a more specific response. In particular, I am only looking at regression, and I want my selection algorithm to keep the variables important to predictive accuracy; and I'm looking for the method or methods best for that purpose. $\endgroup$ – andrewH Jun 4 at 14:51
  • $\begingroup$ @andrewH the conclusion of this answer still holds: there is no "one size fits all" solution. You can go with LASSO and if it doesn't work, try something else. If you have no theoretical grounds on selecting variables, you are left with the machine learning "try it and see what happens" approach. $\endgroup$ – Tim♦ Jun 4 at 14:59
  • 1
    $\begingroup$ @andrewH the elastic net referred to in this answer (+1) pretty much does what you ask in the last paragraph of the present version of your question, combining L1 regularization for variable selection with L2 penalization. But as this answer and Tim's comment say, there is no one size fits all solution. And if your main interest is in prediction, think about why you need to do any variable selection at all. $\endgroup$ – EdM Jun 4 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.