0
$\begingroup$

This was part of a coding exercise where we had around 10 features, and were asked to find the best pair of features for an OLS model. Best defined as having lowest RMSE. I'm looking to understand if there is an intuition or a definite method for this. I eventually had to fit all of the pairs and then compare the RMSE, given I just had 10 features it didn't take too long to compute but made me think how to approach this problem if we had 100s or 1000s of features.

Improve this question
$\endgroup$
1
  • $\begingroup$ You are right - best subset selection by exhaustive search is typically impractical as the number of subsets grows combinatorially with the number of features. $\endgroup$
    – Don Walpola
    Commented Feb 25, 2020 at 2:52

1 Answer 1

Reset to default
1
$\begingroup$

The technique you used is called Best Subset Selection.

I would say that the most popular technique to reach the same objective is LASSO.

See these techniques and other friends here.

You may also select features considering the importance of the features for out of sample prediction. In this context, I suggest two very interesting and general methods that you can use:

1) Permutation importance: The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled

Solution in R: Permutation importance in R

2) Shap values: It is not an easy concept since it is based in game theory, but it shows the importance of each feature.

Solution in R: Shap values in R

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thank you for the quick response, how would these methods deal with collinearity? If I'm not mistaken LASSO would just try to minimize the loss function, so would the others? $\endgroup$
    – user23564
    Commented Feb 24, 2020 at 20:20
  • 1
    $\begingroup$ LASSO, Ridge and Elastic-Net has no issues with collinearity. In fact, one of the main objectives of the creation of ridge regression was to deal with ill-conditioned regression. The other methods of stepwise regression will probably include only one of the variables that are multicolinear. $\endgroup$
    – DanielTheRocketMan
    Commented Feb 24, 2020 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.