I would like to predict discrete (mean) values from discrete (mean & std) values of the extracted features. My question is how you should perform feature selection when you are applying K-fold cross-validation in multiple linear regression? Let's say you have 30 features and you should end up with only 3 features. My solution was that I computed all the possible combinations of subsets and then for each subset of features I applied cross-validation and multiple linear regression (train and test the model). I got some results for some feature subsets but I am not sure if these are reliable.

  • 3
    $\begingroup$ Feature selection is often a bad idea, and the selections are not well supported by data. I.e. if you repeat the selection you'll find great instability in the features "selected". Feature selection often hurts predictive discrimination (as opposed to e.g. ridge regression). So provide some reasons why variable selection is desired. $\endgroup$ Aug 13 '19 at 11:35
  • $\begingroup$ Regarding @FrankHarrell's comment: there is indeed a problem with choosing only 3 features arbitrarily, since more could be just as important. I'd suggest best-subset selection, where you'd have to try all possible combinations from using only 1 feature to all 30. That'll be the optimal way, but most likely computationally not feasible. In that case, forward or backward selection may be appropriate. You can do that with CV. Also, use test-RMSE, not r-squared as a measure of goodness since r-squared always increases with the number of features, regardless if they're actually good or not. $\endgroup$
    – PaulG
    Dec 15 '20 at 12:17

Your methodology is correct. Just to clarify, you have a dataset that you split in 3: insample-train, insample-test, and outsample.

You train model parameters (betas for your multiple linear reg) on the insample-train (that's when you train on your k-fold). You train hyperparameters (which 3 feature you end up choosing) on the in-sample-test. You evaluate your final model on the outsample, and use that number as your best guess at go-forward predictive error.

If you open up the outsample more than once, you'd have to discount your final go-forward error.

  • $\begingroup$ For each subset of features I calculate the r-squared and root mean square error, so I open the outsample as you referred to. Could you elaborate more on the last sentence? $\endgroup$
    – owblique
    Feb 14 '16 at 10:24
  • $\begingroup$ You should open the "insample-test", not the outsample. The outsample is only opened last, as a best guess of how your final chosen model will perform with new data. $\endgroup$
    – djma
    Feb 16 '16 at 4:31
  • $\begingroup$ Keep in mind that this is in inefficient procedure, as opposed to for example a double bootstrap. $\endgroup$ Dec 15 '20 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.