I would like to predict discrete (mean) values from discrete (mean & std) values of the extracted features. My question is how you should perform feature selection when you are applying K-fold cross-validation in multiple linear regression? Let's say you have 30 features and you should end up with only 3 features. My solution was that I computed all the possible combinations of subsets and then for each subset of features I applied cross-validation and multiple linear regression (train and test the model). I got some results for some feature subsets but I am not sure if these are reliable.

  • 3
    $\begingroup$ Feature selection is often a bad idea, and the selections are not well supported by data. I.e. if you repeat the selection you'll find great instability in the features "selected". Feature selection often hurts predictive discrimination (as opposed to e.g. ridge regression). So provide some reasons why variable selection is desired. $\endgroup$ Commented Aug 13, 2019 at 11:35
  • $\begingroup$ Regarding @FrankHarrell's comment: there is indeed a problem with choosing only 3 features arbitrarily, since more could be just as important. I'd suggest best-subset selection, where you'd have to try all possible combinations from using only 1 feature to all 30. That'll be the optimal way, but most likely computationally not feasible. In that case, forward or backward selection may be appropriate. You can do that with CV. Also, use test-RMSE, not r-squared as a measure of goodness since r-squared always increases with the number of features, regardless if they're actually good or not. $\endgroup$
    – PaulG
    Commented Dec 15, 2020 at 12:17
  • $\begingroup$ Best subset selection does not solve the selection volatility problem. $\endgroup$ Commented Jan 5, 2022 at 13:44

1 Answer 1


Your methodology is correct. Just to clarify, you have a dataset that you split in 3: insample-train, insample-test, and outsample.

You train model parameters (betas for your multiple linear reg) on the insample-train (that's when you train on your k-fold). You train hyperparameters (which 3 feature you end up choosing) on the in-sample-test. You evaluate your final model on the outsample, and use that number as your best guess at go-forward predictive error.

If you open up the outsample more than once, you'd have to discount your final go-forward error.

  • $\begingroup$ For each subset of features I calculate the r-squared and root mean square error, so I open the outsample as you referred to. Could you elaborate more on the last sentence? $\endgroup$
    – owblique
    Commented Feb 14, 2016 at 10:24
  • $\begingroup$ You should open the "insample-test", not the outsample. The outsample is only opened last, as a best guess of how your final chosen model will perform with new data. $\endgroup$
    – djma
    Commented Feb 16, 2016 at 4:31
  • $\begingroup$ Keep in mind that this is in inefficient procedure, as opposed to for example a double bootstrap. $\endgroup$ Commented Dec 15, 2020 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.