I would like to predict discrete (mean) values from discrete (mean & std) values of the extracted features. My question is how you should perform feature selection when you are applying K-fold cross-validation in multiple linear regression? Let's say you have 30 features and you should end up with only 3 features. My solution was that I computed all the possible combinations of subsets and then for each subset of features I applied cross-validation and multiple linear regression (train and test the model). I got some results for some feature subsets but I am not sure if these are reliable.

  • 1
    $\begingroup$ Feature selection is often a bad idea, and the selections are not well supported by data. I.e. if you repeat the selection you'll find great instability in the features "selected". Feature selection often hurts predictive discrimination (as opposed to e.g. ridge regression). So provide some reasons why variable selection is desired. $\endgroup$ – Frank Harrell Aug 13 '19 at 11:35

Your methodology is correct. Just to clarify, you have a dataset that you split in 3: insample-train, insample-test, and outsample.

You train model parameters (betas for your multiple linear reg) on the insample-train (that's when you train on your k-fold). You train hyperparameters (which 3 feature you end up choosing) on the in-sample-test. You evaluate your final model on the outsample, and use that number as your best guess at go-forward predictive error.

If you open up the outsample more than once, you'd have to discount your final go-forward error.

| cite | improve this answer | |
  • $\begingroup$ For each subset of features I calculate the r-squared and root mean square error, so I open the outsample as you referred to. Could you elaborate more on the last sentence? $\endgroup$ – owblique Feb 14 '16 at 10:24
  • $\begingroup$ You should open the "insample-test", not the outsample. The outsample is only opened last, as a best guess of how your final chosen model will perform with new data. $\endgroup$ – djma Feb 16 '16 at 4:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.