I am trying to know the right way of doing feature selection. I am kind of mixed up with feature selection and cross-validation. I am not even sure if I should perform feature selection inside cross-validation or if I should do feature selection first and run cross-validation for model building with selected features.

What is the right to do feature selection when building a model?


1 Answer 1


Always perform feature selection inside the cross-validation loop on the training data only. The alternative, selecting features first and then splitting into folds, will lead to biased results. If you do that, you are using both the training and the test data to find features that are associated with your target variable - you are using the test data to find features that will work well in the test data, at least in part.

Your test data should never be used in the model training process, it should only be used to test models learned on the training data only. Selecting features on the whole data first integrates all of your potential test data into the model building process, which entirely defeats the purpose of having a test set in the first place.

It's a bit more work, since you need to run feature selection K times instead of just once (once for each of the K folds), but it is the only unbiased way to do it.

  • $\begingroup$ Could you tell me the details about how feature selection is dome inside cross-validation? $\endgroup$
    – oceanus
    Jan 14, 2020 at 17:38
  • 1
    $\begingroup$ @ChangheeKang It's done the same way you'd do it outside the cross-validation, except you only use the samples in each training fold as your input data, rather than the entire sample set. There are many, many dimensionality reduction methods, which could be looking for the variables with the strongest association with a target, the variables that explain the most variance, variables that are uncorrelated with one another, or combined variables that aggregate multiple raw features. Feature selection is a huge topic, but the link to cross validation is that you only run it on the training data. $\endgroup$ Jan 14, 2020 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.