In order to test a potential classification set, usually some data is kept as a holdout set, and not used for inner-cross-validation or model training.

However, what happens if too many features exist, and it would be impossible to do iterative cross-validated feature selection on all of them.

Is it acceptable (and scientific) to first reduce the overall number of features using Filter Based Feature Selection (e.g. Pearson Correlation)?

Is it also acceptable to further reduce the number of features / dimensions using Fisher Linear Discriminant Analysis (LDA)?

As for acceptable and scientific, I mean, if cross-validated feature selection is still done on the remaining features, then would information still be leaked into the test set causing a classification performance bias?


You can reduce the features by performing filter feature selection methods on the training set(s). If you reduce the features prior to splitting you will have data leakage even if you cross validate afterwards.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.