I need to perform feature selection over a high dimensional dataset, this is p >> N, and later perform a classification using a 2-class response. Which of this 2 options is correct? Assuming I initially split my data into training an test sets:
- Perform FS over the training data and with the same training data with the subset of candidate selected features, train a classifier doing regular CV?
- Perform nested CV, this is, do CV over training data, in each fold perform FS and later train the classification model with the current fold data using the subset of candidate features, also with an inner CV loop.