I need to perform feature selection over a high dimensional dataset, this is p >> N, and later perform a classification using a 2-class response. Which of this 2 options is correct? Assuming I initially split my data into training an test sets:

  1. Perform FS over the training data and with the same training data with the subset of candidate selected features, train a classifier doing regular CV?
  2. Perform nested CV, this is, do CV over training data, in each fold perform FS and later train the classification model with the current fold data using the subset of candidate features, also with an inner CV loop.
  • $\begingroup$ Can you tell us about how large is your $p$? About how large is your $N$? That migfht matter. $\endgroup$ – kjetil b halvorsen Sep 22 '16 at 12:21
  • $\begingroup$ Yes, sorry, p ~ 250.000, it can even be more and N = 178. $\endgroup$ – mgvaldes Sep 23 '16 at 13:05
  • $\begingroup$ With that low $N$ there is no good idea to split the sample! Sample splitting is really an idea that comes from situations with very much data. You could try tenfold cross validation. Or some kind of bootstrapping. $\endgroup$ – kjetil b halvorsen Sep 23 '16 at 14:51
  • $\begingroup$ regardless of the split, what's most important of this question is the FS topic. Which way is the right way to do it, if the final goal is to create a predictive model? $\endgroup$ – mgvaldes Sep 29 '16 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.