I have a dataset of n = 55 samples with 300,000 - 600,000 features for each sample. I am trying to train a model (ksvm classification) to predict the class of each sample. N = 26 of the samples are class 1, N = 29 are class 2.

Due to the large number of features, I planned to pre-filter the features to include only features that are significantly different between the two classes. When I have done this in practice, the number of features can be reduced to 1,000 - 4,000.

My plan for training the model is to partition the dataset 1/4 for validation, 3/4 for training/testing, and use the 3/4 training set to perform a 10-fold cross validation to evaluate model performance. By 10-fold CV I am referring to the definition of using 9/10 of the training data to train and then assess performance on 1/10 of the training data, and repeating this process 10 times, using each slice (1/10) of the training data once as the test set for assessing performance during each iteration.

The part I am unsure about is whether I need to perform the pre-filtering/feature selection for each fold of the 10-fold CV. My intuition is that it is necessary to pre-filter significant features for each unique 9/10 slice of the training data for each iteration of cross validation. Is this correct?

It seems like it will be very labor intensive to do it this way because most of the R packages I have seen for performing k-fold CV seem to operate on a single dataset as opposed to 10 unique datasets for each fold of CV.

One possibility is to perform pre-filtering 10-times, once for each set of 9/10 training data, and then combine each of the pre-filtered datasets into a single dataset for 10-fold CV, but I am not sure if this is a valid method.

Alternatively does anyone know of any R packages that allow you to supply a new dataset/featureset for each fold of cross validation?

Please let me know if any of this is unclear, I am still learning the statistical jargon and it is quite possible I've misused some vocabulary. Thanks in advance.

  • 2
    $\begingroup$ Not a full answer, but your intuition is correct. You should not pre-select features on the whole dataset, since you will be learning (in part) which features are discriminative in the test data. This will give you a biased (overly optimistic) estimation of how your feature selection + classification method performs. The correct way to do it is as you suggest, selecting a new feature subset for each training fold. $\endgroup$ – Nuclear Wang Mar 24 '17 at 18:36
  • $\begingroup$ Thanks for your help @Matt, do you happen to know of any workflows/packages that would help with handling the unique feature sets during cross validation? $\endgroup$ – Reilstein Mar 24 '17 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.