I have a dataset of n = 55 samples with 300,000 - 600,000 features for each sample. I am trying to train a model (ksvm classification) to predict the class of each sample. N = 26 of the samples are class 1, N = 29 are class 2.

Due to the large number of features, I planned to pre-filter the features to include only features that are significantly different between the two classes. When I have done this in practice, the number of features can be reduced to 1,000 - 4,000.

My plan for training the model is to partition the dataset 1/4 for validation, 3/4 for training/testing, and use the 3/4 training set to perform a 10-fold cross validation to evaluate model performance. By 10-fold CV I am referring to the definition of using 9/10 of the training data to train and then assess performance on 1/10 of the training data, and repeating this process 10 times, using each slice (1/10) of the training data once as the test set for assessing performance during each iteration.

The part I am unsure about is whether I need to perform the pre-filtering/feature selection for each fold of the 10-fold CV. My intuition is that it is necessary to pre-filter significant features for each unique 9/10 slice of the training data for each iteration of cross validation. Is this correct?

It seems like it will be very labor intensive to do it this way because most of the R packages I have seen for performing k-fold CV seem to operate on a single dataset as opposed to 10 unique datasets for each fold of CV.

One possibility is to perform pre-filtering 10-times, once for each set of 9/10 training data, and then combine each of the pre-filtered datasets into a single dataset for 10-fold CV, but I am not sure if this is a valid method.

Alternatively does anyone know of any R packages that allow you to supply a new dataset/featureset for each fold of cross validation?

Please let me know if any of this is unclear, I am still learning the statistical jargon and it is quite possible I've misused some vocabulary. Thanks in advance.

  • 2
    $\begingroup$ Not a full answer, but your intuition is correct. You should not pre-select features on the whole dataset, since you will be learning (in part) which features are discriminative in the test data. This will give you a biased (overly optimistic) estimation of how your feature selection + classification method performs. The correct way to do it is as you suggest, selecting a new feature subset for each training fold. $\endgroup$ – Nuclear Wang Mar 24 '17 at 18:36
  • $\begingroup$ Thanks for your help @Matt, do you happen to know of any workflows/packages that would help with handling the unique feature sets during cross validation? $\endgroup$ – Reilstein Mar 24 '17 at 18:44
  • $\begingroup$ I dont think you can do it, if I understood you correctly. You need to filter and then do CV. Check out this video $\endgroup$ – Alex Nikiforov Jul 11 at 23:02
  • $\begingroup$ You also should keep in mind that euclidean distance wont work well due to curse of dimensionality. Im not sure if a similar discussion happened on SO, but it def was on Quora $\endgroup$ – Alex Nikiforov Jul 11 at 23:03
  • $\begingroup$ @AlexNikiforov This was a couple of years ago now, and I figured it out: in short, you actually DO need to select features for each individual training fold IF you are trying to estimate true model performance on outside data, which is what I was trying to do. If you select features at the beginning using information on the grouping variable you are trying to predict then it will appear as if you have a highly performing model, but in reality your model is useless, because you preselected features that differentiate the two classes, and you won't have that benefit on outside data. $\endgroup$ – Reilstein Jul 12 at 3:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.