I have a dataset of n = 55 samples with 300,000 - 600,000 features for each sample. I am trying to train a model (ksvm classification) to predict the class of each sample. N = 26 of the samples are class 1, N = 29 are class 2.
Due to the large number of features, I planned to pre-filter the features to include only features that are significantly different between the two classes. When I have done this in practice, the number of features can be reduced to 1,000 - 4,000.
My plan for training the model is to partition the dataset 1/4 for validation, 3/4 for training/testing, and use the 3/4 training set to perform a 10-fold cross validation to evaluate model performance. By 10-fold CV I am referring to the definition of using 9/10 of the training data to train and then assess performance on 1/10 of the training data, and repeating this process 10 times, using each slice (1/10) of the training data once as the test set for assessing performance during each iteration.
The part I am unsure about is whether I need to perform the pre-filtering/feature selection for each fold of the 10-fold CV. My intuition is that it is necessary to pre-filter significant features for each unique 9/10 slice of the training data for each iteration of cross validation. Is this correct?
It seems like it will be very labor intensive to do it this way because most of the R packages I have seen for performing k-fold CV seem to operate on a single dataset as opposed to 10 unique datasets for each fold of CV.
One possibility is to perform pre-filtering 10-times, once for each set of 9/10 training data, and then combine each of the pre-filtered datasets into a single dataset for 10-fold CV, but I am not sure if this is a valid method.
Alternatively does anyone know of any R packages that allow you to supply a new dataset/featureset for each fold of cross validation?
Please let me know if any of this is unclear, I am still learning the statistical jargon and it is quite possible I've misused some vocabulary. Thanks in advance.