I am using a linear SVM classifier for binary classification problem in a small dataset (60 samples total, balanced i.e. 30+30). My features are 60 in number. I am using an aggregate of filter methods for ranking my features before running a grid search to simultaneously optimize number of features and the C value for linear SVM (all of this in a nested CV framework).
With respect to grid search, I am varying my number of features from 1:p; for each feature size (p), I am checking 11 values of C. For each CV split, I generate feature ranking, choose the top p values, and for each C value, find the misclassification error.
- Take training data from outer CV
- For each inner CV loop:
- split into training and test data
- generate ranking of features using this inner training data
- for each feature size, for each value of C, find misclassification error on the inner test data
- Find average misclassification error across all inner CV
- Choose the values of C and p which gave least average misclassification error
- Using these values, report unbiased estimate of performance on test data for outer CV
My questions are:
Does this seem like a valid strategy to do simultanous optimization of number of features and other parameters for ML?
Since my overall dataset is quite small, how do I minimize chances of over-fitting while simultaneously optimizing all parameters? Specifically, since I am choosing the model parameters which give me the least misclassification error, it is quite possible that I end up over-fitting; are there strategies similar to 1 standard error rule that could be applied here?