I have a small data set (about 120 rows) with a large number of features (about 600). I'm attempting to train a linear support vector machine and using the following approach:
1) Apply recursive feature elimination with 10-fold cross-validation to reduce the number of features 2) Fit the SVM on the reduced feature space and test with 10-fold cross-validation again.
I'm slightly concerned because the outputs of these steps seem somewhat unstable. Step 1) can produce an "optimal" feature space with anywhere from 6 to 300 features, and seems to produce the best CV scores when it settles on a feature space with about 100 features. The variation is due to the random partitions of the training data during cross-validation, of course.
The best CV scores are in the neighborhood of 0.9, and the worst around 0.6-0.7.
I have read a fair bit about over-fitting during the model selection step, eg http://jmlr.org/papers/v11/cawley10a.html
My question is, should I be concerned? Can I safely accept the better models, or am I subtly overfitting to my dataset? All of the research I've read suggests that overfitting during model selection is attributable to hyperparameter tuning, but I am not tuning anything here. Yet, I feel like the feature selection step may have a similar effect.