I am trying to classify a dataset with ~1000 points. 90/10 is the class ratio - super imbalanced.
Here are the following steps I did:
Use 20 relevant features from previous knowledge
Remove highly correlated features
Perform backwards feature selection (use all features, take one out)
Estimate which feature-set is the best and repeat (3) until it doesn't get any better
To evaluate (3) accurately I do nested CV as follows:
a. Split dataset into the two classes (say 0 and 1) and split each class into $X$ folds and merge folds together (stratification).
b. Take $X-1$ folds for training and one for testing. Use training data to split again into $Y$ folds (again train and test) and perform SVM to find optimal parameters
c. Once parameter found, use the test data to estimate accuracy. To estimate accuracy I used the F2 score to account for the imbalanced dataset.
After each outer fold has completed, I end up with $X$ F2 scores that I average to get my final score to be able to estimate (4).
Here is the problem: If I do (3) and (4) with multiple iterations I have around ~10% variability in my average F2 score. So it makes it hard to say which feature set is the best. I think this is because of the randomly chosen folds in (a) and (b)
Now here are my questions:
What do you think is the best number of folds for this particular dataset? I have tried with 5 for outer and 5 inner. But maybe I should decrease the number of folds in the outer CV since I have a few data points in the inner CV?! Maybe the best thing to do is to try different combinations and see what is best and stick to it?
Alternatively, I thought to do iterative nested CV (around 10-50 times) and to take the average of that to be able to choose the best feature set. Do you think this is OK to do?
Is overall the approach that I do for this classification legit?
Thoughts and comments are very much appreciated.