Choosing fold size for highly Imbalanced dataset + nested CV + SVM

Question

I am trying to classify a dataset with ~1000 points. 90/10 is the class ratio - super imbalanced.

Here are the following steps I did:

1. Use 20 relevant features from previous knowledge

2. Remove highly correlated features

3. Perform backwards feature selection (use all features, take one out)

4. Estimate which feature-set is the best and repeat (3) until it doesn't get any better

To evaluate (3) accurately I do nested CV as follows:

a. Split dataset into the two classes (say 0 and 1) and split each class into $X$ folds and merge folds together (stratification).

b. Take $X-1$ folds for training and one for testing. Use training data to split again into $Y$ folds (again train and test) and perform SVM to find optimal parameters

c. Once parameter found, use the test data to estimate accuracy. To estimate accuracy I used the F2 score to account for the imbalanced dataset.

After each outer fold has completed, I end up with $X$ F2 scores that I average to get my final score to be able to estimate (4).

Here is the problem: If I do (3) and (4) with multiple iterations I have around ~10% variability in my average F2 score. So it makes it hard to say which feature set is the best. I think this is because of the randomly chosen folds in (a) and (b)

Now here are my questions:

1. What do you think is the best number of folds for this particular dataset? I have tried with 5 for outer and 5 inner. But maybe I should decrease the number of folds in the outer CV since I have a few data points in the inner CV?! Maybe the best thing to do is to try different combinations and see what is best and stick to it?

2. Alternatively, I thought to do iterative nested CV (around 10-50 times) and to take the average of that to be able to choose the best feature set. Do you think this is OK to do?

3. Is overall the approach that I do for this classification legit?

Thoughts and comments are very much appreciated.

Answer 1

The problem as you have set it up would be better served using the bootstrap than using cross-validation. The bootstrap has more precision and there is only one choice (the number of bootstrap repetitions). With cross-validation you have to choose the fold size and the number of repetitions of the whole process. For example, with your sample size you would need 100 repeats of 10-fold cross-validation to achieve stability.