I am trying to classify a dataset with ~1000 points. 90/10 is the class ratio - super imbalanced.

Here are the following steps I did:

  1. Use 20 relevant features from previous knowledge

  2. Remove highly correlated features

  3. Perform backwards feature selection (use all features, take one out)

  4. Estimate which feature-set is the best and repeat (3) until it doesn't get any better

To evaluate (3) accurately I do nested CV as follows:

a. Split dataset into the two classes (say 0 and 1) and split each class into $X$ folds and merge folds together (stratification).

b. Take $X-1$ folds for training and one for testing. Use training data to split again into $Y$ folds (again train and test) and perform SVM to find optimal parameters

c. Once parameter found, use the test data to estimate accuracy. To estimate accuracy I used the F2 score to account for the imbalanced dataset.

After each outer fold has completed, I end up with $X$ F2 scores that I average to get my final score to be able to estimate (4).

Here is the problem: If I do (3) and (4) with multiple iterations I have around ~10% variability in my average F2 score. So it makes it hard to say which feature set is the best. I think this is because of the randomly chosen folds in (a) and (b)

Now here are my questions:

  1. What do you think is the best number of folds for this particular dataset? I have tried with 5 for outer and 5 inner. But maybe I should decrease the number of folds in the outer CV since I have a few data points in the inner CV?! Maybe the best thing to do is to try different combinations and see what is best and stick to it?

  2. Alternatively, I thought to do iterative nested CV (around 10-50 times) and to take the average of that to be able to choose the best feature set. Do you think this is OK to do?

  3. Is overall the approach that I do for this classification legit?

Thoughts and comments are very much appreciated.

  • 1
    $\begingroup$ Frankly, I would use a different feature selection approach (or none, actually). SVMs are quite robust against uninformative features. Additionally, in the context of class imbalance it is better to use class-weighted SVM (different value of C per class, higher for the minority class). Your setup seems extremely complex to solve a fairly basic problem. $\endgroup$ – Marc Claesen Aug 20 '14 at 13:50

The problem as you have set it up would be better served using the bootstrap than using cross-validation. The bootstrap has more precision and there is only one choice (the number of bootstrap repetitions). With cross-validation you have to choose the fold size and the number of repetitions of the whole process. For example, with your sample size you would need 100 repeats of 10-fold cross-validation to achieve stability.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.