I have a dataset with 121 features and 7176 data points. Only 11 of these are positive, the rest is negative. If I want to train a SVM on this data set, what would be the best strategy to do this? Does this even make sense?

For now I decided to 5-fold cross validation with stratified sampling. Would this approach be reasonable?


You have a positive class prior of $0.0015$, which is way to low to train a classifier from. You need at least an excess of $100$ positive cases, your most rare class. You will be able to start with a few (most) discriminative features.

Next, a support vector machine is not a suited classifier for such a skewed prior distribution. A recommended approach is to train for example a random forest classifier, a decision tree (C4.5 or CART) and discriminant-analysis/logistic-regression. Use a balanced training set and correct the posterior probabilities post hoc, see the approach for this correcting for a skewed prior after training.

After note

The set of $100$ required positive cases is chosen based on experience. With $121$ different feature values, this number is at an absolute minimum. A classifier like C4.5 does feature selection while building the decision tree (starting with each individual feature variable).

More theoretical work can begin with the paper by E.B. Baum, D. Haussler, What size net gives valid generalization? Neural Computation, Vol. 1, No. 1, March 1989. Having computed with the guidelines from their Corollary 9, to ensure an error rate of $0.015$, $121$ feature variables, you need $125$ training cases. I assumed one parameter/weight per feature variable (one hidden node - the simplest neural network classifier).

Beware that their results are based on the nonparametric VC-dimension. In practice, their bounds are below what is needed to fit well-generalizing decision boundaries. Using more model parameters than training cases, that is discouraged.

  • $\begingroup$ Where is an excess of 100 positive cases recommended? Can you provide some literature? Your answer is very valuable, thanks. I actually have the same problem with a OneClass SVM (see my other question on cross validated). I would appreciate it very much, if you could have a look at my other question here: stats.stackexchange.com/questions/478565/… $\endgroup$ Jul 23 '20 at 14:51
  • 1
    $\begingroup$ I have added some literature. Much later research has been built on the results by Baum & Haussler (1989). $\endgroup$ Jul 23 '20 at 17:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.