Considering class imbalance, why does random oversampling, in general, improve the performance of a linear SVM? Is it because the number of support vectors for the minority class are increased as a result of the oversampling (even though the data is just replicated)?

Thanks for listening.


What sort of SVM do you use?. If you are using $\nu$-SVM, then you can find the answer in this paper.

The idea is the following: "v is an upper bound on the fraction of margin errors, a lower bound on the fraction of support vectors, and both quantities approach v asymptotically". Additionally, this number cannot exceed the quantity 2*lmin/l, where l is the total number of SVs and lmin is the minimum between the number of positive and negative SVs (labels +/-1).

Notice that this means that for inbalanced problems, you will have to work with lower values of v, so you will tend to overfit data. Why? Because it forces you to use an eventually very small value of nu, that is, you allow very small number of errors during training, what leads to overfitting.

Oversampling is a way to dodge this problem. This intuition is also valid for C-SVM, though the meaning of the C parameter is different from that of the $\nu$ parameter. Still, it also acts as a parameter controlling the amount of regularization.

The idea is that C weights a term which can be understood as the number of errors your classifier does on the training data. If a class has a much higher density of samples, then the optimization algorithm tends to reduce the error by pushing the separating hyperplane to the minority class. The idea is that you decrease the total error by reducing the mistakes on the big class. Overall looks better, but as a result you have a higher error rate on the smaller class. Consider the extreme case when you have a ratio 97%-3%. More details about the problem and techniques to overcome it here.

Hope this helps.

| cite | improve this answer | |
  • $\begingroup$ Thanks for that. It's a c-SVM. Based on the 2nd paper (Batuwita & Palade) the reason SVMs suffer from class imbalance is due to the "weakness of the soft margin optimization problem". This basically describes how the separating hyperplane (SH) becomes skewed towards the minority class. We want to maximize the margin (either side of the SH) and minimise misclassifications. The low number of minority samples makes them appear further from the SH resulting in the skew towards this class. With oversampling, the misclassification costs increase causing the sep hyperplane to shift back? $\endgroup$ – dubby Apr 25 '13 at 18:24
  • $\begingroup$ yes, the idea is that samples from both classes tend to be weighted equally. If you are using C-SVM, there is a more principled approach to deal with this problem. See "A User’s Guide to Support Vector Machines" (www.cs.colostate.edu/~asa/pdfs/howto.pdf). It is available in the libSVM package. $\endgroup$ – jpmuc Apr 25 '13 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.