10
$\begingroup$

I'm trying to build a prediction model with SVMs on fairly unbalanced data. My labels/output have three classes, positive, neutral and negative. I would say the positive example makes about 10 - 20% of my data, neutral about 50 - 60%, and negative about 30 - 40%. I'm trying to balance out the classes as the cost associated with incorrect predictions among the classes are not the same. One method was resampling the training data and producing an equally balanced dataset, which was larger than the original. Interestingly, when I do that, I tend to get better predictions for the other class (e.g. when I balanced the data, i increased the number of examples for the positive class, but in out of sample predictions, the negative class did better). Anyone can explain generally why this occurs? If I increase the number of example for the negative class, would I get something similar for the positive class in out of sample predictions (e.g., better predictions)?

Also very much open to other thoughts on how I can address the unbalanced data either through imposing different costs on misclassification or using the class weights in LibSVM (not sure how to select/tune those properly though).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

Having different penalties for the margin slack variables for patterns of each class is a better approach than resampling the data. It is asymptotically equivalent to resampling anyway, but is esier to implement and continuous, rather than discrete, so you have more control.

However, choosing the weights is not straightforward. In principal you can work out a theoretical weighting that takes into account the misclassification costs and the differences between training set an operational prior class probabilities, but it will not give the optimal performance. The best thing to do is to select the penalties/weights for each class via minimising the loss (taking into account the misclassification costs) by cross-validation.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Is there an automated way on how to do that, or do there exist learners which have this functionality incorporated? $\endgroup$
    – Vam
    Dec 8, 2013 at 19:01
  • $\begingroup$ I usually just write a matlab function to evaluate the loss for a particular set of penalties and then minimize it using the Nelder-Mead simplex algorithm. I don't know of any libraries that has this built in. $\endgroup$
    – Dikran Marsupial
    Dec 9, 2013 at 9:04
  • $\begingroup$ @DikranMarsupial Would a grid search of the two margin slack variables in a two-class problem be equivalent to what you are doing with the simplex algorithm? $\endgroup$
    – Spacey
    Nov 25, 2014 at 1:13
  • $\begingroup$ @Tarantula yes, the precise optimisation method is relatively unimportant, the key point is to make sure that you are optimising the cross-validation statistic that you are really interested in for the purposes of your application (i.e. the same class frequencies encountered in operational use and taking into account misclassification costs if known). $\endgroup$
    – Dikran Marsupial
    Nov 25, 2014 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.