3
$\begingroup$

This question already has an answer here:

I'm using libSVM for binary classification and my training data is very unbalanced (-1:90%, +1:10%). According to libSVM's documentation, it's better to set different penalties for positive and negative classes. For example, the SVM problem is:

$\min\limits_{w,b,\xi} \frac{1}{2}{\bf w^Tw} + C^+\sum\limits_{y_i=1} \xi_i + C^-\sum\limits_{y_i=-1} \xi_i$

My question is which penalty should be larger and why. Thanks

$\endgroup$

marked as duplicate by Tim, Michael Chernick, Peter Flom Jun 3 '17 at 12:16

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Check out this paper. It shines some light on ideas on how to deal with unbalanced data. $\endgroup$ – user46739 Jun 5 '14 at 0:09
  • $\begingroup$ Welcome to the site, @nickb. Would you mind adding a brief summary of the information in that paper in case the link goes dead, &/or so readers can know if they want to pursue it further? $\endgroup$ – gung Jun 5 '14 at 0:30
7
$\begingroup$

The larger the penalty, the more an error on the training set (which is what is measured by $\xi_i$) for a pattern of that class influences the model. So if you have more negative patterns than positive patterns then you probably want to make $C^+$ larger than $C^-$. Personally if there is a class imbalance problem then it usually means that the costs of false-positive and false-negative errors are not the same, and the relative costs of the errors is an important criterion for adjusting the penalties. I would suggest using cross-validation to estimate the expected loss and choose the penalties to minimise that.

$\endgroup$
  • $\begingroup$ A small addition: In practice, I have found that often in unbalanced problems, different penalties yield the same CV performance. So, OP, keep in mind that changing penalties will not necessarily improve your results in CV. $\endgroup$ – Bitwise Oct 25 '12 at 19:27
  • $\begingroup$ @Bitwise, what would you do in this case then? $\endgroup$ – user11869 Oct 25 '12 at 19:39
  • $\begingroup$ @user11689 there is not much to do, this is just another parameter to play with to try and improve your results (with proper CV, of course). $\endgroup$ – Bitwise Oct 25 '12 at 19:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.